Bug 361021 - CSV importer: The transaction has missing assignment of...
Summary: CSV importer: The transaction has missing assignment of...
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Applications
Component: importer (show other bugs)
Version: git (master)
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-26 19:45 UTC by NSLW
Modified: 2017-07-01 21:22 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In: 4.8.0


Attachments
CSV Test File (329 bytes, text/plain)
2016-03-26 19:49 UTC, NSLW
Details
Column Assignment for CSV file (73.47 KB, image/png)
2016-03-26 19:49 UTC, NSLW
Details
Problem description (307.83 KB, application/pdf)
2016-03-26 19:53 UTC, NSLW
Details
[PATCH] Add fee for buy and sell operation in the right way (5.60 KB, patch)
2016-03-26 19:54 UTC, NSLW
Details

Note You need to log in before you can comment on or make changes to this bug.
Description NSLW 2016-03-26 19:45:38 UTC
Buy transactions imported by CSV importer always have missing assignment (see attachment).

Reproducible: Always

Steps to Reproduce:
1. file->import csv
2. choose investment
3. create new profile
4. open "test file.csv" and assign columns to values (see attachment)
5. FieldDelimiter to comma
6. TextDelimiter to double quotes
7. DecimalSymbol to comma
8. ImportCSV

Actual Results:  
In ledger:
All buy operations have missing assignment
On home page:
Balance for banking and investment account are equal.

Expected Results:  
In ledger:
All operations should have assignment
On home page:
Balance for banking and investment account should be equal only in special cases. Correct difference in balance is shown in attachment.

To get good balance without the need of patching one has to double click every operation in ledger and press enter button. Upon completion column value will have exact same values as column from patched KMM.
Comment 1 NSLW 2016-03-26 19:49:07 UTC
Created attachment 98103 [details]
CSV Test File
Comment 2 NSLW 2016-03-26 19:49:36 UTC
Created attachment 98104 [details]
Column Assignment for CSV file
Comment 3 NSLW 2016-03-26 19:53:36 UTC
Created attachment 98105 [details]
Problem description
Comment 4 NSLW 2016-03-26 19:54:29 UTC
Created attachment 98106 [details]
[PATCH] Add fee for buy and sell operation in the right way
Comment 5 Jack 2016-03-26 19:59:00 UTC
Without looking at the details, I believe this is not a problem with the CSV importer, but with any KMM import of an investment transaction which requires a brokerage account for transfer of funds.  (I have it with OFX import.) The issue is that when KMM imports an investment transaction, it does not specify the brokerage account, so the missing assignment refers to the amount which would go to that account.  When you edit the transaction, KMM automatically enters the brokerage account, so the error disappears.
Comment 6 NSLW 2016-03-26 20:02:37 UTC
(In reply to Jack from comment #5)
> Without looking at the details, I believe this is not a problem with the CSV
> importer, but with any KMM import of an investment transaction which
> requires a brokerage account for transfer of funds.  (I have it with OFX
> import.) The issue is that when KMM imports an investment transaction, it
> does not specify the brokerage account, so the missing assignment refers to
> the amount which would go to that account.  When you edit the transaction,
> KMM automatically enters the brokerage account, so the error disappears.

It makes no sense for me that sell operations are properly assigned and buy operation not. Which error do you refer to Jack?
Comment 7 allan 2016-03-27 01:12:27 UTC
(In reply to Jack from comment #5)
> Without looking at the details, I believe this is not a problem with the CSV
> importer, but with any KMM import of an investment transaction which
> requires a brokerage account for transfer of funds.

That's not exactly true.  I've just done a QIF import of a Buy transaction, as a test, and that correctly identified the checking account to be used.

There is also another issue, with fees sometimes getting the wrong sign, which I identified in https://bugs.kde.org/show_bug.cgi?id=360129.  I think the patch in this current bug may be related.

>  (I have it with OFX
> import.) The issue is that when KMM imports an investment transaction, it
> does not specify the brokerage account, so the missing assignment refers to
> the amount which would go to that account.  When you edit the transaction,
> KMM automatically enters the brokerage account, so the error disappears.
Comment 8 allan 2016-03-27 01:14:40 UTC
(In reply to NSLW from comment #0)
> Buy transactions imported by CSV importer always have missing assignment

It's not correct to say that they '...always have missing assignment'  it is only under certain conditions.

> (see attachment).
> 
> Reproducible: Always
> 
> Steps to Reproduce:
> 1. file->import csv
> 2. choose investment
> 3. create new profile
> 4. open "test file.csv" and assign columns to values (see attachment)
> 5. FieldDelimiter to comma
> 6. TextDelimiter to double quotes
> 7. DecimalSymbol to comma
> 8. ImportCSV
> 
> Actual Results:  
> In ledger:
> All buy operations have missing assignment
> On home page:
> Balance for banking and investment account are equal.
> 
> Expected Results:  
> In ledger:
> All operations should have assignment
> On home page:
> Balance for banking and investment account should be equal only in special
> cases. Correct difference in balance is shown in attachment.
> 
> To get good balance without the need of patching one has to double click
> every operation in ledger and press enter button. Upon completion column
> value will have exact same values as column from patched KMM.
Comment 9 NSLW 2016-03-27 08:01:34 UTC
(In reply to allan from comment #8)
> (In reply to NSLW from comment #0)
> > Buy transactions imported by CSV importer always have missing assignment
> 
> It's not correct to say that they '...always have missing assignment'  it is
> only under certain conditions.

For me it always has missing assignment during import from CSV and empty ledger. Do you know conditions under which it doesn't happen?

(In reply to allan from comment #7)
> There is also another issue, with fees sometimes getting the wrong sign,
> which I identified in https://bugs.kde.org/show_bug.cgi?id=360129.  I think
> the patch in this current bug may be related.

According to my research bug #360129 can be independently fixed from this bug and this bug can be independently fixed from bug #360129.
Moreover through simple sign changes in my patch I can cause both operations to display warning about assignment and not only for sell operations.
How do you see them correlated?

Nevertheless, Allan please analyze this and another bug with patches for them.
Comment 10 allan 2016-03-27 11:29:13 UTC
(In reply to NSLW from comment #9)
> (In reply to allan from comment #8)
> > (In reply to NSLW from comment #0)
> > > Buy transactions imported by CSV importer always have missing assignment
> > 
> > It's not correct to say that they '...always have missing assignment'  it is
> > only under certain conditions.
> 
> For me it always has missing assignment during import from CSV and empty
> ledger. Do you know conditions under which it doesn't happen?

Yes.  Often, the problem is that a Buy/Sell/ReinvDiv, which involve funds transfers, does not have the name of the relevant checking/brokerage account provided.  During CSV import of these types, an extra dialog opens that asks for the name of the checking/brokerage account that is to be used.  If this is correctly entered, then the transaction is not unbalanced. In general, I do not have a problem, over many years,  with missing assignments.

> (In reply to allan from comment #7)
> > There is also another issue, with fees sometimes getting the wrong sign,
> > which I identified in https://bugs.kde.org/show_bug.cgi?id=360129.  I think
> > the patch in this current bug may be related.
> 
> According to my research bug #360129 can be independently fixed from this
> bug and this bug can be independently fixed from bug #360129.
> Moreover through simple sign changes in my patch I can cause both operations
> to display warning about assignment and not only for sell operations.
> How do you see them correlated?

I don't see the two bugs as related, except that https://bugs.kde.org/show_bug.cgi?id=361029 highlighted the fee sign issue.  I'm assuming/hoping that your patch here is for that same problem.  I haven't yet had a chance to look into it.

> Nevertheless, Allan please analyze this and another bug with patches for
> them.
Comment 11 NSLW 2016-03-27 13:17:18 UTC
(In reply to allan from comment #10)
> (In reply to NSLW from comment #9)
> > (In reply to allan from comment #8)
> > > (In reply to NSLW from comment #0)
> > > > Buy transactions imported by CSV importer always have missing assignment
> > > 
> > > It's not correct to say that they '...always have missing assignment'  it is
> > > only under certain conditions.
> > 
> > For me it always has missing assignment during import from CSV and empty
> > ledger. Do you know conditions under which it doesn't happen?
> 
> Yes.  Often, the problem is that a Buy/Sell/ReinvDiv, which involve funds
> transfers, does not have the name of the relevant checking/brokerage account
> provided.  During CSV import of these types, an extra dialog opens that asks
> for the name of the checking/brokerage account that is to be used.  If this
> is correctly entered, then the transaction is not unbalanced. In general, I
> do not have a problem, over many years,  with missing assignments.

I enter the name of checking/brokerage account always correctly and only once for every CSV file. I always have wrong amounts, even for transactions that show as balanced.
Can you provide investment statement, you don't have problems with?
 
> > (In reply to allan from comment #7)
> > > There is also another issue, with fees sometimes getting the wrong sign,
> > > which I identified in https://bugs.kde.org/show_bug.cgi?id=360129.  I think
> > > the patch in this current bug may be related.
> > 
> > According to my research bug #360129 can be independently fixed from this
> > bug and this bug can be independently fixed from bug #360129.
> > Moreover through simple sign changes in my patch I can cause both operations
> > to display warning about assignment and not only for sell operations.
> > How do you see them correlated?
> 
> I don't see the two bugs as related, except that
> https://bugs.kde.org/show_bug.cgi?id=361029 highlighted the fee sign issue. 
> I'm assuming/hoping that your patch here is for that same problem.  I
> haven't yet had a chance to look into it.

Yes, that's the same issue but the other report supposed to concern other bug, so patch for this bug doesn't fix the other bug.
Comment 12 NSLW 2016-05-08 07:12:02 UTC
Git commit 7d7e398cf180b21f3ab9e90c5184290aced161a0 by Łukasz Wojniłowicz.
Committed on 08/05/2016 at 07:02.
Pushed by wojnilowicz into branch 'master'.

Add fee for buy and sell operation in the right way

Fee should be added to buy and substracted from sell
operations. Difference between balance for banking and investment
account should include fees.
REVIEW: 127711

M  +18   -18   kmymoney/converter/mymoneystatementreader.cpp
M  +6    -4    kmymoney/plugins/csvimport/investprocessing.cpp

http://commits.kde.org/kmymoney/7d7e398cf180b21f3ab9e90c5184290aced161a0
Comment 13 NSLW 2016-05-08 07:15:13 UTC
Git commit 9b054dac7c81131a31c4afe57b589750208ddd6f by Łukasz Wojniłowicz.
Committed on 08/05/2016 at 07:14.
Pushed by wojnilowicz into branch 'frameworks'.

Add fee for buy and sell operation in the right way

Fee should be added to buy and substracted from sell
operations. Difference between balance for banking and investment
account should include fees.
REVIEW: 127711

M  +18   -18   kmymoney/converter/mymoneystatementreader.cpp
M  +6    -4    kmymoney/plugins/csvimport/investprocessing.cpp

http://commits.kde.org/kmymoney/9b054dac7c81131a31c4afe57b589750208ddd6f
Comment 14 allan 2016-05-16 23:07:50 UTC
Just to note that I discovered a possible problem with this, and documented it, in error, in https://bugs.kde.org/show_bug.cgi?id=362139, comment #7.  See there for follow up.