Bug 327495 - CSV Import causes KMM to recreate every single payee even if they already exist in the payee database
Summary: CSV Import causes KMM to recreate every single payee even if they already exi...
Status: RESOLVED WORKSFORME
Alias: None
Product: kmymoney
Classification: Applications
Component: general (show other bugs)
Version: 4.6.4
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-11-12 12:38 UTC by lp.allard.1
Modified: 2013-11-13 22:06 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description lp.allard.1 2013-11-12 12:38:41 UTC
When I import a CSV file (generated from LibreOffice Calc), KMM screws up the amounts.

For example, the CSV file has 3 columns in LibreOffice Calc

Date  -  Payee  -  Amount

In the exported CSV file, for example the first few lines would look like:

2012-12-04,"Seller ABC",-$16
2012-12-14,"The corner store",-$13.50
2013-01-16,"Best restaurant in town",-$8.00

Once imported, KMM would create a new payee called "Seller ABC" even if this payee already exists and has the IDENTICAL same name.  Then KMM will map the transaction of $16 to it. Because of that, importing a large CSV file is impractical as it basically doubles the payee list.

Reproducible: Always

Steps to Reproduce:
1. Create a CSV file with a program such as Libreoffice calc or MS excel
2. Import the CSV in KMM
3.KMM will recreate the payees even if they already exist
Actual Results:  
KMM will recreate the payees even if they already exist.  It doubles the payees..

Expected Results:  
KMM should do a cross reference and reuse the existing payees to match the transactions, not recreate identical ones..
Comment 1 allan 2013-11-12 12:53:43 UTC
Matching is not a function of the CSV importer.  Matching occurs after the importer has finished its job, i.e. in KMM itself.

If I import your sample file into a KMM file which doesn't have those payees, then the payees are created correctly.  If I then re-import the same file, duplicate payees are NOT created.

However, as in your case, those payees already exist (but not having been imported), when you do the import duplicates are created.

I think the difference may be that in your file, those payees are set to 'don't match' and that is why duplicates are created.  However, when the payees are created by KMM, they are set to match.  I think the logic is that, if they already exist but are not set to match, then that is what the user wants.
Comment 2 lp.allard.1 2013-11-12 15:22:16 UTC
"I think the difference may be that in your file, those payees are set to 'don't match' and that is why duplicates are created"

If there is some sort of switch somewhere to match or not match payees, then I fully agree, this is not a bug but an unclear feature.

When you say "in your file" do you mean in the CSV file or in the kmy file?
Comment 3 allan 2013-11-12 17:32:52 UTC
(In reply to comment #2)
> "I think the difference may be that in your file, those payees are set to
> 'don't match' and that is why duplicates are created"
> 
> If there is some sort of switch somewhere to match or not match payees, then
> I fully agree, this is not a bug but an unclear feature.
> 
> When you say "in your file" do you mean in the CSV file or in the kmy file?

Sorry, I meant in your kmy file.  If you look in Payees view on the left then select a payee, on the right there is a 'Matching' tab where you may select whether/how to match.  No Match is pretty obvious, as is 'Payee name'.  Then, you may import a file where the payee has additional characters appended, so the third choice allows for these, with, regex-type matching.  The easiest way to handle these is to select the 'new' payee and select delete.  If the payee is actually being used, it will ask if you want to substitute another payee.  If so, select that payee and its entries will be moved over, and the original payee's matching detail will have been amended appropriately.
Comment 4 lp.allard.1 2013-11-12 21:46:23 UTC
OK I have seen this "Transaction matching" in the payee list.  I can modify one payee at a time and select "Match on payee name" and click update, but can I do it for all payees in one shot?

That will be long to modify 968 payees one by one....
Comment 5 allan 2013-11-13 11:23:13 UTC
(In reply to comment #4)
> OK I have seen this "Transaction matching" in the payee list.  I can modify
> one payee at a time and select "Match on payee name" and click update, but
> can I do it for all payees in one shot?
> 
> That will be long to modify 968 payees one by one....

Yes, indeed.

I don't know of a way from within KMM to do this.  It is possible to multi-select payees, which gives the impression that it might work, but it doesn't. I'm afraid.

The way I would attempt to tackle this is with search and replace in an editor that can handle XML format, such as Kate or Kwrite.

First, and of utmost importance, TAKE a BACKUP!  Then save your file in XML format.  Open it in your editor, and search for <PAYEE matchingenabled="0", and replace with <PAYEE matchingenabled="1".  Just try it with one or two payees then save that file and load into KMM, which recognises XML format.  If those payees look right, proceed to do them all.  Once KMM look correct save again, but in .kmy format.  I've just tested the procedure and it works, but do take care.
Comment 6 lp.allard.1 2013-11-13 19:42:06 UTC
Crystal clear explanations!!!!
Comment 7 allan 2013-11-13 22:06:20 UTC
(In reply to comment #6)
> Crystal clear explanations!!!!
I suppose you could achieve the same via your SQL.