Bug 340956 - Duplicated statements in KMM-4.7.1 when importing transactions
Summary: Duplicated statements in KMM-4.7.1 when importing transactions
Status: RESOLVED WORKSFORME
Alias: None
Product: kmymoney
Classification: Applications
Component: onlinebanking (show other bugs)
Version: 4.7.0
Platform: Other Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords: triaged
Depends on:
Blocks:
 
Reported: 2014-11-14 13:23 UTC by Dr.Edgar Alwers
Modified: 2018-10-27 04:03 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments
Test.kmy performed with KMM-4.6.4 (6.45 KB, application/gzip)
2014-11-15 16:28 UTC, Dr.Edgar Alwers
Details
Test2.kmy performed with KMM-4.7.1 (6.58 KB, application/gzip)
2014-11-15 16:29 UTC, Dr.Edgar Alwers
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dr.Edgar Alwers 2014-11-14 13:23:06 UTC
After succesfully building  KMM-4.7.1 on an new Linux From Scratch Systemd with KDE-4.14.1, I  used the file "edgar.kmy" from an older SystemP running KMM  4.6.4 . This file was accepted, KMM-4.7.1 works so far OK.
But when I start importing transactions from my bank, a large number of statements become 
duplicated: KMM-4.7.1 does not recognize that some of the imported statements are already in the database. The only difference I see is an additional "blank" anywhere under "notes". Not all statements become duplicated.
The problem is persistent and can be reproduced. The behaviour is the same if I work with the file "edgar.kmy" or with MySQL.

It looks like the files generated by KMM-4.6.4 are not fully compatible with  KMM 4.7.1.
Comment 1 Cristian Oneț 2014-11-14 14:07:52 UTC
See the discussion at BUG 315072. Do you have payee matching setup for the transactions that become duplicated?

For example if you enter manually a transaction:
Payee: A
Amount: 10
Date: 14/11/2014

Then import a transaction:
Payee: Aa
Amount: 10
Date: 14/11/2014

It won't match because the two transactions have different payees.

But if you setup that payee 'A' will match string 'Aa' you'll obtain the expected behavior. This is also specified in the announcement of 4.7.1 https://kmymoney.org/news.php#itemKMyMoney471released
Comment 2 Dr.Edgar Alwers 2014-11-15 14:27:22 UTC
Thank you for your hint to the  Bug 315072, which I have studied now  a little. 
However, concerning your example:
I did not enter manually a transaction. The first transaction is the result of a bank OFX actualisation, say on November 10th. 2014.
The duplicate transaction is the result of the OFX actualisation of today. I told the program to start the actualisation on the, let me say, first of november.
The result is independent of what I setup on the payee matching. Furthermore: the payees are in both statements identical ( not A and Aa as in your example ).
Of course I can delete all duplcate statements by hand, but I did not have this behaviour in KMM-6.4. Is it not possible to instruct the program to look only to the date and to the amount in the matching process ?
Comment 3 Jack 2014-11-15 15:04:37 UTC
I am assuming that by actualisation you mean download.  (I seem to recall someone once using that term for reconciliation, which is a completely different issue.)  A downloaded transaction will only ever match a manually entered transaction, not another downloaded one.  However, it sound like in your case, it is not that the new transaction should MATCH the previously downloaded transaction, but should be detected as a DUPLICATE, and not imported at all.  Duplicate detection is based on a bank ID in the transaction.  If that is not there, KMM cannot know it is a duplicate.  

Can you create a simple sample file which shows the problem?  For example - create a new KMM file with one account, do an import, and then repeat the same import.
Comment 4 Dr.Edgar Alwers 2014-11-15 16:28:39 UTC
Created attachment 89598 [details]
Test.kmy performed with KMM-4.6.4
Comment 5 Dr.Edgar Alwers 2014-11-15 16:29:52 UTC
Created attachment 89599 [details]
Test2.kmy performed with KMM-4.7.1
Comment 6 Dr.Edgar Alwers 2014-11-15 16:42:52 UTC
exactly OK, what you say.. The program is not detecting duplicate statements.
I send two test files. Test1.kmy was created with KMM-4.6.4., starting from November 10th. Test2.kmy is Test1.kmy read by KMM-4.7.1 and updated online from the bank, starting also with November 10th.
You can see the duplicated statements included, there are no differences neither in payee nor in the date or so. Only an additional blank in the note.
I hope you can find out, why the program is not finding the bank id. I did not have this problem under KMM-4.6.4. However, I cannot say at this moment how would perform KMM-4.7.1 when both tests would be run from KMM-4.7.1, but I would not expect problems for this case.
Comment 7 Jack 2014-11-15 23:47:34 UTC
Thanks for posting the files.  I have not looked at them yet, but it would also help to have the OFX that you imported.  If you use direct connect, you can create an empty file called ofx.log in your home directory, and then after the download, it will contain what the bank sent.

What happens if you just repeat the import immediately, not waiting for another day?
Comment 8 Dr.Edgar Alwers 2014-11-16 09:44:37 UTC
> What happens if you just repeat the import immediately, not waiting for another day?

The import  ( Test2.kmy ) was performed immediatly after Test1.kmy ( 2 minutes perhaps ), but of course, in the second PC.
I tried to get the ofx.log, but the file remained empty. The protocol of AqBanking v5.4.2 does not show any data coming  from the bank. Only general comments like connected, selecting iTAN mode, sending jobs etc., you probably knows this better than I do
Thanks for your help
Comment 9 Jack 2014-11-16 17:14:22 UTC
Sorry I either forgot or didn't know you were using aqbanking.  What I was suggesting was to repeat the import on the second PC.  With libofx, this would download the same transactions again, and we could see if they were recognized as duplicates or added again as new.  Unfortunately, I don't know whether this would work using aqbanking, so now we just have to wait for a response from someone who knows aqbanking.
Comment 10 Dr.Edgar Alwers 2014-11-16 20:04:25 UTC
I  understand.  If KMM-4.7.1 is not recognizing the bank ID and therefore cannot identify duplicate statements, it could be, that the format used by aqbanking-5.0.27beta ( used by KMM-4.6.4 ) is not the same as the format used by aqbanking-5.4.2beta used in my built KMM-4.7.1. It seems, that it could be a problem of the OFX imported data.
I am going to  link this Bug to Martin Preuss, who is the author of aqbanking, and who for sure is the expert on this topic. Perhaps he can point us to a solution
Comment 11 Marko Käning 2014-11-16 20:22:08 UTC
(In reply to Dr.Edgar Alwers from comment #0)
> It looks like the files generated by KMM-4.6.4 are not fully compatible with
> KMM 4.7.1.

I also see this every now and then - when there has been a change on client or server side!

I usually clean up the overlapping transactions bit by bit and then go on downloading transactions with the latest aqbanking library from then on.
Comment 12 Dr.Edgar Alwers 2014-11-19 11:19:29 UTC
> I usually clean up the overlapping transactions bit by bit and then go on
> downloading transactions with the latest aqbanking library from then on.

I do not have news from the Aqbanking side, but I share your point of view, and I shall continue in this way, until a better solution shows up. The remaining problem is the exchange of data between PC's with different  KMM systems, wich is a little more complicated, but such is life .-)
Comment 13 Andrew Crouthamel 2018-09-25 03:55:56 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days, the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please set the bug status as REPORTED so that the KDE team knows that the bug is ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 14 Andrew Crouthamel 2018-10-27 04:03:41 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 30 days. The bug is now closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!