Bug 400465

Summary: Imported transactions are not matched against reconciled transactions
Product: [Applications] kmymoney Reporter: rustonm2005
Component: importerAssignee: KMyMoney Devel Mailing List <kmymoney-devel>
Status: RESOLVED WORKSFORME    
Severity: normal CC: KDE, phil
Priority: NOR    
Version: 5.1.3   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: Version Fixed In:

Description rustonm2005 2018-10-29 23:47:07 UTC
SUMMARY
When importing QIF file, transactions that should match existing transactionsare not matched and instead created duplicates

STEPS TO REPRODUCE
1. Import a QIF file into an account
2. Mark the transactions as reconciled
3. Re-Import the QIF file

OBSERVED RESULT
Duplicate transactions are created

EXPECTED RESULT
Duplicated transactions should not be created

SOFTWARE VERSIONS
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
Comment 1 Jack 2018-10-30 16:41:20 UTC
If you do not mark the transactions from the first import as reconciled, are they detected as duplicates or not in a second import?  (It would help to be sure whether or not the reconciled status is actually part of the problem or not.)
Comment 2 Justin Zobel 2022-12-02 01:23:06 UTC
Thank you for reporting this issue in KDE software. As it has been a while since this issue was reported, can we please ask you to see if you can reproduce the issue with a recent software version?

If you can reproduce the issue, please change the status to "REPORTED" when replying. Thank you!
Comment 3 Bug Janitor Service 2022-12-17 05:14:09 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 4 Phillip Vuchetich 2022-12-19 18:49:46 UTC
I am not original reporter, but this has frequently affected importing at least on the MS Windows 10 platform at least in previous versions (The one I was previously using appears to have been labeled 5.1-bbdd7eb6f).  In previous versions, imports from bank and credit cards in OFX format would typically import duplicates that required manual cleanup.  This would typically happen in the following scenario:
1. Import (OFX) a partial month of current transactions (perhaps the first to 15th of the month).
2. Process these (accept, match, edit categories, etc.)
3. After the month completes, import a new file with the entire month (to match a billing/statement cycle), including both known duplicates (days 1-15) and known new transactions (days 16-end of month).
4. The outcome is that duplicates would typically be close to 100% of transactions that should have been able to match. 

I have just upgraded to the 5.1.3  (stable: file labeled kmymoney5-mingw64-5.1.3-3.1.12-setup.exe) version and will be importing things over the next few weeks.  I plan to report back if it looks like imports are generally clean or if it can be easily reproduced.  I have Windows 10, Ubuntu 20.04 and FreeBSD 13.1 as readily available desktops in case the issue affects only some platforms.

Suggestion: Leave this open for another month so I can get multiple imports of valid data in the current version.

I have an financial account that uses QIF, but have few transactions.  Anyone working on the programming for QIF import functionality knows that the QIF is very limited in its data fields for deduplication. The QIF I receive has only Date (D), Amount (T), and Payee (P) fields.
Comment 5 Jack 2022-12-19 19:14:09 UTC
Given the original report was for QIF files, I'm not going to change the status, and will let this one get closed if the OP doesn't respond.

OFX duplicate detection generally relies on a unique transaction ID provided by the bank.  I'll have to go hunting, but I do remember a previous report about this, where it turned out the bank was not producing a stable transaction ID, but creating a new one every time the user downloaded the same transaction.  I also believe that a change was made to do a better job of detecting when that happened, but I don't remember the details.  How recently did you have this problem?  If it wasn't within the past few  months, there's a good chance it has been fixed.

If the problem does happen again for you, please let us know (in a new bug) and please include which bank you are using and whether or not it is platform specific.  Saving the transaction section from the OFX file from each download would also be helpful.
Comment 6 Brendan 2022-12-21 18:35:53 UTC
Thomas changed this several years ago when I ran into a problem with KMM not being able to detect duplicate transactions. He added a second method to detect duplicates. You can find the setting in both the manual OFX import file selector and in the direct connect setup for an account.

The setting is called "Method to detect duplicate transactions during import". There are 2 options, "OFX FITID" and "KMyMoney Hash". The KMM hash fixed my import problem when the bank did not sent the same FITID each time you downloaded a transaction.

This is not an immediate fix since switching this setting to KMM Hash will result in none of the transactions matching the previous transactions since they used FITID. 

It's also not perfect, I still get duplicate transactions from one bank where the transactions don't always match. I can limit the extent of the problem by limiting the date range of the download so I haven't dug in to figure out what is going wrong.
Comment 7 Jack 2022-12-21 21:20:36 UTC
Brendon:  Thanks.  That was what I was thinking about.
Phillip:  If you still get duplicates after following Brendan's advice, please open a new ticket.  Thanks.
Comment 8 Bug Janitor Service 2023-01-05 05:26:10 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 9 Bug Janitor Service 2023-01-20 05:09:23 UTC
This bug has been in NEEDSINFO status with no change for at least
30 days. The bug is now closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!