Bug 336320

Summary: OFX file import duplicates the last 12 transactions
Product: [Applications] skrooge Reporter: rodsingleton <rodsingleton1>
Component: generalAssignee: Stephane MANKOWSKI <stephane>
Status: RESOLVED FIXED    
Severity: normal CC: stephane
Priority: NOR    
Version: 1.9.0   
Target Milestone: ---   
Platform: Kubuntu   
OS: Linux   
Latest Commit: http://commits.kde.org/skrooge/3363ada6227a54338bfa5634ab4eb385fbe3612f Version Fixed In:
Sentry Crash Report:
Attachments: Sample OFX file with 36 transactions
Result of ofxdump on your file

Description rodsingleton 2014-06-16 13:31:10 UTC 
When importing OFX files downloaded from my bank containing multiple transactions I've notcied that the last 12 transactions in any account I import get duplicated with the import preference checked to say "only import operations since the last imported one". The same import using QIF files correctly rejects all previously imported transactions.

Reproducible: Always

Steps to Reproduce:
1. Import and OFX file containing more than 12 transactions into a new or existing account.
2. Try importing the same OFX file again.
3. See how 12 fewer transactions are rejected as already being imported.
4. The last 12 transactions ARE imported and duplicated.

Actual Results:  
The last 12 transactions in the OFX file are not checked as being already imported and are imported again and may cause duplicate transactions to appear in an account if you are doing regular OFX file imports with over-lapping dates.

Expected Results:  
All transactions contained in an OFX file should be checked for duplicates when importing when the Import check option is enabled.
Comment 1 Stephane MANKOWSKI 2014-06-16 14:01:23 UTC 
Could you provide a sample OFX file ?
!!!!! Do not forget to remove all confidential information !!!!!
Comment 2 rodsingleton 2014-06-16 23:13:56 UTC 
(In reply to comment #1)
> Could you provide a sample OFX file ?
> !!!!! Do not forget to remove all confidential information !!!!!

Hi Stephane.
Can do in about 12 hrs when I return home. 
Rod
Comment 3 rodsingleton 2014-06-17 11:41:40 UTC 
Created attachment 87234 [details]
Sample OFX file with 36 transactions

Also I just noticed a positive amount for 1,111.27 (with comma) was imported as 1.11. I'll look at this and test with QIF and also by removing the thousand separator and see if it warrants a separate bug. :)
Comment 4 Stephane MANKOWSKI 2014-06-17 20:53:06 UTC 
Created attachment 87237 [details]
Result of ofxdump on your file

Hi,

Some operations are not detected "already imported" because the unique identifiers (<FITID>) are missing in your file. Nevertheless, I should be able to do a correction to correctly treat this kind of "non correct" file.

For the value 1,111.27 interpreted as 1.11, you can see in the attached file that ofxdump does the same interpretation. This is normal because ofxdump and Skrooge are both using an external library for reading oxf file. The issue is not in the tools but in your file nor respecting the OFX file. You should open a but to your bank ;-).
Comment 5 Stephane MANKOWSKI 2014-06-18 09:33:54 UTC 
Git commit 3363ada6227a54338bfa5634ab4eb385fbe3612f by Stephane Mankowski.
Committed on 18/06/2014 at 09:33.
Pushed by smankowski into branch 'master'.
OFX file import duplicates the last 12 transactions

M  +1    -0    CHANGELOG
M  +1    -1    skgbankmodeler/skgimportexportmanager.cpp
A  +338  -0    tests/input/skgtestimportofx/336320.ofx
M  +30   -0    tests/skgbankmodelertest/skgtestimportofx.cpp

http://commits.kde.org/skrooge/3363ada6227a54338bfa5634ab4eb385fbe3612f
Comment 6 rodsingleton 2014-06-18 13:15:15 UTC 
(In reply to comment #4)
> Created attachment 87237 [details]
> Result of ofxdump on your file
> 
> Hi,
> 
> Some operations are not detected "already imported" because the unique
> identifiers (<FITID>) are missing in your file. Nevertheless, I should be
> able to do a correction to correctly treat this kind of "non correct" file.
> 
> For the value 1,111.27 interpreted as 1.11, you can see in the attached file
> that ofxdump does the same interpretation. This is normal because ofxdump
> and Skrooge are both using an external library for reading oxf file. The
> issue is not in the tools but in your file nor respecting the OFX file. You
> should open a but to your bank ;-).

Hi, 
Just spent a bit of time viewing the dump and OFX specs around FITID. I very much doubt I'll be able to request my bank change their format given the online and paper based statement doesn't seem to have any form of unique transaction identifier that I can see at least. So I guess I'm stuck with what I have from the bank and its imperfect file format. 
Thank you for making the mod and commiting it. I look forward to the next update via the PPA. With the value containing thousand separator I guess I'll have to parse this first before importing to Skrooge. For me it's not often a transaction is over $1000. :)