Bug 428156 - OFX import goes to the wrong account
Summary: OFX import goes to the wrong account
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Unclassified
Component: importer (show other bugs)
Version: 5.1.0
Platform: Archlinux Packages Linux
: NOR normal (vote)
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-10-23 19:59 UTC by Isaac Wismer
Modified: 2021-10-05 18:43 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.1.3


Attachments
The import selection and result screen (59.20 KB, image/png)
2020-10-23 19:59 UTC, Isaac Wismer
Details
attachment-23286-0.html (2.56 KB, text/html)
2021-10-05 16:16 UTC, Dawid Wróbel
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Isaac Wismer 2020-10-23 19:59:25 UTC
Created attachment 132669 [details]
The import selection and result screen

SUMMARY
When I use the OFX import wizard from an OFX file downloaded from the internet, it asks me which account to import it into, but then imports it into a different account.

This only seems to happen with my credit cards, not with my bank accounts.

STEPS TO REPRODUCE
1. Have multiple credit card accounts
2. Import OFX for one of them
3. In the account selection wizard, select the one you want
4. Click OK

OBSERVED RESULT

The transactions are imported into the wrong account

EXPECTED RESULT

The transactions are imported into the correct account.


SOFTWARE/OS VERSIONS
Windows: 
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 5.20.1
KDE Frameworks Version: 5.75.0
Qt Version: 5.15.1

ADDITIONAL INFORMATION

This may be similar to: https://bugs.kde.org/show_bug.cgi?id=405206 except in this case it allows me to choose the target, but then seemingly ignores my choice.
Comment 1 Maxime Vincent 2020-12-22 10:06:06 UTC
I have the exact same issue here. I'm also on Arch Linux, KMyMoney version 5.1.0.

When importing an OFX file generated by `ofxstatement`, KMyMoney asks which account to import to, and then still goes on to import to the wrong account (the default, I suspect).
Comment 2 Jack 2021-09-03 21:18:37 UTC
Is this still happening in 5.1.2?  Also, what are the account types, both of the chosen account and the one which actually gets the transaction?  I do not have this problem, and I import many accounts by OFX (usually direct connect, but on occasion by file import.)
Comment 3 Thomas Baumgart 2021-09-05 07:16:06 UTC
Adjusting state
Comment 4 Isaac Wismer 2021-09-12 12:04:26 UTC
Yes, this still occurs in 5.1.2.
Both accounts are credit cards.
Comment 5 Isaac Wismer 2021-09-12 12:09:29 UTC
Also, for a little more info:
Credit card account #1 imports fine. It gives me a dialog, which defaults to importing into card #1, and then I hit OK and it imports into account #1.
Credit card account #2 doesn't import correctly. It gives me a dialog which defaults to card #1, so I change it to card #2, but then when I hit OK, the status dialog shows that it was imported into account #1 (so I think maybe it just always imports into the default account?).
Unfortunately I only have 2 credit cards that offer OFX export, so I can't test with anything else.
Comment 6 Isaac Wismer 2021-09-12 12:13:49 UTC
As a workaround, I just download the transactions as a CSV and import it that way, which doesn't have any issues, it just takes a little more work.
Comment 7 Jack 2021-09-12 17:48:47 UTC
Does the problem happen with both direct-connect and importing downloaded OFX files?  If the bank offers to download MsMoney or QIF files, that should be less work for you than using csv, until we figure out the underlying problem.
Are the credit cards from the same or different banks?
On the Edit account dialog, does all the information on all subtabs of the Online settings tab look correct?
One thing you might try is to unmap and then remap the account for card #2.  (this is only if you already have the account mapped and use direct-connect)
Another thing to try is to create a new credit card account and try to import the OFX for card #2 into the new account.
Comment 8 Isaac Wismer 2021-09-12 17:52:36 UTC
The credit cards are from different banks.

All the details in the account info seems correct.

Unfortunately neither bank offers direct connect, so that's not an option to test.

I will try setting up a new card in KMyMoney and see what happens.

The really weird thing is that I manually select for it to be imported into the correct account, but it still ends up importing into the wrong account. I guess it doesn't respect the manual selection.
Comment 9 Dawid Wróbel 2021-09-12 17:56:53 UTC
Have you previously accidentally imported into that account that KMM now insist on importing to? Can you inspect your .kmy file and see if the accounts in speech (the one you want to import to and the one KMM imports to) have a 'StatementKey' value assigned?
Comment 10 Isaac Wismer 2021-09-12 19:15:53 UTC
What is the file type/encoding of the .kmy file? I can't seem to figure out out to inspect it.
Comment 11 Dawid Wróbel 2021-09-12 19:24:38 UTC
Try changing the description to .zip and decompress it.
Comment 12 Dawid Wróbel 2021-09-12 19:25:29 UTC
*extension, obviously :)
Comment 13 Isaac Wismer 2021-09-12 20:37:08 UTC
The correct credit card account has a StatementKey, the account it goes into has a StatementKey, but it's just 2 spaces.
Comment 14 Isaac Wismer 2021-09-12 20:39:11 UTC
(In reply to Isaac Wismer from comment #13)
> The correct credit card account has a StatementKey, the account it goes into
> has a StatementKey, but it's just 2 spaces.

Actually, I'm sorry, the account that the import should go into does not have a statement key, the account that it does go into (but shouldn't) has a statementkey of just 2 spaces.

<PAIR value="  " key="StatementKey"/>
Comment 15 Thomas Baumgart 2021-09-14 06:34:16 UTC
Detailed information regarding the file format and encoding can be found in the manual at https://docs.kde.org/stable5/en/kmymoney/kmymoney/details.formats.html.

Seeing two blanks as the StatementKey value is strange. The StatementKey is used as a second indicator to identify an account. It is set as part of the statement import when the account number received from the bank differs from the one KMyMoney knows about. Since we're talking OFX here, this information comes straight from libOFX. It is strange though to see two blanks here. Let's see what we can do about it or if there is a bug somewhere down the line.

@Isaac: Please proceed as follows:

- use the settings/general/support dialog to
  a) turn on "Log imported statements"
  b) turn on "Log OFX transactions"

a) will leave a "kmm-statement-<date> <time>.txt" in the 'logpath'
b) will write the communication with the OFX server to the file "ofxlog.txt" in the 'logpath'

'logpath' can also be set in the above dialog but must exist. The users home dir is a good choice.

Delete a possibly existing ofxlog.txt file in the logpath so that we have a fresh copy.

Then run the import process that fails until the end.

Next I am interested in the value of accountid, accountnumber and type found in the statement file. You can see those with the following command:

  grep version kmm-statement-....

Next I want to know which account number was provided by the bank. You can get that with the following command

  grep ACCTID ofx-log.txt | od -c -Ax

If you have a tool called hexdump available, please provide the output of the following command:

  grep ACCTID ofx-log.txt | hexdump -C


Maybe you can repeat the above for a bank account (the one that is working) so that we can see what the difference is.

Don't forget to turn off logging.

p.s. Taking a look at the screen shot attached, the account number seems empty (but that could well be two blanks which we can't see).
Comment 16 Isaac Wismer 2021-09-15 15:28:21 UTC
When importing the OFX file it doesn't create an ofxlog.txt (yes, I checked to make sure my log path was correct and that I had checked "Log OFX transactions"). But I think I can get the same information by just opening up the OFX file I downloaded from my bank.

from kmm-statement:
<STATEMENT routingnumber="" type="creditcard" enddate="2021-09-15" begindate="2021-09-13" closingbalance="-5049/50" version="1.1" skipCategoryMatching="0" accountname="Bank account " accountnumber="  " currency="CAD" accountid="">

from the actual OFX file:
<CCSTMTRS><CURDEF>CAD<CCACCTFROM><ACCTID>4525501001994321<ACCTTYPE>CREDITLINE</CCACCTFROM>

I have not used the other credit card in the past few days, so I unfortunately can't really import anything right now.
Comment 17 Jack 2021-09-15 15:45:31 UTC
ofxlog.txt only gets created when downloading with directconnect, not file import, so there's no problem there.  

Assuming you did not edit the kmm-statement extract, I'd say the two blanks value of the accountnumber is the issue, although I don't see how that could be derived from the ACCTID in the source OFX.  Also, is the KMM Account name really "Bank account " (with trailing space?)
Comment 18 Isaac Wismer 2021-09-15 19:22:40 UTC
I didn't edit the files, so yeah, definitely weird output.
Maybe the problem is in the OFX library? Or maybe my bank is giving me malformed OFX files?
Comment 19 Bug Janitor Service 2021-09-30 04:35:45 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 20 Isaac Wismer 2021-10-01 21:26:43 UTC
Is there anything else you need from me to try and reproduce it? Or anything I can do to reset the account number?
Comment 21 Thomas Baumgart 2021-10-04 08:41:19 UTC
I looked into this one again today since it is so strange. I located where the values of accountname="Bank account " and accountnumber="  " are set: it is in OfxAccountContainer::gen_account_id(), so way inside LibOFX. Here's the code:

    STRNCPY(data.account_id, string(data.account_id) + m_bankid + " " + m_branchid + " " + m_acctid);
    STRNCPY(data.account_name, string(data.account_name) + "Bank account " + m_acctid);

and with m_bankid, m_branchid and m_acctid being empty, the values we see get generated. Also, account_type must be differing from credit card and investment account at that point (I suspect, it is also unset).

This now raises the following questions:

- which version of LibOFX do you have installed and is used by KMyMoney?
- can you run the utility ofxdump to see if it provides some more details about what is happening esp. when comparing your two cc accounts? Check "ofxdump --help" for info about options.

Note: Regarding the difference between "OFX file import" and "OFX direct connect": OFX direct connect downloads an OFX file, saves it into a temp file (and writes info about it into the log file) and then calls the OFX file importer just like you would do when you have the OFX file. So no difference there.
Comment 22 Isaac Wismer 2021-10-05 00:07:54 UTC
I have libOFX 0.9.15
I will email the outputs of ofxdump.
Comment 23 Thomas Baumgart 2021-10-05 14:55:55 UTC
Git commit 791ddcfebd24f842cff1052ae0eba40bb06fcc52 by Thomas Baumgart.
Committed on 05/10/2021 at 14:55.
Pushed by tbaumgart into branch 'master'.

Discard empty account information provided by LibOFX

Account information may inaccurate when the provided OFX data does not
comply with the OFX specification.

M  +26   -3    kmymoney/plugins/ofx/import/ofximporter.cpp

https://invent.kde.org/office/kmymoney/commit/791ddcfebd24f842cff1052ae0eba40bb06fcc52
Comment 24 Thomas Baumgart 2021-10-05 14:58:06 UTC
The source of the problem has been detected: the OFX file (created by the bank) contains invalid OFX structures. In

<CCSTMTRS><CURDEF>CAD<CCACCTFROM><ACCTID>4525501001994321<ACCTTYPE>CREDITLINE</CCACCTFROM>

the part

<ACCTTYPE>CREDITLINE

is not specified in the OFX spec and the parser rejects all data received. Removing that element from the OFX file presents correct data and everything is working.
Comment 25 Thomas Baumgart 2021-10-05 15:14:23 UTC
Git commit bd835cb1cd0dce7e215aab3f6813ff754a48c596 by Thomas Baumgart.
Committed on 05/10/2021 at 15:14.
Pushed by tbaumgart into branch '5.1'.

Discard empty account information provided by LibOFX

Account information may be inaccurate when the provided OFX data does
not comply with the OFX specification.
FIXED-IN: 5.1.3

(cherry picked from commit 791ddcfebd24f842cff1052ae0eba40bb06fcc52)

M  +26   -3    kmymoney/plugins/ofx/import/ofximporter.cpp

https://invent.kde.org/office/kmymoney/commit/bd835cb1cd0dce7e215aab3f6813ff754a48c596
Comment 26 Dawid Wróbel 2021-10-05 16:16:47 UTC
Created attachment 142174 [details]
attachment-23286-0.html

Wouldn’t it be better if this was instead fixed upstream in both aqbanking
and libofx?

On Tue, Oct 5, 2021 at 5:14 PM Thomas Baumgart <bugzilla_noreply@kde.org>
wrote:

> https://bugs.kde.org/show_bug.cgi?id=428156
>
> Thomas Baumgart <thb@net-bembel.de> changed:
>
>            What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>              Status|NEEDSINFO                   |RESOLVED
>       Latest Commit|                            |
> https://invent.kde.org/offi
>                    |
> |ce/kmymoney/commit/bd835cb1
>                    |
> |cd0dce7e215aab3f6813ff754a4
>                    |                            |8c596
>          Resolution|WAITINGFORINFO              |FIXED
>    Version Fixed In|                            |5.1.3
>
> --- Comment #25 from Thomas Baumgart <thb@net-bembel.de> ---
> Git commit bd835cb1cd0dce7e215aab3f6813ff754a48c596 by Thomas Baumgart.
> Committed on 05/10/2021 at 15:14.
> Pushed by tbaumgart into branch '5.1'.
>
> Discard empty account information provided by LibOFX
>
> Account information may be inaccurate when the provided OFX data does
> not comply with the OFX specification.
> FIXED-IN: 5.1.3
>
> (cherry picked from commit 791ddcfebd24f842cff1052ae0eba40bb06fcc52)
>
> M  +26   -3    kmymoney/plugins/ofx/import/ofximporter.cpp
>
>
> https://invent.kde.org/office/kmymoney/commit/bd835cb1cd0dce7e215aab3f6813ff754a48c596
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
Comment 27 Thomas Baumgart 2021-10-05 18:00:55 UTC
I agree, it should be fixed upstream. But really upstream, which is the bank. Why do we have a specification? I am sick of cleaning up behind those institutions.
Comment 28 Isaac Wismer 2021-10-05 18:43:02 UTC
I'll definitely send a note off to my bank about their malformed OFX export, but knowing Canadian banks, it'll just be ignored...

Thanks for all your hard work on this application, it's great to use!