Bug 405206 - OFX import targets the wrong account
Summary: OFX import targets the wrong account
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Applications
Component: importer (show other bugs)
Version: 5.0.3
Platform: Debian unstable Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-03-08 07:58 UTC by Dominique Dumont
Modified: 2019-07-10 05:58 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In: 5.0.5
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique Dumont 2019-03-08 07:58:53 UTC
SUMMARY

I'm trying to import OFX data. OFX transactions are imported, but not in the selected account.

STEPS TO REPRODUCE
1. Open dialog to import ofx data, select file
2. select Asset -> usual bank account
3. click ok

OBSERVED RESULT

Transaction are imported in "Cartes de Credits La Poste" account

EXPECTED RESULT

Transaction are imported in selected bank account

SOFTWARE/OS VERSIONS
Windows: 
MacOS: 
Linux/KDE Plasma: Debian/unstable, plasma 
(available in About System)
KDE Plasma Version: 5.14.5.1
KDE Frameworks Version: 5.54.0
Qt Version: 5.11.3

ADDITIONAL INFORMATION

Here's an extract from the STDOUT during an import:

Process on: '2019-02-27', id: 'ID PX0ZKOLLT%', amount: '7.09', fees: '0.00'
"Start matching payee" "VIREMENT DE AMAZON PAYMENTS EURO"
Found match with 'VIREMENT DE AMAZON PAYMENTS EURO' on 'VIREMENT'
Found match with 'VIREMENT DE AMAZON PAYMENTS EURO' on 'AMAZON PAYMENTS'
Multiple matches
Adding second split to Loisirs(A000020)
Looking for a match with transaction:  "2019-02-27" , "AMAZON PAYMENTS" , "7.09" (referenced account:  "Cartes de Credits La Poste" )
Considering 0 existing transaction(s) for matching
Looking for a match with transaction:  "2019-02-27" , "AMAZON PAYMENTS" , "7.09" (referenced account:  "Cartes de Credits La Poste" )
Considering 0 schedule(s) for matching the transaction

Note the list of account shown in the first step of import does not show "Cartes de Credits La Poste" (probably because it's a liability account)

All the best
Comment 1 Dominique Dumont 2019-03-08 09:31:18 UTC
Note: this is the first time I use kmyney 5. The kmy file I'm using was written by kmymoney 4.8.1.1. IIRC, the last import operation was using account  "Cartes de Credits La Poste".

HTH
Comment 2 Thomas Baumgart 2019-03-09 07:29:58 UTC
What is the output of the following command run on your <QFX-FILE>

grep ACCT <QFX-FILE>

Is your "Cartes de Credits La Poste" mapped to an online OFX account?
Comment 3 Thomas Baumgart 2019-03-09 07:30:16 UTC
Setup ticket state
Comment 4 Dominique Dumont 2019-03-09 14:34:18 UTC
Hi

I hope you meant to grep the OFX file I get from the bank. 

All transaction of the OFX file are stored on one line, so the grep command is pretty much useless. However, I hope this (formatted and redacted) extract from the OFX file contains the information you need:

<CURDEF>EUR
<BANKACCTFROM>
  <BANKID>20041
  <BRANCHID>01xxx
  <ACCTID>00xxxxxxxx
  <ACCTTYPE>CHECKING
  <ACCTKEY>xx
</BANKACCTFROM>

The "Cartes de Credits La Poste" account is not mapped to an online account. All transactions were imported from OFX files with kmymoney 4

All the best
Comment 5 Bug Janitor Service 2019-03-24 04:33:11 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 6 Thomas Baumgart 2019-03-24 12:35:30 UTC
Sorry, did not see your answer because the status did not change.

Can you do a

   zgrep kmmofx-acc-ref <your-kmy-file>

and check if the value assigned to this key resembles information in your OFX file (should by BANKID and ACCTID)?
Comment 7 Dominique Dumont 2019-03-29 18:06:24 UTC
ok, I've used a copy of a v4 file and imported an ofx file with kmymoney 5.0.3.

But the result of the grep command is empty:

$ ll domi-v5.kmy
-rw------- 1 domi domi 227885 Mar 29 19:00 domi-v5.kmy
$ zgrep kmmofx-acc-ref domi-v5.kmy
$ zgrep -i kmmofx-acc-ref domi-v5.kmy
$ 

All the best
Comment 8 Thomas Baumgart 2019-03-30 06:30:52 UTC
Good, since the parameter is normally only provided when using OFX direct connect but might have caused interference in this case.
Comment 9 Dominique Dumont 2019-04-24 10:10:58 UTC
BTW, ofx import begins with:

OfxImporterPlugin::slotImportFile
setup callback routines
process data
LibOFX INFO: libofx_proc_file(): File format not specified, autodetecting...
LibOFX INFO: libofx_proc_file(): Detected file format: OFX (Open Financial eXchange (OFX or QFX))
LibOFX STATUS: find_dtd():DTD found: /usr/share/libofx7/libofx/dtd/opensp.dcl
LibOFX STATUS: find_dtd():DTD found: /usr/share/libofx7/libofx/dtd/ofx160.dtd
LibOFX INFO: Created OfxDummyContainer to hold unsupported aggregate SIGNONMSGSRSV1
LibOFX INFO: Created OfxDummyContainer to hold unsupported aggregate SONRS
LibOFX INFO: Created OfxDummyContainer to hold unsupported aggregate BANKMSGSRSV1
LibOFX INFO: Created OfxDummyContainer to hold unsupported aggregate STMTTRNRS
process data done
OfxImporterPlugin::storeStatements() with 1 statements called
OfxImporterPlugin::importStatement start
KMyMoneyPlugin::KMMStatementInterface::import start
Importing statement for 'Cartes de Credits La Poste'
Processing transactions (Cartes de Credits La Poste)


I've no idea why the import is done on 'Cartes de Credits La Poste' even though I've selected another account in the import menu that shows account list.
Comment 10 Dominique Dumont 2019-05-05 16:31:47 UTC
I've found that the right account is selected if <ACCTID> tag (with its data) is removed from the OFX file before import.

Hope this helps
Comment 11 Thomas Baumgart 2019-05-11 13:30:44 UTC
This is really strange. Can you provide us with the output of the following commands, please:

  zgrep StatementKey domi-v5.kmy
  zgrep ### domi-v5.kmy

where your replace ### with the information following <ACCTID> that you remove when things work.
Comment 12 Dominique Dumont 2019-05-11 16:19:11 UTC
Well, the information provided by <ACCTID> is my bank account number
which I'm quite reluctant to post on a public forum :-). So this
number is replaced with a dummy value. Hopefully, this will not impact your
investigation of this issue.

And the zgrep command provides 2 other real account numbers which are
also masked (with 1234567b and 1234567c).

This information provided by <ACCTID> is replaced with 1234567a

$ zgrep StatementKey domi-v5.kmy
    <PAIR value="20041 01017 1234567a" key="StatementKey"/>
    <PAIR value="20041 01017 1234567a" key="StatementKey"/>
    <PAIR value="10011 00020 1234567b" key="StatementKey"/>
    <PAIR value="10011 00020 1234567c" key="StatementKey"/>
$ zgrep 1234567a domi-v5.kmy
    <PAIR value="20041 01017 1234567a" key="StatementKey"/>
  <ACCOUNT lastreconciled="" opened="2013-12-14" id="A000063" name="Bank account 1234567a" lastmodified="2019-03-29" number="" type="1" institution="I000001" parentaccount="AStd::Asset" currency="EUR" description="Autogenerated by QIF importer">
    <PAIR value="20041 01017 1234567a" key="StatementKey"/>

All the best
Comment 13 Thomas Baumgart 2019-05-11 18:33:07 UTC
I fully understand your privacy concerns. Thanks for providing the information in obfuscated form, which is helpful.

The problem seems to be that

  <PAIR value="20041 01017 1234567a" key="StatementKey"/>

shows up twice. I looks to me that the second occurrence in your output contains to the <ACCOUNT> element of the account in question. Does the second one show up in the context of the account "Cartes de Credits La Poste"? This would explain what is happening.  I wonder how it got there, though?
Comment 14 Jack 2019-05-17 19:35:08 UTC
As I follow this, all imports have been of downloaded files, and not direct connect.  Is it possible a file was once imported to the wrong account, and the statement key was associated with that account?  Is there any way to clear this pair through the GUI?  
(I'm changing status to be sure Thomas notices, since I suspect Dominique doesn't have an answer to the question of how the info got associated with the "wrong" account.)
Comment 15 Dominique Dumont 2019-05-18 08:05:28 UTC
Hello

Yes, the "StatementKey" also appears in the context of "Cartes de Credits La Poste":

$ zgrep -B 3 0048775Z028 domi-v5.kmy
  <ACCOUNT lastreconciled="" opened="2013-12-15" id="A000062" name="Cartes de Credits La Poste" lastmodified="2019-03-29" number="" type="10" institution="I000001" parentaccount="A000060" currency="EUR" description="">
   <KEYVALUEPAIRS>
    <PAIR value="" key="IBAN"/>
    <PAIR value="20041 01017 1234567a" key="StatementKey"/>
--
    <PAIR value="1160321/50" key="lastStatementBalance"/>
   </KEYVALUEPAIRS>
  </ACCOUNT>
  <ACCOUNT lastreconciled="" opened="2013-12-14" id="A000063" name="Bank account 1234567a" lastmodified="2019-03-29" number="" type="1" institution="I000001" parentaccount="AStd::Asset" currency="EUR" description="Autogenerated by QIF importer">
   <KEYVALUEPAIRS>
    <PAIR value="20041 01017 1234567a" key="StatementKey"/>

Jack, Yes, the statements are imported manually with ofx files. And I've created manually the accounts. 

Thomas, It's possible that I've created the "Cartes de Credits La Poste" account using the same number as the main account: On "labanquepostale" site, the credit card transactions are associated to the main account and I did not realize that this number is key in kmymoney internal data structure.

Should I manually change this number in my .kmy file  ?

All the best
Comment 16 Dominique Dumont 2019-06-28 17:23:22 UTC
And now I have the opposite problem: once <ACCTID> tag is removed, all transactions go the main bank account. I cannot import in the Credit card account.

Assuming there's something wrong with my kmy file, do you have an idea how I can get out of this mess ?

All the best
Comment 17 Thomas Baumgart 2019-06-29 11:58:24 UTC
This sounds very similar to a problem we had with the AqBanking importer. This was handled in bug #408494. In fact, this could be the identical problem.
Comment 18 Thomas Baumgart 2019-06-29 12:16:50 UTC
Git commit ab628d659ebcbea9265b62e8dc419c3bb20bbcb7 by Thomas Baumgart.
Committed on 29/06/2019 at 12:16.
Pushed by tbaumgart into branch '5.0'.

Use mapped account only if information is present

Using the OFX (file) importer may not provide information about the
account the imported data relates to. In this case, it does not make
sense to search for an online mapping. Doing so leads to false results,
as without values for account or routing number the first mapped account
will be selected which is wrong.

This change does not try the mapping if both values are empty and allows
the user to select the account as part of the process.

M  +4    -1    kmymoney/plugins/ofx/import/ofximporter.cpp

https://commits.kde.org/kmymoney/ab628d659ebcbea9265b62e8dc419c3bb20bbcb7
Comment 19 Dominique Dumont 2019-07-05 16:23:03 UTC
Hi

I've been testing kmymoney from 5.0 branch. The only change I've seen (although I might be wrong) is ofx importer proposes a different account when <ACCTID> is present or not:
- with <ACCTID> importer proposes "cartes de crédit"
- without <ACCTID>, importer proposes main account

Unfortunately, the ofx importer ignores the target account choice and imports data in the wrong (or right) account depending on whether <ACCTID> is present or not.

All the best
Comment 20 Thomas Baumgart 2019-07-06 13:19:17 UTC
Clear out the BANKID field as well. Only if both fields are empty you will be asked for a manual account assignment.