Bug 313237 - No payee matching with aqbanking import (HBCI)
Summary: No payee matching with aqbanking import (HBCI)
Status: RESOLVED INTENTIONAL
Alias: None
Product: kmymoney
Classification: Applications
Component: general (show other bugs)
Version: 4.6.3
Platform: openSUSE Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-01-14 13:30 UTC by Maarten
Modified: 2014-02-20 11:08 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
aqbank.txt (967 bytes, text/plain)
2013-01-26 13:11 UTC, Maarten
Details
KBanking patch to keep line breaks in purpose during regexp evaluation (824 bytes, patch)
2013-03-21 14:56 UTC, Christian David
Details
Keeps new lines in purpose during regexp evaluation and sets hash as before (1.26 KB, patch)
2013-03-24 11:34 UTC, Christian David
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Maarten 2013-01-14 13:30:46 UTC
I import my bank account ledgers with the aqbanking interface. However, payees defined in the payee view are not recognized at all. 

For testing I downloaded a csv file of the same transaction period and the payees were recognized correctly.

Since the manual states "When you import transactions using the QIF import, or using a plugin such as OFX or AqBanking, the Match Keys you have set will be considered.", I assume this is a bug or missing feature.

Cheers,
Maarten

Reproducible: Always

Steps to Reproduce:
1. define payees (known to work)
2. import via aqbanking (HBCI)
3. check ledger view for recognized payees
Comment 1 Christian David 2013-01-23 09:31:05 UTC
Is the payee field empty?

If the first answer is "Yes": Some banks do not fully comply with the HBCI standard and use thier own dialect, then aqbanking is not able to detect the field for the payee and so it is not able to pass it to KMyMoney (resulting in empty payee fields). This is a missing feature of aqbanking.
Comment 2 Thomas Baumgart 2013-01-25 08:07:07 UTC
To further nail down the problem, we need to know which method your bank uses. An indication on which bank you connect to is also helpful. How do the lines starting with ':86:' look like in the aqbanking logs? You can find them at "~/.aqbanking/backends/aqhbci/data/banks/de/<BLZ>/logs and extract them with 'grep :86: <name-of-log-file>'. Please make sure to replace sensitive information with 'X' for char and '#' for digits before you present them here.

BTW: KMyMoney has some built-in features to work around the format problem.
Comment 3 Maarten 2013-01-25 08:51:41 UTC
Thank you both for answering.

I will look after this and mail again.

Maarten

On 01/25/2013 09:07 AM, Thomas Baumgart wrote:
> https://bugs.kde.org/show_bug.cgi?id=313237
>
> --- Comment #2 from Thomas Baumgart <ipwizard@users.sourceforge.net> ---
> To further nail down the problem, we need to know which method your bank uses.
> An indication on which bank you connect to is also helpful. How do the lines
> starting with ':86:' look like in the aqbanking logs? You can find them at
> "~/.aqbanking/backends/aqhbci/data/banks/de/<BLZ>/logs and extract them with
> 'grep :86: <name-of-log-file>'. Please make sure to replace sensitive
> information with 'X' for char and '#' for digits before you present them here.
>
> BTW: KMyMoney has some built-in features to work around the format problem.
>
Comment 4 Maarten 2013-01-26 13:11:50 UTC
Created attachment 76735 [details]
aqbank.txt

Hi,

I attached one ledger which is not matched. The regular expression I
look for is "AUFLADUNG ####-########" where the #'s denote a phone number.

I hope from that is already seen what is going wrong, if not, I will
send more information.

Regards,
Maarten



On 01/25/2013 09:07 AM, Thomas Baumgart wrote:
> https://bugs.kde.org/show_bug.cgi?id=313237
>
> --- Comment #2 from Thomas Baumgart <ipwizard@users.sourceforge.net> ---
> To further nail down the problem, we need to know which method your bank uses.
> An indication on which bank you connect to is also helpful. How do the lines
> starting with ':86:' look like in the aqbanking logs? You can find them at
> "~/.aqbanking/backends/aqhbci/data/banks/de/<BLZ>/logs and extract them with
> 'grep :86: <name-of-log-file>'. Please make sure to replace sensitive
> information with 'X' for char and '#' for digits before you present them here.
>
> BTW: KMyMoney has some built-in features to work around the format problem.
>
Comment 5 Christian David 2013-02-28 20:36:47 UTC
Someone else had the same problem before: https://www.willuhn.de/bugzilla/show_bug.cgi?id=887
Comment 6 Christian David 2013-03-21 14:56:02 UTC
Created attachment 78267 [details]
KBanking patch to keep line breaks in purpose during regexp evaluation

The added line breaks are removed after evaluating the regexp, so most user should not notice any difference.
Comment 7 Christian David 2013-03-21 15:08:31 UTC
Hi, with that patch the regular expression

^(?:Ueberweisung|SEPA-Ueberweisung|Dauerauftrag) an[^\n]*\n([^\n]+)\n

is able to extract the payee in a few cases (transfers initiated by you, as you can read from the expression). Unforunatly more is not possible for accounts at Deutsche Bank as all other transactions do not follow a rule a program could use to extract the payee.

The transfer descibed in aqbank.txt is a debit note (Lastschrift) the described problem prevents automatical detection here.

For memo you should use (.*) (with brackets!) as regexp, then the memo stays unchanged compared to earlier imports in all cases. The patch as well as the regexp are tested.

By the way: I can confim this bug.
Comment 8 Thomas Baumgart 2013-03-24 08:17:54 UTC
Christian writes: "The added line breaks are removed after evaluation of the regexp". I don't currently see where this happens. Maybe, you can point me in the right direction. Nevertheless, changing the string will certainly change the hash that is used for duplicate detection. So adding it as simple as proposed in comment 6 might cause trouble for users. Changing

  h = MyMoneyTransaction::hash(s.trimmed(), h);

into

  h = MyMoneyTransaction::hash(s.replace("\n", " ").trimmed(), h);

could solve this problem. Please give it a try.
Comment 9 Christian David 2013-03-24 11:34:03 UTC
Created attachment 78344 [details]
Keeps new lines in purpose during regexp evaluation and sets hash as before

You are right with the hash, Thomas. I did this test:

1) Used currend git master to receive many transactions. Saved filed and quit.

2) Used my older patch and downloaded the transactions again (actually even more) by manualy setting the start-date. KMyMoney could not detect dublicates, they were listed twice :( .

3) I added your line (this patch), loaded file from (1) and downloaded the transactions again. Now all dublicates were detected. Works as desired.

Before importing the transactions the memo and payee field QString::simplified() [1] is used. This function removes any whitespace character including newlines with a space. So behaviour after the regexp is as before.

Btw: It should be considerd that removing the newlines in the Memo is not a preferable behaviour. But for now changing this is uncomfortable as in ledger view the transfer form at the bottom is not scrolable if not in editing mode.

[1] http://qt-project.org/doc/qt-4.8/qstring.html#simplified
Comment 10 Christian David 2013-12-07 21:05:42 UTC
Git commit c836c3dec4509fb229cf2f71e31afe23512dac4e by Christian Dávid.
Committed on 07/12/2013 at 17:02.
Pushed by christiand into branch 'add-onlinebanking'.

Started adding onlinebanking including sending credit transfers

Started to add a system for creating jobs which can be executed by a
plugin.

Tasks for german and sepa credit transfers were ceated including ui.
KBanking (aqbanking) can use them to create credit transfers.

This is work in progress.
Related: bug 238105

M  +3    -0    .gitignore
M  +3    -1    CMakeLists.txt
M  +6    -5    kmymoney.doxygen.in
M  +5    -3    kmymoney/CMakeLists.txt
M  +6    -0    kmymoney/dialogs/CMakeLists.txt
A  +198  -0    kmymoney/dialogs/konlinetransferform.cpp     [License: GPL (v2+)]
A  +120  -0    kmymoney/dialogs/konlinetransferform.h     [License: GPL (v2+)]
A  +254  -0    kmymoney/dialogs/konlinetransferformdecl.ui
M  +131  -2    kmymoney/kmymoney.cpp
M  +40   -0    kmymoney/kmymoney.h
M  +5    -0    kmymoney/models/CMakeLists.txt
M  +1    -1    kmymoney/models/accountsmodel.cpp
M  +15   -1    kmymoney/models/models.cpp
M  +2    -0    kmymoney/models/models.h
A  +139  -0    kmymoney/models/onlinejobmodel.cpp     [License: UNKNOWN]  *
A  +49   -0    kmymoney/models/onlinejobmodel.h     [License: UNKNOWN]  *
M  +64   -7    kmymoney/mymoney/CMakeLists.txt
A  +21   -0    kmymoney/mymoney/accountidentifier.cpp     [License: GPL (v2+)]
A  +65   -0    kmymoney/mymoney/accountidentifier.h     [License: GPL (v2+)]
A  +1    -0    kmymoney/mymoney/bankaccountidentifier.cpp     [License: Trivial file]
A  +53   -0    kmymoney/mymoney/bankaccountidentifier.h     [License: UNKNOWN]  *
A  +114  -0    kmymoney/mymoney/germanaccountidentifier.cpp     [License: UNKNOWN]  *
A  +61   -0    kmymoney/mymoney/germanaccountidentifier.h     [License: UNKNOWN]  *
A  +161  -0    kmymoney/mymoney/germanonlinetransfer.cpp     [License: GPL (v2+)]
A  +154  -0    kmymoney/mymoney/germanonlinetransfer.h     [License: GPL (v2+)]
M  +69   -0    kmymoney/mymoney/mymoneyfile.cpp
M  +47   -1    kmymoney/mymoney/mymoneyfile.h
M  +156  -0    kmymoney/mymoney/mymoneyfiletest.cpp
M  +7    -0    kmymoney/mymoney/mymoneyfiletest.h
M  +8    -0    kmymoney/mymoney/mymoneyobjectcontainer.cpp
M  +4    -0    kmymoney/mymoney/mymoneyobjectcontainer.h
A  +174  -0    kmymoney/mymoney/onlinejob.cpp     [License: GPL (v2+)]
A  +289  -0    kmymoney/mymoney/onlinejob.h     [License: GPL (v2+)]
A  +213  -0    kmymoney/mymoney/onlinejobadministration.cpp     [License: UNKNOWN]  *
A  +134  -0    kmymoney/mymoney/onlinejobadministration.h     [License: UNKNOWN]  *
A  +50   -0    kmymoney/mymoney/onlinejobadministrationtest.cpp     [License: UNKNOWN]  *
A  +24   -0    kmymoney/mymoney/onlinejobadministrationtest.h     [License: UNKNOWN]  *
A  +5    -0    kmymoney/mymoney/onlinejobfolder.cpp     [License: UNKNOWN]  *
A  +42   -0    kmymoney/mymoney/onlinejobfolder.h     [License: UNKNOWN]  *
A  +113  -0    kmymoney/mymoney/onlinejobknowntask.h     [License: UNKNOWN]  *
A  +49   -0    kmymoney/mymoney/onlinejobknowntasktest.cpp     [License: UNKNOWN]  *
A  +20   -0    kmymoney/mymoney/onlinejobknowntasktest.h     [License: UNKNOWN]  *
A  +9    -0    kmymoney/mymoney/onlinejobmessage.cpp     [License: UNKNOWN]  *
A  +73   -0    kmymoney/mymoney/onlinejobmessage.h     [License: UNKNOWN]  *
A  +57   -0    kmymoney/mymoney/onlinejobtest.cpp     [License: UNKNOWN]  *
A  +21   -0    kmymoney/mymoney/onlinejobtest.h     [License: UNKNOWN]  *
A  +28   -0    kmymoney/mymoney/onlinetask.cpp     [License: UNKNOWN]  *
A  +240  -0    kmymoney/mymoney/onlinetask.h     [License: UNKNOWN]  *
A  +32   -0    kmymoney/mymoney/onlinetransfer.cpp     [License: GPL (v2+)]
A  +71   -0    kmymoney/mymoney/onlinetransfer.h     [License: GPL (v2+)]
A  +116  -0    kmymoney/mymoney/sepaonlinetransfer.cpp     [License: UNKNOWN]  *
A  +96   -0    kmymoney/mymoney/sepaonlinetransfer.h     [License: UNKNOWN]  *
M  +2    -1    kmymoney/mymoney/storage/CMakeLists.txt
M  +3    -0    kmymoney/mymoney/storage/imymoneyserialize.h
M  +34   -0    kmymoney/mymoney/storage/imymoneystorage.h
M  +10   -0    kmymoney/mymoney/storage/mymoneydatabasemgr.h
M  +60   -0    kmymoney/mymoney/storage/mymoneyseqaccessmgr.cpp
M  +38   -0    kmymoney/mymoney/storage/mymoneyseqaccessmgr.h
M  +21   -0    kmymoney/mymoney/storage/mymoneyseqaccessmgrtest.cpp
M  +1    -0    kmymoney/mymoney/storage/mymoneyseqaccessmgrtest.h
A  +2    -0    kmymoney/mymoney/swiftaccountidentifier.cpp     [License: UNKNOWN]  *
A  +39   -0    kmymoney/mymoney/swiftaccountidentifier.h     [License: UNKNOWN]  *
M  +16   -5    kmymoney/plugins/CMakeLists.txt
M  +9    -0    kmymoney/plugins/kbanking/CMakeLists.txt
A  +52   -0    kmymoney/plugins/kbanking/aqbankingkmmoperators.cpp     [License: UNKNOWN]  *
A  +44   -0    kmymoney/plugins/kbanking/aqbankingkmmoperators.h     [License: UNKNOWN]  *
A  +28   -0    kmymoney/plugins/kbanking/aqbankingqtoperators.h     [License: UNKNOWN]  *
M  +6    -1    kmymoney/plugins/kbanking/dialogs/CMakeLists.txt
A  +1    -0    kmymoney/plugins/kbanking/gwenhywfarqtoperatiors.cpp     [License: Trivial file]
A  +21   -0    kmymoney/plugins/kbanking/gwenhywfarqtoperators.cpp     [License: UNKNOWN]  *
A  +29   -0    kmymoney/plugins/kbanking/gwenhywfarqtoperators.h     [License: UNKNOWN]  *
M  +325  -36   kmymoney/plugins/kbanking/mymoneybanking.cpp
M  +33   -6    kmymoney/plugins/kbanking/mymoneybanking.h
M  +5    -0    kmymoney/plugins/kbanking/views/CMakeLists.txt
M  +4    -2    kmymoney/plugins/kbanking/views/kbjobview.cpp
M  +4    -1    kmymoney/plugins/kbanking/views/kbjobview.h
M  +1    -0    kmymoney/plugins/kmymoneyplugin.h
A  +11   -0    kmymoney/plugins/onlinepluginextended.cpp     [License: UNKNOWN]  *
A  +69   -0    kmymoney/plugins/onlinepluginextended.h     [License: UNKNOWN]  *
M  +2    -0    kmymoney/views/CMakeLists.txt
M  +9    -0    kmymoney/views/kmymoneyview.cpp
M  +5    -1    kmymoney/views/kmymoneyview.h
A  +79   -0    kmymoney/views/konlinejoboutbox.cpp     [License: UNKNOWN]  *
A  +35   -0    kmymoney/views/konlinejoboutbox.h     [License: UNKNOWN]  *
A  +93   -0    kmymoney/views/konlinejoboutbox.ui
M  +11   -0    kmymoney/widgets/CMakeLists.txt
A  +4    -0    kmymoney/widgets/bankidentifieredit.h     [License: UNKNOWN]  *
M  +4    -0    kmymoney/widgets/kmymoney.widgets
A  +92   -0    kmymoney/widgets/onlinejobedit/germancredittransferedit.cpp     [License: UNKNOWN]  *
A  +43   -0    kmymoney/widgets/onlinejobedit/germancredittransferedit.h     [License: UNKNOWN]  *
A  +176  -0    kmymoney/widgets/onlinejobedit/germancredittransferedit.ui
A  +4    -0    kmymoney/widgets/onlinejobedit/germancredittransferedit_p.h     [License: UNKNOWN]  *
A  +135  -0    kmymoney/widgets/onlinejobedit/sepacredittransferedit.cpp     [License: UNKNOWN]  *
A  +35   -0    kmymoney/widgets/onlinejobedit/sepacredittransferedit.h     [License: UNKNOWN]  *
A  +173  -0    kmymoney/widgets/onlinejobedit/sepacredittransferedit.ui
A  +32   -0    kmymoney/widgets/onlinejobedit/sepacredittransferedit_p.h     [License: UNKNOWN]  *

The files marked with a * at the end have a non valid license. Please read: http://techbase.kde.org/Policies/Licensing_Policy and use the headers which are listed at that page.


http://commits.kde.org/kmymoney/c836c3dec4509fb229cf2f71e31afe23512dac4e
Comment 11 Christian David 2014-02-06 17:28:01 UTC
Shall we mark this bug as "won't fix"? We have no chance to solve this (see comment 7).
Comment 12 Christian David 2014-02-20 11:08:00 UTC
Cannot be fixed in KMyMoney as it is an issue of the bank. (see comment #7)