Bug 315072

Note: This appears to be an issue with the OFX data sent from the bank.  When comparing the OFX log dumps from either of them, the only difference is that the duplicate OFX log has an extra "<CORRECTACTION>REPLACE" tag in each of the transactions.

According to the OFX specification, "<CORRECTACTION>" fields are only present with a preceding "<CORRECTFITID>" field.  This is an out of specification field that's being sent by the bank.

However, the correct action in this situation would probably be for KMyMoney to ignore a "<CORRECTACTION>" field without a corresponding "<CORRECTFITID>" field.

Could you attach a test OFX file for this? Otherwise I don't see how I could fix it.

Is there a way to get an OFX log for KMyMoney on Windows?  I'm using version 4.7.0 on Windows 7 now.

(In reply to Mike Wolfe from comment #3)
> Is there a way to get an OFX log for KMyMoney on Windows?  I'm using version
> 4.7.0 on Windows 7 now.

It should work the same way as on Linux (the $HOME is %USERPROFILE% on Windows) so just create the file "%USERPROFILE%\ofxlog.txt".

Created attachment 89008 [details]
OFX text log for non-matching transaction

This is a sanitized (hopefully) OFX log file for a single imported transaction that does not match to an entered one.  If manually matched and accepted, updating using OFX direct connect will import another transaction just like it that is unmatched.  Ideally, the new transaction would be considered a "duplicate" and not imported again.

This seems to happen to every transaction in this account in version 4.7.0, though other accounts are working correctly.

I'll take a look using the provided data.

Created attachment 89038 [details]
The ledger after importing twice

I've tested this on Windows importing the OFX data twice, the transaction is matched with the old entry.

Could you take a look at the attached picture, I don't see the transaction being created twice, or at least it's matched with the previous transaction. Could you create a KMyMoney file from scratch, try to import the response you attached and see if it still happens? If it does could you send us that file?

How do I import the log?  I made a new KMyMoney file from scratch, set up a default account, renamed the log as a .ofx file, and when I import it gets zero transactions.  Is that the right way to do it?

Hmm, even stranger.  I have a new OFX download, which includes two non-matching transactions and one that matches; technically, all three are wrong because all of them should be considered "duplicates", but maybe there's some useful information there?

I attached a new sanitized OFX log, hopefully it's helpful...

Created attachment 89072 [details]
OFX text log with two non-matching and one matching transaction

You can import the data from the log by copying everything between "response:" and "Completed" in a file.ofx and import that file.

I am having the same problem as Mike.  Transactions are not matching even though match on Payee name has several permutations of names that should provide a match.   I have attached a (fairly large) qfx file and a screen shot that shows some transactions that matched well under 4.6.4.  Unfortunately, I had previously deleted some of the original transactions that did not get a match.

Created attachment 89187 [details]
Wschaub Ledger

Screen shot of ledger

Created attachment 89188 [details]
WSchaub  test qfx file

A little large, sorry.

Need to investigate this information.

Thank you for you help and to you & the team for KMyMoney.

I tried the attached files and based on the attached ledger screenshot I guess that the only transactions which are not matched are the ones that have different and not empty payees. The payee criteria was added in this change https://git.reviewboard.kde.org/r/109913/. 

Could you check if you have payee matching properly setup?

After setting up rules so that the imported transaction will get the proper payee (the same as the one that's entered) matching works.
Another possibility to make sure the transactions match is to remove the payee from the entered transactions before importing.

Could you check if this is true in your case?

Bank's online register ( a Citgo gas station): POS PURCHASE - FOOD BAG #517 BETHEL CT 5740 00464289624563243

For the Payee "Citgo" it is set to match on the name below (ignore case is checked):
Citgo
FOOD BAG

Banks online register (a Gulf gas station):
POS PURCHASE - ASSOCIATED BUSINESS BETHEL CT 8471 00384288814155842

For the payee "Gulf" (ignore case is checked):
POS PURCHASE - NESS BETHEL CT
POS PURCHASE - ASSOCIATED BUSINESS BETHEL
ASSOCIATED BUSINESS BETHEL

(In reply to warren_schaub from comment #19)
> Bank's online register ( a Citgo gas station): POS PURCHASE - FOOD BAG #517
> BETHEL CT 5740 00464289624563243
> 
> For the Payee "Citgo" it is set to match on the name below (ignore case is
> checked):
> Citgo
> FOOD BAG

For this to work you need to delete the existing payee 'FOOD BAG' otherwise it will be matched first.

> 
> Banks online register (a Gulf gas station):
> POS PURCHASE - ASSOCIATED BUSINESS BETHEL CT 8471 00384288814155842
> 
> For the payee "Gulf" (ignore case is checked):
> POS PURCHASE - NESS BETHEL CT
> POS PURCHASE - ASSOCIATED BUSINESS BETHEL
> ASSOCIATED BUSINESS BETHEL

In the screenshot the payee 'CIA RESTAURANTS' is the payee which differs from 'Gulf'.

OK, deleting the payee "Food Bag" fixed that match.

I do have a payee "CIA Restaurants" which has a name match to CIA only.  (Culinary Institute of America, I'm not a spy!  :-)   But I don't know why KMyMoney would plug that payee in there.  That (plugging CIA Restaurants in to Payee) has happened with previous releases of KMyMoney too.

If I want more extensive matching do I need to set up more name match parameters relative to 4.6.4?  That's no problem.  Thanks

(In reply to warren_schaub from comment #21)
> OK, deleting the payee "Food Bag" fixed that match.
Nice,  it seems we're getting somewhere now.

> 
> I do have a payee "CIA Restaurants" which has a name match to CIA only. 
> (Culinary Institute of America, I'm not a spy!  :-)   But I don't know why
> KMyMoney would plug that payee in there.  That (plugging CIA Restaurants in
> to Payee) has happened with previous releases of KMyMoney too.
Because the name in the OFX file is "ASSOCIATED BUSI" which contains "CIA". "CIA" is a pretty short substring so you should work on this match setup. 

> 
> If I want more extensive matching do I need to set up more name match
> parameters relative to 4.6.4?  That's no problem.  Thanks

It seems that that's what you need to do. Mike could you confirm if payee mismatch is causing the match failure in your case also?

OK, after adding new matching names and cleaning up some old matching names and payees matching using 4.7 seems to be working well now.   Now, I'm on to testing the csv import with investments!  Thanks

>> If I want more extensive matching do I need to set up more name match
>> parameters relative to 4.6.4?  That's no problem.  Thanks
> It seems that that's what you need to do. Mike could you confirm if payee
> mismatch is causing the match failure in your case also?
>
Sorry, changing the matching name does not fix the problem with mine.  
Selecting "match on payee name" with my payee name being "Nicor Gas" and 
the downloaded name "NICOR GAS ONLINE PMT 141014" doesn't fix it.

I think we might be talking about two separate problems, too; generally, 
I haven't needed to enable matching by name in Payees. In the past, 
KMyMoney has been able to match a downloaded transaction for exactly the 
same amount as a transaction in the register without having to specify a 
matching name.

By the way, thanks for the help!

-Mike

(In reply to Mike Wolfe from comment #24)
> >> If I want more extensive matching do I need to set up more name match
> >> parameters relative to 4.6.4?  That's no problem.  Thanks
> > It seems that that's what you need to do. Mike could you confirm if payee
> > mismatch is causing the match failure in your case also?
> >
> Sorry, changing the matching name does not fix the problem with mine.  
> Selecting "match on payee name" with my payee name being "Nicor Gas" and 
> the downloaded name "NICOR GAS ONLINE PMT 141014" doesn't fix it.
> 

So you are saying that you have an entered transaction for "Nicor Gas" with amount 10.15 (for example) and importing a transaction with payee "NICOR GAS ONLINE PMT 141014" with amount 10.15 will not match the two?

> I think we might be talking about two separate problems, too; generally, 
> I haven't needed to enable matching by name in Payees. In the past, 
> KMyMoney has been able to match a downloaded transaction for exactly the 
> same amount as a transaction in the register without having to specify a 
> matching name.

That was the case in previous versions, in the current version if payees are present in both entered and imported transactions they should match.

> 
> By the way, thanks for the help!
> 
> -Mike

On 10/19/2014 1:31 AM, Cristian Oneț wrote:
> https://bugs.kde.org/show_bug.cgi?id=315072
>
> --- Comment #25 from Cristian Oneț <onet.cristian@gmail.com> ---
> (In reply to Mike Wolfe from comment #24)
>>>> If I want more extensive matching do I need to set up more name match
>>>> parameters relative to 4.6.4?  That's no problem.  Thanks
>>> It seems that that's what you need to do. Mike could you confirm if payee
>>> mismatch is causing the match failure in your case also?
>>>
>> Sorry, changing the matching name does not fix the problem with mine.
>> Selecting "match on payee name" with my payee name being "Nicor Gas" and
>> the downloaded name "NICOR GAS ONLINE PMT 141014" doesn't fix it.
>>
> So you are saying that you have an entered transaction for "Nicor Gas" with
> amount 10.15 (for example) and importing a transaction with payee "NICOR GAS
> ONLINE PMT 141014" with amount 10.15 will not match the two?

Yes, that's exactly it.  In this case, the transaction for "Nicor Gas" 
is one that I had previously matched and accepted.  Updating the account 
again creates a new yellow transaction with the name "NICOR GAS ONLINE 
PMT 141014", when what it should do is detect it as a "duplicate" and 
not make a new one at all.
>> I think we might be talking about two separate problems, too; generally,
>> I haven't needed to enable matching by name in Payees. In the past,
>> KMyMoney has been able to match a downloaded transaction for exactly the
>> same amount as a transaction in the register without having to specify a
>> matching name.
> That was the case in previous versions, in the current version if payees are
> present in both entered and imported transactions they should match.
>
>> By the way, thanks for the help!
>>
>> -Mike

(In reply to Cristian Oneț from comment #18)
> I tried the attached files and based on the attached ledger screenshot I
> guess that the only transactions which are not matched are the ones that
> have different and not empty payees. The payee criteria was added in this
> change https://git.reviewboard.kde.org/r/109913/. 

Cristian, I haven't introduced payee matching criteria after all - it's been there as a proposal for quite some time and I recently discarded it after Christian's comment on likely "collision" with HBCI.

As far as I know (but haven't checked in the code) there were no changes in transaction matching code for a long time and the payee is NOT a criteria for matching.

(In reply to Łukasz Maszczyński from comment #27)
> (In reply to Cristian Oneț from comment #18)
> > I tried the attached files and based on the attached ledger screenshot I
> > guess that the only transactions which are not matched are the ones that
> > have different and not empty payees. The payee criteria was added in this
> > change https://git.reviewboard.kde.org/r/109913/. 
> 
> Cristian, I haven't introduced payee matching criteria after all - it's been
> there as a proposal for quite some time and I recently discarded it after
> Christian's comment on likely "collision" with HBCI.
> 
> As far as I know (but haven't checked in the code) there were no changes in
> transaction matching code for a long time and the payee is NOT a criteria
> for matching.

My reference to the review request is wrong, I was referring to this line:
https://projects.kde.org/projects/extragear/office/kmymoney/repository/revisions/master/entry/kmymoney/converter/transactionmatchfinder.cpp#L122

It seems that payee is a criteria for matching after all. Since 4.7.0, the first version containing the rewritten matching mechanism, was released there have been reports of matches which no longer work as in previous versions.

See this thread:
https://forum.kde.org/viewtopic.php?f=69&t=123443

Łukasz could you please join this investigation so that we can help the user's setup their data to work with the stricter matching?

*** Bug 340471 has been marked as a duplicate of this bug. ***

Found the commit that adds the payee as a match criteria https://projects.kde.org/projects/extragear/office/kmymoney/repository/revisions/597c2b8ba69cd2cc9edef9c25fd7e414642ead8f
it went trough this review https://git.reviewboard.kde.org/r/107759/ so it seems the current behaviour is correct. The users that have troubles matching transactions need to make sure that:
1) Both imported and existing transaction have the same payee (imported payee can be customized using payee matching)
or
2) The payee of the imported or the existing transaction is empty

Please let us know if this fixes your import problems.

Yes, this seems to be the case.  That is, more 'matching payee names' are needed than previously but once those are set up the matching seems to work as expected.  I think the 'match payee from beginning' setting (or something like that, I'm not at my PC now)  needs to be taken in to account when setting up matching payee names.

Then I'm closing this as it works as expected.

I'm not sure if the behavior is what is expected; I don't think that warren_schaub's problem is the same as the one I'm having.

I'm exclusively using OFX direct download now.  ONE of my accounts will import automatically and attempt to match transactions repeatedly, even after they've already been accepted.  A different account will automatically and correctly ignore transactions that have already been imported that way.

This seems to me like two separate issues.  The first is the one that warren_schaub mentioned where matching works differently than it used to; I may or may not like the new way that it works, but I think that there's some unexpected behavior when using the "Payee's name is based on the contents of the OFX tag" option as anything other than "PAYEEID". (Edit->Edit Account->Online Settings->Import Details).

The second is that automatic download will sometimes (i.e. on some accounts, possibly due to differences in OFX formats) not recognize transactions as duplicates.  This may be compensated for by matching, but on other accounts the already imported transactions are listed as "duplicates" in the OFX direct download report.  This issue is still outstanding.

Sorry Mike, I did not mean to hi-jack your issue.

I do note that check number does not seem to be part of the matching algorithm.  Since many checks are hand-written the bank (at least Wells-Fargo) will populate the Name field with "CHECK".  So, I don't see how a match can ever happen.  Should I submit an enhancement to include check number in the matching algorithm?  Maybe as the primary matching criterion.

<STMTTRN>
						<TRNTYPE>CHECK
						<DTPOSTED>20141028120000.000
						<TRNAMT>-90.00
						<FITID>201410285
						<CHECKNUM>1217
						<NAME>CHECK
					</STMTTRN>

Today I imported my transactions from my bank account and the algorithm was neither able to match the imported transactions nor did it recognize duplikates. It worked pretty much better with release 4.6.

(In reply to Florian from comment #36)
> Today I imported my transactions from my bank account and the algorithm was
> neither able to match the imported transactions nor did it recognize
> duplikates. It worked pretty much better with release 4.6.

OK, I get that, but the above statement isn't really helpful to figure out why matching does not work. 

Duplicate detection works the following way:
- amounts must match (considering variation for schedules)
- bankids must match or the existing transaction must have an empty bankid

The payee was added as a matching criteria to avoid a wrong match scenario described here https://git.reviewboard.kde.org/r/107759/.

Could you apply these conditions to your data and report back why is the matching and duplicate detection failing?

(In reply to Cristian Oneț from comment #37)
> - bankids must match or the existing transaction must have an empty bankid
correction: "bankids must match and the imported transaction's bankid must not be empty "

This is the original refactoring review request https://git.reviewboard.kde.org/r/107137 and from what I can see duplicates detection was not changed at all.

More information:

In the account I'm having problems with, I have a single transaction downloading that is not detected as a duplicate.  First time I download it, it matches (turns green) with an existing transaction that I imported previously.  If I accept and download again, the match happens again.  If I do NOT accept and download again, a new unmatched transaction is created in addition to the original transaction (which is still green since I haven't accepted it).

The transaction in the OFX log file seems fairly benign.  I can confirm that the BANKID field and amount are the same across multiple downloads, so if there's a mismatch problem it's not due to the OFX message that KMyMoney is receiving.  

The only oddness that I think might be an issue is the  name of the payee:

          <STMTTRN>
            <TRNTYPE>POS
            <DTPOSTED>20141024120000
            <TRNAMT>-125.08
            <FITID>xxxx
            <CORRECTACTION>REPLACE
            <NAME>WOODMAN'S FOOD MRKT #03 ROCKFORD
            <MEMO>WOODMAN'S FOOD MRKT #03 ROCKFORD

Is the '#' some sort of escape sequence in the comparison being used?  It doesn't seem to be an issue in any of the other accounts, though.

What might be relevant too is that this account is the only one that has formatted line spacing like the text shows (i.e. "LF" at the end of each spaced line).  Could there be an issue with "LF" not being recognized properly on a non-UNIX-y system as it's expecting a "CR LF" instead?  This has bitten me in the past before

With the indentical bankid it worked. I left the field of the comming planned transactions blank and I will see what will happen with the next transactions

Any news on this?  Is there any information that I can provide to help troubleshoot the issue(s)?

(In reply to Mike Wolfe from comment #42)
> Any news on this?  Is there any information that I can provide to help
> troubleshoot the issue(s)?

Mike, since you reopened this issue could you provide information on why? Do you still have troubles matching transactions? If so please pick one such transaction that you think should match but it doesn't and let's go trough your settings together so we can finally figure out where is the problem if there is any.

I'm marking this as needs info until we do that.

Sorry, I haven't been able to see the matching problem consistently; it's still there, but it's hard to find examples.  

I did find one thing that might be a clue: In the "Payees" section, for each particular payee there's a "Matching" tab, and a setting for "Match on a name listed below", with a configurable list of names.  This didn't work the last time I imported, even though the payee name and the amount of the transaction were the same. 

Could there be a mismatch between this setting and the one for online update where you can use PAYEEID, NAME, or MEMO fields as the imported name?

(In reply to Mike Wolfe from comment #45)
> Sorry, I haven't been able to see the matching problem consistently; it's
> still there, but it's hard to find examples.  

No problem.

> I did find one thing that might be a clue: In the "Payees" section, for each
> particular payee there's a "Matching" tab, and a setting for "Match on a
> name listed below", with a configurable list of names.  This didn't work the
> last time I imported, even though the payee name and the amount of the
> transaction were the same. 

This is probably what is causing you problems, we need to figure out why this matching is not working.

> Could there be a mismatch between this setting and the one for online update
> where you can use PAYEEID, NAME, or MEMO fields as the imported name?

Or you might have expressions which match other payees, another similar problem in this area is that a payee has a match with an empty string causing it to "absorb" all transactions.

Is the imported transaction created with an empty payee or an wrong payee?

Are we  mixing up matching an imported transaction to a payee with noting an imported transaction is a duplicate of an existing transaction?  Does it matter whether an imported transaction is checked for being a duplicate before or after it is matched to a payee?

Christian: The imported transaction had a payee that exactly matched the single "Match on Payee Name" entry I had in the table.  I had actually copied it from the last time I imported from that payee.  So it's not empty or wrong, it just wasn't matched with an entered transaction.

Jack: Yes, I am mixing them up a bit.  I've seen both of them, but at different times.  This particular issue that Christian is responding to is that an imported transaction didn't match with its manually input transaction, but after manually matching the two and re-downloading, it's working fine (i.e. not falsely input as a "new" transaction).

(In reply to Jack from comment #47)
> Are we  mixing up matching an imported transaction to a payee with noting an
> imported transaction is a duplicate of an existing transaction?  Does it
> matter whether an imported transaction is checked for being a duplicate
> before or after it is matched to a payee?

No we are not mixing up this. AFAIK before the imported transaction is created payee matching is performed and after that the imported transaction is matched to existing transactions. The second match considers the payee as a match criteria so in order for it to work correctly the first match must work as expected. I'll check the code to make sure that this is actually true but as I recall this is the way it's done.

(In reply to Mike Wolfe from comment #48)
> Christian: The imported transaction had a payee that exactly matched the
> single "Match on Payee Name" entry I had in the table.  I had actually
> copied it from the last time I imported from that payee.  So it's not empty
> or wrong, it just wasn't matched with an entered transaction.

So the imported transaction was created with the correct payee and it didn't match an existing transaction which had the same amount and the same payee? If this is true the only thing I can think of is that the transactions must have had different bank ids.
 
> Jack: Yes, I am mixing them up a bit.  I've seen both of them, but at
> different times.  This particular issue that Christian is responding to is
> that an imported transaction didn't match with its manually input
> transaction, but after manually matching the two and re-downloading, it's
> working fine (i.e. not falsely input as a "new" transaction).

I think Jack was referring to my statements.

(In reply to Cristian Oneț from comment #49)
> No we are not mixing up this. AFAIK before the imported transaction is created payee matching is performed and after that the imported transaction is matched to existing transactions. The second match considers the payee as a match criteria so in order for it to work correctly the first match must work as expected. I'll check the code to make sure that this is actually true but as I recall this is the way it's done.

So there are actually three steps?  First, payee matching is performed and the transaction is created.  Then, the imported transaction is checked for matching an existing manually entered transation.  Separately (second or third?) the newly imported transaction is checked to see if it is a duplicate of a previously imported transaction.

(In reply to Jack from comment #51)
> So there are actually three steps?  First, payee matching is performed and
> the transaction is created.  Then, the imported transaction is checked for
> matching an existing manually entered transation.  Separately (second or
> third?) the newly imported transaction is checked to see if it is a
> duplicate of a previously imported transaction.

No, there are two steps, the first step is collecting all possible data for the imported transaction, this includes finding the payee based on the payee matching rules. Then, after all data is collected, the second step is performed which matches the imported transaction to existing transactions and this match can have several results:
1. Not found - there is no existing transaction related to the imported one
2. Duplicate - there is an existing transaction which is a duplicate of the imported transaction
3. Imprecise/precise match - there is an existing transaction which matches the imported transaction imprecisely/precisely

Okay, the behavior I've noticed in one of my accounts is that after import, some transactions are matched (meaning they turn green and have to be accepted) and some are not matched (meaning that are added and yellow and have to be manually matched then accepted).  This is repeatable after matching and accepting as necessary.

(I wasn't seeing this for a while because my settings were set up on the account to only retrieve data back to the last time I updated.  I changed it to retrieving for the last four days, and saw the issue occur again.)

My question then is: is this import matching behavior expected?  I would have thought that if the FITID field matches an existing transaction it wouldn't be imported, since it's the same transaction.

Do you mean transactions which have already been imported are imported again as new transactions instead of being detected as duplicates?  If this is the case, can you confirm that the IDs are really the same?  I vaguely recall someone having this problem because the bank was not using consistent and stable IDs for the transactions.  What happens if you import twice in a  row - where you should get an identical set of transactions each time.  Would you expect all transactions to be detected as duplicates the second time?  Are they all imported as new transactions?

"Do you mean transactions which have already been imported are imported again as new transactions instead of being detected as duplicates?"

Yes.

"If this is the case, can you confirm that the IDs are really the same?"

Yes, they are the same.  I'm using the <FITID> field to judge this, and they're consistent between updates.

"What happens if you import twice in a  row - where you should get an identical set of transactions each time."

Yes, I do.

"Would you expect all transactions to be detected as duplicates the second time?  Are they all imported as new transactions?"

There are five transactions posted in the last four days that I'm updating with.  Four are matched (transaction turns green) and one is unmatched (transaction added in yellow).  Matching and accepting them works, but next time I update the same thing happens. 

I would expect (without knowing how the software is written) that transactions with matching FITIDs (and several other fields) would be ignored for an import.  I don't know if this is what KMyMoney is written to do, but maybe it should be?

<quote>
I would expect (without knowing how the software is written) that transactions with matching FITIDs (and several other fields) would be ignored for an import. I don't know if this is what KMyMoney is written to do, but maybe it should be?
</quote>

That is exactly how it should work. I am using the KBanking plugin for online downloads and that works exactly the way you expect it to work. Since you're using the OFX plugin I assume that we have to look for the culprit in that area.

Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days, the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please set the bug status as REPORTED so that the KDE team knows that the bug is ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!

Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 30 days. The bug is now closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!