Created attachment 110781 [details] Screenshot During an OFX import in Kmymoney, the operations already mapped continue to be added instead of being taken for duplicates ... example: I import 30 days of operations from my account and match the all operations, then a week later I do the same (30 days) which should logically no longer add me operations already in correspondence (thanks to the tag <FITID> unique for each operation) but this is not the case, so everything to do again each time. Currently here is what gives importing an operation: <STMTTRN> <TRNTYPE>FLOW <DTPOSTED>20180118230000,000 <TRNAMT>-10.85 <FITID>0083a29895f5-1c6d-4e58-a774-059a01964111 <NAME>INVOICE CARD <MEMO>OF 180118 GEANT CG302 PARIS CARD 49XXXX </STMTTRN> No matching is done. The tag <FITID> does not work during the import unlike my old bank (the postal bank) which for example, has this type of tag in its OFX files <FITID>PSMF@$LL1P Hoping that a patch is created for this.
Can you check that the FITID can be found in the .kmy file? If not, try to find the transaction it should match and provide it here together with the one found in the OFX file. Please remove any sensitive information and replace it with a placeholder character. Information about the fileformat can be found in the manual at https://docs.kde.org/stable4/en/extragear-office/kmymoney/details.formats.html
Created attachment 110785 [details] Two identical operations but two different OFX imports via my bank I think I found the source of my problem ... The FITID changes with each new OFX import from my bank (see OFX screenshot), in order to overcome this, Kmymoney could not he do the correspondence with the amount of operations? Because there I am condemned to make everything correspond to the hand with each import OFX ...
From (an older version of) the OFX specifications: ---8<--- An FI (or its Service Provider) assigns an <FITID> to uniquely identify a financial transaction that can appear in an account statement. Its primary purpose is to allow a client to detect duplicate responses. Open Financial Exchange intends <FITID> for use in statement download applications, where every transaction (not just those that are client-originated or server-originated) requires a unique ID. An <FITID> also uniquely identifies the closing statement in <CLOSINGRS> and <CCCLOSINGRS>. Again, the OFX client should detect repeated closing statements (duplicate downloads) using these identifiers. FITIDs must be unique within the scope of an account but need not be sequential or even increasing. Clients should be aware that FITIDs are not unique across FIs. If a client performs the same type of request within the same scope at two different FIs, clients will need to use FI + <ACCTID> + <FITID> as a globally unique key in a client database. That is, the <FITID> value must be unique within the account and Financial Institution (independent of the service provider). ---8<--- You may want to get in touch with your bank to get this fixed.
Although this is not a bug in the importing, I'm marking as NEEDSINFO in case it might turn into a wishlist for a way for KMM to detect such "duplicates" although I'm not sure what would be the correct action. It can't match the transactions without ignoring one of the FITID's, and it would probably be annoying to have to "OK" if each were popped up as a separate error message. Listing all such transactions as errors in the import summary could be an option, but I'm not sure if there are cases where they are actually desired as new transactions.
Here's what I found for a commercial product on their web-site (product name removed): ---8<--- If your bank and/or their service provider is making changes to how they present data to the application, there is the possibility that they may transmit the same transaction with two different IDs. Solution: Contact the Online Banking Support for your bank to request they escalate to their OFX team or service provider in order to resolve this issue for all their customers. ---8<--- I think this is the only valid solution to the problem
Here's what I found for a well-known commercial product with OFX support on their support web-site (product name removed by me): ---8<--- If your bank and/or their service provider is making changes to how they present data to the application, there is the possibility that they may transmit the same transaction with two different IDs. Solution: Contact the Online Banking Support for your bank to request they escalate to their OFX team or service provider in order to resolve this issue for all their customers. ---8<--- I think this is the only valid solution to this problem.
Dear Bug Submitter, This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days, the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information. For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging If you have already provided the requested information, please set the bug status as REPORTED so that the KDE team knows that the bug is ready to be confirmed. Thank you for helping us make KDE software even better for everyone!
Dear Bug Submitter, This bug has been in NEEDSINFO status with no change for at least 30 days. The bug is now closed as RESOLVED > WORKSFORME due to lack of needed information. For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging Thank you for helping us make KDE software even better for everyone!
Git commit 65e3837697c9378616a7eda2ba3387d1208f288e by Thomas Baumgart. Committed on 07/04/2019 at 07:13. Pushed by tbaumgart into branch '5.0'. Provide method to construct unique transaction ID for OFX import User reports show, that the FITID attribute provided by some institutions during OFX statement download is not as unique as it is specified. In fact, some banks issue a different FITID every time you download the same statement. This makes the FITID attribute absolutely worthless for duplicate detection. Since asking banks to fix their software has no reasonable chance to lead to success on short call, this change allows to switch the duplicate detection to be based on a method provided by KMyMoney. In fact it uses the same algorithm which works well in the KBanking importer for years already. Note: switching from "OFX FITID" to "KMyMoney Hash" may result in duplicates for one more time. At the same time, the OFX import options are now available during file import and not only for mapped accounts (OFX direct connect). FIXED-IN: 5.0.4 M +8 -0 kmymoney/plugins/interfaces/kmmimportinterface.cpp M +2 -0 kmymoney/plugins/ofx/import/dialogs/konlinebankingstatus.cpp M +56 -44 kmymoney/plugins/ofx/import/dialogs/konlinebankingstatusdecl.ui M +108 -29 kmymoney/plugins/ofx/import/importoption.ui M +68 -11 kmymoney/plugins/ofx/import/ofximporter.cpp https://commits.kde.org/kmymoney/65e3837697c9378616a7eda2ba3387d1208f288e
Git commit 9fca9e3661ce2d03e472ab1d520b59493ec110cb by Thomas Baumgart. Committed on 19/04/2019 at 08:10. Pushed by tbaumgart into branch '5.0'. Provide internal OFX id generation also for WebConnect The change provided in commit 65e3837697c93786 is not available when importing an OFX statement file via WebConnect. For this case, a new option has been added (no UI yet) to allow selection of the default for the unique transaction id to be either the OFX FITID field or our internal hash. The default is to use OFX FITID, so no change in prior behavior. This default settings will also be used to preset the combo box in the file selection dialog for OFX import and the account setting for mapped accounts. If users set this option, they need to know that some duplicate transactions may appear since the method to detect them changed. One way to deal with that is to remove both transactions and redo the import. To set the method to the internal hash create a file named ~/.config/kmymoney/ofximporterrc with the following content: [General] useOwnFITID=true Future versions of KMyMoney will provide a UI to change this setting. M +36 -11 kmymoney/plugins/ofx/import/ofximporter.cpp https://commits.kde.org/kmymoney/9fca9e3661ce2d03e472ab1d520b59493ec110cb
I tested this and it works. This is not ideal as a global setting since I have to decide if I want to fix the one account that does not handle the FITID numbers correctly or deal with every other account that has previous imported transactions. First guess is that will require me to go throut hundreds of new duplicates and delete the ones that are already reconciled / cleared. Is there any chance that this will end up being an account based setting or is global the only way to do this? ---- Brendan Coupe On Fri, Apr 19, 2019 at 2:11 AM Thomas Baumgart <bugzilla_noreply@kde.org> wrote: > > https://bugs.kde.org/show_bug.cgi?id=390681 > > --- Comment #10 from Thomas Baumgart <thb@net-bembel.de> --- > Git commit 9fca9e3661ce2d03e472ab1d520b59493ec110cb by Thomas Baumgart. > Committed on 19/04/2019 at 08:10. > Pushed by tbaumgart into branch '5.0'. > > Provide internal OFX id generation also for WebConnect > > The change provided in commit 65e3837697c93786 is not available when > importing an OFX statement file via WebConnect. For this case, a new > option has been added (no UI yet) to allow selection of the default for > the unique transaction id to be either the OFX FITID field or our > internal hash. The default is to use OFX FITID, so no change in prior > behavior. This default settings will also be used to preset the combo > box in the file selection dialog for OFX import and the account setting > for mapped accounts. > > If users set this option, they need to know that some duplicate > transactions may appear since the method to detect them changed. One way > to deal with that is to remove both transactions and redo the import. > > To set the method to the internal hash create a file named > > ~/.config/kmymoney/ofximporterrc > > with the following content: > > [General] > useOwnFITID=true > > Future versions of KMyMoney will provide a UI to change this setting. > > M +36 -11 kmymoney/plugins/ofx/import/ofximporter.cpp > > https://commits.kde.org/kmymoney/9fca9e3661ce2d03e472ab1d520b59493ec110cb > > -- > You are receiving this mail because: > You are the assignee for the bug.
I wanted to follow up on the change that added the option to use the KMM Hash to generate FITIDs globally. At first this worked for the Citi credit card and then it stopped working toward the end of the statement period. I decided not to judge it until it was used for an entire statement period and my patience paid off. The first full month worked perfectly. I have checked the Citi OFX file and the FITID number is still not following the OXF standard and is simply the date followed by a sequential number starting with the first transaction in the file. Since this is a global setting I had to shorten the OFX download time frame for the rest of my accounts. After enough time passed most of my accounts are working fine with the KMM hash method and I have extended the ofx download time frame for all of them. Is this setting going to be added to the KMM settings? Is there any chance it will be account based rather than global? The global setting does have the temporary side effect of not matching any previously downloaded transactions that are downloaded again. ---- Brendan Coupe On Sun, Apr 21, 2019 at 1:35 PM <bugzilla_noreply@kde.org> wrote: > > https://bugs.kde.org/show_bug.cgi?id=390681 > > --- Comment #11 from Brendan@CoupeWare.com --- > I tested this and it works. This is not ideal as a global setting > since I have to decide if I want to fix the one account that does not > handle the FITID numbers correctly or deal with every other account > that has previous imported transactions. First guess is that will > require me to go throut hundreds of new duplicates and delete the ones > that are already reconciled / cleared. > > Is there any chance that this will end up being an account based > setting or is global the only way to do this? > > ---- > Brendan Coupe > > On Fri, Apr 19, 2019 at 2:11 AM Thomas Baumgart > <bugzilla_noreply@kde.org> wrote: > > > > https://bugs.kde.org/show_bug.cgi?id=390681 > > > > --- Comment #10 from Thomas Baumgart <thb@net-bembel.de> --- > > Git commit 9fca9e3661ce2d03e472ab1d520b59493ec110cb by Thomas Baumgart. > > Committed on 19/04/2019 at 08:10. > > Pushed by tbaumgart into branch '5.0'. > > > > Provide internal OFX id generation also for WebConnect > > > > The change provided in commit 65e3837697c93786 is not available when > > importing an OFX statement file via WebConnect. For this case, a new > > option has been added (no UI yet) to allow selection of the default for > > the unique transaction id to be either the OFX FITID field or our > > internal hash. The default is to use OFX FITID, so no change in prior > > behavior. This default settings will also be used to preset the combo > > box in the file selection dialog for OFX import and the account setting > > for mapped accounts. > > > > If users set this option, they need to know that some duplicate > > transactions may appear since the method to detect them changed. One way > > to deal with that is to remove both transactions and redo the import. > > > > To set the method to the internal hash create a file named > > > > ~/.config/kmymoney/ofximporterrc > > > > with the following content: > > > > [General] > > useOwnFITID=true > > > > Future versions of KMyMoney will provide a UI to change this setting. > > > > M +36 -11 kmymoney/plugins/ofx/import/ofximporter.cpp > > > > https://commits.kde.org/kmymoney/9fca9e3661ce2d03e472ab1d520b59493ec110cb > > > > -- > > You are receiving this mail because: > > You are the assignee for the bug. > > -- > You are receiving this mail because: > You are the assignee for the bug.
The per account setting is available for online mapped accounts. The problem for all other imports is, that the current processing reads the file and converts it into a general MyMoneyStatement file which is independent from the source format (QIF, OFX, ...) before the account is selected. The setting though is needed in the first step. This order has been chosen, because with some formats the account can be chosen automatically. So its more of chicken-and-egg problem.
Created attachment 122489 [details] attachment-19417-0.html This problem came back. My Citi Visa issued a credit for an old charge and it moved to the beginning of my OFX file. From that point forward nothing is matching. I have the following in ~/.config/kmymoney/ofximporterrc [General] useOwnFITID=true Has something changed that stopped the FITID value from being used instead of the bank supplied FTID? I'm having a matching problem in another account that may also be related. It feels like the code reverted to using the bank supplied FITID value. I'm running KMM compiled very recently from the master branch. *----Brendan Coupe* On Sun, Jun 23, 2019 at 11:14 AM Thomas Baumgart <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=390681 > > --- Comment #13 from Thomas Baumgart <tbaumgart@kde.org> --- > The per account setting is available for online mapped accounts. The > problem > for all other imports is, that the current processing reads the file and > converts it into a general MyMoneyStatement file which is independent from > the > source format (QIF, OFX, ...) before the account is selected. The setting > though is needed in the first step. This order has been chosen, because > with > some formats the account can be chosen automatically. So its more of > chicken-and-egg problem. > > -- > You are receiving this mail because: > You are the assignee for the bug.
Nothing has changed since early July in the OFX parts, so it must be something else. Maybe, the bank sends slightly different data than before so that a different hash is computed.