When clicking update, it will constantly create duplicate transactions even though the transaction originated from a previous update. It does not matter if the transactions are reconciled, cleared or none of the above. It appears to be applied to all transactions up to a month previous to the latest transaction. This regression appeared in 4.6.3 and has continued in 4.6.4. Reproducible: Always Steps to Reproduce: Configure the account to communicate with the bank. Click update and every transaction listed for the last month will always be marked as a duplicate transaction. Actual Results: Every transaction for the last month is marked as a duplicate. Expected Results: If the transaction originated from a previous update, it should not be marked as a duplicate.
I believe that matching is based on a transaction id provided by the bank/credit card. Is there any chance your institution made any changes to their system in the past month? Also, does each transaction duplicate just once, or does it create a new one each time you update the account? Additional question - how do you have your online settings for that account set? You can set how far back it tries to update each time it connects. Setting it to the last time you updated might at least eliminate some of the duplication.
(In reply to comment #1) > I believe that matching is based on a transaction id provided by the > bank/credit card. Is there any chance your institution made any changes to > their system in the past month? Also, does each transaction duplicate just > once, or does it create a new one each time you update the account? > > Additional question - how do you have your online settings for that account > set? You can set how far back it tries to update each time it connects. > Setting it to the last time you updated might at least eliminate some of the > duplication. I'm not aware of any changes my bank has made with transactions. What I do recall is as soon as I upgraded from 4.6.2 to 4.6.3, I immediately started getting duplicate transactions every time I click update. This occurs every time on the same transactions assuming I do it again the same day and there hasn't been any new transactions. It doesn't even matter if I previously accepted it, they will be marked as duplicates again on a subsequent update. I changed the setting so it checks the last 30 days instead of 60 days which has limited the number of duplicate transactions but this is still rather annoying since I still have to accept the last 30 days of duplicate transactions every time I update.
I've been having the same problem with one of my banks for a while. I'm running the latest version of KMM from git and this problem started a year or two ago. I only get duplicate transactions from one bank (Charles Schwab), it happens with both accounts I have at Charles Schwab. My many other accounts work as expected. I get the duplicates as long as the start date for the download covers the transaction. I've shortened it to 3 days to minimize the pain or deleting or constantly matching it to previous the previous transactions. If I approve the duplicate match each time then the next time it comes in matched (green) and if I approve the match it's ready for the next duplicate. If I don't approve the match then I get a new transaction, I have to approve the previous match, then I have to manually match the new with the old and approve it. I hate using KMM for 3 days each month:-) I have worked with Thomas Baumgart on OFX problems in the past and he told me about a simple trick to make it easy to see the actual OFX files after you download them. Make a directory called thb in /home and give your user write access to it. If KMM sees /home/thb it will save the OFX files in that directory (kmm-statement-0.txt, kmm-statement-1.txt...) I finally took the time to do this and I think I found the problem. Charles Schwab does not include a bankid with the transactions. Matthew, I'm guessing your bank(s) may not include a bankid value. Everything else looks the similar in the OFX that works and the one that does not work. Charles Schwab does include a unique value in both the payee field and the memo field, and they are the same value. For example: BANK INT 091613-101513 BANK INT 081613-091513 BANK INT 071613-081513 BANK INT 061613-071513 This appears to be the category plus the date range of the interest payment. I'm hoping there is a way to use this information to skip duplicate transactions when there is no unique bankid provided. * ---- Brendan* On Wed, Oct 16, 2013 at 7:55 AM, Matthew Schultz <mattsch@gmail.com> wrote: > https://bugs.kde.org/show_bug.cgi?id=326100 > > --- Comment #2 from Matthew Schultz <mattsch@gmail.com> --- > (In reply to comment #1) > > I believe that matching is based on a transaction id provided by the > > bank/credit card. Is there any chance your institution made any changes > to > > their system in the past month? Also, does each transaction duplicate > just > > once, or does it create a new one each time you update the account? > > > > Additional question - how do you have your online settings for that > account > > set? You can set how far back it tries to update each time it connects. > > Setting it to the last time you updated might at least eliminate some of > the > > duplication. > > I'm not aware of any changes my bank has made with transactions. What I do > recall is as soon as I upgraded from 4.6.2 to 4.6.3, I immediately started > getting duplicate transactions every time I click update. This occurs > every > time on the same transactions assuming I do it again the same day and there > hasn't been any new transactions. It doesn't even matter if I previously > accepted it, they will be marked as duplicates again on a subsequent > update. > > I changed the setting so it checks the last 30 days instead of 60 days > which > has limited the number of duplicate transactions but this is still rather > annoying since I still have to accept the last 30 days of duplicate > transactions every time I update. > > -- > You are receiving this mail because: > You are the assignee for the bug. > _______________________________________________ > KMyMoney-devel mailing list > KMyMoney-devel@kde.org > https://mail.kde.org/mailman/listinfo/kmymoney-devel >
Brendan, thanks for the detailed information. It happens that I also bank with Charles Schwab so that would explain the issue since we're both having the same problem with the same bank.
I'm pretty sure it used to work and you appear to be confirming that it did. I always assumed that Schwab did something to screw it up but it sounds like a change in KMM may be the problem. I'm probably moving my money out of Schwab due to their pathetic online security so it may not affect me much longer. Please confirm that 4.6.2 worked correctly and 4.6.3 does not work correctly? I'm on Version 4.6.90-acd1b9cbeb. * ---- Brendan* On Wed, Oct 16, 2013 at 10:32 AM, Matthew Schultz <mattsch@gmail.com> wrote: > https://bugs.kde.org/show_bug.cgi?id=326100 > > --- Comment #4 from Matthew Schultz <mattsch@gmail.com> --- > Brendan, thanks for the detailed information. It happens that I also bank > with > Charles Schwab so that would explain the issue since we're both having the > same > problem with the same bank. > > -- > You are receiving this mail because: > You are the assignee for the bug. > _______________________________________________ > KMyMoney-devel mailing list > KMyMoney-devel@kde.org > https://mail.kde.org/mailman/listinfo/kmymoney-devel >
(In reply to comment #5) > I'm pretty sure it used to work and you appear to be confirming that it > did. I always assumed that Schwab did something to screw it up but it > sounds like a change in KMM may be the problem. I'm probably moving my > money out of Schwab due to their pathetic online security so it may not > affect me much longer. > > Please confirm that 4.6.2 worked correctly and 4.6.3 does not work > correctly? > > I'm on Version 4.6.90-acd1b9cbeb. > > * > ---- > Brendan* > > > On Wed, Oct 16, 2013 at 10:32 AM, Matthew Schultz <mattsch@gmail.com> wrote: > > > https://bugs.kde.org/show_bug.cgi?id=326100 > > > > --- Comment #4 from Matthew Schultz <mattsch@gmail.com> --- > > Brendan, thanks for the detailed information. It happens that I also bank > > with > > Charles Schwab so that would explain the issue since we're both having the > > same > > problem with the same bank. > > > > -- > > You are receiving this mail because: > > You are the assignee for the bug. > > _______________________________________________ > > KMyMoney-devel mailing list > > KMyMoney-devel@kde.org > > https://mail.kde.org/mailman/listinfo/kmymoney-devel > > It appears you are correct again. This is not a version regression as I thought and I suspect that as you said, Schwab must not be sending the correct values in this case. I tested with 4.6.1, 4.6.2 and 4.6.3 and they all have the same problem.
So then the question is how can kmymoney work around this issue?
What's the current status of this issue? I'm asking because it seems as an issue with your service provider. Anyway it's definitely not a major KMyMoney issue.
There has been a similar issue recently discussed on the mailing list. It would help if someone who has this problem could save two updates and then check if the <FITID> field is different in the two updates for the same transaction. If that is the case, the problem is with the institution and there is nothing that KMM can do, since that is the field used to recognize a transaction as unique. (This is true for OFX imports, either direct connect or file download. I do not know if it also holds for QIF.)
It would be cool to have a copy of these duplicates so that I can investigate. AFAICS, only the transaction container (the stuff between <STMTTRN> and the next </STMTTRN> of the duplicates is of interest. Feel free to send them via private e-mail to me.
Dear Bug Submitter, This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days, the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information. For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging If you have already provided the requested information, please set the bug status as REPORTED so that the KDE team knows that the bug is ready to be confirmed. Thank you for helping us make KDE software even better for everyone!
Sorry for the delay in a response. I tried creating a /home/thb directory and giving it full write permissions. I did not see any files created in that directory after performing an account update. I am currently using kmymoney 5.0.0. Is there another way to get this information that kmymoney is downloading in raw format?
Matthew, please check if there is a "Support" tab on "Settings/Configure KMyMoney/General"? If yes, then use them to create the logs. These settings supersede the /home/thb hidden way of producing logs.
I've sent Thomas Baumgart an email with the data he requested.
moving back to REPORTED so it doesn't get prematurely closed
Is this still happening? I have a vague memory of a change being made to recognize duplicate transactions even if the institution is not correctly and consistently identifying them.
(In reply to Jack from comment #16) > Is this still happening? I have a vague memory of a change being made to > recognize duplicate transactions even if the institution is not correctly > and consistently identifying them. No clue. My bank doesn't allow me to download transactions with kmymoney and as far as I know, there is no plan by kde/kmymoney developers to make that possible.
Sorry, but I don't follow. As I read the history of the bug, this started by you getting duplicate transactions when downloading from your bank. Has something changed? If you cannot currently replicate the problem, there is no point in keeping it open, since we can't test any potential fixes.
Yes, there's no way for me to test it because my bank now only allows secure or signed programs to download transactions. From what I understand, it's technically feasible for kmymoney to get this capability but for some reason is not a priority. I can't imagine the update button will be useful for much longer for anyone if most banks end up doing this.
Can you provide any more information about what the bank has actually done? Do you know of any software which is still able to download from your bank, and are you willing to identify the bank? I see Charles Schwab mentioned in several of the comments. It sounds to me like the problem of duplicate transactions might still exist, but because your bank has introduced extra barriers to downloading transactions, we can't really test to confirm.
(In reply to Jack from comment #20) > Can you provide any more information about what the bank has actually done? > Do you know of any software which is still able to download from your bank, > and are you willing to identify the bank? I see Charles Schwab mentioned in > several of the comments. It sounds to me like the problem of duplicate > transactions might still exist, but because your bank has introduced extra > barriers to downloading transactions, we can't really test to confirm. You can find the information about what needs to be done to kmymoney for the update button to work again with my bank (Schwab) and quite possibly with other banks as well: https://bugs.kde.org/show_bug.cgi?id=428146
OK. I see I shouldn't have bothered with Comment #16. This bug can just live as unfixable (or any fix untestable) and perhaps it should depend on that other bug, and that one should be opened up as a wishlist.
> From what I understand, it's technically feasible for kmymoney to get this capability but for some reason is not a priority. It's rather the opposite: because KMyMoney is an open source software, anyone can build it from the source - that includes Linux distributions which do it themselves. However, in order to communicate with Schwab and other banks who implement similar changes, we would instead have to ship it exclusively as a binary package, signed with a unique cryptographic key which Schwab would be able to recognize, deeming the KMyMoney app as a secure one per their terms. So no, it is almost impossible to have this done.
(In reply to Dawid Wróbel from comment #23) > > From what I understand, it's technically feasible for kmymoney to get this capability but for some reason is not a priority. > > It's rather the opposite: because KMyMoney is an open source software, > anyone can build it from the source - that includes Linux distributions > which do it themselves. However, in order to communicate with Schwab and > other banks who implement similar changes, we would instead have to ship it > exclusively as a binary package, signed with a unique cryptographic key > which Schwab would be able to recognize, deeming the KMyMoney app as a > secure one per their terms. > > So no, it is almost impossible to have this done. Wait, why couldn't both ways be possible? Obviously if it was compiled by the user/distro, then it wouldn't work with Schwab.
Why is this still open? I'm pretty sure the issue of duplicate transactions has been dealt with, and it it's still a problem with a bank not providing consistent FITID, I thought we had actually created a way to deal with that. The issue of whether or not we (or any distribution) provide a binary version of the program acceptable to Schwab (or any other bank with similar rules) is a totally separate issue, and should have it's own wishlist, if it doesn't already.
When I download transactions manually and import them as CSV, I don't notice any duplicate transactions. I'm going to assume the process is the same assuming it could still be obtained with the update button.
Thanks for the follow-up. Note that the csv import probably does not contain a FITID, which is pretty OFX specific. It might be interesting if Schwab allowed downloading of an OFX export file, but I would doubt they do. Most banks outsource the OFX support, so when they eliminate OFX direct connect download, the also eliminate any OFX download at all.
The CSV importer as well as KBanking produce their own id to detect duplicates because there is no unique ID provided by the bank for them. They are stored as "bankid" with the split. The same applies for the OFX importer (also used for direct connect) where you can choose between three different methods to construct this id (one of them being the FITID). Here are examples of KBanking bankid="A000076-2022-07-28-fa28c20-1" the CSV Importer bankid="2021-05-31-146598088-0" and the OFX Importer (KMyMoney Hash method). bankid="KMM 2022-08-06-814ef05-1"