Bug 246108 - Consistency Check does not catch duplicate IDs
Summary: Consistency Check does not catch duplicate IDs
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Applications
Component: general (show other bugs)
Version: unspecified
Platform: openSUSE Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
: 248546 (view as bug list)
Depends on:
Blocks:
 
Reported: 2010-07-29 02:47 UTC by Fernando Vilas
Modified: 2010-09-10 09:22 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Fernando Vilas 2010-07-29 02:47:20 UTC
Version:           unspecified (using KDE 4.4.4) 
OS:                Linux

The file consistency check does not find/fix duplicate IDs. This prevents users from using the "Save as database" feature, since the IDs are used as primary keys in the database backend.

Reproducible: Didn't try

Steps to Reproduce:
Create a file with at least 2 transactions. Ensure that there is a duplicate transactionid. Run the consistency check to try to fix the duplication. Attempt to save to a database.

Actual Results:  
The consistency check reports that no problems were found. Then an error message occurs, complaining about a duplicate entry for a primary key in the kmmTransactions table.

Expected Results:  
The consistency check should fix the duplicate transactionid issue by changing all except one transaction to a new transactionid and update the transactionid counter to avoid future duplication issues.

This was reported on the KMM2 developer list by upscope@nwi.net on 2010-07-24. Tony B. attempted to resolve the database issue by recommending a consistency check, but that did not solve the problem.

The fix will likely involve a string change, since the errors are reported to the user.
Comment 1 Alvaro Soliverez 2010-07-29 03:05:56 UTC
The check could be added somewhere around line 1602 of mymoneyfile.cpp. That's where transactions are checked.
Comment 2 Thomas Baumgart 2010-07-29 13:44:41 UTC
My proposal is to assign new transaction IDs when loading an XML file. This way, we avoid duplicates. I have no idea how one could get into this situation w/o manually screwing around with the file contents.

Assigning IDs while loading the file does not hurt.

I suggest to resolve the problem in this way. First it's transparent and does not have the problem to overlook any corner cases. I do have a fix sitting here, but wanted to get some feedback first.
Comment 3 Cristian Oneț 2010-07-29 14:54:45 UTC
(In reply to comment #2)
> My proposal is to assign new transaction IDs when loading an XML file. This
> way, we avoid duplicates. I have no idea how one could get into this situation
> w/o manually screwing around with the file contents.
> 
> Assigning IDs while loading the file does not hurt.
> 
> I suggest to resolve the problem in this way. First it's transparent and does
> not have the problem to overlook any corner cases. I do have a fix sitting
> here, but wanted to get some feedback first.

I think we should do it this way. It seems more cleaner than the fix in the consistency check version.
Comment 4 Jack 2010-07-29 15:21:08 UTC
I can't comment on implementation issues, but if there is any chance a user might rely on internal IDs for anything then this should be carefully documented, although I'm not sure where.  (And I do think any doc changes for this can wait until after release.)
Would an alternate be to check for duplicate IDs on loading, and then just change the second one?
Comment 5 Thomas Baumgart 2010-07-29 16:25:42 UTC
SVN commit 1156689 by tbaumgart:

Make sure that each transaction has a unique ID while loading a file.

BUG: 246108

 M  +33 -4     mymoneyseqaccessmgr.cpp  
 M  +27 -0     mymoneyseqaccessmgrtest.cpp  
 M  +1 -0      mymoneyseqaccessmgrtest.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1156689
Comment 6 Alvaro Soliverez 2010-08-20 21:58:57 UTC
*** Bug 248546 has been marked as a duplicate of this bug. ***
Comment 7 Alvaro Soliverez 2010-08-20 21:59:47 UTC
Version:           unspecified (using KDE 4.5.0) 
OS:                Linux

openSUSE 11.3

KMyMoney
Version 4.5.0
Using KDE Development Platform 4.5.00 (KDE 4.5.0)

Only seems to be problem using MySQL database. If just use KMyMoney.kmy appears
to not effect it.As stated in Bug 246108, there is a duplicated transaction
(actually about 6). They all appear to be tranactions that were carried over
from my old bank to the new one because they had not cleared before then. 

Problem defined in Bug 246108 still exists. Bug marked resolved. 



Reproducible: Always

Steps to Reproduce:
Not sure how to reproduce. But here's output looking at KMyMoney.kmy of the
first dup. transaction. There are about 6 of them I found, all from the time
around switching banks, etc. (Aug 2005)

see actual results. This still causes error shown in the above bug report.

Actual Results:  
<TRANSACTION postdate="2004-11-16" commodity="USD" memo=""
id="T000000000000000001" entrydate="2005-03-12">
   <SPLITS>
    <SPLIT payee="P000001" reconcileflag="2" shares="-2500/100"
reconciledate="2005-03-07" action="Check" bankid="" account="A000192"
number="11088" value="-2500/100" memo="" id="S0001"/>
    <SPLIT payee="" reconcileflag="0" shares="2500/100" reconciledate=""
action="Check" bankid="" account="A000262" number="" value="2500/100"
memo="" id="S0002"/>
   </SPLITS>
  </TRANSACTION>


  <TRANSACTION postdate="2010-06-26" commodity="USD" memo=""
id="T000000000000000001" entrydate="2010-
07-23">
   <SPLITS>
    <SPLIT payee="P000108" reconcileflag="0" shares="-6036/100"
reconciledate="" action="" bankid="" a
ccount="A000294" number="2024" value="-6036/100" memo="" id="S0001"/>
    <SPLIT payee="P000108" reconcileflag="0" shares="6036/100"
reconciledate="" action="" bankid="" ac
count="A000041" number="2024" value="6036/100" memo="" id="S0002"/>
   </SPLITS>
  </TRANSACTION>

Expected Results:  
Different transaction numbers. 

Also different Hardware from original early transaction. Moved from 32 bit
system to 64 bit system. Can I just edit the transaction numbers to something
else and not mess up the reconciliation etc? 

If you need additional information please let me know.
Comment 8 Russ Fineman 2010-08-20 22:04:42 UTC
Looks like system automatically reopened this. i was going to but decided it was new version of KDE and KMyMoney (4).
Comment 9 Russ Fineman 2010-08-21 00:42:33 UTC
Did some additional testing today.

If I edit the .kmy file and change the later T000000000000000001 transaction to T123000000000000001, and then say save as database. it goes until the next duplicate transaction (T000000000000000003) and give message for this transaction. Changing it per above, it goes to next duplicate before error again. I did not go any further but I assume it will eventually point out all duplicates. I did not keep the changes until I here what effects it will have and what your recommended solution is. In the mean time I can use the .kmy local version instead of the MySQL database.
Comment 10 Fernando Vilas 2010-08-21 05:34:51 UTC
The fix as implemented should change the transaction id values on opening the file. What happens if you open your existing file, then save it? If the transaction ids are now unique, you can then save to the database.

If this fixes the issue, we probably need something in the FAQ about it, and a way to make it more automatic for the users.
Comment 11 Russ Fineman 2010-08-21 15:54:42 UTC
Tried did not work. What I did was open checking 1 (started 7-19-05) in KMyMoney and then did save as KMyMoney.kmy. Then open Checking (old bank started 11-16-04) (Last entry and reconciliation was 10-23-05 due to interest payments). 

Looking at KMyMoney.kmy looks like transaction numbering started over on

<TRANSACTION postdate="2010-06-25" commodity="USD" memo="" id="T000000034359738651" entrydate="2010-06-25">
   <SPLITS>
    <SPLIT payee="P000015" reconcileflag="0" shares="-2000/100" reconciledate="" action="" bankid="" account="A000294" number="2023" value="-2000/100" memo="For use in Moses Lake, WA." id="S0001"/>
    <SPLIT payee="P000015" reconcileflag="0" shares="2000/100" reconciledate="" action="" bankid="" account="A000059" number="2023" value="2000/100" memo="For use in Moses Lake, WA." id="S0002"/>
   </SPLITS>
  </TRANSACTION>
  <TRANSACTION postdate="2010-06-26" commodity="USD" memo="" id="T000000000000000001" entrydate="2010-07-23">
   <SPLITS>
    <SPLIT payee="P000108" reconcileflag="0" shares="-6036/100" reconciledate="" action="" bankid="" account="A000294" number="2024" value="-6036/100" memo="" id="S0001"/>
    <SPLIT payee="P000108" reconcileflag="0" shares="6036/100" reconciledate="" action="" bankid="" account="A000041" number="2024" value="6036/100" memo="" id="S0002"/>
   </SPLITS>
  </TRANSACTION>

Last Tranaction is T000000000000000054 entered on 8-13-10.

The renumbering appears to have started around the time I restored my system after loosing a hard drive. Restore was from KMyMoney backup saved externally.

I'm wondering if the version of KMyMoney I have has the changes you mentioned.

 kmymoney4-4.5-42.pm.42.1.x86_64
 libkmm_kdchart4-4.5-42.pm.42.1.x86_64
 libkmm_widgets4-4.5-42.pm.42.1.x86_64
 libkmm_plugin4-4.5-42.pm.42.1.x86_64
 libkmm_mymoney4-4.5-42.pm.42.1.x86_64

What would I need to download and compile from your site?
Comment 12 Alvaro Soliverez 2010-08-21 16:24:31 UTC
The version you have should have the fixes, and it sure seems like it's not working.
Comment 13 Russ Fineman 2010-08-30 18:07:43 UTC
I've been looking at this closely this last week. What I found was a change in the transaction ids on 6-26-2010. The last entry Transaction on 6-25 was id="T000000034359738651" postdate 6-25-2010. On 6-26-2010 the next transaction was number "T000000000000000001". Transactions since then are continuing from xxx0001 thru xxx0074. This means they are all duplicate transactions. When I try to reconcile it does weid things. Example:
  First three to be reconciled
      1. Deposit  xxxxxx
      2. withdrawal  company A
      3. Withdrawal  Company B

if i click #1 and set it to C it turns the C on in col.
if I click #2 and set it to C, it puts the C in number 3. 

All three of these transactions occured after 7-1-10. The ones let over from the previous month set the C correctly. The difference is the one's after 6-26 are duplicate transactions.

Also consistency check says :Finished: Data is Consistent.

Is there a valve somewhere I can reset to the next sequential number after the one on 6-25-2010 and then delete and re-enter all the transaction from that point with the right numbers and sequential transactions? 

As far as I know I have the latest version of KMyMoney4 (Version 4.5.0)
  kmymoney4-4.5-42.pm.42.1.x86_64 (from Suse packman repo.) Is there a version I can download from your site and deinstall the version I have.

Thanks.
Comment 14 Thomas Baumgart 2010-09-10 08:46:58 UTC
SVN commit 1173699 by tbaumgart:

Make sure transactions have a unique ID after loading from file

BUG: 246108

 M  +3 -31     mymoneyseqaccessmgr.cpp  
 M  +0 -25     mymoneyseqaccessmgrtest.cpp  
 M  +0 -1      mymoneyseqaccessmgrtest.h  
 M  +17 -5     mymoneystoragexml.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1173699
Comment 15 Thomas Baumgart 2010-09-10 09:22:25 UTC
SVN commit 1173703 by tbaumgart:

Make sure transactions have a unique ID after loading from file

Backported to 4.5 branch

BUG: 246108

 M  +3 -31     mymoneyseqaccessmgr.cpp  
 M  +0 -25     mymoneyseqaccessmgrtest.cpp  
 M  +0 -1      mymoneyseqaccessmgrtest.h  
 M  +17 -5     mymoneystoragexml.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1173703