Bug 346810

Summary: CSV importer fails to correctly parse values
Product: [Applications] kmymoney Reporter: bugs
Component: importerAssignee: KMyMoney Devel Mailing List <kmymoney-devel>
Status: RESOLVED WORKSFORME    
Severity: major CC: agander93, bugs, onet.cristian
Priority: NOR    
Version: 4.7.2   
Target Milestone: ---   
Platform: Compiled Sources   
OS: Linux   
Latest Commit: Version Fixed In:
Attachments: Initial CSV dialog setup
Second dialog section setup
Results
Test csv (final screenshot shows result of importing this)
Results of importing test file

Description bugs 2015-04-27 18:58:37 UTC
I've tried this in the latest version of kmymoney on Fedora 21 and then downloaded 4.7.2 and compiled it, but experience the same behavior. When importing from CSV, the Payment column isn't updated correctly. Additionally, the details column sometimes has a 0 in it, but sometimes has the correct value.

Reproducible: Always

Steps to Reproduce:
1. Configure CSV Import as per screenshots
2. Run import
3. Be sad

Actual Results:  
Payment column is empty. Details column has 0 in it many times.

Expected Results:  
Properly imported transactions.
Comment 1 bugs 2015-04-27 18:59:18 UTC
Created attachment 92286 [details]
Initial CSV dialog setup
Comment 2 bugs 2015-04-27 18:59:45 UTC
Created attachment 92287 [details]
Second dialog section setup
Comment 3 bugs 2015-04-27 19:00:25 UTC
Created attachment 92288 [details]
Results
Comment 4 bugs 2015-04-27 19:02:45 UTC
Created attachment 92289 [details]
Test csv (final screenshot shows result of importing this)
Comment 5 bugs 2015-04-27 19:05:19 UTC
Created attachment 92290 [details]
Results of importing test file
Comment 6 allan 2015-04-27 22:54:40 UTC
(In reply to bugs from comment #0)
> I've tried this in the latest version of kmymoney on Fedora 21 and then
> downloaded 4.7.2 and compiled it, but experience the same behavior. When
> importing from CSV, the Payment column isn't updated correctly.
> Additionally, the details column sometimes has a 0 in it, but sometimes has
> the correct value.

I have to admit that you have me confused!  Above, you give your version as 4.7.2, but all the screen shots show a very old version of the importer plugin.
When you switch between different versions, it is very import that you uninstall the previous version, otherwise you will have a bit of a mongrel on your hands.  I'm not saying that you didn't uninstall, but definitely, you are not using the current one, but about two versions older.
The latest, 4.7.2, plugin has two separate windows, one for the wizard and one for the result table.

The next problem is with your data file.  There is a, possibly unwritten, convention that when dealing with debits and credits, a transaction may have either one, but not both.  I have only once before met your version, and I introduced a coping mechanism.  The user is asked whether he wishes to accept either the zero field or the non-zero field.
I input your data into my 4.7.2 and waded through the interrogation, and the outcome was correct.  Whilst it works, it can be a bit labour intensive, and it may be better to edit the file before import.
Comment 7 bugs 2015-04-28 01:24:13 UTC
Interesting. I didn't uninstall the old version of kmymoney (which came by default with Fedora 21) before compiling the new, so perhaps there's some conflict occurring. I'm not at that machine at the moment but will try it again tomorrow. Regarding the 0's in the debit/credit section, I did that manually because it was complaining that certain rows had an improper number of columns. Probably because it was the older version as well ;-). Thanks for responding and I'll update with my results.
Comment 8 Cristian OneČ› 2015-04-28 04:16:29 UTC
(In reply to bugs from comment #7)
> Interesting. I didn't uninstall the old version of kmymoney (which came by
> default with Fedora 21) before compiling the new, so perhaps there's some
> conflict occurring. I'm not at that machine at the moment but will try it
> again tomorrow. Regarding the 0's in the debit/credit section, I did that
> manually because it was complaining that certain rows had an improper number
> of columns. Probably because it was the older version as well ;-). Thanks
> for responding and I'll update with my results.

To be able to load the proper plugins (csvimporter is one of them) in a self-compiled setup the proper environment needs to be setup.

I use the following script:

#!/bin/bash
export KDEDIR=/home/cristi/dezvoltare/kmymoney-install
export KDEDIRS=$KDEDIR
export PATH=$KDEDIR/bin:$PATH
export LD_LIBRARY_PATH=$KDEDIR/lib64:$LD_LIBRARY_PATH
export QT_PLUGIN_PATH=$KDEDIR/lib64/kde4/plugins:$QT_PLUGIN_PATH
/home/cristi/dezvoltare/kmymoney-install/bin/kmymoney -n --config kmymoneydevelopmentrc --caption "KMyMoney Development Version" --lang=C

Where '/home/cristi/dezvoltare/kmymoney-install' is the path were KMyMoney compiled from sources is installed .
Comment 9 allan 2015-04-28 10:49:48 UTC
(In reply to bugs from comment #7)

> Regarding the 0's in the debit/credit section, I did that
> manually because it was complaining that certain rows had an improper number
> of columns. 

> Probably because it was the older version as well ;
No not necessarily.  What sometimes/often happens is the producer of the file
doesn't always put the same number of fields in the header as in the data -
possibly because of an extra field delimiter.
The plugin detects that and suggests there is a problem, which could be just 
that an incorrect start line has been selected.  The best thing is to cancel and
check.  If you are happy that all looks OK, just continue.  You will be given the
option to accept that row, or all similar.
Comment 10 bugs 2015-04-28 14:32:21 UTC
Is there a trick to get the plugin to show up? I've recompiled, installed kmymoney to /home/myuser/kmymoney, and verified that kmymoney/lib64/kde4/kmm_csvimport.so exists in that directory. However, when I run, CSV Import doesn't show up in the File->Import menu, and it doesn't show up under Settings->Configure KMyMoney->Plugins either. This happens whether I just run the executable directly, or if I use the following script:

#!/bin/bash
export KDEDIR=/home/myuser/kmymoney
export KDEDIRS=$KDEDIR
export PATH=$KDEDIR/bin:$PATH
export LD_LIBRARY_PATH=$KDEDIR/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$KDEDIR/lib64/kde4:$LD_LIBRARY_PATH
export QT_PLUGIN_PATH=$KDEDIR/lib64/kde4/plugins:$QT_PLUGIN_PATH
$KDEDIR/bin/kmymoney
Comment 11 bugs 2015-04-28 14:42:16 UTC
Seems my output shows: kmymoney(7660)/kdecore (trader) KServiceTypeTrader::defaultOffers: KServiceTypeTrader: serviceType  "KMyMoneyPlugin"  not found 
. That's probably the problem. However, none of the fixes here (https://www.kubuntuforums.net/archive/index.php/t-61827.html) seem to resolve it.
Comment 12 bugs 2015-04-28 14:50:25 UTC
I've tried running the following: 

$ kbuildsycoca4 --noincremental
kbuildsycoca4 running...
kbuildsycoca4(7892) VFolderMenu::loadDoc: Parse error in  "/home/myuser/.config/menus/applications-merged/xdg-desktop-menu-dummy.menu" , line  1 , col  1 :  "unexpected end of file" 

But they still don't show up after starting KMyMoney again.
Comment 13 Jack 2015-04-28 15:09:52 UTC
What about trying kbuildsycoca4 without the '--noincremental' ?
Comment 14 allan 2015-04-28 15:14:34 UTC
(In reply to bugs from comment #10)
> Is there a trick to get the plugin to show up? I've recompiled, installed
> kmymoney to /home/myuser/kmymoney, and verified that
> kmymoney/lib64/kde4/kmm_csvimport.so exists in that directory. However, when
> I run, CSV Import doesn't show up in the File->Import menu, and it doesn't
> show up under Settings->Configure KMyMoney->Plugins either. This happens
> whether I just run the executable directly, or if I use the following script:

I do get that occasionally after a change.  The official fix is run kbuildsycoca4.  However, without fail, I just restart KMM and that always works.

For a quick trick, copy kmymoney/lib64/kde4/kmm_csvimport.so to /usr

However, I always install to /usr, like "cmake .. -DCMAKE_INSTALL_PREFIX=/usr", no script.

Do the uninstall again first, make sure your kmymoney/lib64/kde4/kmm_csvimport.so has gone, then install to /usr, and check that /usr/lib64/kde4 has the plugin.

I must admit I've had problems installing elsewhere, and haven't had time to get to the bottom of it.

So, it's /usr for me.

> #!/bin/bash
> export KDEDIR=/home/myuser/kmymoney
> export KDEDIRS=$KDEDIR
> export PATH=$KDEDIR/bin:$PATH
> export LD_LIBRARY_PATH=$KDEDIR/lib64:$LD_LIBRARY_PATH
> export LD_LIBRARY_PATH=$KDEDIR/lib64/kde4:$LD_LIBRARY_PATH
> export QT_PLUGIN_PATH=$KDEDIR/lib64/kde4/plugins:$QT_PLUGIN_PATH
> $KDEDIR/bin/kmymoney
Comment 15 bugs 2015-04-28 15:49:24 UTC
Thanks. I built it the Fedora way and everything seems to be working fine. I appreciate your help!
Comment 16 allan 2015-04-28 17:00:20 UTC
(In reply to bugs from comment #15)
> Thanks. I built it the Fedora way and everything seems to be working fine. I
> appreciate your help!

OK, I'll bite!  What's 'the Fedora way'?
Comment 17 bugs 2015-04-28 17:26:58 UTC
Well, I spoke with the kmymoney package maintainer for Fedora on #kde-devel. His instructions:

 yum install @buildsys-build fedpkg; fedpkg clone -a kmymoney; cd kmymoney;

Edit the spec file to point to 4.7.2 (it points at 4.7.1 today, but he said he planned to fix).

Then, either drop the tarball in that directory or run spectool -g kmymoney.spec, and then, finally:

sudo yum-builddep kmymoney.spec;  fedpkg local

That will leave you with a (in my case) x86_64 directory full of installable rpms, and a noarch directory which has the kmymoney doc rpm.

HTH.
Comment 18 allan 2015-04-28 22:40:27 UTC
(In reply to bugs from comment #17)
> Well, I spoke with the kmymoney package maintainer for Fedora on #kde-devel.
> His instructions:
> 
>  yum install @buildsys-build fedpkg; fedpkg clone -a kmymoney; cd kmymoney;
> 
> Edit the spec file to point to 4.7.2 (it points at 4.7.1 today, but he said
> he planned to fix).
> 
> Then, either drop the tarball in that directory or run spectool -g
> kmymoney.spec, and then, finally:
> 
> sudo yum-builddep kmymoney.spec;  fedpkg local
> 
> That will leave you with a (in my case) x86_64 directory full of installable
> rpms, and a noarch directory which has the kmymoney doc rpm.
> 
> HTH.

Ok, so Fedora-specific then.  Nice to know.  So, that will keep you going until
the development branch starts to move on.