Bug 360747

Summary: CSV Importer detects more columns than are assigned
Product: [Applications] kmymoney Reporter: NSLW <lukasz.wojnilowicz>
Component: importerAssignee: KMyMoney Devel Mailing List <kmymoney-devel>
Status: RESOLVED FIXED    
Severity: normal CC: agander93, ralf.habacker
Priority: NOR    
Version: git (master)   
Target Milestone: ---   
Platform: Compiled Sources   
OS: Linux   
Latest Commit: http://commits.kde.org/kmymoney/70b85864c91151bc68fe8fdae9c432bb93a44ba9 Version Fixed In: 5.0.0
Attachments: CSV Test File
Column Assignment for CSV file
Warning about columns number
[PATCH] CSV Importer detects more columns than are assigned

Description NSLW 2016-03-19 19:36:13 UTC 
My file is correct, KMM displays correct preview of CSV file, but I get warning about incorrect column number (see attachment)

Reproducible: Always

Steps to Reproduce:
1. file->import csv
2. choose investment
3. create new profile
4. open "test file.csv" and assign columns to values (see attachment)
5. FieldDelimiter to comma
6. TextDelimiter to double quotes
7. DecimalSymbol to comma
8. ImportCSV

Actual Results:  
Get warning that file doesn't have expected number of columns.

Expected Results:  
No warning at all.

FieldDelimiter=DecimalSymbol !
Comment 1 NSLW 2016-03-19 19:36:58 UTC 
Created attachment 97978 [details]
CSV Test File
Comment 2 NSLW 2016-03-19 19:37:30 UTC 
Created attachment 97979 [details]
Column Assignment for CSV file
Comment 3 NSLW 2016-03-19 19:38:14 UTC 
Created attachment 97980 [details]
Warning about columns number
Comment 4 NSLW 2016-03-19 19:54:31 UTC 
Created attachment 97981 [details]
[PATCH] CSV Importer detects more columns than are assigned

The place where warning is displayed is this:
>>>>>>>>>>>>>>>>>>>>>>
  if (m_columnList.count() < m_endColumn) {
    if (!m_csvDialog->m_accept) {
      QString row = QString::number(m_row);
      int ret = KMessageBox::questionYesNoCancel(0, i18n("<center>Row number %1 does not have the expected number of columns.</center>"
                "<center>This might not be a problem, but it may be a header line.</center>"
                "<center>You may accept all similar items, or just this one, or cancel.</center>",
                row), i18n("CSV import"),
                KGuiItem(i18n("Accept All")),
                KGuiItem(i18n("Accept This")),
                KGuiItem(i18n("Cancel")));
      if (ret == KMessageBox::Cancel) {
        return ret;
      }
      if (ret == KMessageBox::Yes) {
        m_csvDialog->m_accept = true;
      }
    }
>>>>>>>>>>>>>>>>>>>>>>

Where m_columnList:
m_columnList = m_parse->parseLine(data);

and

m_endColumn = m_maxColumnCount = colCount = data.count(m_parse->m_fieldDelimiterCharList[count]) + 1;


Data.count doesn't handle following lines well:
2016-01-25,MONNARI,Sprzedaż,20,"12,7",254,MON,3

It return 9 columns, while in fact it should return 8 (single value in double quotes "12,7" is treated as two columns). parseLine(data) returns correct value, which is 8 columns. Attached patch fixes this issue. Please revise it and apply to master
branch.
Comment 5 NSLW 2016-05-08 07:12:02 UTC 
Git commit 4debce51fe4afa867db9d33baaa1a0185b031379 by Łukasz Wojniłowicz.
Committed on 08/05/2016 at 07:11.
Pushed by wojnilowicz into branch 'master'.

Use parseLine() to determine most likely fieldDelimiter

Current routine doesn't calculate columns well when
FieldDelimiter=DecimalSymbol. parseLine() from csvutil.cpp does it
properly.
REVIEW: 127712

M  +3    -3    kmymoney/plugins/csvimport/investprocessing.cpp

http://commits.kde.org/kmymoney/4debce51fe4afa867db9d33baaa1a0185b031379
Comment 6 NSLW 2016-05-08 07:15:13 UTC 
Git commit 70b85864c91151bc68fe8fdae9c432bb93a44ba9 by Łukasz Wojniłowicz.
Committed on 08/05/2016 at 07:14.
Pushed by wojnilowicz into branch 'frameworks'.

Use parseLine() to determine most likely fieldDelimiter

Current routine doesn't calculate columns well when
FieldDelimiter=DecimalSymbol. parseLine() from csvutil.cpp does it
properly.
REVIEW: 127712

M  +3    -3    kmymoney/plugins/csvimport/investprocessing.cpp

http://commits.kde.org/kmymoney/70b85864c91151bc68fe8fdae9c432bb93a44ba9