Bug 360747 - CSV Importer detects more columns than are assigned
Summary: CSV Importer detects more columns than are assigned
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Applications
Component: importer (show other bugs)
Version: git (master)
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-19 19:36 UTC by wojnilowicz
Modified: 2018-03-28 08:00 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.0.0
Sentry Crash Report:


Attachments
CSV Test File (329 bytes, text/plain)
2016-03-19 19:36 UTC, wojnilowicz
Details
Column Assignment for CSV file (73.47 KB, image/png)
2016-03-19 19:37 UTC, wojnilowicz
Details
Warning about columns number (15.66 KB, image/png)
2016-03-19 19:38 UTC, wojnilowicz
Details
[PATCH] CSV Importer detects more columns than are assigned (2.28 KB, patch)
2016-03-19 19:54 UTC, wojnilowicz
Details

Note You need to log in before you can comment on or make changes to this bug.
Description wojnilowicz 2016-03-19 19:36:13 UTC
My file is correct, KMM displays correct preview of CSV file, but I get warning about incorrect column number (see attachment)

Reproducible: Always

Steps to Reproduce:
1. file->import csv
2. choose investment
3. create new profile
4. open "test file.csv" and assign columns to values (see attachment)
5. FieldDelimiter to comma
6. TextDelimiter to double quotes
7. DecimalSymbol to comma
8. ImportCSV

Actual Results:  
Get warning that file doesn't have expected number of columns.

Expected Results:  
No warning at all.

FieldDelimiter=DecimalSymbol !
Comment 1 wojnilowicz 2016-03-19 19:36:58 UTC
Created attachment 97978 [details]
CSV Test File
Comment 2 wojnilowicz 2016-03-19 19:37:30 UTC
Created attachment 97979 [details]
Column Assignment for CSV file
Comment 3 wojnilowicz 2016-03-19 19:38:14 UTC
Created attachment 97980 [details]
Warning about columns number
Comment 4 wojnilowicz 2016-03-19 19:54:31 UTC
Created attachment 97981 [details]
[PATCH] CSV Importer detects more columns than are assigned

The place where warning is displayed is this:
>>>>>>>>>>>>>>>>>>>>>>
  if (m_columnList.count() < m_endColumn) {
    if (!m_csvDialog->m_accept) {
      QString row = QString::number(m_row);
      int ret = KMessageBox::questionYesNoCancel(0, i18n("<center>Row number %1 does not have the expected number of columns.</center>"
                "<center>This might not be a problem, but it may be a header line.</center>"
                "<center>You may accept all similar items, or just this one, or cancel.</center>",
                row), i18n("CSV import"),
                KGuiItem(i18n("Accept All")),
                KGuiItem(i18n("Accept This")),
                KGuiItem(i18n("Cancel")));
      if (ret == KMessageBox::Cancel) {
        return ret;
      }
      if (ret == KMessageBox::Yes) {
        m_csvDialog->m_accept = true;
      }
    }
>>>>>>>>>>>>>>>>>>>>>>

Where m_columnList:
m_columnList = m_parse->parseLine(data);

and

m_endColumn = m_maxColumnCount = colCount = data.count(m_parse->m_fieldDelimiterCharList[count]) + 1;


Data.count doesn't handle following lines well:
2016-01-25,MONNARI,Sprzedaż,20,"12,7",254,MON,3

It return 9 columns, while in fact it should return 8 (single value in double quotes "12,7" is treated as two columns). parseLine(data) returns correct value, which is 8 columns. Attached patch fixes this issue. Please revise it and apply to master
branch.
Comment 5 wojnilowicz 2016-05-08 07:12:02 UTC
Git commit 4debce51fe4afa867db9d33baaa1a0185b031379 by Łukasz Wojniłowicz.
Committed on 08/05/2016 at 07:11.
Pushed by wojnilowicz into branch 'master'.

Use parseLine() to determine most likely fieldDelimiter

Current routine doesn't calculate columns well when
FieldDelimiter=DecimalSymbol. parseLine() from csvutil.cpp does it
properly.
REVIEW: 127712

M  +3    -3    kmymoney/plugins/csvimport/investprocessing.cpp

http://commits.kde.org/kmymoney/4debce51fe4afa867db9d33baaa1a0185b031379
Comment 6 wojnilowicz 2016-05-08 07:15:13 UTC
Git commit 70b85864c91151bc68fe8fdae9c432bb93a44ba9 by Łukasz Wojniłowicz.
Committed on 08/05/2016 at 07:14.
Pushed by wojnilowicz into branch 'frameworks'.

Use parseLine() to determine most likely fieldDelimiter

Current routine doesn't calculate columns well when
FieldDelimiter=DecimalSymbol. parseLine() from csvutil.cpp does it
properly.
REVIEW: 127712

M  +3    -3    kmymoney/plugins/csvimport/investprocessing.cpp

http://commits.kde.org/kmymoney/70b85864c91151bc68fe8fdae9c432bb93a44ba9