Bug 299637 - Difficulty importing small CSV file following large file, using same profile.
Summary: Difficulty importing small CSV file following large file, using same profile.
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Applications
Component: general (show other bugs)
Version: git (master)
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-05-08 16:41 UTC by allan
Modified: 2012-05-27 12:56 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description allan 2012-05-08 16:41:49 UTC
If only a few entries are imported from a large file, the resulting "TrailerLines" value can prevent the
next small file from importing most of its lines.  It could even be that just a single line gets imported, and, if that is a header line with just a column or two, it becomes impossible to advance into the wizard to access and change the 'last line' widget setting, and the config file would need to be edited.

Reproducible: Sometimes

Steps to Reproduce:
1. Import a large csv file, but when importing, opt to import only, say 10 lines.
2. Proceed with the import,  clicking 'Finish' at the end, to save the settings.
3. The m_trailerLines value will be high (the difference between the last line imported and the true last line).
4. Now import a small csv file.
5. The number of lines displayed should be the true last line minus the m_trailerLines value, but because m_trailerLines has such a high value, only the first line might be displayed. Even if some lines are visible, selecting a different field separator, usually produces the symptoms.
6. Click 'Next'. If only one column is present on the visible line, it is not possible to select columns for the various fields.  Because no columns have been selected, the conditions to advance to the next page have not been met, so it is not possible to advance.  Because it is not possible to advance, it is not possible to change the start/end line settings.

Actual Results:  
The file cannot be imported using the present profile.  It is necessary either to edit this profile's trailer line setting or to create a new profile.

Expected Results:  
It should not be possible for one import to leave behind a condition which can prevent certain other files from being imported.
Comment 1 Jack 2012-05-08 19:54:54 UTC
I was going to suggest that the importer actuall store the number of trailer lines to skip rather than the actual last line to import, but apparently this is how it does work.  However, this can only lead to this problem if the curren file to import both is relatively short and also has fewer trailer lines to ignore than specified in the profile.  In this case, putting a limit on the number of lines trimmed, possibly only if the new file is shorter than some arbitrary length or maybe twice the number of trimmed lines.

Another option might be to continue to display all lines, so as to not prevent the user from increasing the number of the last line to import.
Comment 2 allan 2012-05-08 21:31:53 UTC
Presently, I obtain the actual number of lines in the file, and subtract from this the saved  number of trailer lines.  The resulting 'last line' value gets loaded into the UI, and the remainder get dropped.  The unfortunate consequence of this is that if the saved  number of trailer lines is greater than or close to the actual number of lines in the next file,(using that same profile), then all or most of that file will get dropped.  Ironically, if, as originally done, it were the user's input last line that was saved, then this current problem couldn't arise.

Almost invariably, there is just one trailer line, so what I'm contemplating is to limit the saved trailer lines to perhaps just a maximum of two or three, so that it is more likely that a subsequent small file is less likely to get decimated.  Better still, perhaps, is to ignore the trailer lines setting if it is greater than the true last line, so that the full file is read in. 

I've now tried this later suggestion and it looks promising, so far.

Going back to the symptoms of this bug, rathet than the cure, the situation can be even worse.  To attempt to avoid an incorrect choice of field delimiter, I have in place a check for the number of columns detected, the rationale being that an incorrect choice is likely to curtail the number of columns.  If this check fails, it's impossible to advance from the field delimiter page.  Selecting the correct delimiter was expected to solve that situation.  However, if the trailer line issue results in just a single line being imported, and if that line has, say, just a date stamp, then there is no chance to proceed at all.

The new method handles this, too.
Comment 3 allan 2012-05-27 12:46:14 UTC
Git commit 07084e5af960435aceac276523986cc218213791 by Allan Anderson.
Committed on 27/05/2012 at 14:02.
Pushed by allananderson into branch 'master'.
using same profile.
Correct error message when checking decimal symbol.
Moved some code from readSettings() to slotFileDialogClicked() to ensure initial display shows start of file.

M  +4    -0    kmymoney/plugins/csvimport/completionwizardpage.ui
M  +155  -100  kmymoney/plugins/csvimport/csvdialog.cpp
M  +12   -1    kmymoney/plugins/csvimport/csvdialog.h
M  +172  -122  kmymoney/plugins/csvimport/csvdialog.ui
M  +71   -31   kmymoney/plugins/csvimport/investprocessing.cpp
M  +3    -1    kmymoney/plugins/csvimport/investprocessing.h
M  +2    -1    kmymoney/plugins/csvimport/lines-datewizardpage.ui
M  +1    -2    kmymoney/plugins/csvimport/redefinedlg.cpp
M  +2    -3    kmymoney/plugins/csvimport/symboltabledlg.cpp
M  +4    -4    kmymoney/plugins/csvimport/symboltabledlg.h

http://commits.kde.org/kmymoney/07084e5af960435aceac276523986cc218213791
Comment 4 allan 2012-05-27 12:56:19 UTC
To be clear about what now happens with 'trailer lines'.  Instead of dropping unwanted trailer lines during setup, they are now highlighted, to give the user a clearer view of the end of the file, to allow him to change the end line setting if need be.  Unwanted lines now get dropped during import.