If only a few entries are imported from a large file, the resulting "TrailerLines" value can prevent the next small file from importing most of its lines. It could even be that just a single line gets imported, and, if that is a header line with just a column or two, it becomes impossible to advance into the wizard to access and change the 'last line' widget setting, and the config file would need to be edited. Reproducible: Sometimes Steps to Reproduce: 1. Import a large csv file, but when importing, opt to import only, say 10 lines. 2. Proceed with the import, clicking 'Finish' at the end, to save the settings. 3. The m_trailerLines value will be high (the difference between the last line imported and the true last line). 4. Now import a small csv file. 5. The number of lines displayed should be the true last line minus the m_trailerLines value, but because m_trailerLines has such a high value, only the first line might be displayed. Even if some lines are visible, selecting a different field separator, usually produces the symptoms. 6. Click 'Next'. If only one column is present on the visible line, it is not possible to select columns for the various fields. Because no columns have been selected, the conditions to advance to the next page have not been met, so it is not possible to advance. Because it is not possible to advance, it is not possible to change the start/end line settings. Actual Results: The file cannot be imported using the present profile. It is necessary either to edit this profile's trailer line setting or to create a new profile. Expected Results: It should not be possible for one import to leave behind a condition which can prevent certain other files from being imported.
I was going to suggest that the importer actuall store the number of trailer lines to skip rather than the actual last line to import, but apparently this is how it does work. However, this can only lead to this problem if the curren file to import both is relatively short and also has fewer trailer lines to ignore than specified in the profile. In this case, putting a limit on the number of lines trimmed, possibly only if the new file is shorter than some arbitrary length or maybe twice the number of trimmed lines. Another option might be to continue to display all lines, so as to not prevent the user from increasing the number of the last line to import.
Presently, I obtain the actual number of lines in the file, and subtract from this the saved number of trailer lines. The resulting 'last line' value gets loaded into the UI, and the remainder get dropped. The unfortunate consequence of this is that if the saved number of trailer lines is greater than or close to the actual number of lines in the next file,(using that same profile), then all or most of that file will get dropped. Ironically, if, as originally done, it were the user's input last line that was saved, then this current problem couldn't arise. Almost invariably, there is just one trailer line, so what I'm contemplating is to limit the saved trailer lines to perhaps just a maximum of two or three, so that it is more likely that a subsequent small file is less likely to get decimated. Better still, perhaps, is to ignore the trailer lines setting if it is greater than the true last line, so that the full file is read in. I've now tried this later suggestion and it looks promising, so far. Going back to the symptoms of this bug, rathet than the cure, the situation can be even worse. To attempt to avoid an incorrect choice of field delimiter, I have in place a check for the number of columns detected, the rationale being that an incorrect choice is likely to curtail the number of columns. If this check fails, it's impossible to advance from the field delimiter page. Selecting the correct delimiter was expected to solve that situation. However, if the trailer line issue results in just a single line being imported, and if that line has, say, just a date stamp, then there is no chance to proceed at all. The new method handles this, too.
Git commit 07084e5af960435aceac276523986cc218213791 by Allan Anderson. Committed on 27/05/2012 at 14:02. Pushed by allananderson into branch 'master'. using same profile. Correct error message when checking decimal symbol. Moved some code from readSettings() to slotFileDialogClicked() to ensure initial display shows start of file. M +4 -0 kmymoney/plugins/csvimport/completionwizardpage.ui M +155 -100 kmymoney/plugins/csvimport/csvdialog.cpp M +12 -1 kmymoney/plugins/csvimport/csvdialog.h M +172 -122 kmymoney/plugins/csvimport/csvdialog.ui M +71 -31 kmymoney/plugins/csvimport/investprocessing.cpp M +3 -1 kmymoney/plugins/csvimport/investprocessing.h M +2 -1 kmymoney/plugins/csvimport/lines-datewizardpage.ui M +1 -2 kmymoney/plugins/csvimport/redefinedlg.cpp M +2 -3 kmymoney/plugins/csvimport/symboltabledlg.cpp M +4 -4 kmymoney/plugins/csvimport/symboltabledlg.h http://commits.kde.org/kmymoney/07084e5af960435aceac276523986cc218213791
To be clear about what now happens with 'trailer lines'. Instead of dropping unwanted trailer lines during setup, they are now highlighted, to give the user a clearer view of the end of the file, to allow him to change the end line setting if need be. Unwanted lines now get dropped during import.