463246 2022-12-19 18:51:17 +0000 Importing CSV's with double-sets of quotes ("") splits the field 2024-12-19 10:13:09 +0000 1 1 2 Applications kmymoney importer 5.1.3 Other Linux RESOLVED FIXED NOR normal --- 1 matt kmymoney-devel alx.kuzza matt https://invent.kde.org/office/kmymoney/commit/fc8f557ef56d07faf369ecc5ab917f5ef119493d 5.2 0 oldest_to_newest 2187085 0 matt 2022-12-19 18:51:17 +0000 SUMMARY I'm attempting to import Amazon Item Exports for reconciliation. Amazon's Description column uses `""` to indicate a " symbol (since " is used for surrounding a field). While LibreOffice handles this fine, KMyMoney treats that as a field delimiter, even though I've specified that fields are delimited by a comma (,). This forces me to have to perform manual manipulation before importing, and is against spec. Thank you! STEPS TO REPRODUCE 1. Create a CSV file with a line where the description field has a "" (eg. "The thing we purchased measures 4""x6""." ) 2. Import that CSV into KMyMoney 3. Look at the fields, and you'll see the "x6" and "." have their own fields. OBSERVED RESULT A few of my hundreds of lines parsed incorrectly, with "additional fields" that don't exist in other transactions, because the " symbol is necessary for data and the imported CSV uses "" to cause a " symbol to be part of the data. EXPECTED RESULT I expect the CSV to be parsed correctly, where fields are only separated by a comma, if the comma is not within "-symbols, and "" causes a " symbol to be part of the data. SOFTWARE/OS VERSIONS Windows: macOS: Linux/KDE Plasma: (available in About System) KDE Plasma Version: KDE Frameworks Version: Qt Version: ADDITIONAL INFORMATION 2187088 1 alx.kuzza 2022-12-19 19:44:50 +0000 Could you, please, attach a fragment of the CSV file you are talking about? 2187095 2 matt 2022-12-19 21:10:25 +0000 https://www.rfc-editor.org/rfc/rfc4180 ``` 2. Definition of the CSV Format ... 7. If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote. For example: "aaa","b""bb","ccc" ``` 2187103 3 ostroffjh 2022-12-19 21:36:02 +0000 It really would make it easier if you provide a sample csv file, even if it's only a header and one data row. It may be obvious to you what is wrong, but debugging to find exactly where in the code the problem occurs is easier if the developers have a real example to trace through. 2187306 4 matt 2022-12-20 15:20:13 +0000 (In reply to alx.kuzza from comment #1) > Could you, please, attach a fragment of the CSV file you are talking about? Sure! 2187307 5 154711 matt 2022-12-20 15:25:23 +0000 Created attachment 154711 Test CSV This is a test CSV. It includes a header (so the fields are understandable) directly from Amazon's Item Download. Two entries are included, left mostly unmodified. The first entry should parse fine. The second line will break up the string in field 3 and stick "650-Sheet Capacity" into field 4. Please accept my apologies, it appears my original description is likely off. It seems that, while the fields are not breaking at the double-quote (""), KMyMoney appears to "stop protecting commas" such that commas after a "" are treated as delimiters, not string text. 2187308 6 matt 2022-12-20 15:26:11 +0000 (In reply to Jack from comment #3) > It really would make it easier if you provide a sample csv file, even if > it's only a header and one data row. It may be obvious to you what is > wrong, but debugging to find exactly where in the code the problem occurs is > easier if the developers have a real example to trace through. That's totally reasonable. And a good request, since I realized that my original description was likely misleading. Please see test file and accompanying note. 2187310 7 matt 2022-12-20 15:27:35 +0000 Thank you for fast responses, and more thanks for creating such a powerful, beautiful, and friendly piece of F/OSS money-management software! 2187734 8 alx.kuzza 2022-12-21 08:53:04 +0000 (In reply to matt from comment #0) > SUMMARY > I'm attempting to import Amazon Item Exports for reconciliation. > Amazon's Description column uses `""` to indicate a " symbol (since " is > used for surrounding a field). While LibreOffice handles this fine, > KMyMoney treats that as a field delimiter, even though I've specified that > fields are delimited by a comma (,). > This forces me to have to perform manual manipulation before importing, and > is against spec. > > Thank you! > > STEPS TO REPRODUCE > 1. Create a CSV file with a line where the description field has a "" > (eg. "The thing we purchased measures 4""x6""." ) > 2. Import that CSV into KMyMoney > 3. Look at the fields, and you'll see the "x6" and "." have their own fields. > > OBSERVED RESULT > A few of my hundreds of lines parsed incorrectly, with "additional fields" > that don't exist in other transactions, because the " symbol is necessary > for data and the imported CSV uses "" to cause a " symbol to be part of the > data. > > EXPECTED RESULT > I expect the CSV to be parsed correctly, where fields are only separated by > a comma, if the comma is not within "-symbols, and "" causes a " symbol to > be part of the data. > > SOFTWARE/OS VERSIONS > Windows: > macOS: > Linux/KDE Plasma: > (available in About System) > KDE Plasma Version: > KDE Frameworks Version: > Qt Version: > > ADDITIONAL INFORMATION Good news, it is reproducible :) Could you, please, also provide the OS/Version you have a problem with. 2187772 9 bug-janitor 2022-12-21 13:03:36 +0000 A possibly relevant merge request was started @ https://invent.kde.org/office/kmymoney/-/merge_requests/180 2187775 10 matt 2022-12-21 13:31:36 +0000 This is reproducible on Kubuntu 22.04 running the KMyMoney-5.1-322-linux-gcc--x86_64.AppImage AppImage (Version 5.1.3-71876c371) 2189887 11 tbaumgart 2022-12-28 15:46:53 +0000 Git commit 33879f13b955753214385bc1c67e77c6d510a7d5 by Thomas Baumgart, on behalf of Alexander Kuznetsov. Committed on 28/12/2022 at 15:46. Pushed by tbaumgart into branch 'master'. Fix CSV parser to correctly support quoted cells M +60 -33 kmymoney/plugins/csv/import/core/csvutil.cpp M +23 -11 kmymoney/plugins/csv/import/core/tests/parsedata-test.cpp https://invent.kde.org/office/kmymoney/commit/33879f13b955753214385bc1c67e77c6d510a7d5 2189888 12 tbaumgart 2022-12-28 15:50:26 +0000 Git commit fc8f557ef56d07faf369ecc5ab917f5ef119493d by Thomas Baumgart, on behalf of Alexander Kuznetsov. Committed on 28/12/2022 at 15:50. Pushed by tbaumgart into branch '5.1'. Fix CSV parser to correctly support quoted cells Backported from master FIXED-IN: 5.1.4 M +2 -0 .gitignore M +40 -13 kmymoney/plugins/csv/import/core/csvutil.cpp M +21 -11 kmymoney/plugins/csv/import/core/tests/parsedata-test.cpp https://invent.kde.org/office/kmymoney/commit/fc8f557ef56d07faf369ecc5ab917f5ef119493d 2195143 13 matt 2023-01-09 22:16:15 +0000 Downloaded AppImage KMyMoney-5.1-348-linux-gcc-x86_64.AppImage and parsing appears to work correctly. Thank you! 154711 2022-12-20 15:25:23 +0000 2022-12-20 15:25:23 +0000 Test CSV test.csv text/csv 1416 matt T3JkZXIgRGF0ZSxPcmRlciBJRCxUaXRsZSxDYXRlZ29yeSxBU0lOL0lTQk4sVU5TUFNDIENvZGUs V2Vic2l0ZSxSZWxlYXNlIERhdGUsQ29uZGl0aW9uLFNlbGxlcixTZWxsZXIgQ3JlZGVudGlhbHMs TGlzdCBQcmljZSBQZXIgVW5pdCxQdXJjaGFzZSBQcmljZSBQZXIgVW5pdCxRdWFudGl0eSxQYXlt ZW50IEluc3RydW1lbnQgVHlwZSxQdXJjaGFzZSBPcmRlciBOdW1iZXIsUE8gTGluZSBOdW1iZXIs T3JkZXJpbmcgQ3VzdG9tZXIgRW1haWwsU2hpcG1lbnQgRGF0ZSxTaGlwcGluZyBBZGRyZXNzIE5h bWUsU2hpcHBpbmcgQWRkcmVzcyBTdHJlZXQgMSxTaGlwcGluZyBBZGRyZXNzIFN0cmVldCAyLFNo aXBwaW5nIEFkZHJlc3MgQ2l0eSxTaGlwcGluZyBBZGRyZXNzIFN0YXRlLFNoaXBwaW5nIEFkZHJl c3MgWmlwLE9yZGVyIFN0YXR1cyxDYXJyaWVyIE5hbWUgJiBUcmFja2luZyBOdW1iZXIsSXRlbSBT dWJ0b3RhbCxJdGVtIFN1YnRvdGFsIFRheCxJdGVtIFRvdGFsLFRheCBFeGVtcHRpb24gQXBwbGll ZCxUYXggRXhlbXB0aW9uIFR5cGUsRXhlbXB0aW9uIE9wdC1PdXQsQnV5ZXIgTmFtZSxDdXJyZW5j eSxHcm91cCBOYW1lCjA2LzA4LzEyLDAwMi0xMjM0ODUwLTQzMTI0NjQsIlBoeXNpY3Mgb2YgdGhl IEltcG9zc2libGU6IEEgU2NpZW50aWZpYyBFeHBsb3JhdGlvbiBpbnRvIHRoZSBXb3JsZCBvZiBQ aGFzZXJzLCBGb3JjZSBGaWVsZHMsIFRlbGVwb3J0YXRpb24sIGFuZCBUaW1lIFRyYXZlbCIsQUJJ U19CT09LLCIwMzA3Mjc4ODI0IiwiNTUxMDE1MDAiLEFtYXpvbi5jb20sMjAwOS0wNC0wN1QwMDow MDowMSxuZXcsQW1hem9uLmNvbSwsJDE1Ljk1LCQ5LjQ4LDEsIkNyZWRpdENhcmQgLSAyMDAwIiws LG1hdHRAbWUuY29tLDA2LzA4LzEyLE1hdHQgQ3Rlciw0NTUgV2VzdCBOb3doZXJlLCxBcmFnb25p YSxNSSw0MzQzNCxTaGlwcGVkLFVTUFMoOTM5MzkzOTQ5NTk1OTU5NDkzOTMwMiksJDkuNDgsJDAu MDAsJDkuNDgsLCwsTWF0dEN0ZXIsVVNELAowNi8xMS8xMiwwMDItMTIzNDUxNy00MzIxNjEyLCJD YXJkaW5hbMKuIFByZW1pZXIgMTEiIiB4IDE3IiIgQmluZGVyLCBFYXN5IE9wZW7CriBMb2NraW5n IFNsYW50LUTCriBSaW5ncywgMyIiLCA2NTAtU2hlZXQgQ2FwYWNpdHksIEJsYWNrIHdpdGggU3Bp bmUgTGFiZWwgSG9sZGVyIixTVE9SQUdFX0JJTkRFUiwiQjAwMTNDQk82SSIsIjQ0MDAwMDAwIixB bWF6b24uY29tLCx1c2VkIG1pbnQsQW1hem9uIFdhcmVob3VzZSwsJDM1Ljk5LCQxMy43OCwyLCJD cmVkaXRDYXJkIC0gMjAwMCIsLCxtYXR0QG1lLmNvbSwwNi8xMi8xMixNYXR0IEN0ZXIsNDU1IFdl c3QgTm93aGVyZSwsQXJhZ29uaWEsTUksNDkzMzcsU2hpcHBlZCxVU1BTKDQzMjUzMjM2MjU0MjU0 MjMyMzM0NDEpLCQyNy41NiwkMC4wMCwkMjcuNTYsLCwsTWF0dCBDdGVyLFVTRCwK