SUMMARY I tried to import a CSV file but the app refused to see it. Passing the file through LibreOffice Calc and copy/pasting the data led to text columns and trying to convert to numerical crashed the app. I then tried to read the same file in a Python script and that's when I realized that it was encoded in UTF-16. Opening with Kate and saving as UTF-8 solved all the issues. The file in question was either generated as such by a scientific instrument's software (it's spectrometer data), or it was passed through Microsoft Office and somehow ended up with UTF-16. Either way, I think it's possibly a common occurrence and potential source of frustration for scientists who aren't very code savvy and have to work in a Microsoft/Origin dominated ecosystem. It would be nice to have at least a more verbose error message, if not full support for files with such weird encodings. STEPS TO REPRODUCE 1. Import or copy/paste from CSV file with UTF16 encoding. OBSERVED RESULT Either failure to load data with "File does not exist" error, or loading gibberish data that cannot be converted or parsed further without crashing the app. EXPECTED RESULT Error message pointing out incompatible encoding or support for said encoding. SOFTWARE/OS VERSIONS Windows: macOS: (available in the Info Center app, or by running `kinfo` in a terminal window) Linux/KDE Plasma: KDE Plasma Version: KDE Frameworks Version: Qt Version: ADDITIONAL INFORMATION
Git commit 16a3ceb0ab95a5ea47f32e8edc55e64a2ce81b5a by Alexander Semke. Committed on 20/11/2024 at 07:21. Pushed by asemke into branch 'master'. [ascii import] notify the user in the import dialog about the unsupported UTF16 encoding. FIXED-IN: 2.12 M +1 -0 ChangeLog M +40 -0 src/backend/datasources/filters/AsciiFilter.cpp M +1 -0 src/backend/datasources/filters/AsciiFilter.h M +1 -0 src/backend/datasources/filters/AsciiFilterPrivate.h M +15 -0 tests/import_export/ASCII/AsciiFilterTest.cpp M +2 -1 tests/import_export/ASCII/AsciiFilterTest.h https://invent.kde.org/education/labplot/-/commit/16a3ceb0ab95a5ea47f32e8edc55e64a2ce81b5a
(In reply to Alex E from comment #0) > SUMMARY > > I tried to import a CSV file but the app refused to see it. Passing the file > through LibreOffice Calc and copy/pasting the data led to text columns and > trying to convert to numerical crashed the app. I then tried to read the > same file in a Python script and that's when I realized that it was encoded > in UTF-16. Opening with Kate and saving as UTF-8 solved all the issues. > > The file in question was either generated as such by a scientific > instrument's software (it's spectrometer data), or it was passed through > Microsoft Office and somehow ended up with UTF-16. Either way, I think it's > possibly a common occurrence and potential source of frustration for > scientists who aren't very code savvy and have to work in a Microsoft/Origin > dominated ecosystem. > > It would be nice to have at least a more verbose error message, if not full > support for files with such weird encodings. Thank you for reporting this issue. We implemented an error message that is shown in the import dialog for such files, it will be part of the next release 2.12. Above you mentioned a crash of the application when working with utf16. In case this is reproducible for you, can you please provide the steps for how to reproduce this issue?
(In reply to Alexander Semke from comment #2) > (In reply to Alex E from comment #0) > > SUMMARY > > > > I tried to import a CSV file but the app refused to see it. Passing the file > > through LibreOffice Calc and copy/pasting the data led to text columns and > > trying to convert to numerical crashed the app. I then tried to read the > > same file in a Python script and that's when I realized that it was encoded > > in UTF-16. Opening with Kate and saving as UTF-8 solved all the issues. > > > > The file in question was either generated as such by a scientific > > instrument's software (it's spectrometer data), or it was passed through > > Microsoft Office and somehow ended up with UTF-16. Either way, I think it's > > possibly a common occurrence and potential source of frustration for > > scientists who aren't very code savvy and have to work in a Microsoft/Origin > > dominated ecosystem. > > > > It would be nice to have at least a more verbose error message, if not full > > support for files with such weird encodings. > Thank you for reporting this issue. We implemented an error message that is > shown in the import dialog for such files, it will be part of the next > release 2.12. > > Above you mentioned a crash of the application when working with utf16. In > case this is reproducible for you, can you please provide the steps for how > to reproduce this issue? I tried but I haven't been able to reproduce it at all. Now after copy/pasting from LibreOffice I just get text columns and converting to double/integer just makes them non-values, using the same UTF16 files. The crash I reported might have had more to do with trying to import the same files over and over again for something like 30 minutes with varying combinations of value separators. Possibly other apps working in the background were also an influence, but I cannot recall the exact conditions.