Bug 468209 - Cannot import massive data
Summary: Cannot import massive data
Status: RESOLVED FIXED
Alias: None
Product: LabPlot2
Classification: Applications
Component: general (show other bugs)
Version: 2.9.0
Platform: Neon Linux
: NOR major
Target Milestone: ---
Assignee: Alexander Semke
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-04-06 14:29 UTC by Martin Tlustos
Modified: 2023-04-08 15:26 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In: 2.10.1
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Tlustos 2023-04-06 14:29:16 UTC
SUMMARY
I have a csv with weather data. 141601 lines, 18 columns, 20MB. I can't import it with "import data" (see my other bug report), and when trying to open the data in libreoffice, copying it and pasting it into labplot, all the memory is being filled (15GB), until labplot crashes.

STEPS TO REPRODUCE
1. start labplot
2. create table
3. copy and paste data from libreoffice

OBSERVED RESULT
massive memory usage and final crash (no error report).

EXPECTED RESULT
should be able to import the data

SOFTWARE/OS VERSIONS
Windows: 
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
Comment 1 Alexander Semke 2023-04-08 15:26:14 UTC
Thank you for reporting this issue! 

I did a small fix now to reduce the peak memory consumption and also not to crash when running out of memory. We'll bring the fix into the next patch release of 2.10. More optimizations are possible here, we'll track these activities in https://invent.kde.org/education/labplot/-/issues/549.

My test data was a bit bigger than in your example (150k rows and 20 columns) and the peak memory consumption was at 514MB. If labplot is crashing with your data, it means the memory consumption on your system is already very high or you're working maybe with a lot of text data.

The file import is more efficient. So, ideally we address the other bug ticket that you raised. Please check my reply there.