Bug 240475 - example netcdf file complains "You cannot load this much data"
Summary: example netcdf file complains "You cannot load this much data"
Status: RESOLVED FIXED
Alias: None
Product: kst
Classification: Applications
Component: datasources (show other bugs)
Version: 1.7.0
Platform: Debian unstable Linux
: NOR normal
Target Milestone: ---
Assignee: kst
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-06-02 16:11 UTC by Timo Lindfors
Modified: 2010-06-03 08:41 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
netcdf testcase to reproduce the bug (740 bytes, application/octet-stream)
2010-06-02 16:12 UTC, Timo Lindfors
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Timo Lindfors 2010-06-02 16:11:21 UTC
Version:           1.7.0 (using KDE 4.4.3) 
OS:                Linux

Steps to reproduce:
1) cat > testcase1.cdl <<EOF
netcdf foo {  // an example netCDF specification in CDL

dimensions:
     lat = 10, lon = 5, time = unlimited ;

variables:
     long    lat(lat), lon(lon), time(time);
     float   Z(time,lat,lon), t(time,lat,lon);
     double  p(time,lat,lon);
     long    rh(time,lat,lon);

     // variable attributes
     lat:long_name = "latitude";
     lat:units = "degrees_north";
     lon:long_name = "longitude";
     lon:units = "degrees_east";
     time:units = "seconds since 1992-1-1 00:00:00";
     Z:units = "geopotential meters";
     Z:valid_range = 0., 5000.;
     p:_FillValue = -9999.;
     rh:_FillValue = -1;

data:
     lat   = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90;
     lon   = -140, -118, -96, -84, -52;
}
EOF
2) ncgen -b -o testcase1.netcdf testcase1.cdl
3) kst
4) Data Wizard
5) select testcase1.netcdf
6) select all data columns
7) Next, Next, Finish

Expected results:
7) kst opens testcase1.netcdf

Actual results:
7) kst prints

"You requested to read in over 16777215 TB of data but it seems that you have approximately only 0 TB of usable memory available. You cannot load this much data."

More info:
1) The example is from the manual page of the ncgen command that I use to generate netcdf files.
2) If I select only lat and lon columns then kst does not crash and opens the data successfully. Maybe the crash is related to the fact that other columns do not have data points in this file?
3) I am not quite sure which version of KDE is in debian unstable:

$ COLUMNS=100 dpkg -l | grep kde
ii  kde-icons-oxygen    4:4.4.4-1           transitional package for oxygen icon theme
ii  kdebase-data        4:4.4.4-1           shared data files for the KDE 4 base module
rc  kdebase-runtime     4:4.3.4-2           runtime components from the official KDE 4 release
ii  kdebase-runtime-dat 4:4.4.4-1           shared data files for the KDE 4 base runtime module
ii  kdelibs-data        4:3.5.10.dfsg.1-3   core shared data for all KDE applications
ii  kdelibs4-dev        4:3.5.10.dfsg.1-3   development files for the KDE core libraries
ii  kdelibs4c2a         4:3.5.10.dfsg.1-3   core libraries and binaries for all KDE applications
rc  kdelibs5            4:4.3.4-3           core libraries for all KDE 4 applications
rc  kdelibs5-data       4:4.3.4-3           core shared data for all KDE 4 applications
rc  libkdeedu4          4:4.3.4-1+b1        libraries for KDE 4 educational applications
rc  liblockdev1         1.0.3-1.2+b1        Run-time shared library for locking devices

suggests 4.4.4 but that option is not available in bugs.kde.org.

Reproducible: Didn't try
Comment 1 Timo Lindfors 2010-06-02 16:12:11 UTC
Created attachment 47611 [details]
netcdf testcase to reproduce the bug
Comment 2 Nicolas Brisset 2010-06-02 16:21:53 UTC
Hi Timo,

just my first idea: in kst1, the netcdf datasource does not properly check for dimensions. It can load only vectors, but shows all types of variables, and it is possible that the computation of the number of available samples gives a completely wrong number. But it needs to be investigated a bit more to make sure.
You could also try to use a recent kst2 snapshot to see how it behaves there. A couple of days ago the netcdf datasource for kst2 has been substantially improved. It is not 100% done, but may work better.
You can find the links and info on the kst mailing list if you need support for that.
Comment 3 Andrew Walker 2010-06-02 20:58:30 UTC
From what I understand of netcdf only the lon and lat fields actually have data, all the rest are empty. The correct solution would seem to be to remove the empty vectors from the field list - so that the user cannot ask for them to be plotted.

I have tried a patch for this in 1.10.0-devel which appears to achieve the desired effect.

Let me know if there is any reason not to take this approach.
Comment 4 Andrew Walker 2010-06-02 22:58:27 UTC
SVN commit 1133924 by arwalker:

BUG:240475 return 0 for the samplesForFrame for any empty field. This effectively creates a vector with a single element with a NaN value. This fixes the problem described in this bug report for the kst 1.10 development branch.

 M  +10 -0     netcdf.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1133924
Comment 5 Timo Lindfors 2010-06-03 08:41:35 UTC
Thank you! I can confirm that after I apply

http://websvn.kde.org/branches/extragear/kde3/graphics/kst/src/datasources/netcdf/netcdf.cpp?r1=826410&r2=1133924&pathrev=1133924&view=patch

I can successfully read testcase1.netcdf. The vectors that are not in use just show a single nan value.