Bug 330032 - Cannot import NumPy in the Python backend
Summary: Cannot import NumPy in the Python backend
Status: RESOLVED FIXED
Alias: None
Product: cantor
Classification: Applications
Component: python2-backend (show other bugs)
Version: unspecified
Platform: Kubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: Filipe Saraiva
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-01-16 13:29 UTC by Tuukka Verho
Modified: 2014-11-10 18:37 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.14.3


Attachments
Patch to load python symbols at runtime with ExportExternalSymbols (1015 bytes, patch)
2014-02-03 07:28 UTC, Tuukka Verho
Details
Unit test (1.31 KB, text/plain)
2014-02-03 07:43 UTC, Tuukka Verho
Details
Unit test (.h file) (1.03 KB, text/x-chdr)
2014-02-03 07:44 UTC, Tuukka Verho
Details
Unit test (.cpp file) (1.16 KB, text/x-c++src)
2014-02-03 07:47 UTC, Tuukka Verho
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tuukka Verho 2014-01-16 13:29:11 UTC
Only occurs when using Cantor, not with ipython for example.

Reproducible: Always

Steps to Reproduce:
execute 'import numpy' in a python2 worksheet
Actual Results:  
The following backtrace:

Traceback (most recent call last):
File "", line 1, in 
File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 137, in 
import add_newdocs
File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 9, in 
from numpy.lib import add_newdoc
File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 4, in 
from type_check import *
File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 8, in 
import numpy.core.numeric as _nx
File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 5, in 
import multiarray
ImportError: /usr/lib/python2.7/dist-packages/numpy/core/multiarray.so: undefined symbol: PyExc_SystemError


Cantor version 0.5 from Kubuntu-ppa backports
Comment 1 Filipe Saraiva 2014-01-16 14:16:14 UTC
Thanks for your feedback Tuukka. Could you try import another modules, like scipy, matplotlib, sieve, and more?
Comment 2 Tuukka Verho 2014-01-16 14:29:44 UTC
Importing scipy and matplotlib fails because they try to import numpy (same backtrace). I don't have sieve. I can import some modules (such as PyKDE4, PIL, random) but many others fail.

'import serial' gives the following
ImportError: /usr/lib/python2.7/lib-dynload/termios.x86_64-linux-gnu.so: undefined symbol: PyExc_TypeError

'import vtk' gives the following
ImportError: /usr/lib/libvtkPythonCore.so.5.8: undefined symbol: PyBool_Type

'import ctypes':
ImportError: /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so: undefined symbol: PyFloat_Type

Looks like the CPython symbols can't be found?
Comment 3 Tuukka Verho 2014-01-16 15:16:38 UTC
I compiled and ran the following test code:

#include <Python.h>

int
main(int argc, char *argv[])
{
  Py_Initialize();
  PyRun_SimpleString("import numpy\n"
	                  "print dir(numpy)\n");
  PyErr_Print();
  Py_Finalize();
  return 0;
}

And the output was as expected. So there doesn't seem to be anything fundamentally wrong with my setup.
Comment 4 RJVB 2014-01-22 22:45:31 UTC
I discovered exactly the same issue in code of mine (a data visualisation app with an embedded Python interpreter loadable via plugin) that works fine on OS X, Cygwin and used to work under Linux (still does on the current Sabayon version). The app contains a few C++ modules and is thus linked using g++ but the Python plugin is pure C.
When I step through the plugin loader in the debugger, "p PyExc_SystemError" shows that the symbols is available, but munged by a C++ compiler. I presume this is the source of the error, with the multiarray module being pure C.
Looking at my build logs for Python (admittedly on OS X), I see that module/python.c is compiled with g++, and I suspect it's that module that has the definition for PyExc_SystemError.

What bugs me is that other Python packages are not affected, and that the issue doesn't arise either when loading numpy from within Python itself...
Comment 5 Tuukka Verho 2014-01-23 06:58:55 UTC
I tried compiling the above test code with g++ instead of gcc, it still worked.

Also, I tried the python backend on my other Kubuntu machine and had the same problem.
Comment 6 RJVB 2014-01-25 10:49:29 UTC
I've been looking into this more, and I'm now convinced it's an issue with having various python libraries around, due to the way the distribution is conceived (with "minimal" Python packages for use by system applications, and add-on packages allowing more advanced uses). 

Observations:
- the same binaries of my application work perfectly fine under Sabayon Linux
- they worked on the Debian Testing VM I had even after converting it into Linux Mint Debian by simply changing the appropriate entries in apt/sources.list
- they give the import error on a pure Linux Mint Debian system

Conclusion: my code and the way I build it on Linux is not in error.

Observations:
- I built Python 2.7.6 from the official sources and installed it in /usr/local/Python/2.7.6
- running the resulting python2.7 gave erratic behaviour because the dynamic loader linked in the system's libpython.so rather than the appropriate one in /usr/local/Python/2.7.6; I had to launch it with the proper lib dir added to LD_LIBRARY_PATH (empty otherwise!)
- when launching my own app with LD_LIBRARY_PATH set, behaviour was as intended when built using my own python 2.7.6
- without LD_LIBRARY_PATH set, the import error reappeared.

Conclusion: this is an issue to be raised with the distribution maintainers. I think it should be considered a bug.

On OS X, dynamic library paths are hard-coded (but can be relative), which is often a PITA but also means you can have basically as many versions or even copies in parallel as you wish, and always be sure of the one being used. I haven't yet found a way to obtain the same behaviour under linux ...
Comment 7 Tuukka Verho 2014-01-25 11:50:11 UTC
I tried compiling Cantor and the backend from latest sources, but that did not get rid of the error. I first removed the packages with apt-get so there's no risk it could've be using the old version of the plugin.
Comment 8 RJVB 2014-01-25 13:29:39 UTC
You could try to do the same as I did. Get the Python 2.7.6 tarball from python.org, configure and install it like so

```
configure --prefix=/usr/local/Python/2.7.6 --enable-shared
sudo make install
```

and then create a wrapper in /usr/local/bin/python2.7 that reads

```
#!/bin/sh

LD_LIBRARY_PATH="/usr/local/Python/2.7.6/lib:${LD_LIBRARY_PATH}"
exec /usr/local/Python/2.7.6/bin/python2.7 "$@"
```

and then figure out what packages you'll have to install in that Python installation. Finally, launch Cantor with LD_LIBRARY_PATH containing /usr/local/Python/2.7.6, and see if things work as they should.

Of course you can replace the 2.7.6 version by the Python version you need...
Comment 9 Tuukka Verho 2014-02-02 22:44:50 UTC
Ok I've found out what this is about. Basically, the python plugin is loaded in the RTLD_LOCAL mode, which means the symbols in the plugin won't be made available to any other object, e.g. numpy. This is the same problem as what is described here: https://github.com/pv/pythoncall. Also, the same fix works: if you add

dlopen("libpython2.7.so", RTLD_NOW|RTLD_GLOBAL);

to the backend, the bug no longer occurs. Obviously this is a hack that is not suited as a permanent solution, though.

It seems QPluginLoader is supposed to be somehow intelligent about how it opens plugins. Maybe there is something in Kubuntu that makes it not work correctly. Perhaps worth asking someone knowledgeable about KPluginLoader/QPluginLoader...
Comment 10 Filipe Saraiva 2014-02-03 01:43:21 UTC
Thanks Tuukka, could you provide a patch or indicate where add 'dlopen' call?
Comment 11 Tuukka Verho 2014-02-03 07:28:46 UTC
Created attachment 84959 [details]
Patch to load python symbols at runtime with ExportExternalSymbols

Here's a patch that does what I described above but using QLibrary instead of calling dlopen() directly. This should work on any platform as long as Python 2.7 is present.
Comment 12 Tuukka Verho 2014-02-03 07:43:27 UTC
Created attachment 84963 [details]
Unit test

Here's a test for this bug. The following lines will be needed in CMakeLists.txt:

kde4_add_unit_test(testpython2 testpython2.cpp)
target_link_libraries( testpython2
  ${KDE4_KDECORE_LIBS}
  ${QT_QTTEST_LIBRARY}
  cantorlibs
  cantortest
)

Note: this backend is affected by https://bugs.kde.org/show_bug.cgi?id=330665, so the fix needs to be applied before the test can work.
Comment 13 Tuukka Verho 2014-02-03 07:44:08 UTC
Created attachment 84964 [details]
Unit test (.h file)
Comment 14 Tuukka Verho 2014-02-03 07:47:00 UTC
Created attachment 84965 [details]
Unit test (.cpp file)

Oops, I left some garbage in the original version I submitted. Here's the cleaned up version.
Comment 15 Filipe Saraiva 2014-11-01 12:56:22 UTC
Git commit 0ecc94c70e841099620a0dff1f2e9afc7a958f15 by Filipe Saraiva.
Committed on 01/11/2014 at 12:56.
Pushed by filipesaraiva into branch 'master'.

Add fix to import numpy in python backend
python symbols is loaded again.
CCMAIL: tuukka.verho@aalto.fi

M  +3    -0    src/backends/python2/CMakeLists.txt
M  +10   -0    src/backends/python2/python2backend.cpp
A  +43   -0    src/backends/python2/testpython2.cpp     [License: GPL (v2+)]
A  +38   -0    src/backends/python2/testpython2.h     [License: GPL (v2+)]

http://commits.kde.org/cantor/0ecc94c70e841099620a0dff1f2e9afc7a958f15
Comment 16 Filipe Saraiva 2014-11-01 12:59:29 UTC
Hi all, sorry for this big delay. I sent the fixes developed by Tuukka for master branch. Thank you, Tuukka!
Please, could someone test the fix again?
Thank you for all and best regards.
Comment 17 Tuukka Verho 2014-11-01 13:32:20 UTC
Oops, the fix doesn't seem to work on my current system, which is Kubuntu 14.10. I'll try to investigate.
Comment 18 Tuukka Verho 2014-11-01 13:57:53 UTC
Never mind, it was because it kept loading the old version of the plugin. Now it works. Thanks!
Comment 19 Filipe Saraiva 2014-11-01 14:26:41 UTC
Uff! =)

Just more one thing Tuukka, could you test the code changing from QLibrary pythonLib("python2.7"); to QLibrary pythonLib("python2"); ?

I know for now all Linux distributions are using 2.7, but if a more generic code works, it will be preferable.

Thanks!
Comment 20 Tuukka Verho 2014-11-01 15:07:48 UTC
> Just more one thing Tuukka, could you test the code changing from QLibrary
> pythonLib("python2.7"); to QLibrary pythonLib("python2"); ?
> 
> I know for now all Linux distributions are using 2.7, but if a more generic
> code works, it will be preferable.

Unfortunately doesn't work that way, I guess the version number is a part of the library name in this case.