Thanks for your feedback Tuukka. Could you try import another modules, like scipy, matplotlib, sieve, and more?

Importing scipy and matplotlib fails because they try to import numpy (same backtrace). I don't have sieve. I can import some modules (such as PyKDE4, PIL, random) but many others fail.

'import serial' gives the following
ImportError: /usr/lib/python2.7/lib-dynload/termios.x86_64-linux-gnu.so: undefined symbol: PyExc_TypeError

'import vtk' gives the following
ImportError: /usr/lib/libvtkPythonCore.so.5.8: undefined symbol: PyBool_Type

'import ctypes':
ImportError: /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so: undefined symbol: PyFloat_Type

Looks like the CPython symbols can't be found?

I compiled and ran the following test code:

#include <Python.h>

int
main(int argc, char *argv[])
{
  Py_Initialize();
  PyRun_SimpleString("import numpy\n"
	                  "print dir(numpy)\n");
  PyErr_Print();
  Py_Finalize();
  return 0;
}

And the output was as expected. So there doesn't seem to be anything fundamentally wrong with my setup.

I discovered exactly the same issue in code of mine (a data visualisation app with an embedded Python interpreter loadable via plugin) that works fine on OS X, Cygwin and used to work under Linux (still does on the current Sabayon version). The app contains a few C++ modules and is thus linked using g++ but the Python plugin is pure C.
When I step through the plugin loader in the debugger, "p PyExc_SystemError" shows that the symbols is available, but munged by a C++ compiler. I presume this is the source of the error, with the multiarray module being pure C.
Looking at my build logs for Python (admittedly on OS X), I see that module/python.c is compiled with g++, and I suspect it's that module that has the definition for PyExc_SystemError.

What bugs me is that other Python packages are not affected, and that the issue doesn't arise either when loading numpy from within Python itself...

I tried compiling the above test code with g++ instead of gcc, it still worked.

Also, I tried the python backend on my other Kubuntu machine and had the same problem.

I've been looking into this more, and I'm now convinced it's an issue with having various python libraries around, due to the way the distribution is conceived (with "minimal" Python packages for use by system applications, and add-on packages allowing more advanced uses). 

Observations:
- the same binaries of my application work perfectly fine under Sabayon Linux
- they worked on the Debian Testing VM I had even after converting it into Linux Mint Debian by simply changing the appropriate entries in apt/sources.list
- they give the import error on a pure Linux Mint Debian system

Conclusion: my code and the way I build it on Linux is not in error.

Observations:
- I built Python 2.7.6 from the official sources and installed it in /usr/local/Python/2.7.6
- running the resulting python2.7 gave erratic behaviour because the dynamic loader linked in the system's libpython.so rather than the appropriate one in /usr/local/Python/2.7.6; I had to launch it with the proper lib dir added to LD_LIBRARY_PATH (empty otherwise!)
- when launching my own app with LD_LIBRARY_PATH set, behaviour was as intended when built using my own python 2.7.6
- without LD_LIBRARY_PATH set, the import error reappeared.

Conclusion: this is an issue to be raised with the distribution maintainers. I think it should be considered a bug.

On OS X, dynamic library paths are hard-coded (but can be relative), which is often a PITA but also means you can have basically as many versions or even copies in parallel as you wish, and always be sure of the one being used. I haven't yet found a way to obtain the same behaviour under linux ...

I tried compiling Cantor and the backend from latest sources, but that did not get rid of the error. I first removed the packages with apt-get so there's no risk it could've be using the old version of the plugin.

You could try to do the same as I did. Get the Python 2.7.6 tarball from python.org, configure and install it like so

```
configure --prefix=/usr/local/Python/2.7.6 --enable-shared
sudo make install
```

and then create a wrapper in /usr/local/bin/python2.7 that reads

```
#!/bin/sh

LD_LIBRARY_PATH="/usr/local/Python/2.7.6/lib:${LD_LIBRARY_PATH}"
exec /usr/local/Python/2.7.6/bin/python2.7 "$@"
```

and then figure out what packages you'll have to install in that Python installation. Finally, launch Cantor with LD_LIBRARY_PATH containing /usr/local/Python/2.7.6, and see if things work as they should.

Of course you can replace the 2.7.6 version by the Python version you need...

Ok I've found out what this is about. Basically, the python plugin is loaded in the RTLD_LOCAL mode, which means the symbols in the plugin won't be made available to any other object, e.g. numpy. This is the same problem as what is described here: https://github.com/pv/pythoncall. Also, the same fix works: if you add

dlopen("libpython2.7.so", RTLD_NOW|RTLD_GLOBAL);

to the backend, the bug no longer occurs. Obviously this is a hack that is not suited as a permanent solution, though.

It seems QPluginLoader is supposed to be somehow intelligent about how it opens plugins. Maybe there is something in Kubuntu that makes it not work correctly. Perhaps worth asking someone knowledgeable about KPluginLoader/QPluginLoader...

Thanks Tuukka, could you provide a patch or indicate where add 'dlopen' call?

Created attachment 84959 [details]
Patch to load python symbols at runtime with ExportExternalSymbols

Here's a patch that does what I described above but using QLibrary instead of calling dlopen() directly. This should work on any platform as long as Python 2.7 is present.

Created attachment 84963 [details]
Unit test

Here's a test for this bug. The following lines will be needed in CMakeLists.txt:

kde4_add_unit_test(testpython2 testpython2.cpp)
target_link_libraries( testpython2
  ${KDE4_KDECORE_LIBS}
  ${QT_QTTEST_LIBRARY}
  cantorlibs
  cantortest
)

Note: this backend is affected by https://bugs.kde.org/show_bug.cgi?id=330665, so the fix needs to be applied before the test can work.

Created attachment 84964 [details]
Unit test (.h file)

Created attachment 84965 [details]
Unit test (.cpp file)

Oops, I left some garbage in the original version I submitted. Here's the cleaned up version.

Git commit 0ecc94c70e841099620a0dff1f2e9afc7a958f15 by Filipe Saraiva.
Committed on 01/11/2014 at 12:56.
Pushed by filipesaraiva into branch 'master'.

Add fix to import numpy in python backend
python symbols is loaded again.
CCMAIL: tuukka.verho@aalto.fi

M  +3    -0    src/backends/python2/CMakeLists.txt
M  +10   -0    src/backends/python2/python2backend.cpp
A  +43   -0    src/backends/python2/testpython2.cpp     [License: GPL (v2+)]
A  +38   -0    src/backends/python2/testpython2.h     [License: GPL (v2+)]

http://commits.kde.org/cantor/0ecc94c70e841099620a0dff1f2e9afc7a958f15

Hi all, sorry for this big delay. I sent the fixes developed by Tuukka for master branch. Thank you, Tuukka!
Please, could someone test the fix again?
Thank you for all and best regards.

Oops, the fix doesn't seem to work on my current system, which is Kubuntu 14.10. I'll try to investigate.

Never mind, it was because it kept loading the old version of the plugin. Now it works. Thanks!

Uff! =)

Just more one thing Tuukka, could you test the code changing from QLibrary pythonLib("python2.7"); to QLibrary pythonLib("python2"); ?

I know for now all Linux distributions are using 2.7, but if a more generic code works, it will be preferable.

Thanks!

> Just more one thing Tuukka, could you test the code changing from QLibrary
> pythonLib("python2.7"); to QLibrary pythonLib("python2"); ?
> 
> I know for now all Linux distributions are using 2.7, but if a more generic
> code works, it will be preferable.

Unfortunately doesn't work that way, I guess the version number is a part of the library name in this case.