Bug 335965

Summary: mathematical functions like sin, exp,... unusable in python scripts
Product: [Applications] kig Reporter: Maurizio Paolini <maurizio.paolini>
Component: generalAssignee: David E. Narvaez <david.narvaez>
Status: REOPENED ---    
Severity: normal CC: aspotashev, encukou, hrvoje.senjan, kevin.kofler, rdieter
Priority: NOR    
Version: unspecified   
Target Milestone: ---   
Platform: Fedora RPMs   
OS: Linux   
URL: https://bugzilla.redhat.com/show_bug.cgi?id=1238113
Latest Commit: Version Fixed In:
Attachments: kig file that exposes the problem
explicitly use QLibrary to load libpython (like pykde does)

Description Maurizio Paolini 2014-06-08 18:19:32 UTC
with kig compiled from the latest git source.
Error detected when creating a new python script that contains some mathematical function.  Here is an example:

- create a numerical label (e.g. x coordinate of a point)
- open python script creation dialog, selecting the label as argument
- enter the following script:
------------------------------------------
def calc( arg1 ):
  x = arg1.value()
  y = sin(x)
  return DoubleObject(y)
----------------------------------------
When the "Finish" button is pressed I get the following error message:

The Python interpreter caught an error during the execution of your script. Please fix the script and click the Finish button again.

   Details

The Python Interpreter generated the following error output:
NameError: global name 'sin' is not defined
  File "<string>", line 9, in calc
Traceback (most recent call last) [...]

Reproducible: Always

Steps to Reproduce:
see above



Maybe this has something to do with comment 9 in bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=1101626
Comment 1 David E. Narvaez 2014-06-09 05:29:00 UTC
Does this work?

def calc( arg1 ):
  x = arg1.value()
  y = math.sin(x)
  return DoubleObject(y)
Comment 2 Maurizio Paolini 2014-06-09 12:22:35 UTC
It does not work either:

The Python Interpreter generated the following error output:
NameError: global name 'math' is not defined
  File "<string>", line 9, in calc
Traceback (most recent call last):

(In reply to comment #1)
> Does this work?
> 
> def calc( arg1 ):
>   x = arg1.value()
>   y = math.sin(x)
>   return DoubleObject(y)
Comment 3 David E. Narvaez 2014-06-09 12:36:39 UTC
How about importing math directly?

def calc( arg1 ):
  import math
  x = arg1.value()
  y = math.sin(x)
  return DoubleObject(y)

I cannot reproduce this in Gentoo so I am pretty sure this will boil down to something specific to Fedora. Technically, Kig's code already imports math and its contents but for some reason it is not sticking when the code is executed.
Comment 4 Maurizio Paolini 2014-06-09 12:50:20 UTC
I still get an error, now it reads:

The Python Interpreter generated the following error output:
ImportError: /usr/lib/python2.7/lib-dynload/math.so: undefined symbol: PyFloat_Type
  File "<string>", line 8, in calc
Traceback (most recent call last):

In addition, I noticed also an error appearing in the console running "kig":

$ Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /usr/lib/python2.7/lib-dynload/math.so: undefined symbol: PyFloat_Type

The exact same error is displayed also for the previous scripts.
Trying "ldd -r /usr/lib/libboost_python.so"
gives a long list of undefined symbols, among which I also find
      PyFloat_Type
(https://bugzilla.redhat.com/show_bug.cgi?id=1101626, comment 6)

(In reply to comment #3)
> How about importing math directly?
> 
> def calc( arg1 ):
>   import math
>   x = arg1.value()
>   y = math.sin(x)
>   return DoubleObject(y)
> 
> I cannot reproduce this in Gentoo so I am pretty sure this will boil down to
> something specific to Fedora. Technically, Kig's code already imports math
> and its contents but for some reason it is not sticking when the code is
> executed.
Comment 5 David E. Narvaez 2014-06-09 13:01:51 UTC
OK, then it is a linking error so I'll close this as downstream. If Fedora investigates and finds out it is something to do with Kig I will reopen and see what can be fixed.
Comment 6 Rex Dieter 2014-06-09 18:02:52 UTC
David, what version of boost and python are you using to test?  does your libboost_python contain undefined symbols too?
Comment 7 Christoph Feck 2014-07-21 19:35:47 UTC
David, any idea regarding comment #6?
Comment 8 David E. Narvaez 2014-07-22 06:42:11 UTC
(In reply to Rex Dieter from comment #6)
> David, what version of boost and python are you using to test?  does your
> libboost_python contain undefined symbols too?

Python 2.7.7
Boost 1.55.0

My Boost-Python library is properly linked.
Comment 9 Kevin Kofler 2015-03-15 14:57:58 UTC
Boost linkage was fixed:
https://bugzilla.redhat.com/show_bug.cgi?id=1102667
and the problem still occurs.
Comment 10 David E. Narvaez 2015-03-16 00:48:17 UTC
(In reply to Kevin Kofler from comment #9)
> Boost linkage was fixed:
> https://bugzilla.redhat.com/show_bug.cgi?id=1102667
> and the problem still occurs.

Please post the output of

$ ldd -r  /usr/lib64/python2.7/lib-dynload/math.so

and your boost-python version.
Comment 11 Maurizio Paolini 2015-03-16 09:45:21 UTC
I have a 32 bit architecture, here is the result for me:

$ rpm -q boost-python
boost-python-1.55.0-8.fc21.i686

$ ldd -r  /usr/lib/python2.7/lib-dynload/math.so
        linux-gate.so.1 =>  (0xb7800000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb77b3000)
        libc.so.6 => /lib/libc.so.6 (0xb75e6000)
        /lib/ld-linux.so.2 (0x49ceb000)
undefined symbol: PyFloat_Type  (/usr/lib/python2.7/lib-dynload/math.so)
undefined symbol: sinh  (/usr/lib/python2.7/lib-dynload/math.so)
undefined symbol: ceil  (/usr/lib/python2.7/lib-dynload/math.so)
[...]
undefined symbol: log1p (/usr/lib/python2.7/lib-dynload/math.so)
undefined symbol: PyMem_Malloc  (/usr/lib/python2.7/lib-dynload/math.so)

for a total of 64 undefined symbols
Comment 12 David E. Narvaez 2015-03-16 15:28:20 UTC
Thanks Murizio, please post that output in the bug report in Comment 9. As a workaround while RedHat fixes linkage problems, you can run kig from the console with

 $ LD_PRELOAD=/usr/lib64/libpython2.7.so.1.0 kig --nofork

or the lib folder if you are using 32 bits.
Comment 13 Rex Dieter 2015-03-16 15:49:13 UTC
I cannot reproduce the problem now, trying the example from Comment #1, I get:

The Python Interpreter generated the following error output:
TypeError: calc() takes exactly 1 argument (0 given)

The Python interpreter caught an error during the execution of your script. Please fix the script and click the Finish button again.


Which at least implies stuff is getting loaded properly now.
Comment 14 David E. Narvaez 2015-03-16 15:51:19 UTC
(In reply to Rex Dieter from comment #13)
> Which at least implies stuff is getting loaded properly now.

No, it means you didn't follow the steps on Comment 1: you need to click on the numeric label so that it will be passed as the first argument.
Comment 15 Rex Dieter 2015-03-16 15:53:48 UTC
I'm not a kig user, sorry, the directions didn't mention how to create a label specifically.  How to do that?
Comment 16 Rex Dieter 2015-03-16 15:54:03 UTC
Or give some other simple test case?
Comment 17 David E. Narvaez 2015-03-16 16:04:04 UTC
(In reply to Rex Dieter from comment #15)
> I'm not a kig user, sorry, the directions didn't mention how to create a
> label specifically.  How to do that?

To create a numeric label press N and then click anywhere in the drawing area.
Comment 18 Rex Dieter 2015-03-16 16:05:27 UTC
OK , I see I make a valid example now (and get no error), but I do see in console output still:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /usr/lib/python2.7/lib-dynload/math.so: undefined symbol: PyFloat_Type

and do not get that when using LD_PRELOAD=/usr/lib64/libpython2.7.so.1.0  , odd though, since kig also is linking libpython
Comment 19 Maurizio Paolini 2015-03-16 16:06:01 UTC
You can create a numeric label by first creating a point, then right-click on the point and
select "Add Text Label" and then "X-coordinate".
a number should appear below the point, that is a numeric label that can be used
as an argument (see comment 1)
Comment 20 Maurizio Paolini 2015-03-16 16:13:57 UTC
Sorry David for the cross-post... and thank you for the workaround, it works fine.
However, unlike Rex, without the "LD_PRELOAD=" I cannot use mathematical functions!
Comment 21 Maurizio Paolini 2015-06-30 15:04:04 UTC
Created attachment 93431 [details]
kig file that exposes the problem

To aid in investigating the problem, this kig file exposes it.
the command "kig mathbug.kig" should display a sinusoidal curve (as a locus)
it works with "LD_PRELOAD /usr/lib/libpython2.7.so kik mathbug.kig"
but it produces an "[invalid]" object otherwise.
Comment 22 Rex Dieter 2015-07-01 14:05:37 UTC
https://projects.kde.org/projects/kde/kdeedu/kig/repository/revisions/master/entry/scripting/python_scripter.cc#L384

Essentially does:

Py_Initialize();
...	
s = newstring( "import math; from math import *;" );
...
PyRun_SimpleString( s );

and this is where it fails to load anything from math (per comment #1 above).

Comparing with pykde4, I see it (essentially) does an extra dlopen on libpython to ensure symbols are available at runtime, does kig need to do something similar here?  See also:

https://projects.kde.org/projects/kde/kdebindings/python/pykde4/repository/revisions/master/entry/kpythonpluginfactory/kpythonpluginfactory.cpp#L98
Comment 23 Rex Dieter 2015-07-01 14:41:40 UTC
Created attachment 93445 [details]
explicitly use QLibrary to load libpython (like pykde does)

This patch seems to fix the issue for me.  Would you prefer I post to reviewboard?
Comment 24 David E. Narvaez 2015-07-01 14:59:59 UTC
I am not going to include workarounds for distro bugs in Kig. You can either bug Fedora about their broken linkage or apply that patch in the Kig package for Fedora. In any case, stop reopening this bug since there is nothing to fix in Kig.
Comment 25 Rex Dieter 2015-07-01 15:42:09 UTC
It's not a distro bug, python modules do not link libpython by default.
Comment 26 Rex Dieter 2015-07-01 15:43:25 UTC
Are you saying how pykde works is wrong then? :)
Comment 27 Hrvoje Senjan 2015-07-01 16:06:52 UTC
i can reproduce the problem in openSUSE FWIW
Comment 28 David E. Narvaez 2015-07-01 16:19:04 UTC
(In reply to Hrvoje Senjan from comment #27)
> i can reproduce the problem in openSUSE FWIW

Do you have undefined symbols in math.so? (See comment 11).
Comment 29 Rex Dieter 2015-07-01 16:20:19 UTC
Let me correct comment #18, kig doesn't link python itself, only kigpart does, which may be part of the problem here.
Comment 30 Petr Viktorin 2015-07-01 16:43:29 UTC
Hi, one of Python maintainers in Fedora here (though I've never heard of kig before).

math.so is an extension module *for Python*. Loading Python's math module before loading Python itself doesn't make much sense to me.
Comment #22 mentions that kig calls "Py_Initialize" before importing the math library. Where does the symbol "Py_Initialize" come from if libpython is not yet loaded?


Note that Fedora does build math, among others, as a shared module. This is not the upstream default, but I do believe it's valid configuration. (See https://hg.python.org/cpython/file/53975668c9e9/Modules/Setup.dist -- Fedora has the lines "*shared*" and "math mathmodule.c _math.c" uncommented, among others.)
Comment 31 David E. Narvaez 2015-07-02 05:09:45 UTC
(In reply to Petr Viktorin from comment #30)
> Note that Fedora does build math, among others, as a shared module. This is
> not the upstream default, but I do believe it's valid configuration. (See
> https://hg.python.org/cpython/file/53975668c9e9/Modules/Setup.dist -- Fedora
> has the lines "*shared*" and "math mathmodule.c _math.c" uncommented, among
> others.)

So distro plays around with the default installation settings, distro breaks an application, nothing new here. I still do not understand why are we discussing this in the KDE bug tracker.
Comment 32 Petr Viktorin 2015-07-02 07:52:15 UTC
(In reply to David E. Narvaez from comment #31)
> I still do not understand why are we
> discussing this in the KDE bug tracker.

I believe we're discussing to find out why there's unexpected behavior in systems where Python is configured differently. We assumed you might be interested; if not, just remove yourself or reassign. No one is forcing you to investigate or fix this; Fedora already includes the patch.

To get Fedora out of the picture, I reproduced the same `ldd -r` output with Python-2.6.9 source tarball, with math as a shared library. If you believe building math as a shared library this counts as misconfiguring Python, I can open a Python bug to improve docs or remove the option.
As a Python developer, I'd be interested in knowing the root cause, so that possibly Python can be improved.
Comment 33 David E. Narvaez 2015-07-02 12:21:44 UTC
(In reply to Petr Viktorin from comment #32)
> I believe we're discussing to find out why there's unexpected behavior in
> systems where Python is configured differently. We assumed you might be
> interested; if not, just remove yourself or reassign. No one is forcing you
> to investigate or fix this; Fedora already includes the patch.

Can you add the URL to where this specific bug is being tracked in Fedora to the URL field of this bug?
Comment 34 Rex Dieter 2015-07-02 13:06:17 UTC
Adding reference to downstream bug (fedora 22)
https://bugzilla.redhat.com/show_bug.cgi?id=1238113

as requested

There's also an older (fedora 21)
https://bugzilla.redhat.com/show_bug.cgi?id=1101626

which is essentially the same issue.
Comment 35 Rex Dieter 2015-12-28 17:37:23 UTC
Reviewboard submitted,

https://git.reviewboard.kde.org/r/126549/
Comment 36 Rex Dieter 2017-12-28 14:55:14 UTC
*** Bug 388241 has been marked as a duplicate of this bug. ***