Bug 174806 - python stuff pretends to be installed in the system python prefix
Summary: python stuff pretends to be installed in the system python prefix
Status: RESOLVED FIXED
Alias: None
Product: bindings
Classification: Developer tools
Component: general (show other bugs)
Version: unspecified
Platform: Compiled Sources Linux
: VHI critical
Target Milestone: ---
Assignee: kde-bindings
URL:
Keywords:
: 183291 198927 (view as bug list)
Depends on:
Blocks:
 
Reported: 2008-11-10 19:58 UTC by Tom Albers
Modified: 2012-01-17 17:32 UTC (History)
22 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Patch to honour the PYTHON_SITE_PACKAGES_DIR path passed in the command line. (1.11 KB, patch)
2009-03-14 19:26 UTC, Raphael Kubo da Costa
Details
This patches up kdelibs to provide the new CMake variables and improved python helper script. (8.84 KB, patch)
2009-03-15 04:17 UTC, Michael Witten
Details
This updates kdebase to make use of the changes in kdelibs, so that python modules are handled better (785 bytes, patch)
2009-03-15 04:22 UTC, Michael Witten
Details
Patches for kdelibs, kdebase, and kdebindings. (11.81 KB, application/x-bzip)
2009-03-25 03:32 UTC, Michael Witten
Details
add some comments about setting PYTHON_SITE_PACKAGES_DIR (406 bytes, patch)
2009-12-13 21:55 UTC, Marcos Dione
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tom Albers 2008-11-10 19:58:56 UTC
because of:

CMake Error at workspace/plasma/scriptengines/python/cmake_install.cmake:68 (FILE):
  file INSTALL cannot copy file
  "/mnt/data/kde/source/kdebase/workspace/plasma/scriptengines/python/plasmascript.py"
  to "/usr/lib64/python2.5/site-packages/PyKDE4/plasmascript.py".
Call Stack (most recent call first):
  workspace/plasma/scriptengines/cmake_install.cmake:40 (INCLUDE)
  workspace/plasma/cmake_install.cmake:43 (INCLUDE)
  workspace/cmake_install.cmake:62 (INCLUDE)
  cmake_install.cmake:39 (INCLUDE)

it is not possible to install kdebase, a similar error appears in kdeedu. 

Would it be possible to detect that and ask for the sudo password in these cases?

It seems to be a limitation in Python < 2.6.
Comment 1 Pino Toscano 2008-11-15 12:39:24 UTC
This is not kdesvn-build specific, but due to the Python plasma stuff in kdebase.
Comment 2 Pino Toscano 2008-11-15 12:53:33 UTC
Ideally it should just install in the chosen installation prefix, and in case it is different than the Python one, give a message saying to setup PYTHONPATH properly.
Comment 3 Pino Toscano 2009-02-05 12:59:29 UTC
*** Bug 183291 has been marked as a duplicate of this bug. ***
Comment 4 Sebastian Sauer 2009-03-04 22:39:50 UTC
Severity=>major though for me it's even a blocker cause it does break my automated build-process and hence "Blocks development and/or testing work" :-/
Comment 5 Raphael Kubo da Costa 2009-03-14 19:26:05 UTC
Created attachment 32108 [details]
Patch to honour the PYTHON_SITE_PACKAGES_DIR path passed in the command line.

This patch for kdelibs honours the PYTHON_SITE_PACKAGES_DIR value passed in the command line with the -D parameter and only sets it to the value returned by distutils.sysconfig.get_python_lib(plat_specific=1) (which is usually /usr/lib/python2.N/site-packages) if a value is not specified in the command line.

It may be useful to do the same to PYTHON_INCLUDE_PATH too.
Comment 6 Raphael Kubo da Costa 2009-03-14 19:30:14 UTC
Additional comments about the previous patch to kdelibs:

If you're using kdesvn-build, adding 'cmake-options -DPYTHON_SITE_PACKAGES_DIR:PATH=/path/to/local/directory' to the 'module kdebase' directive should work. If you're compiling trunk manually, just pass the -D switch to cmake when compiling kdebase.

As I said, there may be some case in which PYTHON_INCLUDE_PATH should be modified too (it defaults to the global Python include path, something like /usr/include/python2.6), even though this has not happened to me yet.
Comment 7 Michael Witten 2009-03-15 03:10:00 UTC
I've got a patch ready to submit. I just need a moment to get it ready.
Comment 8 Michael Witten 2009-03-15 04:17:01 UTC
Created attachment 32117 [details]
This patches up kdelibs to provide the new CMake variables and improved python helper script.

I hope the comment I wrote is already attached to this. If not, see below.
Comment 9 Michael Witten 2009-03-15 04:22:30 UTC
Created attachment 32118 [details]
This updates kdebase to make use of the changes in kdelibs, so that python modules are handled better

PLEASE SEE MY ABOVE PATCH FOR kdelibs AS WELL. THEY BOTH APPLY TO revision 939041

I've got *2* patch that tries to make this whole enterprise as transparent as possible.

The crux of the problem is that KDE developers like to configure CMAKE_INSTALL_PREFIX so that it points to some directory under ${HOME}, while python has traditinally dictated that modules be installed to a system-wide site-packages directory. This causes a clash of permissions: `make install' can't install to this system-wide site-packages directory without special priveleges (for instance, `sudo make install').

For those with python>=2.6, the attached patches provides a complete fix. For those with python<2.6 installed, the attached patches provide help via cmake control.

Python>2.6 implements PEP 370, which provides for a per-user site-packages directory. Here's how the patches make use of this new functionality:

If the user has configured CMAKE_INSTALL_PREFIX such that it points to a directory that is not underneath ${HOME}, then cmake selects the system-wide site-packages directories for installing the python modules; this behavior is the same as it has always been.

If the user has configured CMAKE_INSTALL_PREFIX such that it points to a directory underneath the user's ${HOME}, then cmake assumes the user is a KDE developer and therefore intends to run `make install' without special priveleges, so cmake asks python for a path to the user's site-packages directory. If this succeeds, then everything is installed as expected, and the whole process is transparent.

However, if cmake finds that python doesn't know about user-specific site-packages directories, then cmake will message the user to explain the situation and to offer a partial solution: The user can force a directory to be used as the user's site-packages directory by specifying -DPYTHON_SITE_PACKAGES_INSTALL_DIR=$some_path on the command line.

The attached patches preserve PYTHON_SITE_PACKAGES_DIR and add 2 new variables: PYTHON_USER_SITE_PACKAGES_DIR to hold the user-specific site-packages directory of the user configuring KDE, and PYTHON_SITE_PACKAGES_INSTALL_DIR to multiplex between those 2 variables (it can always be overridden on the command line).

I have tested every scenario but 2: (1) systems without python and (2) systems with python<2.6. However, I'm fairly sure the code will work perfectly ;-)
Comment 10 Tom Albers 2009-03-17 19:30:40 UTC
Michael, note that this patch is now being discussed on the build-system mailinglist. You might want to comment there on questions or remarks from Alexander, our build system expert.

http://lists.kde.org/?t=123712361900003&r=1&w=2
Comment 11 Michael Witten 2009-03-25 03:32:49 UTC
Created attachment 32379 [details]
Patches for kdelibs, kdebase, and kdebindings.

Sorry for getting back WAY too late on this one. I've got excuses, but who cares... ;-)

I've attached a tarball that contains 3 pataches: kdelibs.patch, kdebase.patch, kdebindings.patch.

The main patch is kdelibs.patch, which cleans up some cruft and adds 3 new CMake commands: PARSE_ARGS(), ADD_PYTHON(), and INSTALL_PYTHON().

If I may boast, these are very cool and they do The Right Thing.

ADD_PYTHON() creates a target for compiling python modules (as well as whole packages) and INSTALL_PYTHON() allows for easily installing the results. Whereas before, there was just one always-out-of-date target to which all compilation commands (everywhere) were appended, it is now possible to create any number of named targets with outputs that only need to be rebuilt when necessary. More interesting is that the handling of packages and the preservation of relative paths might allow for a more pythonic organization of the python source code. In short, It's Awesome.

The patches to kdebase and kdebindings simply make use of the new commands where CURRENTLY appropriate, which---unfortunately---is not that many places: Both kdebase and kdelibs (and probably a lot of others) handle python modules improperly; python byte-code is specific to each version of the python interpreter. This is why python has interpreter-specific site-packages directories. Until many of the python modules are converted to modules that can be properly installed in versioned site-packages directories, you can't byte-compile the python and feel proud about it (and more importantly, you can't make use of the shiny new commands provided by kdelibs.patch ;-) As it is now, the misused compilation has been converted to direct installation of the .py files only.

Because of the large additions, the patches are kind of hard to read. I recommend applying them and then reading the affected files. The comments/documentation are pretty darn thorough. I would particularly point people to read about PARSE_ARGS(), which I think could be very helpful for further cmakery.

Also, as a very small footnote that can hopefully be disregarded, I didn't actually test my changes to kdebindings. I've compiled my qt-copy out of source, and my guess is that kdebindings makes a lot of egregious assumptions to the contrary (written in lurid tab-and-space-mixed perl, no less! Probably some emacs user). Needless to say, I couldn't get kdebindings to compile and install, but at least cmake ran and it's output looks suitable. Furthermore---and perhaps worse---I imagine that there are many more projects out there that will break, because I have thrown out the old PYTHON_INSTALL(). This is a good thing, for the path to the future is laid on the ruins of the past!
Comment 12 Detlev Casanova 2009-05-08 13:32:14 UTC
For me, after applying the patch, it tries to install in /PyKDE4 but can't of course.
Comment 13 eva 2009-05-17 20:56:07 UTC
The following patch has been applied to KDE trunk (rev. 965738): 
http://reviewboard.kde.org/r/553/diff/#index_header

So you should try setting -DPYTHON_SITE_PACKAGES_DIR=~/.local/lib/python2.6/site-packages

Worked fine here with python 2.6.0
Comment 14 David Faure 2009-05-20 23:45:59 UTC
So by default kde isn't installable into a custom prefix anymore, one needs to find this bug report in order to find out how to do that.... That's not really great for developers.

Could the cmake stuff detect that the location is not writable and use ~/.local in that case? [Not sure how this interacts with DESTDIR though]. Or at least telling the user what to do in such a case. Currently all a user sees is:
  file INSTALL cannot copy file
  "[...]"
  to "/usr/lib/python2.6/dist-packages/PyKDE4/plasmascript.py".

Which looks very buggy when the chosen prefix is not /usr....
Comment 15 Raphael Kubo da Costa 2009-05-21 00:41:30 UTC
There was a heated discussion on the kde-buildsystem mailing list some time ago about informing the user, or automatically choosing a local path (and how that only plays nice with Python 2.6+).

IIRC, Michael Witten's patch does something like that, but it needs some work so it hasn't been integrated.

Currently, I don't really know what to tell the user: if he's using Python <= 2.5 he needs to install somewhere and export PYTHONPATH to include his installation path (and then someone in the thread said exporting PYTHONPATH was not good, and someone else argued it was or something ;), if the user is using Python 2.6+ he should install to ~/.local/lib/python2.6/site-packages. The lowest common denominator here is just telling the user to pass -DPYTHON_SITE_PACKAGES_DIR to cmake, with which I agree, but I don't know if it's usual.
Comment 16 Pino Toscano 2009-07-04 18:36:37 UTC
*** Bug 198927 has been marked as a duplicate of this bug. ***
Comment 17 David Faure 2009-08-18 18:10:50 UTC
Michael's patches look good overall indeed, it's a shame they're not in svn ;)
I do like the autodetection solution, and I think this should definitely go in.
(and if not, we should at least tell people which var to set in order to fix this, currently it's unclear to everyone but those who find comment 13 in this report)

Has this been posted to kde-buildsystem@kde.org so that Alex (well, let's cc him) and the other cmake guys can have a look?

In particular, ParseArgs.cmake looks very generic, maybe it should go to cmake itself?
Comment 18 Raphael Kubo da Costa 2009-08-18 18:52:08 UTC
The patches have been discussed in kde-buildsystem (there has been some discussion over k-c-d as well), Alex asked for some changes and then the discussion stalled ;)

The main thread can be seen here: http://lists.kde.org/?l=kde-buildsystem&m=123722881329874&w=2
Comment 19 Beat Wolf 2009-08-24 12:00:09 UTC
I just stumbled over this bug yesterday and today over the bugreport. I hope this patch gets into svn, at least for 4.4
Comment 20 David Faure 2009-09-30 18:11:02 UTC
Michael: ping?
Comment 21 David Faure 2009-09-30 18:49:07 UTC
Hmm, re-reading http://lists.kde.org/?l=kde-buildsystem&m=123958336819732&w=2
I'm afraid that Thiago alienated Michael a bit :{

It looks like the small unrelated changes made "on the way" are the reason for this big fix not being committed, which is a bit sad.

IMHO the stuff should be committed as is and then we can talk about polishing  and/or detect breakages if the needs be, because that would be better than the current breakage going on forever and nothing happening for lack of an actual fix in svn. But I don't know anything about python so I certainly don't want to commit it myself, since I wouldn't really be able to make fixes if needed (nor fully explain what the patch does).

Michael: please commit your fix. You can blame me if someone says you should have posted a Nth version of the patch first.

I see benefits in reviews, but not when the result is discouragement and an important fix not going in at all.
Comment 22 Georg Grabler 2009-12-13 06:39:12 UTC
I just wanted to state: Still not committed, and quite annoying for automated building :-(.

Michael? You're still on CC, are you around as well?

Currently, this problem also fails kdesvn-build if you don't run it with sudo (and risk killing your stable copy).

I'll try to get fix the kdesvn-build problem meaning at least in a sandbox it would be safe (which is related to pathing most likely), but my python knowledge is much more limited (does not really exist) than my perl knowledge, so i'd really appreciate this if it was fixed.
Comment 23 Raphael Kubo da Costa 2009-12-13 16:59:03 UTC
(In reply to comment #22)
> Currently, this problem also fails kdesvn-build if you don't run it with sudo
> (and risk killing your stable copy).
> 
> I'll try to get fix the kdesvn-build problem meaning at least in a sandbox it
> would be safe (which is related to pathing most likely), but my python
> knowledge is much more limited (does not really exist) than my perl knowledge,
> so i'd really appreciate this if it was fixed.

Erm, you just have to pass "cmake-options -DPYTHON_SITE_PACKAGES_DIR:PATH=/foo/bar" inside the "module kdebase" section inside kdesvn-build...
Comment 24 Marcos Dione 2009-12-13 21:55:04 UTC
Created attachment 39030 [details]
add some comments about setting PYTHON_SITE_PACKAGES_DIR

meanwhile we're still deciding whether comitting the patch or not, would some documentation at least mitigate the problem?
Comment 25 g111 2009-12-13 22:55:58 UTC
Yes, a documentation would be good. I e.g. do not know where to set the path best. Why don't you set it to $KDEDIRS/lib/python2.6/site-packages per default?
Comment 26 Ben Cooksley 2010-06-12 13:46:23 UTC
Raising priority. Recent changes in kdebase mean that it *ALWAYS* tries to install the Plasma bindings now, regardless of the presence of PyQt4 or PyKDE4 ( which I don't have installed ). 

Requesting that the Plasma bindings be kicked out of kdebase to kdebindings, or the appropriate checks put in place.
Comment 27 Ben Cooksley 2010-06-12 14:52:29 UTC
Ugh. One shouldn't do these things late at night...

kicked out -> moved from.
Comment 28 Matthew Woehlke 2010-06-18 01:58:16 UTC
So, I'm confused, *why* aren't we simply using CMAKE_INSTALL_PREFIX with a big warning to set PYTHONPATH? After all that's exactly how *everything else* works; if CMAKE_INSTALL_PREFIX is not / or /usr, you have to set PATH, LD_LIBRARY_PATH, QT_PLUGIN_PATH, etc. in order to use the installation. What's wrong with adding PYTHONPATH to that list? Why is Python being given special treatment?
Comment 29 Yury G. Kudryashov 2012-01-17 17:32:54 UTC
Python stuff goes to $CMAKE_INSTALL_PREFIX for quite some time. The only trick is that one has to set SIP_DEFAULT_SIP_DIR.