Noted, we will need to add support for Python 3.10 by next release.

As I mentioned previously, we tend to leave this until late in the Python release cycle because the API is prone to breaking changes even in late betas.

This time Python upstream has made life particularly inconvenient for us (c.f.https://bugzilla.redhat.com/show_bug.cgi?id=1898116#c12) and we'll have to work around their deliberate obstruction. C'est la vie.

A heads up that Arch Linux is now affected with their recent rebuild to Python 3.10.1. The package was removed from their repositories:

https://pkgbuild.com/~foutrelis/failed-py310-builds/kdevelop-python.log
https://github.com/archlinux/svntogit-packages/commits/packages/kdevelop-python
https://archlinux.org/todo/remaining-rebuilds-for-python-310/

Hei,

I use kdevelop for python development in my day-to-day job. It seem now that kdev-python has stopped working on arch, which I am currently using. I might in any case like to change distros, but it seems inevitable that this particular python version change will become an issue down the road for other distros as well.

Is there any way I could help out? If at all possible, I would really like to fund someone directly to work on this. I don't think I would have time to devote myself, in particular since I'm not very familiar with c++.

(In reply to Øystein S. Haaland from comment #3)
> I use kdevelop for python development in my day-to-day job. It seem now that
> kdev-python has stopped working on arch, which I am currently using.

I use KDevelop on Arch nearly every day too. A workaround I found is to install Python 3.9 (`python39` in AUR) and install `kdevelop-python` (now in AUR). Syntax highlighting and parsing works again as before, and so far, it is processing modules installed in /usr/lib/python3.10 for auto-completion, etc. I imagine Python 3.10-specific syntax won't work (I haven't checked) but at least things can continue to work for Python <= 3.9 projects.

(In reply to Luke Horwell from comment #4)
> (In reply to Øystein S. Haaland from comment #3)
> > I use kdevelop for python development in my day-to-day job. It seem now that
> > kdev-python has stopped working on arch, which I am currently using.
> 
> I use KDevelop on Arch nearly every day too. A workaround I found is to
> install Python 3.9 (`python39` in AUR) and install `kdevelop-python` (now in
> AUR). Syntax highlighting and parsing works again as before, and so far, it
> is processing modules installed in /usr/lib/python3.10 for auto-completion,
> etc. I imagine Python 3.10-specific syntax won't work (I haven't checked)
> but at least things can continue to work for Python <= 3.9 projects.

I too found the kdev-python AUR, installed that, and it seems to work as before. On my setup, I have py3.9 installed, not 3.10 yet. So in principle, I guess kdev-python should still have worked.

*** Bug 443167 has been marked as a duplicate of this bug. ***

Unfortunately Python completely removed the Parser module we were using from their public API, so there is no easy fix for this issue.

The most realistic option at this point is probably to write a Python snippet which uses the "ast" module to do the parsing (which is effectively exactly the same) and serializes the result somehow, e.g. into an XML or ProtocolBuffer stream. Then we can execute this snippet from C++ code (this works without creating a process using Python's C API) and deserialize it, giving us the same data structure as before.

From an end-user perspective, you can use the KDevelop AppImage from our website, which we can build against Python 3.9 until this issue is resolved somehow. It will of course not support Python 3.10 syntax, so if you require that, you are out of luck for now.

(In reply to Sven Brauch from comment #7)
> Unfortunately Python completely removed the Parser module we were using from
> their public API, so there is no easy fix for this issue.
> 
> The most realistic option at this point is probably to write a Python
> snippet which uses the "ast" module to do the parsing (which is effectively
> exactly the same) and serializes the result somehow, e.g. into an XML or
> ProtocolBuffer stream. Then we can execute this snippet from C++ code (this
> works without creating a process using Python's C API) and deserialize it,
> giving us the same data structure as before.

Thanks a lot for sketching a possible route for implementing this!

I would be more than willing to put some effort into this if I had the time. But maybe with some guidance it would still be possible to have some progress every now and then.

The first part seem easy enough;  write the code that reads a file with python code and loads it into the ast.

You mention a few ways serialization can be done. Are there any examples of similar approaches already in the code base? Are there already places where de-serialization is done with one of the formats mentioned (or other formats for that matter)?

Any pointers for tools/tests in the repo that can be useful for working with this would be much appreciated.

Btw. would it make things easier if one used cython or mypyc to compile the python part to a c module?

Again, if there is any way I could fund this work more or less directly, I would also be interested in hearing about that.

My own thinking is that it's probably easier to fork (or, so far as possible, vendor in the most recent version of) CPython's parser code.

The amount of code involved is relatively small - see https://github.com/python/typed_ast/tree/master/ast3 which is another fork.

(in the past I'd considered changing kdev-python to use that typed-ast library instead of linking against the CPython parser, but it's now deprecated with the features added to CPython upstream).

Sadly I haven't yet had time to work on this, but I hope to in a few weeks. Right now I'm busy buying and moving onto a houseboat...

To address the development comments:

It's great that you're interested in contributing to kdev-python. As you can see it's kind of needed at the moment. Consider dropping by #kdevelop on Libera IRC if you want to discuss it (ping FLHerne [me] or scummos [Sven]).

Tools/tests:
There are quite a lot of unit/end-to-end tests. Run `ninja test` (or `make test`) after building kdev-python, or for more detailed output run the test binaries individually (e.g. build/parser/tests/pyasttest , build/duchain/tests/pyduchaintest ).

parser/generated.h converts from CPython's AST structures to kdev-python's. It's generated based on parser/python39.sdef by the script conversionGenerator.py.

The script pythonpythonparser.py spits out the AST as an XML representation, using the Python `ast` module API. In recent years it's been used only as a tool for debugging and development. I believe it was once used in kdevelop before it called into the C API instead, and could be resurrected as such to implement Sven's suggestion. I'm not a big fan of this idea, I suspect performance would suffer considerably.

Any help is certainly very welcome. I am not very active on IRC currently, but I definitely read and answer kdevelop-devel@kde.org.

In principle, the task is simple: in parser/ast.h we have our own data structure for an AST, and that needs to be populated. The current way to do that is based on a... let's say, "creative" domain-specific language which describes how CPython's internal AST representation maps to ours from ast.h. C code is generated from this DSL file. You could just as well write that C code by hand -- it's not actually that much, a few hundred lines.

This doesn't work any more, because CPython's representation is gone.

There are certainly several different approaches to fix this, I think my suggestion above is the most straightforward one since it doesn't structurally change anything. Though, if one has to touch all this stuff *anyways*, it might be worth evaluating other parser libraries which support also e.g. proper incomplete parsing. I'd honestly opt for whatever the person doing it feels most comfortable with.

I wouldn't discard approaches involving serializing/deserializing the AST (e.g. to/from XML) because of performance concerns; XML stream serialization/deserialization (i.e. if you avoid building a DOM) is extremely fast, and I'm pretty sure that the impact is negligible compared to the semantic analysis effort later on. The only thing I'd really try to avoid is spawning a process for each parse job.

Hi, I've drafted a possible way forward in the "python310" branch in the kdev-python repo. The AST is written to an XML stream from a tiny Python program and read using QXmlStreamReader in C++. So far, only Module, Assignment, Name and Constant AST nodes can be deserialized, so we can basically parse programs of the form "a = 3" and nothing else ;)

But the whole structure is there, most of what's left is filling in all the 63 or so missing NodeReader<> template specializations for all the AST nodes, see https://invent.kde.org/kdevelop/kdev-python/-/blob/python310/parser/astfromxml.cpp#L164

In addition, we need to deal with 1. error reporting (parse errors, plus invalid XML issues), and 2. running the ast-to-xml Python code in-process, that should make it reasonably fast. But I'm fairly certain that the most difficult step is done.

Let's move any discussion on this topic to https://invent.kde.org/kdevelop/kdev-python/-/merge_requests/16.

Master now supports Python 3.10.

@Sven do you know whether python 3.11 would work and could be enabled as well?

Not just like that, but it shouldn't be a large effort to support it now. I should be able to look into that sometime in the next few weeks, I think.

Master now also supports Python 3.11, where "supports" means "builds and runs without crashing".

It's too late to get into Applications 22.12, but it will be in the next release.

great to know - THX a lot for all your work