Bug 340841

Summary:	crash when parsing files with DOS line endings (\r\n)
Product:	[Developer tools] kdev-python	Reporter:	H L Prasad <hlprasu>
Component:	general	Assignee:	kdevelop-bugs-null
Status:	RESOLVED WORKSFORME
Severity:	grave	CC:	felix_mauch, haizaar, hlprasu, mail
Priority:	NOR
Version:	unspecified
Target Milestone:	---
Platform:	Ubuntu
OS:	Linux
Latest Commit:		Version Fixed In:
Sentry Crash Report:
Attachments:	Colorama package which causes the crash attachment-19772-0.html attachment-9722-0.html tracelog

Description H L Prasad 2014-11-11 07:51:22 UTC

Kdevelop is crashing with assert failure in Python::AstBuilder::parse. 

My guess: 
at https://github.com/kensington/kdevelop-python/blob/master/parser/astbuilder.cpp#L266
because the QList<> "indents" is empty. 

This is reproducible every-time for me. I'm pasting below relevant portion of debug log.
------------------------------------------------------------------------------
.
.
.
kdevelop(7668)/kdevplatform (language) KDevelop::BackgroundParserPrivate::parseDocumentsInternal: creating parse-job KUrl("file:///usr/lib/python2.7/dist-packages/colorama/initialise.py") new count of active parse-jobs: 1
 ====> PARSING ====> parsing file  KUrl("file:///usr/lib/python2.7/dist-packages/colorama/initialise.py") ; has priority 9996 
kdevelop(7668)/kdevelop (python support) Python::LanguageSupport::language: KDevelop::Language(0x47566c0)
 ====> AST     ====>     building abstract syntax tree for  "/usr/lib/python2.7/dist-packages/colorama/initialise.py" 
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: DID NOT RECEIVE A SYNTAX TREE -- probably parse error. 
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Error objects:  0x7f54e6f0f240 0x7f54b5c77470 0x0
('invalid syntax', ('<kdev-editor-contents>', 1, 14, 'import atexit\r\n'))Eventual errors while extracting tuple:  
invalid syntaxinvalid syntaxkdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Problem range:  [(0, 10) ,  (0, 18) ]
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Discarding parts of the code to be parsed because of previous errors 
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: (0)
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'i'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'm'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'p'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'o'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'r'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 't'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: ' '
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'a'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 't'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'e'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'x'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'i'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 't'
'develop(7668)/kdevelop (python support) Python::AstBuilder::parse: '
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: '
'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 't'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: ' '
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 's'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 'y'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: 's'
'develop(7668)/kdevelop (python support) Python::AstBuilder::parse: '
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: '
'
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Start of error code:  0
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: End of error block (current position):  20
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Length:  20
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: indent at error <> current indent: 0 <> 0
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: This is what is left:  "import atexit
import sys

from .ansitowin32 import AnsiToWin32


orig_stdout = sys.stdout
orig_stderr = sys.stderr

wrapped_stdout = sys.stdout
wrapped_stderr = sys.stderr

atexit_done = False


def reset_all():
    AnsiToWin32(orig_stdout).reset_all()


def init(autoreset=False, convert=None, strip=None, wrap=True):

    if not wrap and any([autoreset, convert, strip]):
        raise ValueError('wrap=False conflicts with any other arg=True')

    global wrapped_stdout, wrapped_stderr
    sys.stdout = wrapped_stdout = \
        wrap_stream(orig_stdout, convert, strip, autoreset, wrap)
    sys.stderr = wrapped_stderr = \
        wrap_stream(orig_stderr, convert, strip, autoreset, wrap)

    global atexit_done
    if not atexit_done:
        atexit.register(reset_all)
        atexit_done = True


def deinit():
    sys.stdout = orig_stdout
    sys.stderr = orig_stderr


def reinit():
    sys.stdout = wrapped_stdout
    sys.stderr = wrapped_stdout


def wrap_stream(stream, convert, strip, autoreset, wrap):
    if wrap:
        wrapper = AnsiToWin32(stream,
            convert=convert, strip=strip, autoreset=autoreset)
        if wrapper.should_wrap():
            stream = wrapper.stream
    return stream



"
kdevelop(7668)/kdevelop (python support) Python::ParseSession::parse: Couldn't parse content
kdevelop(7668)/kdevelop (python support) Python::ParseJob::run: ---- Parsing FAILED ---- 
kdevelop(7668)/kdevplatform (language) KDevelop::BackgroundParserPrivate::parseDocumentsInternal: creating parse-job KUrl("file:///usr/lib/python2.7/dist-packages/colorama/ansitowin32.py") new count of active parse-jobs: 1
 ====> PARSING ====> parsing file  KUrl("file:///usr/lib/python2.7/dist-packages/colorama/ansitowin32.py") ; has priority 9996 
kdevelop(7668)/kdevelop (python support) Python::LanguageSupport::language: KDevelop::Language(0x47566c0)
 ====> AST     ====>     building abstract syntax tree for  "/usr/lib/python2.7/dist-packages/colorama/ansitowin32.py" 
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: DID NOT RECEIVE A SYNTAX TREE -- probably parse error. 
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Error objects:  0x7f54e6f0f240 0x7f54b9532c60 0x0
('invalid syntax', ('<kdev-editor-contents>', 1, 1, '\r\n'))Eventual errors while extracting tuple:  
invalid syntaxinvalid syntaxkdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Problem range:  [(0, 0) ,  (0, 5) ]
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Discarding parts of the code to be parsed because of previous errors 
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: ()
ASSERT failure in QList<T>::at: "index out of range", file /usr/include/qt4/QtCore/qlist.h, line 469
KCrash: Application 'kdevelop' crashing...
KCrash: Attempting to start /usr/lib/kde4/libexec/drkonqi from kdeinit
KCrash: Connect sock_file=/home/prasad/.kde/socket-prasad/kdeinit4__0
QSocketNotifier: Invalid socket 7 and type 'Read', disabling...
QSocketNotifier: Invalid socket 10 and type 'Read', disabling...
QSocketNotifier: Invalid socket 19 and type 'Read', disabling...
QSocketNotifier: Invalid socket 25 and type 'Read', disabling...
QSocketNotifier: Invalid socket 22 and type 'Read', disabling...
kdevelop: Fatal IO error: client killed
ClassFunctionDeclarationData::m_defaultParameters There were items left on destruction: 709
SpecialTemplateDeclarationData::m_specializations There were items left on destruction: 332
FunctionDeclarationData::m_decorators There were items left on destruction: 780
ClassDeclarationData::m_decorators There were items left on destruction: 44
DUContextData::m_importers There were items left on destruction: 22786
DUContextData::m_importedContexts There were items left on destruction: 23975
FunctionDeclarationData::m_defaultParameters There were items left on destruction: 20464
ClassDeclarationData::baseClasses There were items left on destruction: 3572
DUContextData::m_uses There were items left on destruction: 47057
TopDUContextData::m_usedDeclarationIds There were items left on destruction: 1288
TopDUContextData::m_problems There were items left on destruction: 1427
DUContextData::m_childContexts There were items left on destruction: 49728
DUContextData::m_localDeclarations There were items left on destruction: 49900
Unable to start Dr. Konqi
Re-raising signal for Apport handling.
Segmentation fault (core dumped)
----------------------------------------------------------------------------------------------------

Reproducible: Always

Comment 1 H L Prasad 2014-11-11 07:56:04 UTC

BTW, there is no component - "Language Support: Python", available in this bugs site, to select for this bug.

Comment 2 Kevin Funk 2014-11-11 08:29:23 UTC

Yep. it's a separate "product". Changed.

Comment 3 Sven Brauch 2014-11-12 23:32:03 UTC

Can't reproduce this here, can you post some code which triggers the crash? I installed the colorama package and created a file which said "import colorama", which didn't trigger the crash.

Comment 4 Sven Brauch 2014-11-15 20:40:57 UTC

Ping? I'd really like to have this fixed, but without a way to reproduce it's not easy to do :(

Comment 5 H L Prasad 2014-11-16 00:10:37 UTC

Its very easy to crash.

1. Install colorama Python package in your system and ensure that it is in default Python path. If its a Linux system, most likely, it is already present in your system. If not, do 'pip install colorama'.
2. In a sample Python file within KDevelop, type
    'from colorama import *'
   and immediately KDevelop will crash.

Comment 6 H L Prasad 2014-11-16 00:17:19 UTC

I have v0.2.5 of colorama installed in my system. 

My OS is Ubuntu 14.04.1, and I used kubuntu-backports to install KDevelop. See https://launchpad.net/~kubuntu-ppa/+archive/ubuntu/backports/+index?batch=75&memo=375&start=375, for KDevelop related PPA which got installed in my system.

Comment 7 H L Prasad 2014-11-16 03:14:44 UTC

Created attachment 89608 [details]
Colorama package which causes the crash

I've attached colorama Python package from my system which is causing the crash. You could probably import this as a local Python package to reproduce the crash.

Comment 8 Sven Brauch 2014-11-16 08:58:40 UTC

I tried with both the Python 2 and Python 3 versions of kdev-python and couldn't reproduce the crash in either. You do use version 1.7, right (the splash screen should say 4.7.0)?

The log indicates that it finds a syntax error in that file, while the one in the package doesn't have any syntax errors if read as either py2 or py3. Is the file you have locally (file:///usr/lib/python2.7/dist-packages/colorama/ansitowin32.py) identical with what is in the package?

Comment 9 H L Prasad 2014-11-16 09:35:10 UTC

I have 1.7 version of kdev-python, and splash screen of KDevelop says 4.7.0.

I've attached the package as is from my system with this bug report. Please use that once as a local package in KDevelop project and see whether this issue occurs for you.

As far as I understand, there is no syntax error in that package. Parsing as seen in the log, fails for two files -
file:///usr/lib/python2.7/dist-packages/colorama/initialise.py
file:///usr/lib/python2.7/dist-packages/colorama/ansitowin32.py

I suspect that because of parse error in the second one, the "indents" QList is empty and hence causing this crash as indicated by part of the log
.
.
.
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: Discarding parts of the code to be parsed because of previous errors 
kdevelop(7668)/kdevelop (python support) Python::AstBuilder::parse: ()
ASSERT failure in QList<T>::at: "index out of range", file /usr/include/qt4/QtCore/qlist.h, line 469
.
.
.


Do I need to provide any additional details from my system?

Comment 10 Sven Brauch 2014-11-16 09:41:45 UTC

Ah! The file you sent me has dos line endings. That is very unusual and might be the reason for the issue. I even think I've seen that crash before. Why does it have dos line endings?
Try converting it to unix line endings and if the issue doesn't occur any more, we know the reason at least.

Comment 11 H L Prasad 2014-11-16 10:02:04 UTC

Bingo! That seems to be the problem. 

The colorama package came from Ubuntu 14.04.1 (Trusty) repository. It has DOS line endings probably because it a cross platform package as claimed in its description.

Can kdev-python be fixed to support DOS line endings? 

Meanwhile I'll report to the Ubuntu developer mailing list about this.

Comment 12 Sven Brauch 2014-11-16 10:04:04 UTC

Yes, it can be fixed I think. I'll look into it (and also change the bug title).
Talking to the Ubuntu guys still sounds like a good idea, the package probably should have Unix line endings here :)

Comment 13 H L Prasad 2014-11-16 10:22:03 UTC

Thanks!

Looking forward to having this fixed in mainstream release!

Comment 14 Hai Zaar 2014-11-17 06:43:58 UTC

Hit this one as well. There were some DOS-style files in project tree in our company, kdevelop 4.7.0 crashed while parsing the project - on empty DOS-style __init__.py file. Changing it to Unix solved the issue for me.

Saw it crashing on colorama package files as well.

Comment 15 Sven Brauch 2014-12-02 12:57:54 UTC

Curiously, I cannot seem to reproduce this problem with a hand-crafted file saved with DOS line endings. So there must be some other factor in it. Can one of you give me a backtrace of the crash? Maybe that will be sufficient to fix it.

Comment 16 Hai Zaar 2014-12-02 13:25:54 UTC

I've tried to revert my dos2unix'ed filed that caused problems early and it does not crash any more.

I'm on latest 1.7 git branch. 

I'm sure I hit this crash constantly after kdevelop 4.7 arrived to backports, until I've found this bug and dos2unix'ed my problematic file (empty __init__.py in my case). But now it does not reproduce for me anymore. Nor did I succeeded in recreating this issue in a sandbox.

Comment 17 H L Prasad 2014-12-02 13:44:13 UTC

I'm getting the crash again. Just open any .py file from the colorama package (colorama.tar.gz) attached with this bug report.

Comment 18 Sven Brauch 2014-12-02 13:46:20 UTC

Not happening here. Eventually this is related to the text editor? Which version of Kate do you have installed? I have 3.14.12.

Comment 19 Sven Brauch 2014-12-02 13:46:33 UTC

... er, 3.14.2, sorry.

Comment 20 H L Prasad 2014-12-02 13:49:30 UTC

Created attachment 89806 [details]
attachment-19772-0.html

I have Kate version 3.14.2 too.

~ Prasad Bhat

On 2 December 2014 at 19:16, Sven Brauch <svenbrauch@googlemail.com> wrote:

> https://bugs.kde.org/show_bug.cgi?id=340841
>
> --- Comment #19 from Sven Brauch <svenbrauch@googlemail.com> ---
> ... er, 3.14.2, sorry.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
> You reported the bug.
>

Comment 21 Hai Zaar 2014-12-02 14:10:28 UTC

Tried on my side with colorama files from the attached tarball - no crashes.

Comment 22 Sven Brauch 2014-12-02 14:13:13 UTC

That is quite strange. What else could this be related to?
In any case, Prasad, can you please install debug symbols and create a proper backtrace? Thanks!

Comment 23 H L Prasad 2014-12-02 14:33:07 UTC

I am using Kubuntu backports PPA for installing Kdevelop 4.7.0 packages. It does not seem to have packages for debug symbols. Is there any way to install the debug symbols manually? I have 64-bit Ubuntu 14.04.1 installed in my system.

Comment 24 Sven Brauch 2014-12-02 15:16:21 UTC

You can uninstall the kdev-python package, clone kdev-python from git,
and install the 1.7 branch with -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=debug if you like.

2014-12-02 15:33 GMT+01:00 H L Prasad <hlprasu@gmail.com>:
> https://bugs.kde.org/show_bug.cgi?id=340841
>
> --- Comment #23 from H L Prasad <hlprasu@gmail.com> ---
> I am using Kubuntu backports PPA for installing Kdevelop 4.7.0 packages. It
> does not seem to have packages for debug symbols. Is there any way to install
> the debug symbols manually? I have 64-bit Ubuntu 14.04.1 installed in my
> system.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.

Comment 25 H L Prasad 2014-12-02 16:48:36 UTC

I'm using Kdevelop for active development and would be difficult to try this experiment till weekend. I'll try this and email back by Dec 06th.

Comment 26 Sven Brauch 2015-10-08 18:30:43 UTC

If you still hit this issue, please reopen and provide a backtrace, otherwise I'm going to assume it's somehow magically gone :)

Comment 27 Hai Zaar 2015-10-08 19:08:34 UTC

Created attachment 94898 [details]
attachment-9722-0.html

Somehow magically gone indeed :)
On Oct 8, 2015 9:30 PM, "Sven Brauch" <mail@svenbrauch.de> wrote:

> https://bugs.kde.org/show_bug.cgi?id=340841
>
> Sven Brauch <mail@svenbrauch.de> changed:
>
>            What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>          Resolution|---                         |WORKSFORME
>              Status|CONFIRMED                   |RESOLVED
>
> --- Comment #26 from Sven Brauch <mail@svenbrauch.de> ---
> If you still hit this issue, please reopen and provide a backtrace,
> otherwise
> I'm going to assume it's somehow magically gone :)
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>

Comment 28 felix_mauch 2016-08-08 07:57:05 UTC

Created attachment 100488 [details]
tracelog

Comment 29 Sven Brauch 2016-08-08 08:06:37 UTC

Can you give some details please? What version of kdev-python is this? Python 2 or Python 3? What file(s) did it crash on? Why do you think that trace is this bug? Thanks.

Comment 30 felix_mauch 2016-08-08 08:20:55 UTC

hm, my comment got eaten up somehow...

I'm also facing the same issue with parsing colorama from python2

====> PARSING ====> parsing file  KUrl("file:///usr/lib/python2.7/dist-packages/colorama/initialise.py") ; has priority 9989 
 ====> AST     ====>     building abstract syntax tree for  "/usr/lib/python2.7/dist-packages/colorama/initialise.py" 
kdevelop(25563)/kdevelop (python support) Python::AstBuilder::parse: DID NOT RECEIVE A SYNTAX TREE -- probably parse error. 
('invalid syntax', ('<kdev-editor-contents>', 1, 14, 'import atexit\r\n'))Eventual errors while extracting tuple:  
invalid syntaxinvalid syntaxkdevelop(25563)/kdevelop (python support) Python::AstBuilder::parse: Discarding parts of the code to be parsed because of previous errors 
kdevelop(25563)/kdevelop (python support) Python::ParseJob::run: ---- Parsing FAILED ---- 
 ====> PARSING ====> parsing file  KUrl("file:///usr/lib/python2.7/dist-packages/colorama/ansitowin32.py") ; has priority 9989 
 ====> AST     ====>     building abstract syntax tree for  "/usr/lib/python2.7/dist-packages/colorama/ansitowin32.py" 
kdevelop(25563)/kdevelop (python support) Python::AstBuilder::parse: DID NOT RECEIVE A SYNTAX TREE -- probably parse error. 
('invalid syntax', ('<kdev-editor-contents>', 1, 1, '\r\n'))Eventual errors while extracting tuple:  
invalid syntaxinvalid syntaxkdevelop(25563)/kdevelop (python support) Python::AstBuilder::parse: Discarding parts of the code to be parsed because of previous errors 
ASSERT failure in QList<T>::at: "index out of range", file /usr/include/qt4/QtCore/qlist.h, line 469
KCrash: Application 'kdevelop' crashing...
KCrash: Attempting to start /usr/lib/kde4/libexec/drkonqi from kdeinit

This problem occurred, as soon as I added a file that imports colorama:

from pip._vendor import colorama, pkg_resources

Changing the line endings to unix ones fixes the problem for me, as well.

Without knowing the code, just from the output: Wouldn't it be suitable to check whether the given QList is empty? I guess it is, since no syntax tree was found, right?

Cheers
Felix

Comment 31 Sven Brauch 2016-08-08 08:41:15 UTC

Ah, ok. Is this using the Python 2 or Python 3 version of the plugin?