Bug 408176 - Alternative initialisation? Sagemath (= 8.6-6) on python 2 (unicode issue?) crashes on: exec sage-ipython -i --simple-prompt
Summary: Alternative initialisation? Sagemath (= 8.6-6) on python 2 (unicode issue?) c...
Status: RESOLVED FIXED
Alias: None
Product: cantor
Classification: Applications
Component: sage-backend (show other bugs)
Version: unspecified
Platform: Other Linux
: NOR normal
Target Milestone: ---
Assignee: Alexander Semke
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-06-01 12:03 UTC by vialav
Modified: 2023-04-07 07:37 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 23.08


Attachments
sage-ipython invocation crash (10.54 KB, text/plain)
2019-06-01 14:56 UTC, vialav
Details

Note You need to log in before you can comment on or make changes to this bug.
Description vialav 2019-06-01 12:03:25 UTC
SUMMARY
master-branch Cantor v19.07.70+git20190523

STEPS TO REPRODUCE
1. Notice in src/backends/sage/cantor-execsage, line 15:
exec sage-ipython -i --simple-prompt
2. Try Cantor with Sagemath (= 8.6-6) on python 2 (unicode issue?), available in Ubuntu repos, or, indeed, try the above command

OBSERVED RESULT
Sagemath (rendering Cantor unusable) crashes with a distinctive error message

EXPECTED RESULT
Initialisation of the sage-backend

SOFTWARE/OS VERSIONS
Linux/KDE Plasma: Ubuntu 18.04.2
(available in About System)
KDE Plasma Version: N/A (irrelevant, but) (Unity)
KDE Frameworks Version:  (irrelevant, but) any from v5.44.0 backported to, to v5.58.0 (backported)
Qt Version:  (less relevant, but) v5.9.5 backported to, or any released up to date

ADDITIONAL INFORMATION
Are there any alternative ways of initialising the backend?
Cantor lovely works well with the rest of unproblematic backends, except the julia-backend, which is the subject of another bugreport
Comment 1 Nikita Sirgienko 2019-06-01 13:47:17 UTC
I hadn't exectly understand, what is your problem.
SageMath writed on Python 2, right. And SageMath can't work, if placed on not-ascii path (crash on initialization step, so you can't run it in console).

Could you please post here path to sage executable? And test, could you run sage in console?
Comment 2 vialav 2019-06-01 14:17:24 UTC
Hi, Nikita, 

Thanks for your attention to this report. I run the following: 

$ which sage
/usr/bin/sage

$ dpkg -l sagemath
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  sagemath:amd64 8.6-6        amd64        Open Source Mathematical Software

You could find Sage 8.6 on 
https://packages.ubuntu.com/src:sagemath

$ exec sage-ipython -i --simple-prompt # which Cantor invokes internally
crashes under Cantor v19.07.70+git20190523
# master branch, merged into Applications/19.04 (are you verifying the same?)

I tracked down this to have something related to: 
dpkg -l python-ipython
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-====================================================
ii  python-ipython 5.8.0-1      all          Enhanced interactive Python shell (Python 2 version)

namely, initiating a "single prompt" by Cantor through Python 2, which next goes through some intricacies of the Python 3 compatibility in IPython

So, there was my question: could it have been possibly done via a completely different route, as I'm able to use sage from: 

$ cat /usr/share/applications/sagemath.desktop | grep Exec
Exec=sage --notebook=jupyter

or even simpler: 
$ sage -python
Python 2.7.15+ (default, Nov 27 2018, 23:36:35) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
Comment 3 Antonio Rojas 2019-06-01 14:24:09 UTC
What is the exact, literal output of
$ sage-ipython -i --simple-prompt

This looks like a packaging issue in debian's sagemath
Comment 4 vialav 2019-06-01 14:56:45 UTC
Created attachment 120477 [details]
sage-ipython invocation crash
Comment 5 vialav 2019-06-01 14:57:04 UTC
Hi, Antonio, the below is the step-by-step procedure of `sage-ipython -i --simple-prompt`. Please note, that `sage -python` or `sage --notebook=jupyter` (unconveniently opens the browser) otherwise both work as expected/documented (so, it is why I was asking about alternative avenues for initialisation):


$ cat /usr/share/cantor/sagebackend/cantor-execsage
#!/usr/bin/env bash

export SAGE_ROOT="$($1 --root)"

if [[ -f $SAGE_ROOT"/bin/sage-env" ]]; then
    . "$SAGE_ROOT/bin/sage-env" >&2
elif [[ -f $SAGE_ROOT"/local/bin/sage-env" ]]; then
    . "$SAGE_ROOT/local/bin/sage-env" >&2
elif [[ -f $SAGE_ROOT"/src/bin/sage-env" ]]; then
    . "$SAGE_ROOT/src/bin/sage-env" >&2
else
    exit 1 # We have a problem
fi

exec sage-ipython -i simple-prompt


$ sage --root
/usr/share/sagemath

# sourcing the evironment
$ . /usr/share/sagemath/bin/sage-env

$ sage-ipython -i --simple-prompt
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.6, Release Date: 2019-01-15                     │
│ Using Python 2.7.15. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘

**********************************************************************

Oops, Sage crashed. We do our best to make it stable, but...

A crash report was automatically generated with the following information:
  - A verbatim copy of the crash traceback.
  - A copy of your input history during this session.
  - Data on your current Sage configuration.

It was left in the file named:
	'/home/ubuntu/.ipython/Sage_crash_report.txt'
If you can email this file to the developers, the information in it will help
them in understanding and correcting the problem.

You can mail it to: sage-support at sage-support@googlegroups.com
with the subject 'Sage Crash Report'.

If you want to do it now, the following command will work (under Unix):
mail -s 'Sage Crash Report' sage-support@googlegroups.com < /home/ubuntu/.ipython/Sage_crash_report.txt

In your email, please also include information about:
- The operating system under which the crash happened: Linux, macOS, Windows,
  other, and which exact version (for example: Ubuntu 16.04.3, macOS 10.13.2,
  Windows 10 Pro), and whether it is 32-bit or 64-bit;
- How Sage was installed: using pip or conda, from GitHub, as part of
  a Docker container, or other, providing more detail if possible;
- How to reproduce the crash: what exact sequence of instructions can one
  input to get the same crash? Ideally, find a minimal yet complete sequence
  of instructions that yields the crash.

To ensure accurate tracking of this issue, please file a report about it at:
http://trac.sagemath.org

Hit <Enter> to quit (your terminal may close):
Exiting Sage (CPU time 0m0.08s, Wall time 0m4.39s).

P.S. ~/.ipython/Sage_crash_report.txt attached
Comment 6 Nikita Sirgienko 2019-06-01 15:22:20 UTC
I think, Antonio Rojas right, it's acutually something wrong with sagemath on debian, because it's works for my Sage 8.1, Sage 8.6 a Sage 8.7
Comment 7 Nikita Sirgienko 2019-06-01 15:35:18 UTC
About alternatives: the only reason, that Cantor has own sage executable script is that sage don't allow forward option '--simple-prompt' to sage-ipython.

So, that I could suggest:
1. Copy to sage exectable script (/usr/bin/sage) to /usr/share/cantor/sagebackend/
2. Change function interactive_sage() in /usr/share/cantor/sagebackend/sage by adding '--simple-prompt' to sage-ipython call.
Result will be like this:
```
# Start an interactive Sage session, this function never returns.
interactive_sage() {
    sage_setup
    exec sage-ipython "$@" -i --simple-prompt
}
```
3. Remove all logic in cantor-execsage and just call /usr/share/cantor/sagebackend/sage

It will be work, but only if crash of sage-ipython not related with this option.
Comment 8 vialav 2019-06-01 15:48:42 UTC
Thank you for a suggestion. I gave it a try:

$ /usr/share/cantor/sagebackend/sage
/usr/share/cantor/sagebackend/sage: line 244: /usr/share/cantor/sagebackend/sage-env: No such file or directory
Error setting environment variables by sourcing '/usr/share/cantor/sagebackend/sage-env';
possibly contact sage-devel (see http://groups.google.com/group/sage-devel).
Comment 9 vialav 2019-06-01 17:55:45 UTC
// So, my memory reminded me where I was tracking down it to Python 2 and:

$ cd /usr/lib/python2.7/dist-packages/IPython && grep -rnw simple_prompt
terminal/interactiveshell.py:106:    simple_prompt = Bool(_use_simple_prompt,
terminal/interactiveshell.py:120:        return Pdb if self.simple_prompt else TerminalPdb
terminal/interactiveshell.py:222:        if self.simple_prompt:
terminal/ipapp.py:104:addflag('simple-prompt', 'TerminalInteractiveShell.simple_prompt',
testing/globalipapp.py:91:    config.TerminalInteractiveShell.simple_prompt = True

When `exec sage-ipython -i simple-prompt` goes through those lines, the crash happens. I was able to trace it to things like '\xxx\xxx\xxx' instead of getting the prompt like: [1] …

which is why I'd referred to it as base64/unicode(?) Python 2 humble attempt of being compatible with 'unicoded by default' Python 3. I hadn't have the time to dive deeper than this.
Comment 10 vialav 2019-06-05 14:11:58 UTC
FYI: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852780

The bug is still here though.
Comment 11 vialav 2019-06-05 14:15:35 UTC
Also Bug #375467
Comment 12 vialav 2019-06-05 14:21:02 UTC
$ ipython
Python 2.7.15+ (default, Nov 27 2018, 23:36:35) 
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

 In [1]                                                                                                                                                                                                           
Do you really want to exit ([y]/n)? y

$ ipython --simple-prompt
Python 2.7.15+ (default, Nov 27 2018, 23:36:35) 
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/traitlets/config/application.py", line 831, in launch_instance
    app.start()
  File "/usr/lib/python2.7/dist-packages/IPython/terminal/ipapp.py", line 355, in start
    self.shell.mainloop()
  File "/usr/lib/python2.7/dist-packages/IPython/terminal/interactiveshell.py", line 495, in mainloop
    self.interact()
  File "/usr/lib/python2.7/dist-packages/IPython/terminal/interactiveshell.py", line 478, in interact
    code = self.prompt_for_code()
  File "/usr/lib/python2.7/dist-packages/IPython/terminal/interactiveshell.py", line 230, in prompt
    line = cast_unicode_py2(input(prompt_text))
  File "/usr/lib/python2.7/dist-packages/IPython/utils/py3compat.py", line 220, in input
    return builtin_mod.raw_input(prompt)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 0: ordinal not in range(128)

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

$ dpkg -l python-traitlets
ii  python-traitlets 5.0.0~git20180709-1 all          Lightweight Traits-like package for Python 2
Comment 13 vialav 2019-06-06 17:15:20 UTC
UPDATE: The cause of errors has been identified and half-successfully dealt with:

https://github.com/ipython/ipython/issues/11485# IPython crashes at launch (closed, no action)
https://github.com/powerline/powerline/issues/1953# powerline-status crashes with ipython>=7.0.0
https://github.com/powerline/powerline/pull/1997# Fix #1953 (pending merge)


** First, I'm using `powerline`, which was (and still is) the cause of the reported issue **, so here comes the starting point of yet to be updated RTD: 
https://powerline.readthedocs.io/en/master/usage/other.html#ipython-prompt

#!~/.ipython/profile_default/ipython_config.py
- For IPython>=5.0 (and << 7.0) follow the RTD
- For IPython3>=7.0 after merging locally above referenced the pull `since_7` has to be used.

OBSERVED RESULT: 
as in #1953 (`ipython` with `--single-prompt` crashes, `ipython3` does not)

Resulting `ipython_config.py`, pass for `ipython` (python[2] under Sage):
``` python
import sys
if sys.version_info >= (3, 0):
    from powerline.bindings.ipython.since_7 import PowerlinePrompts
    c = get_config()
    c.TerminalInteractiveShell.simple_prompt = False
    c.TerminalInteractiveShell.prompts_class = PowerlinePrompts
else:
    pass
```

** Cantor is able to log in into Sage (python2) with `exec sage -i --simple-prompt` ! **:
ShortcutOverride 16777220 QFlags<Qt::KeyboardModifiers>(ShiftModifier)
login
found version:  ("8.6", "8", "6")
wsStatusChange 1
evaluating:  "1+2"
wsStatusChange 0
out:  "┌────────────────────────────────────────────────────────────────────┐\r\n│ SageMath version 8.6, Release Date: 2019-01-15                     │\r\n│ Using Python 2.7.15. Type \"help()\" for help.                       │\r\n└────────────────────────────────────────────────────────────────────┘\r\n"
found version:  ("8.6", "8", "6")
using the current set of commands
out:  "┌────────────────────────────────────────────────────────────────────┐\r\n│ SageMath version 8.6, Release Date: 2019-01-15                     │\r\n│ Using Python 2.7.15. Type \"help()\" for help.                       │\r\n└────────────────────────────────────────────────────────────────────┘\r\n\u001B]0;IPython: home/ubuntu\u0007"
found version:  ("8.6", "8", "6")
using the current set of commands
out:  "┌────────────────────────────────────────────────────────────────────┐\r\n│ SageMath version 8.6, Release Date: 2019-01-15                     │\r\n│ Using Python 2.7.15. Type \"help()\" for help.                       │\r\n└────────────────────────────────────────────────────────────────────┘\r\n\u001B]0;IPython: home/ubuntu\u0007sage: "
found version:  ("8.6", "8", "6")
using the current set of commands
out:  "┌────────────────────────────────────────────────────────────────────┐\r\n│ SageMath version 8.6, Release Date: 2019-01-15                     │\r\n│ Using Python 2.7.15. Type \"help()\" for help.                       │\r\n└────────────────────────────────────────────────────────────────────┘\r\n\u001B]0;IPython: home/ubuntu\u0007sage: sage: sage: sage: sage: sage: sage: "
found version:  ("8.6", "8", "6")
using the current set of commands
out:  "┌────────────────────────────────────────────────────────────────────┐\r\n│ SageMath version 8.6, Release Date: 2019-01-15                     │\r\n│ Using Python 2.7.15. Type \"help()\" for help.                       │\r\n└────────────────────────────────────────────────────────────────────┘\r\n\u001B]0;IPython: home/ubuntu\u0007sage: sage: sage: sage: sage: sage: sage: sage: sage: "
found version:  ("8.6", "8", "6")
using the current set of commands
out:  "┌────────────────────────────────────────────────────────────────────┐\r\n│ SageMath version 8.6, Release Date: 2019-01-15                     │\r\n│ Using Python 2.7.15. Type \"help()\" for help.                       │\r\n└────────────────────────────────────────────────────────────────────┘\r\n\u001B]0;IPython: home/ubuntu\u0007sage: sage: sage: sage: sage: sage: sage: sage: sage: ____TMP_DIR____ /home/ubuntu/.sage/temp/work/6787\r\nsage: sage: sage: ....: ....: ....: ....: ....: "
tmp path:  "/home/ubuntu/.sage/temp/work/6787"
found version:  ("8.6", "8", "6")
using the current set of commands
out:  "┌────────────────────────────────────────────────────────────────────┐\r\n│ SageMath version 8.6, Release Date: 2019-01-15                     │\r\n│ Using Python 2.7.15. Type \"help()\" for help.                       │\r\n└────────────────────────────────────────────────────────────────────┘\r\n\u001B]0;IPython: home/ubuntu\u0007sage: sage: sage: sage: sage: sage: sage: sage: sage: ____TMP_DIR____ /home/ubuntu/.sage/temp/work/6787\r\nsage: sage: sage: ....: ....: ....: ....: ....: sage: ____END_OF_INIT____\r\nsage: "
tmp path:  "/home/ubuntu/.sage/temp/work/6787"
found version:  ("8.6", "8", "6")
using the current set of commands
initialized
writing  "1+2\n\n"  to the process
out:  "3\r\n"
out:  "sage: "
got prompt 0    -1
out:  "sage: "
got prompt 0    -1
evaluation finished
"3"
setting result to a type  1  result
update Entry


ARTIFACTS: `python /usr/share/sagemath/bin/sage-cleaner` remains a ghost process for a while during which 10 cleanups happen, and then self-terminates)
Comment 14 Nikita Sirgienko 2019-06-06 17:27:38 UTC
Should I mark this as upstream bug?
Comment 15 vialav 2019-06-06 18:27:58 UTC
If `sage-ipython -i --simple-prompt` is inevitable for initialization of the backend, then yes. 

`sage -python` doesn't have this problem, and the Sage environment is being loaded properly ($SAGE_ROOT and friends), however, I'm not sure this would be enough, or how to proceed further.

So, either it would (or would not) be fixed upstream in Powerline, or one shall disable Powerline for Python[2] only, or Sage soon migrates to Python3, or there may exists an alternative initialization as above.
Comment 16 Antonio Rojas 2019-06-06 18:32:18 UTC
(In reply to vialav from comment #15)
> If `sage-ipython -i --simple-prompt` is inevitable for initialization of the
> backend, then yes. 
> 
> `sage -python` doesn't have this problem, and the Sage environment is being
> loaded properly ($SAGE_ROOT and friends), however, I'm not sure this would
> be enough, or how to proceed further.

sage -python is just a plain python prompt (just using sage's own python instead of the system one) so it's certainly not enough
Comment 17 Nikita Sirgienko 2019-06-06 18:44:00 UTC
(In reply to vialav from comment #15)
> If `sage-ipython -i --simple-prompt` is inevitable for initialization of the
> backend, then yes. 
> 
Yes, it's necesarry, because without the option you will have colourfull Sage. In terminal it's look nice, but on symbol stream level it is pain for handlening, so we don't support colors in sage output.
Comment 18 vialav 2019-06-06 19:32:28 UTC
(as a matter of Statistics and to have as many methods of initialization as possible in your arsenal, the following guy is initializing Python under Julia programmatically, but with its own set of issues): https://github.com/JuliaPy/pyjulia/issues/185
Comment 19 John Scott 2021-01-25 14:19:56 UTC
It's tangentially related to the issue at hand, so I'm posting this here, but Cantor gives trouble with the current (9.2) version of SageMath in Debian Unstable. This is partly due to what I believe is a Sage bug, SAGE_ROOT being an empty string, but I think there's still room to simplify cantor-execsage immensely and this is what we're tentatively going to be shipping for Debian Bullseye.

Please see the commit here: https://salsa.debian.org/qt-kde-team/kde/cantor/-/commit/7b7f610
and perhaps the Debian bug here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=972346#19
I basically turn cantor-execsage into the one-liner
{ echo "sage-ipython -i --simple-prompt"; cat; } | $1 --sh
which pipes the command into a Sage shell. Instead of mangling SAGE_ROOT, opening the Sage shell with 'sage --sh' seems more elegant, but may not be a supported command-line argument with older versions of Sage.

Please let me know if you plan to adopt this upstream. It looks like it will be shipping in Debian Bullseye, but this trick of opening a Sage shell should work with non-Debian-packaged Sage as well.
Comment 20 Alexander Semke 2023-04-07 07:37:01 UTC
Git commit 68d7944c08dc9899d85baffbf4caae8cc932d816 by Alexander Semke.
Committed on 07/04/2023 at 07:35.
Pushed by asemke into branch 'master'.

[sage] bump the minimal supported version of Sage to 9.2  (released Oct 24, 2020) and the recommended version to 9.5 (released Jan 30, 2022)
which allows us to simplify the initialization of the external process.

Most distributions provide already 9.5 or higher. So, we can also get rid of some old and unmaintained code now in Cantor.
FIXED-IN: 23.08

M  +1    -7    src/backends/sage/CMakeLists.txt
D  +0    -23   src/backends/sage/cantor-execsage
M  +1    -1    src/backends/sage/sagebackend.cpp
M  +73   -98   src/backends/sage/sagesession.cpp
M  +5    -7    src/backends/sage/sagesession.h
M  +38   -37   src/backends/sage/testsage.cpp

https://invent.kde.org/education/cantor/commit/68d7944c08dc9899d85baffbf4caae8cc932d816