Bug 405585 - Somewhere between Kate 17.12.3 and 18.12.2 code folding changed
Summary: Somewhere between Kate 17.12.3 and 18.12.2 code folding changed
Status: RESOLVED FIXED
Alias: None
Product: kate
Classification: Applications
Component: folding (show other bugs)
Version: 18.12.2
Platform: OpenSUSE Linux
: NOR normal (vote)
Target Milestone: ---
Assignee: KWrite Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-03-18 04:50 UTC by Russell
Modified: 2019-04-04 05:14 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.57


Attachments
Kate code folding test files and results (170.00 KB, application/gzip)
2019-03-18 04:50 UTC, Russell
Details
Kate 17 (leap) vs Kate 18 (tumbleweed): Image of block highlighting (358.94 KB, image/jpeg)
2019-03-22 04:12 UTC, Russell
Details
Kate 17 (leap) vs Kate 18 (tumbleweed): Image of top level code folding (416.60 KB, image/jpeg)
2019-03-22 04:31 UTC, Russell
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Russell 2019-03-18 04:50:32 UTC
Created attachment 118871 [details]
Kate code folding test files and results

SUMMARY:
Somewhere between Kate 17.12.3 and 18.12.2 code folding changed.

Referring to Code folding Menu:

    View > Code Folding > Fold TopLevel Nodes
    View > Code Folding > UnFold TopLevel Nodes

STEPS TO REPRODUCE
1. Install testkate.xml into one of the locations:

   * Kate 17: $HOME/.local/share/katepart5/syntax
   * Kate 18: $HOME/.local/share/org.kde.syntax-highting/syntax

2. Start Kate
3. Open the file testkate.txt
4. View > Code Folding > Fold TopLevel Nodes

OBSERVED RESULT
In Kate 17, the folding is correct, see (2) in testkate-codefolding.jpg.
In Kate 18, the folding is broken, see (3) in testkate-codefolding.jpg.

EXPECTED RESULT
Kate 18 code folding to look like Kate 17 code folding when using testkate.xml.

SOFTWARE/OS VERSIONS

Tested Environments (About > Kate > Libraries:)

    #1
    Kate 18.12.2 *broke* opensuse_tumbleweed manjaro
    KDE Frameworks 5.55.0
    Qt 5.12.0

    #2
    Kate 17.12.3 *worked* (opensuse_leap_15, linux_mint_19)
    KDE Frameworks 5.45.0
    QT 5.9.4
Comment 1 Nibaldo G. 2019-03-21 21:21:34 UTC
Does it have to do with this?: https://phabricator.kde.org/D18516
Comment 2 Russell 2019-03-22 04:12:21 UTC
Created attachment 118967 [details]
Kate 17 (leap) vs Kate 18 (tumbleweed): Image of block highlighting
Comment 3 Russell 2019-03-22 04:31:13 UTC
Created attachment 118968 [details]
Kate 17 (leap) vs Kate 18 (tumbleweed): Image of top level code folding

I'll take a look tomorrow in more detail, but it looks like it might be. It seems the end is one-off.  The snippet of validated xml below focuses on highlighting blocks/code folding (see original attachment for xml and sample test file):

<context attribute="Normal"           lineEndContext="#stay"        name="_SingleLineTitle">
  <RegExpr attribute="TitleSection0"  context="#stay"               String="^(={1})\s+\S(?:.*[^=])(?:\s+\1)?$"
             beginRegion="SingleLineFolding" endRegion="SingleLineFolding" />
  <RegExpr attribute="TitleSection1"  context="#stay"               String="^(={2})\s+\S(?:.*[^=])(?:\s+\1)?$"
             beginRegion="SingleLineFolding" endRegion="SingleLineFolding" />
  <RegExpr attribute="TitleSection2"  context="#stay"               String="^(={3})\s+\S(?:.*[^=])(?:\s+\1)?$"
             beginRegion="SingleLineFolding" endRegion="SingleLineFolding" />
  <RegExpr attribute="TitleSection3"  context="#stay"               String="^(={4})\s+\S(?:.*[^=])(?:\s+\1)?$"
             beginRegion="SingleLineFolding" endRegion="SingleLineFolding" />
  <RegExpr attribute="TitleSection4"  context="#stay"               String="^(={5})\s+\S(?:.*[^=])(?:\s+\1)?$"
             beginRegion="SingleLineFolding" endRegion="SingleLineFolding" />
</context>

I am running OpenSuSE Leap 15 which has Kate 17.12.3 (folding ok) and OpenSuSE Tumbleweed 20190315 which has Kate 18.12.3 (folding not ok). I am using Kate's Help > About Kate to obtain Kates version, as well as KDE Framework and QT versions as originally reported.
Comment 4 Nibaldo G. 2019-03-22 08:33:56 UTC
The problem occurs when using beginRegion+endRegion in the same rule.
Ex:

<StringDetect String="hello" endRegion="region1" beginRegion="region2" />


In this case, the string "hello" is part of the "region1" and "region2" regions simultaneously, which generates folding problems. The behavior you expect (and the ideal behavior) is that "hello" isn't part of "region1", but of "region2".

I have checked and there are many XML files that use endRegion+beginRegion and in which there is an incorrect folding due to this bug.
Comment 5 Nibaldo G. 2019-03-23 14:30:41 UTC
Proposed patch: https://phabricator.kde.org/D19999
Comment 6 Christoph Cullmann 2019-03-24 13:07:10 UTC
Git commit a24144952688f6e6b8700d5c86c563264715f8a7 by Christoph Cullmann, on behalf of Nibaldo González.
Committed on 24/03/2019 at 13:07.
Pushed by cullmann into branch 'master'.

Fix endRegion folding in rules with beginRegion+endRegion (use length=0)

Summary:

There is a problem of folding in rules with beginRegion + endRegion.

Example:
```
<StringDetect String="hello" endRegion="region1" beginRegion="region2" />
```
In this case, the "hello" string is part of the "region1" and "region2" regions simultaneously, which generates folding problems. The ideal behavior is that "hello" isn't part of "region1", but of "region2". That is, don't publish length of endRegion folding in rules with beginRegion+endRegion.

For example, in `.desktop` files:
{F6715422}

The following XML files use rules with beginRegion + endRegion:
* ansforth94.xml
* apparmor.xml
* asciidoc.xml
* asp.xml c.xml
* cisco.xml
* desktop.xml
* fortran.xml
* haml.xml
* ini.xml
* isocpp.xml
* julia.xml
* lex.xml
* lua.xml
* makefile.xml
* perl.xml
* pony.xml
* rhtml.xml
* rpmspec.xml
* ruby.xml
* template-toolkit.xml
* txt2tags.xml
* yacc.xml

Reviewers: #framework_syntax_highlighting, cullmann, dhaumann, vkrause

Reviewed By: #framework_syntax_highlighting, cullmann

Subscribers: kwrite-devel, kde-frameworks-devel

Tags: #kate, #frameworks

Differential Revision: https://phabricator.kde.org/D19999

M  +2    -2    autotests/folding/Makefile.fold
M  +2    -2    autotests/folding/folding.cpp.fold
M  +1    -1    autotests/folding/highlight.asp.fold
M  +20   -20   autotests/folding/highlight.cpp.fold
M  +1    -1    autotests/folding/highlight.f90.fold
M  +2    -2    autotests/folding/highlight.lex.fold
M  +3    -3    autotests/folding/highlight.pl.fold
M  +1    -1    autotests/folding/highlight.pony.fold
M  +1    -1    autotests/folding/highlight.prg.fold
M  +19   -19   autotests/folding/highlight.rb.fold
M  +3    -3    autotests/folding/highlight.spec.fold
M  +7    -7    autotests/folding/highlight.t2t.fold
M  +2    -2    autotests/folding/highlight.y.fold
M  +15   -1    autotests/folding/test.desktop.fold
M  +3    -3    autotests/folding/test.ini.fold
M  +6    -6    autotests/folding/usr.bin.apparmor-profile-test.fold
M  +14   -0    autotests/html/test.desktop.html
M  +14   -0    autotests/input/test.desktop
M  +14   -0    autotests/reference/test.desktop.ref
M  +9    -2    src/lib/abstracthighlighter.cpp

https://commits.kde.org/syntax-highlighting/a24144952688f6e6b8700d5c86c563264715f8a7
Comment 7 Nibaldo G. 2019-03-24 19:53:08 UTC
The patch will be included in KDE Frameworks 5.57, available approx. on April 13, 2019
Comment 8 Russell 2019-03-24 23:45:48 UTC
How can I test this patch?

OpenSuSE Tumbleweed just updated libKF5SyntaxHighlighting5 to 5.56.

I looked at KDE downloads (https://neon.kde.org/download), unstable and neon via docker. Unstable is dated: 17-Mar-2019. I'm not sure about neon/docker.

Is there doc?
Comment 9 Nibaldo G. 2019-03-25 02:15:48 UTC
The git-unstable Neon ISO should be updated.

You can also compile KSyntaxHighligting and copy "libKF5SyntaxHighlighting.so.5.57.0" in /usr/lib64/ or /usr/lib/x86_64-linux-gnu/,
or install it.

https://phabricator.kde.org/source/syntax-highlighting/
Comment 10 Fabian Vogt 2019-03-31 13:18:03 UTC
(In reply to prusselltechgroup from comment #8)
> How can I test this patch?
> 
> OpenSuSE Tumbleweed just updated libKF5SyntaxHighlighting5 to 5.56.

You can just use the Tumbleweed-based Krypton .iso, it uses KDE:Unstable repos with builds from git master: https://en.opensuse.org/SDB:Krypton_and_Argon
Comment 11 Fabian Vogt 2019-03-31 13:19:01 UTC
(In reply to Fabian Vogt from comment #10)
> (In reply to prusselltechgroup from comment #8)
> > How can I test this patch?
> > 
> > OpenSuSE Tumbleweed just updated libKF5SyntaxHighlighting5 to 5.56.
> 
> You can just use the Tumbleweed-based Krypton .iso, it uses KDE:Unstable
> repos with builds from git master:
> https://en.opensuse.org/SDB:Krypton_and_Argon

Whoops, that was a broken link: https://en.opensuse.org/SDB:Argon_and_Krypton
Comment 12 Russell 2019-04-04 04:45:50 UTC
I tested neon-unstable-20190401-1143 in a VM. 

And the patch fixed it. Kate syntax highlighting works as it has since KDE 4:

View > Code Folding > Fold Top Level Nodes 
and
View > Code Folding > Unfold Top Level Nodes 

I'll definitely look into opensuse tumbleweed repos for testing.

A couple of cleanup items:

1. I had a typo in my first entry. The current directory location for a user's own or override syntax highlight definition file, on linux, is:

     $HOME/.local/share/org.kde.syntax-highlighting/syntax/

The documentation is located here:

    https://docs.kde.org/trunk5/en/applications/katepart/highlight.html#idm45858413298848

The documentation says to use the command below to find your user folder ($HOME/.local/share):

    qtpaths --paths GenericDataLocation

When I ran the command on neon-unstable, I got:

    qtpaths: could not find a QT installation of ''

It did not prevent Kate from finding the syntax highlight definition file.

2. During the learning process, it was very helpful to read, grep/sed the system syntax highlight definition xml files. I read alert.xml to find which elements were available. I searched neon-unstable and didn't find any syntax highlight definition files. Doing an Internet search, I found the link below, which describes that they are now compiled into the KSyntaxHighlighting library.

    https://kate-editor.org/2018/03/10/improving-syntax-highlighting-files/

For myself and folks coming along, where would we find the xml source for the syntax highlight definition files?
Comment 13 Nibaldo G. 2019-04-04 05:08:15 UTC
The KSyntaxHighlighting library (part of KDE Frameworks) is used for the syntax highlighting engine in Kate and KDevelop:

https://kate-editor.org/2016/11/15/ksyntaxhighlighting-a-new-syntax-highlighting-framework/

The repository is located at: 
https://phabricator.kde.org/source/syntax-highlighting/ 

Or in GitHub (mirror): https://github.com/KDE/syntax-highlighting

In the folder "data/syntax/" there are the XML files. Also in "data/schema/" there is a script to validate the XML files
Comment 14 Nibaldo G. 2019-04-04 05:14:55 UTC
Also here (they are grouped according to the KDE Frameworks version): https://kate-editor.org/syntax/