Created attachment 141730 [details] Files for reproduction the issue SUMMARY STEPS TO REPRODUCE 1. Copy files to folder of your choice 2. Un-comment Case A or Case B 3. Through Build plugin, set the Python environment 4. Run script 5. Watch the Task Manager... I have tried the same thing on my Linux machine (Manjaro KDE latest), and Kate work pretty well without any hicks. OBSERVED RESULT EXPECTED RESULT SOFTWARE/OS VERSIONS Windows: 10 Enterprise Python 3.8 ADDITIONAL INFORMATION For comparison, Sublime Text 4 done it on Linux and Windows in 6 sec. On Manjaro KDE latest Kate ran just fine. This is not finger-in-the-eye from Sublime fanboy, rather a Kate admirer for being out-of-the-box so nice and useful and in some aspects even better than ST4 (my long time editor). I also find to extend Kate through Python is rather simpler than in ST4, so I really would like to see Kate on professional level.
Hi, have you some time to profile this a bit? I would like to improve that experience but ATM lack time to do so myself on Windows.
Created attachment 147710 [details] attachment-16792-0.html Hi, Christoph, Sure, myself as well rarely find the time... What I was saying here is that Windows version of KDE Kate has something like "short output buffer" or something that lacking smoothness in running Python scripts. Even when I was trying to stop Python script execution, I wait so long that usually killing the app was faster than wait to see when Kate would stop it itself. IMPORTANT to say, I use Build plugin, so maybe it is something with it ? On Linux it works well...Huh, slowest in comparison to my other 2 editors (SciTE & Sublime Text), but after all works. I also make a little test. Produce some random amount of lines, then copy them to produce duplicates by purpose. When I hit about 1 million lines, I have tried first to sort lines, then to remove duplicates. Sublime did it fastest, hands down. SciTE did it a 2 seconds after ST. Kate halted and go to vacation... Something chokes Kate, but I cannot tell what & why ?... Hope I make this case a little bit more clear. All the best, /*Igor Perković*// / // On 27. 12. 2021. 18:57, Christoph Cullmann wrote: > https://bugs.kde.org/show_bug.cgi?id=442739 > > Christoph Cullmann<cullmann@kde.org> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |cullmann@kde.org > > --- Comment #1 from Christoph Cullmann<cullmann@kde.org> --- > Hi, > > have you some time to profile this a bit? > > I would like to improve that experience but ATM lack time to do so myself on > Windows. >
Firstly, I tried to reproduce your issue on linux but failed. Not sure what happens on windows that makes everything slow. > On Linux it works well...Huh, slowest in comparison to my other 2 editors (SciTE & Sublime Text), but after all works There are things in which we are much faster than sublime etc, but also things where we are slower. Also, there is no one on the team that works on the windows version of Kate so it lacks polish there and bug fixes for windows often take a lot longer. We are slower in some things usually because we don't know that a performance issue exists, because no one pushed the limits, yet. That said, for your last comparison Kate's sort/uniq functionality is implemented via javascript. Not sure why 'uniq' is slow, maybe the algorithm is bad or maybe the overhead of calling a JS function is too much. Maybe you can take a look and make it faster? There's another way you can do it (linux/mac) which will be much faster than anything out there. Hit `Ctrl +\` and type `sort | uniq` and you are done. > Hope I make this case a little bit more clear. yes, but for 1 thing only. Not the whole editor or the problem which this bug report is about.
Created attachment 147716 [details] attachment-30825-0.html > There are things in which we are much faster than sublime etc, but also things where we are slower. Yes, Kate is faster than others in some things. Granted. Also it has out-of-the box experience comparing with SciTE & Sublime (you need to build a lot around them to get the close to Kate) >That said, for your last comparison Kate's sort/uniq functionality is >implemented via javascript. Not sure why 'uniq' is slow, maybe the algorithm is >bad or maybe the overhead of calling a JS function is too much. Maybe you can >take a look and make it faster? I have implemented scripts for sort/unique both in Lua & Python for that sake... and try them on Kate. For reasonable amount of lines it works really decent. Things start to work slower on 1,2,3 millions of lines... It really is not game changer for pure coders, but being Data Engineer, I've got situation where I need to dig deep into raw data source, make some adjustments and the text editor comes handy. Also, my original post was about output speed of Python script execution... I really do not know why this part slows down, thus my example in attachment... Thank you for fast response! *Igor Perković+385 97 619 6240* On Fri, 25 Mar 2022 at 07:15, Waqar Ahmed <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=442739 > > Waqar Ahmed <waqar.17a@gmail.com> changed: > > What |Removed |Added > > ---------------------------------------------------------------------------- > CC| |waqar.17a@gmail.com > > --- Comment #3 from Waqar Ahmed <waqar.17a@gmail.com> --- > Firstly, I tried to reproduce your issue on linux but failed. Not sure what > happens on windows that makes everything slow. > > > On Linux it works well...Huh, slowest in comparison to my other 2 > editors (SciTE & Sublime Text), but after all works > > There are things in which we are much faster than sublime etc, but also > things > where we are slower. Also, there is no one on the team that works on the > windows version of Kate so it lacks polish there and bug fixes for windows > often take a lot longer. We are slower in some things usually because we > don't > know that a performance issue exists, because no one pushed the limits, > yet. > > That said, for your last comparison Kate's sort/uniq functionality is > implemented via javascript. Not sure why 'uniq' is slow, maybe the > algorithm is > bad or maybe the overhead of calling a JS function is too much. Maybe you > can > take a look and make it faster? There's another way you can do it > (linux/mac) > which will be much faster than anything out there. Hit `Ctrl +\` and type > `sort > | uniq` and you are done. > > > Hope I make this case a little bit more clear. > > yes, but for 1 thing only. Not the whole editor or the problem which this > bug > report is about. > > -- > You are receiving this mail because: > You reported the bug.
> I have implemented scripts for sort/unique both in Lua & Python for that > sake... and try them on Kate. > For reasonable amount of lines it works really decent. > Things start to work slower on 1,2,3 millions of lines... It really is not > game changer for pure coders, interesting. It should not be getting slow, but it depends on how the operation is being performed. Right now the built-in `uniq` is probably slow because it has to do thousands of removals which is a lot more work than just doing all the work and then replacing the document's text. Anyways, didn't get what you were saying, did you mean that you used your own scripts in kate via external tools plugin and it was slow? > Also, my original post was about output speed of Python script execution... > I really do not know why this part slows down, thus my example in > attachment... Okay, I can suggest a few things. If you can try them, maybe we can narrow down on what is slowing things down. 1. I assume you have run the same command outside kate? If so, how long does it take to finish. 2. Once you run the script and kate freezes, does it every resume? If it does resume, can you check the "Output" tab of the build plugin view and see if it has a lot of output. And if you can somehow measure the number of lines in the output that would be helpful as well. - 1 is so that we know the command itself is fast - 2 might slow things down if you have a ton of output. What happens is that the ui tries to resize all the lines you see in there, and if you have a million lines of output, it will take a while I guess.
Hmm, now that I have written the above comment I see that you have run the same? script on linux too and it worked fine there which would mean that my above comment may not lead us very far (except for point 1 to ensure the script finishes in time). What would really help diagnose the issue would be that someone profiles this on Windows. @sars I think you have windows, maybe you can check?
Just saw that you already tried the same? script on linux and it worked fine there. @sars I think you have windows, maybe you have an idea why this is slowing down
Created attachment 147720 [details] attachment-30830-0.html > Anyways, didn't get what you were saying, did you mean that you used your own scripts in kate via external tools plugin and it was slow? Exactly. I made a little contest: Running script with example source data: Linux (Manjaro KDE, Acer i5 16 GB RAM) * Sublime 3 sec * Scite 2 sec * Kate 12 sec Windows 10 (XPS i7, 16 GB RAM) * Sublime Case C --> 8.99 s | Case A --> 5.01 s * Scite Case C --> 12.36 s | Case A --> 24.14 s * Kate Case C --> 9.69 s | Case A --> SUCCESS (today) after some 20-ish seconds I saw the result Did not change standard output sizes though... So, guess it is improved after all. :) *Igor Perković* On Fri, 25 Mar 2022 at 12:28, Waqar Ahmed <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=442739 > > --- Comment #5 from Waqar Ahmed <waqar.17a@gmail.com> --- > > I have implemented scripts for sort/unique both in Lua & Python for that > > sake... and try them on Kate. > > For reasonable amount of lines it works really decent. > > Things start to work slower on 1,2,3 millions of lines... It really is > not > > game changer for pure coders, > > interesting. It should not be getting slow, but it depends on how the > operation > is being performed. Right now the built-in `uniq` is probably slow because > it > has to do thousands of removals which is a lot more work than just doing > all > the work and then replacing the document's text. Anyways, didn't get what > you > were saying, did you mean that you used your own scripts in kate via > external > tools plugin and it was slow? > > > Also, my original post was about output speed of Python script > execution... > > I really do not know why this part slows down, thus my example in > > attachment... > > Okay, I can suggest a few things. If you can try them, maybe we can narrow > down > on what is slowing things down. > 1. I assume you have run the same command outside kate? If so, how long > does it > take to finish. > 2. Once you run the script and kate freezes, does it every resume? If it > does > resume, can you check the "Output" tab of the build plugin view and see if > it > has a lot of output. And if you can somehow measure the number of lines in > the > output that would be helpful as well. > > - 1 is so that we know the command itself is fast > - 2 might slow things down if you have a ton of output. What happens is > that > the ui tries to resize all the lines you see in there, and if you have a > million lines of output, it will take a while I guess. > > -- > You are receiving this mail because: > You reported the bug.
Hi, thanks for the measurements. Can you share the configuation of the tool you are running especially the value of "Output"? Also, it would really help if you can share the script & dataset and I can then run it here and see what's making us slow.
Created attachment 147726 [details] 5sQygbqBlf.png [image: 5sQygbqBlf.png] *Igor Perković* I attached that on bug creation date... You need to have Python installed , and those 2 additional libraries/modules pip install pandas tabulate On Fri, 25 Mar 2022 at 13:06, Waqar Ahmed <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=442739 > > --- Comment #9 from Waqar Ahmed <waqar.17a@gmail.com> --- > Hi, thanks for the measurements. Can you share the configuation of the > tool you > are running especially the value of "Output"? > > Also, it would really help if you can share the script & dataset and I can > then > run it here and see what's making us slow. > > -- > You are receiving this mail because: > You reported the bug.
Yeah, sorry missed that again. I ran it, and well I can see where the problem is. To confirm the issue, made some changes and with Case A, it takes "3.2 seconds" where previously it was taking 11s on my machine.
Created attachment 147727 [details] attachment-7162-0.html I can't wait a new release :) Thank you! *Igor Perković* On Fri, 25 Mar 2022 at 13:32, Waqar Ahmed <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=442739 > > --- Comment #11 from Waqar Ahmed <waqar.17a@gmail.com> --- > Yeah, sorry missed that again. > > I ran it, and well I can see where the problem is. To confirm the issue, > made > some changes and with Case A, it takes "3.2 seconds" where previously it > was > taking 11s on my machine. > > -- > You are receiving this mail because: > You reported the bug.
Git commit ec3989ffb209c6a06a81f15c161facc638340c51 by Waqar Ahmed. Committed on 26/03/2022 at 00:39. Pushed by waqar into branch 'master'. build: Append to textEdit directly for better performance This is much faster than appending one line at a time. M +5 -4 addons/katebuild-plugin/plugin_katebuild.cpp M +1 -1 addons/katebuild-plugin/plugin_katebuild.h https://invent.kde.org/utilities/kate/commit/ec3989ffb209c6a06a81f15c161facc638340c51
You don't have to wait for release to try it out. Since you are on manjaro it should super simple to build kate and test this. Let me know if you want to, and I will share the steps. Btw, thanks for reporting this. Wouldn't have been fixed if you didn't report. Since you said you have to work with a lot of text /larger files, if you notice other perf issues do report so that we can fix them.
Created attachment 147732 [details] attachment-9898-0.html Hi Waqar, Strangely enough I'm not sleeping yet:) So, yes, I'd love to try this. What should I do to make build properly ? Would it interfere with installed version ? (So that I can make settings backup ?) Sure, you are the most welcome!. Kate is really great editor and find myself more and more working with it. I even wrote a little intro on LinkedIn about that :) https://www.linkedin.com/pulse/kate-python-igor-perkovi%25C4%2587/?trackingId=o%2Bm2tfaT565yyvYNB36pSg%3D%3D Rest assure, I'll notify you and others if something bugs me along the way. Thank you for your time and good will! /*Igor Perković*// /
Here are the steps. You will need git, cmake, make, gcc or clang compiler, extra-cmake-modules package sudo pacman -S git gcc g++ cmake make extra-cmake-modules git clone https://invent.kde.org/utilities/kate mkdir build && cd build cmake .. && make -j8 QT_PLUGIN_PATH=.bin/ ./bin/kate -n . That should be it I think. This will be a local build i.e., won't interfere with the system one but will reuse the same settings. the last command should start up kate. The "QT_PLUGIN_PATH=./bin" env var is important as it will load the newly built plugins in the "bin" subdir and not your system Kate's plugins which can be confusing. Thanks for the article. I saw you were looking for something like AlignTab. That should actually be not that hard to implement via scripts. In fact, all our indenters are written in scripts. If you want to try, the built-in scripts can be used as reference: https://invent.kde.org/frameworks/ktexteditor/-/tree/master/src/script/data ^ That's where you can find the python indenter, sort, uniq and other stuff.
Created attachment 147752 [details] attachment-18025-0.html > git clone https://invent.kde.org/utilities/kate >mkdir build && cd build > cmake .. && make -j8 changed to cmake ../kate && make -j8 > QT_PLUGIN_PATH=.bin/ ./bin/kate -n . Other than that, it was compiled fine, except one and one only warning: [ 52%] Building CXX object addons/symbolviewer/CMakeFiles/katesymbolviewerplugin.dir/fortran_parser.cpp.o /home/petra/kate/addons/snippets/snippetview.cpp: In member function ‘void SnippetView::slotSnippetToGHNS()’: /home/petra/kate/addons/snippets/snippetview.cpp:370:105: warning: ‘KNS3::UploadDialog::UploadDialog(const QString&, QWidget*)’ is deprecated: Since 5.85. See API documentation [-Wdeprecated-declarations] 370 | KNS3::UploadDialog dialog(QStringLiteral(":/katesnippets/ktexteditor_codesnippets_core.knsrc"), this); | ^ In file included from /home/petra/kate/addons/snippets/snippetview.cpp:30: /usr/include/KF5/KNewStuff3/kns3/uploaddialog.h:75:14: note: declared here 75 | explicit UploadDialog(const QString &configFile, QWidget *parent = nullptr); | ^~~~~~~~~~~~ but, successfully started new Kate! I like some new features (new for me) like breadcrumbs below tags, View controls aligned straight left (below FileBrowser...) Tried on other laptop (just in case) with AMD A4, which is slower, but still, on regular Kate I ran Case A in 52 sec, and on my new build it was 34 sec.. Almost double... Good work! Yeah, about AlignTab... what I use the most is prettify complex SQL code (for better visibility and understanding), thus I use Python script for SQL format beautifier #!/home/igorp/python_venv/data/bin/python # -*- coding: utf8 -*- # 📦 PACKAGE: # pip install sqlparse import sys if len(sys.argv): # Join all arguments into one string sql = ' '.join(sys.argv[1:]) import sqlparse parsed = sqlparse.format(sql, indent_width=4, reindent_aligned=True, use_space_around_operators=True, comma_first=True, keyword_case='upper') print(parsed) This one works well as External tool in Kate. But, what I can do with AlignTab is to select my query and type (i.e.): ON|= and my code is aligned over ON keywords and further (in pipe) where is equal sign... I came to this point, and never actually try to do something like that in JavaScript... I find Lua interesting (resembling on Python) and being really fast, but never dig deep into... # -*- coding: utf8 -*- """ 🏷️ ALIGN TEXT SIMPLE 👔 Igor Perković 🚀 Created: 2021-10-05 23:05:21 📅 Changed: 2022-01-23 19:20:14 💡 Idea is to extend KDE Kate with simple align function. USAGE: 1. Type delimiter character(s)/word(s) in a blank line, before text you want to select and align. 2. Select delimiters along with text 3. Run this script """ import sys sys.path.append(r"/home/igorp/Documents/CODE") from df_utils import du import numpy as np import pandas as pd def align_delimited_text(txt, mode=0): if len(txt): res = txt.split('\n') delimiter = res[0] acc = [] # NOTE 💡 The Idea here is to mark these rows who contains delimiter in a new column at first position [0] # and then calculate max columns length on these rows where delimiter exists. # ---------------------------------------------------------------------------------------------------------- for e,d in enumerate(res[1:]): if delimiter in d: dn = '1' + delimiter + d else: dn = '0' + delimiter + d tmp = dn.split(delimiter) acc.append(tmp) df = pd.DataFrame(acc) # Filter splitted - to see in which row exists delimiter dff = df[df[0]=='1'] # After we choose the rows and subset of data for column width measurements, # we may drop this column with indicator #---------------------------------------------------------------------------- dff = dff.drop(dff.columns[[0]], axis=1) df = df.drop(df.columns[[0]], axis=1) dfl = df.columns.tolist() col_widths = np.vectorize(len) cw = col_widths(dff.values.astype(str)).max(axis=0) cwl = [] for i in cw: cwl.append(i) for x,y in zip(dfl, cwl): df[x]=df[x].str.ljust(y) df = df.replace(np.nan, '') rowres = [] for v in df.values.tolist(): rowstr = '' for e,i in enumerate(v): if e: if len(i.strip()): if mode: rowstr = rowstr + f'{delimiter}' + i else: rowstr = rowstr + ' ' + i else: if len(i.strip()): rowstr = rowstr + i rowres.append(rowstr) for t in rowres: print(t) # Start. Check if we do have any selected text to process ? if len(sys.argv): # Join all arguments into one string tmp = ' '.join(sys.argv[1:]) try: align_delimited_text(tmp) except: print(tmp) print('Please check 1st line delimiter...') Thank you for the assistance :) *Igor Perković* On Sat, 26 Mar 2022 at 02:48, Waqar Ahmed <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=442739 > > --- Comment #16 from Waqar Ahmed <waqar.17a@gmail.com> --- > Here are the steps. > > You will need git, cmake, make, gcc or clang compiler, extra-cmake-modules > package > > sudo pacman -S git gcc g++ cmake make extra-cmake-modules > > git clone https://invent.kde.org/utilities/kate > mkdir build && cd build > cmake .. && make -j8 > QT_PLUGIN_PATH=.bin/ ./bin/kate -n . > > That should be it I think. This will be a local build i.e., won't interfere > with the system one but will reuse the same settings. the last command > should > start up kate. The "QT_PLUGIN_PATH=./bin" env var is important as it will > load > the newly built plugins in the "bin" subdir and not your system Kate's > plugins > which can be confusing. > > Thanks for the article. I saw you were looking for something like AlignTab. > That should actually be not that hard to implement via scripts. In fact, > all > our indenters are written in scripts. If you want to try, the built-in > scripts > can be used as reference: > https://invent.kde.org/frameworks/ktexteditor/-/tree/master/src/script/data > > ^ That's where you can find the python indenter, sort, uniq and other > stuff. > > -- > You are receiving this mail because: > You reported the bug.
Git commit c3f24d49496fa0970fadfad40fe853a3800f1f77 by Waqar Ahmed. Committed on 28/03/2022 at 11:35. Pushed by waqar into branch 'release/22.04'. build: Append to textEdit directly for better performance This is much faster than appending one line at a time. (cherry picked from commit ec3989ffb209c6a06a81f15c161facc638340c51) M +5 -4 addons/katebuild-plugin/plugin_katebuild.cpp M +1 -1 addons/katebuild-plugin/plugin_katebuild.h https://invent.kde.org/utilities/kate/commit/c3f24d49496fa0970fadfad40fe853a3800f1f77