Version: 3.9.98 (svn 1098891) (using KDE 4.4.1) Compiler: gcc version 4.4.3 (GCC) Configured with: ../configure --prefix=/usr --enable-shared --enable-languages=c,c++,fortran,objc,obj-c++,ada --enable-threads=posix --mandir=/usr/share/man --infodir=/usr/share/info --enable-__cxa_atexit --disable-multilib --libdir=/usr/lib --libexecdir=/usr/lib --enable-clocale=gnu --disable-libstdcxx-pch --with-tune=generic OS: Linux Installed from: Archlinux Packages My program generates a lot of data which is saved in the project's folder under "output" folder. While the code is run and afterword, the amount of data freeze kdevelop. Cleaning the folder of the data files unfreeze kdevelop. Because of this, I cannot use kdevelop. I suggest adding a blacklist for parsing to prevent parsing of binary data files. I also don't understand why kdevelop should try to parse my binary files... Or maybe there is an option somewhere already? I did not find one.
kdevelop --version Qt: 4.6.2 KDE Development Platform: 4.4.1 (KDE 4.4.1) KDevelop: 3.9.98 Built from svn revision: 1098891 kdevplatform revision: 1098874
When attaching a strace to a frozen kdevelop, I get floded with: stat("/full/path/to/file.bin", {st_mode=S_IFREG|0644, st_size=1000000, ...}) = 0 (different st_size for each files).
Which project manager are you using? Can you share the project? And no, kdevelop doesn't parse binary files, it only parses files which it has a language plugin for. The type is determined by the shared-mime freedesktop.org specification and uses the file-suffixes most of the time. However we do walk the complete filesystem tree to find all files, but this is done in a background thread.
I'm not sure about the project manager. I wrote my own makefiles and imported them using the custom makefiles. Is that what you meant? The files I output are .bin and .vtk (legacy ascii). There might be a lot of them, but not a ridiculous amount.
(In reply to comment #4) > I'm not sure about the project manager. I wrote my own makefiles and imported > them using the custom makefiles. Is that what you meant? Yes thats what I meant. So would it be possible to share the project so we can test this ourselves and debug it?
I cannot share the source code. The makefiles are quite complex (one makefile including many others for different compilers and machines). As for the files kdevelop generates, there is not much. I can even paste it here: $ cat myproject.git.kdev4 [Project] Manager=KDevCustomMakeManager Name=myproject.git $ cat .kdev4/myproject.kdev4 [Buildset] BuildItems=@Variant(\x00\x00\x00\t\x00\x00\x00\x00\x01\x00\x00\x00\x0b\x00\x00\x00\x00\x01\x00\x00\x00\x1e\x00q\x00u\x00a\x00n\x00t\x00u\x00m\x00f\x00d\x00t\x00d\x00.\x00g\x00i\x00t) and that's it...
(In reply to comment #6) > I cannot share the source code. The makefiles are quite complex (one makefile > including many others for different compilers and machines). Hmm, can you run valgrind with callgrind on kdevelop while it loads your project and provide the callgrind file so we can analyze it?
Sorry for the delay. I tried running kdevelop under callgrind but have an emtpy callgrind.out.pid... I ran it like this: $ valgrind --tool=callgrind --time-stamp=yes kdevelop Will I need to recompile kdevelop? It seems kdevelop is stalling when many small files are created (~20 per seconds). The only way to make kdevelop usable again is to 1) kill it, or 2) clean the output directory. Kdevelop outputs a lot of "lockup: ui is blocking": kdevelop(20901)/kdevelop (cpp support) UIBlockTester::lockup: ui is blocking kdevelop(20901)/kdevelop (cpp support) UIBlockTester::lockup: ui is blocking [...] When I "rm -fr output/*" which clears all data, I get the following output: kdevelop(20901)/kdevelop (custommakefile) ProjectFileSystemWatcher::removeDirectory: Removing Directory from Watcher "[full path]/output/test_bhtree" This bug asside, can't there be an option to just ignore _everything_ ever created in a directory, in my case ./output?
first up: to profile kdevelop, you'd have to do something like this: kdevelop --sessions => get {...HASH...} for your session you want to profile KDEV_SESSION={...HASH...} valgrind --tool=callgrind kdevelop.bin Or if you want to start with a clean session, this would be enough: KDEV_SESSION=asdf valgrind --tool=callgrind kdevelop.bin => open your project, etc. pp. To save you some hassle: make sure you use latest SVN, since I fixed some performance issues recently, that got especially triggered in custom make file projects. That said, I'd be interested in that callgrind file, but I fear that it once again won't show the bottleneck, since some external app like make is called too often.
Ok I'll post the calgrind in a minute. I'm using revision 1098891 for now, but will try with latest afterword. As for "some external app like make is called too often", you mean by kdevelop?
Created attachment 43103 [details] Calgrind -r1116610 Here is a callgrind file. Kdevelop was completely unusable for all of this (~12 minutes). I had to kill it so I don't know if you'll be able to see the problem... Svn -r1116610
Updated to KDevelop 4.2.0 and I have the same issue. If my project creates a lot of files in <project_folder>/output, kdevelop freeze. Attaching an strace reveals MB of: stat("/home/nicolas/fichiers/documents/universite/doc/code/projects/quantumfdtd.git/output/OpenCL/wavefunction_000000001077.bin", {st_mode=S_IFREG|0644, st_size=108, ...}) = 0 stat("/home/nicolas/fichiers/documents/universite/doc/code/projects/quantumfdtd.git/output/OpenCL/wavefunction_000000000361.bin", {st_mode=S_IFREG|0644, st_size=108, ...}) = 0 stat("/home/nicolas/fichiers/documents/universite/doc/code/projects/quantumfdtd.git/output/OpenCL/wavefunction_000000000874.bin", {st_mode=S_IFREG|0644, st_size=108, ...}) = 0 stat("/home/nicolas/fichiers/documents/universite/doc/code/projects/quantumfdtd.git/output/OpenCL/wavefunction_000000000079.bin", {st_mode=S_IFREG|0644, st_size=108, ...}) = 0 stat("/home/nicolas/fichiers/documents/universite/doc/code/projects/quantumfdtd.git/output/OpenCL/wavefunction_000000000386.bin", {st_mode=S_IFREG|0644, st_size=108, ...}) = 0 stat("/home/nicolas/fichiers/documents/universite/doc/code/projects/quantumfdtd.git/output/OpenCL/wavefunction_000000000448.bin", {st_mode=S_IFREG|0644, st_size=108, ...}) = 0 stat("/home/nicolas/fichiers/documents/universite/doc/code/projects/quantumfdtd.git/output/OpenCL/wavefunction_000000000429.bin", {st_mode=S_IFREG|0644, st_size=108, ...}) = 0 stat("/home/nicolas/fichiers/documents/universite/doc/code/projects/quantumfdtd.git/output/OpenCL/wavefunction_000000001443.bin", {st_mode=S_IFREG|0644, st_size=108, ...}) = 0 This is why I havn't been using KDevelop for a long time. I just came back to it to try 4.2.0, but forgot that problem... Can't I just tell KDevelop to ignore the "output" folder (or any other one)? Thanks
Created attachment 56593 [details] strace -p `pidof kdevelop` &> kdevelop_strace.log Here is a complete strace. With kdevelop already opened, I typed: $ strace -p `pidof kdevelop` &> kdevelop_strace.log and in another terminal I run my code. The code creates a folder "output/OpenCL/wavefunctions", output some (binary and ascii) stuff to "output/OpenCL" and at each time step (4800 total) a new binary file in "output/OpenCL/wavefunctions" is created. Until several minutes after the initial calculation launch and end, kdevelop is still frozen. The strace log was at aroung 90MB when I stopped strace. Hopefully something can be done for this... Thanks. 2be2b2ee20c5f6698e0ce66eca5050f9 kdevelop_strace.log 71df414d5c519bb74d63bed728e05c7a kdevelop_strace.log.7z
Forgot to say, using KDE 4.6.0, ArchLinux x86_64, kernel 2.6.36.3-2.
try adding a ".kdev_ignore" file to that folder, assuming it is a cmake folder. but to get it actually fixed, I'd like to have a real callgrind file - the one you provided was not done according to the steps required for KDevelop 4.0.x. Anyways, for KDev 4.2 please do this: valgrind --tool=callgrind kdevelop # reproduce, wait for ~10-20min, ctrl + c => attach the callgrind log here.
Created attachment 56768 [details] Callgrind file (1/2) I've run kdevelop through valgrind using: valgrind --tool=callgrind kdevelop I've let it settle for many minutes, then launched the code's calculation which generates a lot of output files. I've let valgrind run for an hour and a half. I'm attaching the compressed file. 3d7395ef5a8f1c4fefe4ca37b04ea64e callgrind.out.18545 74794673899f91e338e8333b82a3e573 callgrind.out.18545.7z I'm not sure if kdevelop was compiled with debug symbols as it was installed through Arch's package manager. If it needs a recompile, let me know. Hopefully you'll see the behaviour in the file. If you need anything else, let me know. Thank you!
Created attachment 56769 [details] Callgrind file (2/2) Due to file size limitation, I had to split the attachment in two using: split -d -a 1 --bytes=900KB callgrind.out.18545.7z callgrind.out.18545.7z To merge: cat callgrind.out.18545.7z0 callgrind.out.18545.7z1 > callgrind.out.18545.7z 3d7395ef5a8f1c4fefe4ca37b04ea64e callgrind.out.18545 74794673899f91e338e8333b82a3e573 callgrind.out.18545.7z 5429c17d2716c24b1912456c248185fb callgrind.out.18545.7z0 3c9a505fbe4083264131b17edbbdace3 callgrind.out.18545.7z1
I am hit again by this. This is terribly annoying. My project folder has different output* folders contianing ~170 GB of binary data. Opening the project in an empty session or opening a session containing the project renders kdevelop unusable. I'm pretty sure the parse is the culprit. It's probably trying to access the thousands of files binary in the output directories. A simple workaround would be to add a blacklist for folders the parser is acting on. That way I could blacklist all "output*" folders and continue using kdevelop. Right now, I have to use something else... I'm using Kdevelop 4.2.3 (KDE 4.6.7) on Gentoo x86_64.
to workaround this issue, please add a ".kdev_ignore" file to the data folders you want to ignore.
Additionally, I did some changes to the custommake manager in master. if you are using that, and have some time, it would be very helpful if you could re-do the callgrind profiling you did before, as the code paths there are now replaced. Alternatively, it would be interesting if you could come up with a way to artificially construct a project that exhibits this behavior. This would enable me to reproduce it and hence try to improve the situation. Cheers
Thanks Milian for your suggestions. I am indeed using a custom make file. I will try to build KDevelop from git and report back. I added .kdev_ignore files to each output* folder I had (15 of them). Loading the project now half-works. The GUI is responsive, I can continue editing, but the background parser is stucked at 0%, with 100% cpu (on one core) and 28% or RAM (~5GB). Is KDevelop going into subfolders, even if a .kdev_ignore is present in the parent folder? I tried cloning my git repo (so there wouldn't be any data folders) and creating a big file using dd (dd if=/dev/zero and dd if=/dev/null). I went up to 500GB for the empty/sparse file (from /dev/null) and 50GB with zeros but it went fine: kdevelop went over it quite fast. What I could do is build a reduced version of my code that just outputs a lot of data. If I remember correctly, the behavior is worst when many many small files are created. Bigger files outputed less often does not show the problem. I understand though that running an untrusted binary is not something we want to do. I'll create a repo instead with a basic project that can, hopefully, identify the problem.
Hi there. First up, no - the parser should ignore files that are not part of your project - at least if they are not otherwise "#include"'ed - which should not be the case for data files. Which means that you hit a different bug. Please attach GDB and get a "thread apply all bt" backtrace, then attach that to a new bug report. Secondly I doubt that single big files are the issue, but rather as you say lots of small ones. Maybe writing a simple shell script that just creates tons of small files in a given directory, could be enough to trigger the issue and hence give me something to work on. Please try that first, before spending time on creating a specific project repo for this. Cheers
I've uploaded to github a simple project that creates 1 million files of size ~79k (10,000 doubles). I can then reproduce the behavior: KDevelop freezes. See https://github.com/nbigaouette/kdevelop_bug_229602 To clean data, compile and run: $ make cd gcc r WARNING: It will create ~100GB of data. Creating the empty file "output/.kdev_ignore" seems to fix this, at least for that test example.
I forgot to say that this test has been performed using kdevelop and kdevplatform from git master (79e68a4e12fd16535f48637ce2b45150ea056d7a and 893c8ab4229bfd8d7f4cd47fda6c63c580d4ebfb).
this code has improved a lot lately, I'd say it should be fixed. If not please reopen.