Bug 390904 - kapidox_generate deletes sources
Summary: kapidox_generate deletes sources
Status: RESOLVED FIXED
Alias: None
Product: frameworks-kapidox
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: unspecified
Platform: openSUSE Linux
: NOR critical
Target Milestone: ---
Assignee: kdelibs bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-02-22 14:19 UTC by Michael Heidelbach
Modified: 2018-02-25 09:47 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Heidelbach 2018-02-22 14:19:47 UTC
Installed PyYAML jinja2 doxygen with zypper.

$ python setup.py install --prefix ~/.local

!! Careful when reproducing this !!
$ cd ~/kde/src/baloo
$ kapidox_generate .
15:04:59 DEBUG Found cached identities file at ~/.cache/kapidox/kde-accounts                                                                                                   
15:04:59 WARNING Group of Baloo not found: dropped.                                                                                                                                      
15:04:59 INFO # Generating doc for Baloo                                                                                                                                                 
15:04:59 INFO Running Doxygen                                                                                                                                                            
15:05:01 INFO # Rebuilding Baloo for interdependencies                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                       
  File "~/.local/bin/kapidox_generate", line 111, in <module>                                                                                                                  
    main()                                                                                                                                                                               
  File "~/.local/bin/kapidox_generate", line 107, in main                                                                                                                      
    copyright=kde_copyright)                                                                                                                                                             
  File "~/.local/lib/python2.7/site-packages/kapidox/hlfunctions.py", line 111, in do_it
    shutil.rmtree(lib.outputdir)
  File "/usr/lib64/python2.7/shutil.py", line 270, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib64/python2.7/shutil.py", line 268, in rmtree
    os.rmdir(path)
OSError: [Errno 22] Invalid argument: '.'
$ ls .
$
All files/folders gone.
Whatever mistake I made during installation or execution this should not happen.
Comment 1 Nate Graham 2018-02-22 15:41:25 UTC
Yikes, this is terrible. Setting severity to Critical, as it causes unexpected data loss. See https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging#Severity
Comment 2 Scott Harvey 2018-02-23 01:08:10 UTC
My attempt to run kapidox_generate also failed, but my files survived.

bundito@mick:~/kde/src/frameworks/kiconthemes$ kapidox_generate .
19:04:41 INFO Downloading KDE identities
19:04:41 DEBUG Using external svn client to fetch svn://anonsvn.kde.org/home/kde/trunk/kde-common/accounts
19:04:41 ERROR Failed to fetch KDE identities: global name 'FileNotFoundError' is not defined
Traceback (most recent call last):
  File "/usr/bin/kapidox_generate", line 111, in <module>
    main()
  File "/usr/bin/kapidox_generate", line 107, in main
    copyright=kde_copyright)
  File "/usr/lib/python2.7/dist-packages/kapidox/hlfunctions.py", line 67, in do_it
    products, groups, libraries, available_platforms = preprocessing.sort_metainfo(metalist, maintainers)
  File "/usr/lib/python2.7/dist-packages/kapidox/preprocessing.py", line 215, in sort_metainfo
    lib = Library(metainfo, products, platforms, all_maintainers)
  File "/usr/lib/python2.7/dist-packages/kapidox/models.py", line 53, in __init__
    products[utils.serialize_name(metainfo['name'])] = Product(metainfo, all_maintainers)
  File "/usr/lib/python2.7/dist-packages/kapidox/models.py", line 144, in __init__
    self.maintainers = utils.set_maintainers(metainfo.get('maintainer'), all_maintainers)
  File "/usr/lib/python2.7/dist-packages/kapidox/utils.py", line 99, in set_maintainers
    maintainers = [all_maintainers.get(maintainer_keys, None)]
AttributeError: 'NoneType' object has no attribute 'get'
bundito@mick:~/kde/src/frameworks/kiconthemes$ ls -al
total 76
drwxrwxr-x  7 bundito bundito  4096 Feb 12 21:26 .
drwxrwxr-x 80 bundito bundito  4096 Feb 12 21:42 ..
-rw-rw-r--  1 bundito bundito    57 Feb 12 21:26 .arcconfig
drwxrwxr-x  2 bundito bundito  4096 Feb 12 21:26 autotests
-rw-rw-r--  1 bundito bundito  3476 Feb 12 21:26 CMakeLists.txt
-rw-rw-r--  1 bundito bundito 26527 Feb 12 21:26 COPYING.LIB
drwxrwxr-x  3 bundito bundito  4096 Feb 12 21:26 docs
drwxrwxr-x  8 bundito bundito  4096 Feb 20 07:45 .git
-rw-rw-r--  1 bundito bundito   197 Feb 12 21:26 KF5IconThemesConfig.cmake.in
-rw-rw-r--  1 bundito bundito   358 Feb 12 21:26 metainfo.yaml
-rw-rw-r--  1 bundito bundito  2910 Feb 12 21:26 README.md
drwxrwxr-x  3 bundito bundito  4096 Feb 15 16:51 src
drwxrwxr-x  2 bundito bundito  4096 Feb 12 21:26 tests
bundito@mick:~/kde/src/frameworks/kiconthemes$
Comment 3 Scott Harvey 2018-02-23 01:10:41 UTC
In the event it matters, I was able to install kapidox as a package for Kubuntu. I will look for the most current source and try it again.
Comment 4 Scott Harvey 2018-02-23 01:41:11 UTC
I tried the git source version as well. It failed, but didn't delete anything.

I attempted to roll back as much as I could, then got the latest .rpm version from openSUSE. Converted it to .deb with alien (I'm on Kubuntu) and tried again. It continues to fail, but hasn't deleted anything.
Comment 5 Scott Harvey 2018-02-23 01:51:15 UTC
Uh oh - there it is. I didn't have Subversion installed during my previous tests. I read my error logs more closely and installed Subversion. Tried kapidox_generate, and sure enough, my kiconthemes source folder has been emptied.

bundito@mick:~/kde/src/frameworks/kiconthemes$ kapidox_generate .
19:48:14 DEBUG Found cached identities file at /home/bundito/.cache/kapidox/kde-accounts
19:48:14 WARNING Group of KIconThemes not found: dropped.
19:48:14 INFO # Generating doc for KIconThemes
19:48:14 INFO Running Doxygen
sh: 1: dot: not found
19:48:14 INFO # Rebuilding KIconThemes for interdependencies
Traceback (most recent call last):
  File "/usr/local/bin/kapidox_generate", line 111, in <module>
    main()
  File "/usr/local/bin/kapidox_generate", line 107, in main
    copyright=kde_copyright)
  File "/usr/local/lib/python3.6/dist-packages/kapidox/hlfunctions.py", line 111, in do_it
    shutil.rmtree(lib.outputdir)
  File "/usr/lib/python3.6/shutil.py", line 484, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 482, in rmtree
    os.rmdir(path)
OSError: [Errno 22] Invalid argument: '.'
bundito@mick:~/kde/src/frameworks/kiconthemes$ ls -al
total 8
drwxrwxr-x  2 bundito bundito 4096 Feb 22 19:48 .
drwxrwxr-x 80 bundito bundito 4096 Feb 12 21:42 ..
Comment 6 Scott Harvey 2018-02-23 04:54:00 UTC
Definitely confirmed. This script does not like to be run at the "." level of a source package. I've been logging it and tracing it all evening. It tries to use "." as the output directory, and in one of the blocks, it wants to erase what was previous written as it does a second pass. Since it believes "." is the output directory, when it calls "rmtree" to clean house, it takes everything. (See approx line 112 of hlfunctions.py)

It also doesn't properly warn about the lack of the Graphviz package, which is used to generate those nifty dependency diagrams in the API docs. Once I installed it, a certain set of errors ceased. There's code in the argument-parsing module that's supposedly checking for Graphviz, but it seems to get skipped. 

I'm not familiar with how this tool is supposed to be run, if not from the root of a source folder. But running it from "." is definitely broken.
Comment 7 Olivier Churlaud 2018-02-23 06:37:52 UTC
No warning/error for the missing Graphviz is a bug, yes, but the deleting of a the files is intended.

As the Readme says, it should be called in the output dir. There is currently no way to build the apidocs over existing folders....

What shall the fix be?
Comment 8 Michael Heidelbach 2018-02-23 08:54:49 UTC
(In reply to Olivier Churlaud from comment #7)
> No warning/error for the missing Graphviz is a bug, yes, but the deleting of
> a the files is intended.
What??? Monty Python: Spring surprise?

> As the Readme says, it should be called in the output dir. 
The --help of kapidox_generate is very extensive. I did not even think of consulting the readme. I just tried it to see what it does. It is totally unexpected for a documenting tool to kill the items it is supposed to document. One expects it to be harmless.
> There is
> currently no way to build the apidocs over existing folders....
Then it should issue an error message and exit. Or ask the user.

> What shall the fix be?
At least this:
$kapidox_generate --help
09:31:46 DEBUG Found cached identities file at ~/.cache/kapidox/kde-accounts
usage: kapidox_generate [-h] [--depdiagram-dot-dir DIR] [--title TITLE]
                        [--man-pages] [--qhp] [--searchengine]
                        [--api-searchbox] [--indexing] [--qtdoc-dir QTDOC_DIR]
                        [--qtdoc-link QTDOC_LINK] [--qtdoc-flatten-links]
                        [--doxygen DOXYGEN] [--qhelpgenerator QHELPGENERATOR]
                        [--doxdatadir DOXDATADIR] [--keep-temp-dirs]
            ==>         sourcesdir

Plus: a big fat warning that whatever-dir will be emptied.
Comment 9 Olivier Churlaud 2018-02-23 20:46:40 UTC
I have a quick fix in the pipe: if the output dir is not empty, the script will exit in error.

It's offtopic but this tool needs more love
Comment 10 Michael Heidelbach 2018-02-23 20:52:29 UTC
That should be sufficient. Please, in that error message make transparent how to do it right.
And please in --help replace 'sourcesdir' with 'doc dir' or whatever is appropiate.

Thanks.

Off-topic: My heart is full with my love for baloo (and I'm not promiscous) :-)
Comment 11 Olivier Churlaud 2018-02-25 09:45:03 UTC
Git commit 70a7f1fb95fbc99c8a0af89959703c7aab0af37f by Olivier CHURLAUD.
Committed on 25/02/2018 at 09:40.
Pushed by ochurlaud into branch 'master'.

Exit in error if the output directory is not empty.
+ Some other small fixes

M  +5    -1    src/kapidox/argparserutils.py
M  +11   -2    src/kapidox/hlfunctions.py
M  +2    -3    src/kapidox_generate

https://commits.kde.org/kapidox/70a7f1fb95fbc99c8a0af89959703c7aab0af37f
Comment 12 Olivier Churlaud 2018-02-25 09:47:12 UTC
I didn't replace the sourcedir by docdir, as it's really the source dir. You should call it as:

$ cd /your/doc/dir
$ kapidox_generate /your/source/dir

This was an important bug anyway, thx for reporting.