Bug 444579

Summary: releaseme: Exception when running fetchpo.rb script from cmake
Product: [Developer tools] releaseme Reporter: Andrew Shark <ashark>
Component: generalAssignee: Unassigned bugs <unassigned-bugs-null>
Status: RESOLVED NOT A BUG    
Severity: normal CC: rakuco, sitter
Priority: NOR    
Version First Reported In: unspecified   
Target Milestone: ---   
Platform: Other   
OS: Linux   
Latest Commit: Version Fixed/Implemented In:
Sentry Crash Report:

Description Andrew Shark 2021-10-29 04:32:00 UTC 
Trying to configure an application with KDE_L10N_SYNC_TRANSLATIONS option. The underlying scripts fails, and translations are not downloaded.


STEPS TO REPRODUCE
1. git clone https://invent.kde.org/pim/zanshin.git
2. mkdir build
3. cd build
4. LC_ALL=C cmake -B . -S ../zanshin -DBUILD_TESTING=OFF -DKDE_L10N_SYNC_TRANSLATIONS=ON --trace-source=/usr/share/ECM/kde-modules/KDECMakeSettings.cmake --trace-expand 

OBSERVED RESULT
There is an error in downloading translations. The relevant part of output:

```
Cloning into 'releaseme'...
/usr/share/ECM/kde-modules/KDECMakeSettings.cmake(376):  execute_process(COMMAND;ruby;/home/andrew/Development/Lab/build/releaseme/fetchpo.rb;--origin;trunk;--project;pim/zanshin;--output-dir;/home/andrew/Development/Lab/build/po;--output-poqm-dir;/home/andrew/Development/Lab/build/poqm;/home/andrew/Development/Lab/zanshin )
INFO -- ReleaseMe::L10n: Downloading translations for /home/andrew/Development/Lab/zanshin
#<Thread:0x000055f78c23c268 /home/andrew/Development/Lab/build/releaseme/lib/releaseme/translationunit.rb:109 run> terminated with exception (report_on_exception is true):
/home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n/asset.rb:42:in `gsub!': invalid byte sequence in US-ASCII (ArgumentError)
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n/asset.rb:42:in `strip_comments!'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n/asset.rb:16:in `strip!'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n.rb:78:in `block (2 levels) in download'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n.rb:76:in `each'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n.rb:76:in `block in download'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/translationunit.rb:128:in `block (2 levels) in each_language_with_tmpdir'
        from /usr/lib/ruby/3.0.0/tmpdir.rb:96:in `mktmpdir'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/tmpdir.rb:22:in `mktmpdir'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/translationunit.rb:128:in `block in each_language_with_tmpdir'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/translationunit.rb:111:in `block (2 levels) in blocking_thread_pool'
/home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n/asset.rb:42:in `gsub!': invalid byte sequence in US-ASCII (ArgumentError)
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n/asset.rb:42:in `strip_comments!'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n/asset.rb:16:in `strip!'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n.rb:78:in `block (2 levels) in download'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n.rb:76:in `each'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/l10n.rb:76:in `block in download'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/translationunit.rb:128:in `block (2 levels) in each_language_with_tmpdir'
        from /usr/lib/ruby/3.0.0/tmpdir.rb:96:in `mktmpdir'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/tmpdir.rb:22:in `mktmpdir'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/translationunit.rb:128:in `block in each_language_with_tmpdir'
        from /home/andrew/Development/Lab/build/releaseme/lib/releaseme/translationunit.rb:111:in `block (2 levels) in blocking_thread_pool'
-- Found Boost: /usr/lib64/cmake/Boost-1.76.0/BoostConfig.cmake (found version "1.76.0")
```

EXPECTED RESULT
There should be no any errors. There should be only these info lines:

```
INFO -- ReleaseMe::L10n: Downloading translations for /home/andrew/Development/Lab/pim/zanshin
INFO -- ReleaseMe::L10n: The following languages had translations files but they amount to no useful strings so they were removed: ja, zh_TW
INFO -- ReleaseMe::L10n: No translations for: as, ast, az, ar, af, bg, be@latin, bn, be, bn_IN, crh, br, csb, cy, el, en, eo, fa, fy, gd, gu, ha, hi, he, hr, hne, hsb, hy, id, is, ka, kab, kk, kn, km, ku, lb, lv, mai, ml, mk, ms, mt, ne, nso, oc, or, pa, ps, ro, rw, se, si, sq, sr@ijekavian, sr, sr@ijekavianlatin, sr@latin, ta, te, tg, th, tn, tt, uz, uz@cyrillic, vi, wa, xh, zh_HK
```


SOFTWARE/OS VERSIONS
Operating System: Arch Linux
KDE Plasma Version: 5.23.2
KDE Frameworks Version: 5.87.0
Qt Version: 5.15.2
Current releaseme commit: b830efbd (Oct 27, 2021)
Arch packages versions:
 extra-cmake-modules: 5.87.0-1
 ruby: 3.0.2-2

ADDITIONAL INFORMATION
When running that command

```
ruby /home/andrew/Development/Lab/build/releaseme/fetchpo.rb --origin trunk --project pim/zanshin --output-dir /home/andrew/Development/Lab/build/po --output-poqm-dir /home/andrew/Development/Lab/build/poqm /home/andrew/Development/Lab/zanshin
```

directly in konsole, it works normally.

I suspect that it may be related with ja, zh_TW languages. Because their number is two, and that error "invalid byte sequence in US-ASCII (ArgumentError)" was logged two times in cmake output.
Comment 1 Harald Sitter 2021-10-30 10:33:20 UTC 
if you run `LC_ALL=C locale charmap` I'm sure you'll find that C is in fact not capable of handling unicode because it defaults to ANSI. You need to use a .UTF-8 capable locale.
Comment 2 Andrew Shark 2021-10-30 16:10:11 UTC 
Yes, if I run `LC_ALL=C locale charmap`, I get `ANSI_X3.4-1968`.

I used LC_ALL=C to only make messages on English. Because I got some other errors. They are still unresloved, and I will submit proper bug reports for them. 

But with this case, when I use LOCALE=en_US.UTF-8, then translations are downloaded properly.

Thanks for the advise. Closing.