Bug 479266 - Unable to export a RAW to iNaturalist
Summary: Unable to export a RAW to iNaturalist
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Plugin-WebService-iNaturalist (show other bugs)
Version: 8.2.0
Platform: Microsoft Windows Microsoft Windows
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-01-01 16:57 UTC by ThierryA
Modified: 2024-01-05 21:27 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 8.3.0


Attachments
error message (9.71 KB, image/jpeg)
2024-01-01 16:57 UTC, ThierryA
Details

Note You need to log in before you can comment on or make changes to this bug.
Description ThierryA 2024-01-01 16:57:10 UTC
Created attachment 164599 [details]
error message

SUMMARY
***
NOTE: If you are reporting a crash, please try to attach a backtrace with debug symbols.
See https://community.kde.org/Guidelines_and_HOWTOs/Debugging/How_to_create_useful_crash_reports
***


STEPS TO REPRODUCE
1. select a raw (tested with CR2 and ORF)
2. menu: Export - to iNaturalist
3. fill in the form and click start uploading

OBSERVED RESULT
error message : see attached
server error - Unprocessable entity

A record is created on iNaturalist  but the photo is missing

EXPECTED RESULT


SOFTWARE/OS VERSIONS
Windows: win10 - firefox (latest)
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
Comment 1 caulier.gilles 2024-01-01 17:02:24 UTC
Hi,

Did you want to push the original RAW file on the webservice ? Is inaturalist support RAW files import ?

Converting RAW to a demosaiced version as JPEG, TIFF, PNG is not enough to post on the webservice ?

Which kind of RAW format did you use ?

Best regards and happy new year.

Gilles Caulier
Comment 2 caulier.gilles 2024-01-01 17:11:26 UTC
Joerg,

With a quick check to inaturalist plugin code, i cannot see any RawEngine usage to preprocess RAW file before upload.

This is typically very easy to do with digiKam API, and take few line of code. See this one for ex:

https://invent.kde.org/graphics/digikam/-/blob/master/core/dplugins/generic/tools/htmlgallery/generator/galleryelementfunctor.cpp?ref_type=heads#L100

In this function, the HTML Gallery plugin is able to pre-process RAW file and export a JPG version as well.

Gilles Caulier
Comment 3 joergmlpts 2024-01-02 06:18:01 UTC
Gilles,
   do you have instructions for Kubuntu 22.04 to build Digikam? I have not been able to compile DigiKam for a very long time.

I have just been using the package (7.5.0) that comes with Kubuntu 22.04.

In order to make these changes I need to compile DigiKam and also a few raw images for testing.

Joerg
Comment 4 caulier.gilles 2024-01-02 06:41:47 UTC
Hi Joerg and happy new year.

I use Kubuntu too here in VM. It's simple:

- run the script to install dependencies : 

https://invent.kde.org/graphics/digikam/-/blob/master/project/scripts/installdeps-ubuntu.sh?ref_type=heads

Warning: this script disable the Snap mechanism on the system, to only use native packages only.

- When it's done, just run the bootstrap.linuc on the root of the repository :

https://invent.kde.org/graphics/digikam/-/blob/master/bootstrap.linux?ref_type=heads

This will create a build directory where you can start compilation with "make -j".

To install, run "make install/fast" in the build directory.

That alll.

For the RAW files, look this repository :

https://rawsamples.ch/index.php/en/

Best

Gilles
Comment 5 joergmlpts 2024-01-03 02:12:47 UTC
Thanks, Gilles!

I have run your installdeps-ubuntu.sh script. It disables snap, removes the local snap directory - and then installs firefox in a snap. After running the script, the snap daemon has been reinstalled and the snap directory is back:


The following packages will be REMOVED:
  libsnapd-qt1* plasma-discover-backend-snap* snapd* squashfs-tools*
0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
After this operation, 104 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 273417 files and directories currently installed.)
Removing plasma-discover-backend-snap (5.24.7-0ubuntu0.1) ...
Removing libsnapd-qt1:amd64 (1.60-0ubuntu1) ...
Removing snapd (2.58+22.04.1) ...
Warning: Stopping snapd.service, but it can still be activated by:
  snapd.socket
Removing squashfs-tools (1:4.5-3build1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.5) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...
(Reading database ... 273292 files and directories currently installed.)
Purging configuration files for snapd (2.58+22.04.1) ...
rmdir: failed to remove '/etc/systemd/system/snapd.mounts.target.wants': No such file or directory
Discarding preserved snap namespaces
Final directory cleanup
Removing extra snap-confine apparmor rules
Removing snapd cache
Removing snapd state
Purging configuration files for plasma-discover-backend-snap (5.24.7-0ubuntu0.1) ...
./installdeps-ubuntu.sh: line 328: /etc/apt/preferences.d/firefox-no-snap: Permission denied
PPA publishes dbgsym, you may need to include 'main/debug' component
Repository: 'deb https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu/ jammy main'
Description:
Mozilla Team's Firefox stable + 115 ESR and Thunderbird 115 stable builds

Support for Ubuntu 16.04 / 18.04 ESM is included.
More info: https://launchpad.net/~mozillateam/+archive/ubuntu/ppa
Adding repository.
Press [ENTER] to continue or Ctrl-c to cancel.
Adding deb entry to /etc/apt/sources.list.d/mozillateam-ubuntu-ppa-jammy.list
Adding disabled deb-src entry to /etc/apt/sources.list.d/mozillateam-ubuntu-ppa-jammy.list
Adding key to /etc/apt/trusted.gpg.d/mozillateam-ubuntu-ppa.gpg with fingerprint 0AB215679C571D1C8325275B9BDB3D89CE49EC21
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease                               
Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease                                        
Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease                                     
Hit:5 http://security.ubuntu.com/ubuntu xenial-security InRelease                                       
Get:6 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu jammy InRelease [23.8 kB]
Get:7 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu jammy/main i386 Packages [2,184 B]
Get:8 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu jammy/main amd64 Packages [34.2 kB]
Get:9 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu jammy/main Translation-en [7,144 B]
Fetched 67.4 kB in 1s (47.2 kB/s)
Reading package lists... Done
W: http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in 
apt-key(8) for details.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  snapd squashfs-tools
The following NEW packages will be installed:
  firefox snapd squashfs-tools
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 231 kB/24.1 MB of archives.
After this operation, 103 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 squashfs-tools amd64 1:4.5-3build1 [159 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 firefox amd64 1:1snap1-0ubuntu2 [72.3 kB]
Fetched 231 kB in 1s (396 kB/s)
Preconfiguring packages ...
Selecting previously unselected package squashfs-tools.
(Reading database ... 273282 files and directories currently installed.)
Preparing to unpack .../squashfs-tools_1%3a4.5-3build1_amd64.deb ...
Unpacking squashfs-tools (1:4.5-3build1) ...
Selecting previously unselected package snapd.
Preparing to unpack .../snapd_2.58+22.04.1_amd64.deb ...
Unpacking snapd (2.58+22.04.1) ...
Setting up squashfs-tools (1:4.5-3build1) ...
Setting up snapd (2.58+22.04.1) ...
Comment 6 joergmlpts 2024-01-03 04:26:39 UTC
This is the issue:

./installdeps-ubuntu.sh: line 328: /etc/apt/preferences.d/firefox-no-snap: Permission denied

sudo cat > /etc/apt/preferences.d/firefox-no-snap << EOF
Package: firefox*
Pin: release o=Ubuntu*
Pin-Priority: -1
EOF 

The redirection works as this: The non-root shell tries to open the file for writing before it invokes "sudo cat". I changed it to

sudo tee /etc/apt/preferences.d/firefox-no-snap << EOF
Package: firefox*
Pin: release o=Ubuntu*
Pin-Priority: -1
EOF 

and tried again. This second run installed firefox as a regular package, not a snap anymore.

Doesn't the ~/snap directory contain among other things all bookmarks, saved logins and everything else for firefox? Should that directory really be silently deleted by your script?
Comment 7 joergmlpts 2024-01-03 05:51:57 UTC
bootstrap.linux dies with this error:

-- 
CMake Error at /usr/lib/x86_64-linux-gnu/cmake/Qt5/Qt5Config.cmake:28 (find_package):
  Could not find a package configuration file provided by
  "Qt5WebEngineWidgets" with any of the following names:

    Qt5WebEngineWidgetsConfig.cmake
    qt5webenginewidgets-config.cmake

  Add the installation prefix of "Qt5WebEngineWidgets" to CMAKE_PREFIX_PATH
  or set "Qt5WebEngineWidgets_DIR" to a directory containing one of the above
  files.  If "Qt5WebEngineWidgets" provides a separate development package or
  SDK, be sure it has been installed.
Call Stack (most recent call first):
  core/cmake/rules/RulesQtFramework.cmake:23 (find_package)
  core/CMakeLists.txt:37 (include)


-- Configuring incomplete, errors occurred!
Comment 8 caulier.gilles 2024-01-03 06:28:04 UTC
Git commit 1a511dc0d77495a5ee0b0fa1effd212718b5e630 by Gilles Caulier.
Committed on 03/01/2024 at 07:25.
Pushed by cgilles into branch 'master'.

uninstall snap properly with tee instead cat
add missing webenginewidgets package

M  +3    -3    project/scripts/installdeps-ubuntu.sh

https://invent.kde.org/graphics/digikam/-/commit/1a511dc0d77495a5ee0b0fa1effd212718b5e630
Comment 9 caulier.gilles 2024-01-03 06:31:09 UTC
Hi Joerg,

This script is to use to a fresh installed ubuntu system. So usage of snap/firefox is suposed to empty.

I fixed the script about the web engine widgets missing package. Install it with :

sudo apt install qtwebengine5-dev

...and reconfigure with the bootstrap script.

Gilles
Comment 10 joergmlpts 2024-01-04 00:33:07 UTC
Hi Gilles,
  thanks, DigiKam compiles with the library installed.

The fix to the install script was not correct. I have submitted a fix in a merge request.

Joerg
Comment 11 joergmlpts 2024-01-05 04:31:58 UTC
Hi Gilles,
  I have a fix for the issue with raw images. But after I updated my sources to compile and test it, I got stuck these error messages:

/home/user/digikam/core/libs/threadimageio/fileio/loadingcache.cpp:24:10: fatal error: kmemoryinfo.h: No such file or directory
   24 | #include <kmemoryinfo.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.

/home/user/digikam/core/libs/dialogs/libsinfodlg.cpp:38:10: fatal error: kmemoryinfo.h: No such file or directory
   38 | #include <kmemoryinfo.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.

/home/user/digikam/core/libs/dimg/loaders/dimgloader.cpp:29:10: fatal error: kmemoryinfo.h: No such file or directory
   29 | #include <kmemoryinfo.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.

Do you happen to know a workaround?

Thanks,
   Joerg
Comment 12 caulier.gilles 2024-01-05 07:20:37 UTC
Hi Joerg,

this is due to this MR :

https://invent.kde.org/graphics/digikam/-/merge_requests/244

This requires KMemoryInfo class from KDE KCoreAddOns module :

https://api.kde.org/frameworks/kcoreaddons/html/classKMemoryInfo.html

This class exists since  KDE Frameworks 5.95

Gilles
Comment 13 joergmlpts 2024-01-05 15:24:36 UTC
Hi Gilles,

Are you saying that your install-script installed an older version of KDE that digikam now does not compile with anymore?

What do I need to change to compile and test the fix?

Joerg
Comment 14 caulier.gilles 2024-01-05 15:41:41 UTC
No, no, i just used the script to install al dependencies under Kubuntu 23.10 arm64 running in MacOS/UTM/QEmu VM, and all work as expected.

I think the default KF5 version on 22.04 is a little bit older now, and it need to point to a new repository (similar as 23.04 or 23.10)

Which version of KF5 frameworks is installed on your 22.04 ?

Gilles
Comment 15 joergmlpts 2024-01-05 15:47:31 UTC
5.92, I think.
Comment 16 caulier.gilles 2024-01-05 16:59:55 UTC
Hum, if you have a tested code, create a MR, i will check it with the CI...
Gilles
Comment 17 joergmlpts 2024-01-05 18:06:54 UTC
Nope, the fix is not tested because I cannot compile anymore. It might not even compile since I have not used C++ in over a year.

The fix is super simple: Loading raw images and uploading them as jpeg works just fine. The jpeg is uploaded along with the orginal filename. This filename is the issue: When its extension is not jpg or jpeg, lower case or upper case, the iNaturalist server does not accept the jpeg upload. Thus, the fix is to append .jpeg to the filename that is sent to the iNaturalist server. I have tested that sending a filename that ends in .jpg fixes the problem. But I still need to test the code change to be submitted.
Comment 18 caulier.gilles 2024-01-05 18:39:19 UTC
Joerg,

On my github account, you will found a repo dedicated to compile and install Qt6 and KF6 for digiKam.

All is installed to /opt/qt6. Nothing is installed on your system.

https://github.com/cgilles/digikam-install-deps

In config.sh, the script is configured to use /mnt/data to host donwload and compilation files. This requires large space to work, as QtWebEngine is a monster. After installation the space used by the binary is lower of course.

The compilation of QtWebEngine is problematic: it take time and a lots of memory. If you use a VM, set 8Gb of RAM and 4 cores. It's enough, else Chromium code will fill all the memory + swap, and compilation will be broken. Google rules to compile Chromium is not optimized so far: a lots of objects are compiled at the same time and the linking stage fill the memory.

Later, when Qt6.6.1 and current KF6 frameworks from master are installed, you can configure and compile digiKam for Qt6 using the bootstrap.sh. You just need to turn on the right option to use Qt6 instead Qt5.

I run the script since 2 hours on my Arm64 VM (Kubuntu 23.10) and compilation is running under QtWebEngine. It's expected to be completed in 30 minutes.

Gilles
Comment 19 joergmlpts 2024-01-05 20:29:36 UTC
Hi Gilles,
  I installed 23.10 in  a VM and could compile and test my fix. The merge request has been sent.

Joerg
Comment 20 caulier.gilles 2024-01-05 21:27:48 UTC
Git commit 788da64db6ab219aba8364d71be02d6062f4d641 by Gilles Caulier, on behalf of Joerg Lohse.
Committed on 05/01/2024 at 22:26.
Pushed by cgilles into branch 'master'.

Fix for raw images; issue #479266

Fix for raw images; issue #479266.

M  +9    -1    core/dplugins/generic/webservices/inaturalist/inattalker.cpp

https://invent.kde.org/graphics/digikam/-/commit/788da64db6ab219aba8364d71be02d6062f4d641