Bug 452589 - Skrooge 2.27.0 cannot import OFX anymore
Summary: Skrooge 2.27.0 cannot import OFX anymore
Status: RESOLVED UPSTREAM
Alias: None
Product: skrooge
Classification: Applications
Component: general (show other bugs)
Version: unspecified
Platform: Arch Linux Linux
: NOR major
Target Milestone: ---
Assignee: Stephane MANKOWSKI
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-04-13 18:31 UTC by Bennett Piater
Modified: 2022-07-24 12:15 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
logs (2.62 KB, text/x-log)
2022-04-13 18:31 UTC, Bennett Piater
Details
minimal reproducing example (2.23 KB, application/xml)
2022-04-20 18:24 UTC, Bennett Piater
Details
Import (20.89 KB, image/png)
2022-04-24 15:01 UTC, Stephane MANKOWSKI
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bennett Piater 2022-04-13 18:31:43 UTC
Created attachment 148147 [details]
logs

SUMMARY
Importing OFX files fails completely on any file I tried, due to a low-level parse error (logs attached).
This includes known-good files that I have successfully imported in the past with previous releases of skrooge.

STEPS TO REPRODUCE
1. Attempt to import an OFX file

OBSERVED RESULT
The import fails completely.

EXPECTED RESULT
The import succeeds, at least for files that used to work.

SOFTWARE/OS VERSIONS
Windows: 
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: n/a
KDE Frameworks Version: 5.93.0
Qt Version: 5.15.3

ADDITIONAL INFORMATION
I cannot select the correct version in the dropdown, looks like the options are 2 versions behind :)
Comment 1 Stephane MANKOWSKI 2022-04-14 18:52:10 UTC
Hi,

Could you provide me an anonymized ofx sample file to reproduce the issue ?

This is just to identify if the issue is coming from the format of the OFX file or from your installation.
For information, the error is not raised by Skrooge but by libofx. This could be due to an issue in installation of libofx.
Comment 2 Bennett Piater 2022-04-20 18:24:48 UTC
Created attachment 148271 [details]
minimal reproducing example

I did some more digging because my minimal anonymous file didn't trigger the issue.
After playing with 'xmllint' and not finding anything, I tried importing a pretty-printed file and the cause became obvious: The offenders where all non-ascii characters.
I attached a minimal example that I was able to reproduce this with.

The funny thing is that nothing major changed about my system. Every software handles unicode just fine, and this file used to be correctly imported in previous versions (of skrooge and probably it's dependencies).

This is the output of 'locale':
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Comment 3 Stephane MANKOWSKI 2022-04-24 15:01:17 UTC
Created attachment 148340 [details]
Import

Hi,
No issue for me with this file.
The import seems to be correct.
Comment 4 Bennett Piater 2022-04-24 17:55:20 UTC
I will contact my libofx maintainer then, thank you! Do you have any other hints or ideas?
Comment 5 Stephane MANKOWSKI 2022-04-24 18:07:18 UTC
You can play with ofxdump to check if the issue is coming from libofx.
Comment 6 Bug Janitor Service 2022-05-09 04:34:59 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 7 Bug Janitor Service 2022-05-24 04:35:53 UTC
This bug has been in NEEDSINFO status with no change for at least
30 days. The bug is now closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!
Comment 8 Bennett Piater 2022-07-24 12:15:56 UTC
I tracked this down to the source and got a workaround implemented in libofx!

The cause of this issue is in OpenSP, which libofx uses to parse SGML and XML.
That library's quality is suboptimal, and the last release is 20 years ago - obviously predating UTF.
So there are a ton of workarounds in libofx around multi-byte encodings...

Libofx has accepted the bug [0] and merged a workaround [1] for the case described in this ticket, which amounts to treating this "error" as a warning and ignoring it in the return status.
I was able to locally build and LD_PRELOAD that branch and could successfully import previously-failing OFX files into Skrooge.
So I think we can treat this as fixed upstream, as soon as they release a new version with the fix!

I can handle asking for a new release of libofx in arch, but since the Skrooge project releases an AppImage that also didn't work for me, I would suggest that you release a new build with the latest libofx as soon as that is released :)

PS: The "RESOLVED UPSTREAM" status is not described in https://bugs.kde.org/page.cgi?id=fields.html#bug_status.
I hope this is the correct state for this ticket.

[0] https://github.com/libofx/libofx/issues/60
[1] https://github.com/libofx/libofx/pull/64