Bug 450426

Summary: OFX import is broken on MS-Windows
Product: [Applications] kmymoney Reporter: John Zoetebier <john>
Component: importerAssignee: KMyMoney Devel Mailing List <kmymoney-devel>
Status: RESOLVED FIXED    
Severity: normal CC: both, hourra, jdard, LibreStephen, me, tomwiesek, trouxezatteca-8886, vleems
Priority: NOR    
Version: 5.1.2   
Target Milestone: ---   
Platform: Other   
OS: Other   
Latest Commit: Version Fixed In:
Sentry Crash Report:
Attachments: OFX import failed
KMyMoney About screen
image.png
image.png
image.png
image.png
Winmerge screen dump for location of libofx in kmymoney-5.1-1143-windows-msvc2019_64-cl.exe
Winmerge screendump with location of libofx folder in setup file kmymoney-5.1-1305-windows-msvc2019_64-cl.exe
attachment-31035-0.html
attachment-22895-0.html

Description John Zoetebier 2022-02-17 04:55:48 UTC
Created attachment 146846 [details]
OFX import failed

SUMMARY
***
NOTE: If you are reporting a crash, please try to attach a backtrace with debug symbols.
See https://community.kde.org/Guidelines_and_HOWTOs/Debugging/How_to_create_useful_crash_reports
***


STEPS TO REPRODUCE
1. Import OFX file
2. 
3. 

OBSERVED RESULT
OFX import fails

EXPECTED RESULT
OFX file is imported

SOFTWARE/OS VERSIONS
Windows: 10
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 
Qt Version: 

ADDITIONAL INFORMATION
Console shows:
Loading "C:/apps/KMyMoney/bin/kmymoney/ofximporter.dll"
Plugins: ofximporter loaded
Loading "C:/apps/KMyMoney/bin/kmymoney/onlinejoboutboxview.dll"
Plugins: onlinejoboutboxview loaded
Loading "C:/apps/KMyMoney/bin/kmymoney/qifimporter.dll"
Plugins: qifimporter loaded
Loading "C:/apps/KMyMoney/bin/kmymoney/reconciliationreport.dll"
Plugins: reconciliation report loaded
Loading "C:/apps/KMyMoney/bin/kmymoney/reportsview.dll"
Plugins: reportsview loaded
Loading "C:/apps/KMyMoney/bin/kmymoney/sqlstorage.dll"
Plugins: sqlstorage loaded
Loading "C:/apps/KMyMoney/bin/kmymoney/xmlstorage.dll"
Plugins: xmlstorage loaded
Found html dir(s): ("C:/apps/KMyMoney/bin/data/kmymoney//html/")
Found an 'html' folder with CSS files: "C:/apps/KMyMoney/bin/data/kmymoney//html/"
reading file
start parsing file
startDocument
reading securities
endDocument
Cost center model created with items 0
Payees model created with items 0
OfxImporterPlugin::slotImportFile
setup callback routines
process data
LibOFX ERROR: find_dtd():Unable to find the DTD named opensp.dcl
LibOFX ERROR: find_dtd():Unable to find the DTD named ofx160.dtd
LibOFX ERROR: ofx_proc_file(): FATAL: Missing DTD, aborting
Comment 1 John Zoetebier 2022-02-17 04:56:25 UTC
Created attachment 146847 [details]
KMyMoney About screen
Comment 2 John Zoetebier 2022-02-17 04:57:43 UTC
The DTD files are present in folder:
C:\apps\KMyMoney\bin\data\libofx\dtd
Comment 3 Jack 2022-02-17 17:52:51 UTC
Has this worked in previous versions?  If so, how recently?  Which version are you using (per the file downloaded, I see the Help About version)
Comment 4 John Zoetebier 2022-02-17 20:40:49 UTC
Created attachment 146884 [details]
image.png

Hi Jack,

Version 5.0.6 is working fine.
[cid:a11fc139-2d3e-4fc5-a3fa-e9b90c1f43f9]

Setup file KMyMoney 5.0.6: kmymoney5-mingw64-5.0.6-8.1-setup.exe
Setup file KMyMoney 5.1.2: kmymoney-5.1-1305-windows-msvc2019_64-cl.exe

Console errors:
process data
LibOFX ERROR: find_dtd():Unable to find the DTD named opensp.dcl
LibOFX ERROR: find_dtd():Unable to find the DTD named ofx160.dtd
LibOFX ERROR: ofx_proc_file(): FATAL: Missing DTD, aborting
process data done

However, files are present:

[cid:d619d676-4c1d-4ef6-b7b2-c96df0a74ab0]


I have installed another version 5.1.2 of KMyMoney installed on VirtualBox guest Windows 10 whihc is working properly.
[cid:6ec7c50f-308a-41ae-b622-fdff70ff34bb]

The location of DTD files is at:
[cid:f55b0125-2bbf-4676-9844-7ae4b76f2136]







________________________________
From: Jack <bugzilla_noreply@kde.org>
Sent: Friday, 18 February 2022 6:52 am
To: john@zoetebier.net <john@zoetebier.net>
Subject: [kmymoney] [Bug 450426] OFX import fails: Unable to find the DTD named opensp.dcl

https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.kde.org%2Fshow_bug.cgi%3Fid%3D450426&amp;data=04%7C01%7C%7Cfec330596fe74098c8bd08d9f23e537d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637807171798530434%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=%2BKlWIw6BH%2BUGFZ7CfdUfODuGGplPb0HOhzV%2F5c3zQBQ%3D&amp;reserved=0

--- Comment #3 from Jack <ostroffjh@users.sourceforge.net> ---
Has this worked in previous versions?  If so, how recently?  Which version are
you using (per the file downloaded, I see the Help About version)

--
You are receiving this mail because:
You reported the bug.
Comment 5 John Zoetebier 2022-02-17 20:40:49 UTC
Created attachment 146885 [details]
image.png
Comment 6 John Zoetebier 2022-02-17 20:40:49 UTC
Created attachment 146886 [details]
image.png
Comment 7 John Zoetebier 2022-02-17 20:40:49 UTC
Created attachment 146887 [details]
image.png
Comment 8 Jack 2022-02-21 22:17:15 UTC
First,  as a hint, when uploading multiple attachments, please provide a description for each of them, as otherwise it is hard to be sure which si which.

My current understanding is that KMyMoney 5.0.6 from kmymoney5-mingw64-5.0.6-8.1-setup.exe works, and has worked for a long time.  You tried to upgrade to KMyMoney 5.1.2 from kmymoney-5.1-1305-windows-msvc2019_64-cl.exe (from last week) but it fails.  I see the paths of the dtd folder are different for the two instances - are they both installed at the same time?  If they are truly in different locations it might work, but I'm not sure.  Also, your screenshot of Help/About shows version 5.1.2-0af86e098, which is from last September based on finding that commit in git.  Might there be any confusion about which versions you are actually using?

Then, 5.1.2 (you say "another version" but not exactly which version) in a Windows VM does work.  Have you tried that exact same version in your main system?  If it works in the VM but not on the host system, I would first try again, being sure you uninstall all other versions first.
Comment 9 John Zoetebier 2022-02-23 06:48:04 UTC
Tested version 5.1.2-0af86e098 on host Windows 10. OFX import is working.
OFX import fails in 5.1.2-83e127485  (setup file kmymoney-5.1-1305-windows-msvc2019_64-cl.exe).
I suspect that moving the DTD files to an other location may be the reason for the OFX import issue.
Comment 10 Jack 2022-02-23 23:41:14 UTC
Are you saying that you moved the location of those files (in which case I'm not at all surprised that the program couldn't find them) or that between September (5.1.2-0af86e098) and last week (5.1.2-83e127485) the install package moved where the DTD files are installed but the program is still looking in the old location?  I suppose it is possible an upgrade to libofx itself moved the files, but I can't imagine why they wouldn't be found in the new location.  At least it's something to start digging into.
Comment 11 John Zoetebier 2022-02-24 00:28:42 UTC
Some developer, in their ultimate wisdom, decided to move the dtd files to another location.
The OFX import function cannot find the new DTD location. I have tested this on two different PCs.
That the location of the DTD files changed can be proven by extracting the setup files.
Next I did a WinMerge over the extracted setup files.
I will attach the following files:
Winmerge - libofx folder - kmymoney-5.1-1143.jpg
Winmerge - libofx folder - kmymoney-5.1-1305.jpg
Comment 12 John Zoetebier 2022-02-24 00:31:32 UTC
Created attachment 147089 [details]
Winmerge screen dump for location of libofx in kmymoney-5.1-1143-windows-msvc2019_64-cl.exe

This screendump show the location of libofx folder in setup file kmymoney-5.1-1143-windows-msvc2019_64-cl.exe
Comment 13 John Zoetebier 2022-02-24 00:34:06 UTC
Created attachment 147090 [details]
Winmerge screendump with location of libofx folder in setup file kmymoney-5.1-1305-windows-msvc2019_64-cl.exe

This screendump show the location of libofx folder in setup file kmymoney-5.1-1305-windows-msvc2019_64-cl.exe
Comment 14 Thomas Baumgart 2022-02-28 07:40:45 UTC
Git commit 3022d9b0db11d01a96457ecae3a469282d8dd7ab by Thomas Baumgart.
Committed on 28/02/2022 at 07:40.
Pushed by tbaumgart into branch 'master'.

Fix path to DTD on Windows

M  +6    -0    kmymoney/plugins/ofx/import/ofximporter.cpp

https://invent.kde.org/office/kmymoney/commit/3022d9b0db11d01a96457ecae3a469282d8dd7ab
Comment 15 Thomas Baumgart 2022-02-28 10:21:28 UTC
Git commit a025ae3b5724a0c33be3e7a32c788118a72079f5 by Thomas Baumgart.
Committed on 28/02/2022 at 10:21.
Pushed by tbaumgart into branch 'master'.

Make sure to include terminating directory separator in dtd path

M  +1    -1    kmymoney/plugins/ofx/import/ofximporter.cpp

https://invent.kde.org/office/kmymoney/commit/a025ae3b5724a0c33be3e7a32c788118a72079f5
Comment 16 Thomas Baumgart 2022-03-01 06:51:24 UTC
The originally reported location problem has been fixed in master (and will be fixed in 5.1 soon) so far with the recent commits by overriding the default path with the one used for KMyMoney.

Unfortunately, this leads to the next problem which is that the build of libOFX contained in the KDE binary factory build produces weird results. This has to be analyzed and fixed. Therefore, I will update the topic of this bug to a more generic version which also covers this problem.
Comment 17 Thomas Baumgart 2022-03-01 07:29:10 UTC
*** Bug 450939 has been marked as a duplicate of this bug. ***
Comment 18 Thomas Baumgart 2022-03-01 11:57:51 UTC
*** Bug 451000 has been marked as a duplicate of this bug. ***
Comment 19 Tom 2022-04-11 22:40:33 UTC
Has there been any progress on this bug?

Thanks.
Comment 20 Thomas Baumgart 2022-04-13 06:45:45 UTC
To my knowledge, unfortunately, not yet.
Comment 21 Hugo 2022-04-13 08:08:32 UTC
Created attachment 148126 [details]
attachment-31035-0.html

Hello,

I'm unsure about how to read this bug report, so my apologies if this reply
is nonsense :
- I certainly can import manualy OFX on my MS-windows KMM  (I've just done
it)
- I can't Direct Connect since OFX Importer Plugin
<https://docs.kde.org/stable5/en/kmymoney/kmymoney/details.impexp.ofx.html>
seems to be missing in MS-windows KMM

This might be the meaning of this bug report, though...


Hugo

Le mer. 13 avr. 2022 à 08:46, Thomas Baumgart via KMyMoney-devel <
kmymoney-devel@kde.org> a écrit :

> https://bugs.kde.org/show_bug.cgi?id=450426
>
> --- Comment #20 from Thomas Baumgart <tbaumgart@kde.org> ---
> To my knowledge, unfortunately, not yet.
>
> --
> You are receiving this mail because:
> You are the assignee for the bug.
Comment 22 M Both 2022-04-13 11:13:03 UTC
I received this cc from Hugo (which I don't seem to be able to find here)
--- Comment #21 from Hugo <hugoborrell@gmail.com> ---
Hello,
I'm unsure about how to read this bug report, so my apologies if this reply
is nonsense :
- I certainly can import manualy OFX on my MS-windows KMM  (I've just done
it)
- I can't Direct Connect since OFX Importer Plugin
<https://docs.kde.org/stable5/en/kmymoney/kmymoney/details.impexp.ofx.html>
seems to be missing in MS-windows KMM
This might be the meaning of this bug report, though...
Hugo
Reply by Michael (one of those who reported the bug):
I CAN import OFX without difficulty on an installation of 5.0.6 on same machine as I have  version 5.1.7-a75e034b4 if that helps clarify anything.
Comment 23 Tom 2022-04-20 02:51:38 UTC
(In reply to M Both from comment #22)
> I received this cc from Hugo (which I don't seem to be able to find here)
> --- Comment #21 from Hugo <hugoborrell@gmail.com> ---
> Hello,
> I'm unsure about how to read this bug report, so my apologies if this reply
> is nonsense :
> - I certainly can import manualy OFX on my MS-windows KMM  (I've just done
> it)
> - I can't Direct Connect since OFX Importer Plugin
> <https://docs.kde.org/stable5/en/kmymoney/kmymoney/details.impexp.ofx.html>
> seems to be missing in MS-windows KMM
> This might be the meaning of this bug report, though...
> Hugo
> Reply by Michael (one of those who reported the bug):
> I CAN import OFX without difficulty on an installation of 5.0.6 on same
> machine as I have  version 5.1.7-a75e034b4 if that helps clarify anything.

I can import on 5.0.6 as well but that version keeps crashing on my computer.
Comment 24 Tom 2022-05-23 15:58:47 UTC
I was wondering if there are any updates on this issue?

Thank you.
Comment 25 Ful R67 2022-06-13 12:41:42 UTC
(In reply to Tom from comment #24)
> I was wondering if there are any updates on this issue?
> 
> Thank you.

Hello,

+1 about asking an update. I'm also wondering about it because this really annoying for Windows users and OFX daily using.

Could you let us know about a taking into account please ?
Thanks
Comment 26 Vicki Leems 2022-06-21 14:46:09 UTC
I would also like to see this fixed.  I haven't been able to import my transactions for several months and I am following behind in my accounting.
Comment 27 Dawid Wróbel 2022-06-30 14:55:55 UTC
All,

It should be fixed in this build: https://binary-factory.kde.org/view/Windows%2064-bit/job/KMyMoney_Release_win64/1444/
Please test once it's ready.
Comment 28 Dawid Wróbel 2022-06-30 15:00:33 UTC
Git commit 9fffb2d6b006a3600924efc1b800d667f2e12501 by Dawid Wróbel.
Committed on 30/06/2022 at 14:59.
Pushed by wrobelda into branch 'master'.

[libofx] Increase patchlevel to force commit

M  +1    -1    libs/libofx/libofx.py

https://invent.kde.org/packaging/craft-blueprints-kde/commit/9fffb2d6b006a3600924efc1b800d667f2e12501
Comment 29 Dawid Wróbel 2022-06-30 15:26:00 UTC
There appears to be another issue with OFX importer under Windows, but unrelated to this one.

Also:

> Some developer, in their ultimate wisdom, decided to move the dtd files to another location.

The issue was caused by switching to another build system for libofx, a dependency library. Snarky comments like this are not appreciated by developers who develop this software in their free time. Please feel invited to learning the intrinsically complicated process of procuring packages for several platforms, in a project as complex as KMyMoney.
Comment 30 John Zoetebier 2022-06-30 20:53:37 UTC
Created attachment 150304 [details]
attachment-22895-0.html

Don't shoot the messenger!
Don't fix an application if it is not broken. This OFX import was working properly before.
I have no problem with introducing a new build system, but TEST, TEST, TEST !
And it's not like there are not oodles of tools available nowadays to automate the testing either.

  *   Run unit tests
  *   Automated deploys to a build server
  *   Virtual guests for testing on other platforms

Personal rant follows.
I have been using KMyMoney for 20 years, maybe more.
But the last few years, since around 2019,  it has come to the point that I do not even bother anymore to upgrade to a new release as it gets worse and worse.
I have become reluctant to lodge bugs out of fear it will just get worse.
For example:

  *   The tax report has entries at the top that are not even tax related
  *   The order of accounts under Account is not right anymore: reverse order
  *   There is a bug with

If someone feels it is getting too complicated, then stop fiddling with the software until you understand what you are doing.
Rather do NOTHING, than "FIX" something that is NOT BROKEN in the first place.
Hopefully this is a wakeup call.

________________________________
From: Dawid Wróbel <bugzilla_noreply@kde.org>
Sent: Friday, 1 July 2022 3:26 am
To: john@zoetebier.net <john@zoetebier.net>
Subject: [kmymoney] [Bug 450426] OFX import is broken on MS-Windows

https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.kde.org%2Fshow_bug.cgi%3Fid%3D450426&amp;data=05%7C01%7C%7C662b3c3658d74b20076c08da5aacdbdb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637921995733532259%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=vHSJE4zE4B38wJ7z7dDm%2FlGH4WK39Pf%2B1fhJwNfRiUg%3D&amp;reserved=0

--- Comment #29 from Dawid Wróbel <me@dawidwrobel.com> ---
There appears to be another issue with OFX importer under Windows, but
unrelated to this one.

Also:

> Some developer, in their ultimate wisdom, decided to move the dtd files to another location.

The issue was caused by switching to another build system for libofx, a
dependency library. Snarky comments like this are not appreciated by developers
who develop this software in their free time. Please feel invited to learning
the intrinsically complicated process of procuring packages for several
platforms, in a project as complex as KMyMoney.

--
You are receiving this mail because:
You reported the bug.
Comment 31 Dawid Wróbel 2022-07-01 10:59:53 UTC
> I have no problem with introducing a new build system, but TEST, TEST, TEST !
> And it's not like there are not oodles of tools available nowadays to
> automate the testing either.
> 
>   *   Run unit tests
>   *   Automated deploys to a build server
>   *   Virtual guests for testing on other platforms

You realize how condescending you sound? Like we don't know these tools? I made it pretty clear we do this in our free time, as in: time is the factor here, not knowledge, which you so happily imply.

Please, do not comment back here unless you have something helpful to add.
Comment 32 Ful R67 2022-07-04 09:14:12 UTC
Hi Team,

For me same "import OFX error" with build #1444 and more recent. 
I can help with logs or whatever if needed.

Thanks for helping the community Dev team :-)
Comment 33 Ful R67 2022-07-09 15:53:25 UTC
Still not fixed, OFX import is not working on MS Windows.
Comment 34 Dawid Wróbel 2022-07-09 16:14:44 UTC
This very issue has been resolved. Having separate reports for specific issues is on purpose, as it helps tracking changes in the code, so a separate report must be filed if another problem is reported, even within same area.

I filed a new report here:
https://bugs.kde.org/show_bug.cgi?id=456520
Comment 35 Thomas Baumgart 2022-07-16 15:39:34 UTC
*** Bug 456794 has been marked as a duplicate of this bug. ***