Bug 450593

Forgot to mention:  Firewall is not blocking access to fx-rate.net.  Networking issues are eliminated.


(In reply to lp.allard.1 from comment #0)
> SUMMARY
> KMM 5.0.5 (as well as AppImage of 2022-02-19), price updates fails with:
> 
> Fetching URL https://fx-rate.net/CAD/E000007...
> Identifier found: 'CAD/"'
> Date found: 'Sat. Feb. 19 2022'
> Unable to update price for CAD > E000007 (no price or no date)
> Fetching URL https://fx-rate.net/CAD/E000025...
> Identifier found: 'CAD/"'
> Date found: 'Sat. Feb. 19 2022'
> Unable to update price for CAD > E000025 (no price or no date)
> Fetching URL https://fx-rate.net/CAD/E000044...
> Identifier found: 'CAD/"'
> Date found: 'Sat. Feb. 19 2022'
> Unable to update price for CAD > E000044 (no price or no date)
> 
> This used to work, only a few weeks ago...  Unfortunately, I cannot find the
> online source settings in the program options.
> 
> STEPS TO REPRODUCE
> 1. Launch price updater
> 2. Click update all
> 3. All is fine with the exception of some prices from fx-rate.net
> 
> OBSERVED RESULT
> See above
> 
> EXPECTED RESULT
> Update prices
> 
> SOFTWARE/OS VERSIONS
> KDE Frameworks 5.68.0
> Qt 5.12.8 (built against 5.12.8)
> The xcb windowing system
> System: 
> Kernel: 5.4.0-100-generic x86_64 bits: 64 compiler: gcc v: 9.3.0 Desktop:
> Gnome 3.36.9 
> wm: xfwm4 dm: GDM3, LightDM Distro: Linux Mint 20.3 Una base: Ubuntu 20.04
> focal 
> 
> ADDITIONAL INFORMATION
> Data on MySQL server.

Are you really sure you want CAD > E000007 and not the other way around?  That sounds like a problem that has popped up every now and then. but never adequately explained.  Are you really sure you have previously received those price pairs, or did you simply not get any errors when retrieving prices?  Can you determine which equities/stocks use those entries (E000007 E000025 E000044?)  Those are internal identifiers for KMM, and you shouldn't expect to fetch prices for them from the web.  What is needed is to figure out why KMM is trying to do so.

Jack,

I have no idea what you're talking about when you ask if I want CAD > E000007 or the other way around.... Sorry ;)

I've never changed any settings in all of that, and KMM always successfully updated my currencies until a few weeks ago...

Now examining the list of stocks and currencies, I realize that E000007, E000025 and E000044 are not CASH currencies (USD, CAD, EURO, etc) but mutual fund investments.  Actually, out of many investments, these three have NO online pricce update source (I couldnt find any)...

Now the problem seems to be that somehow, three new currencies were added in KMM.   Each one seems to be linked to one of the three mutual fund investments that cannot be updated automatically.  These "currencies" look like this (note that E000007 is ID=CIF7710.CF):

ID: CAD > E000007
Name: CAD units in E000007
Price: CIG7710.CF 77,1070710,771,077,100.0336
Date: 2022-01-21
Internal ID: CAD E000007
Source: Kmymoney Currency

I have NO idea what those are, and how they showed up but they weren't there a while ago and I didnt add them for sure...  Can they be deleted?



(In reply to Jack from comment #2)
> Are you really sure you want CAD > E000007 and not the other way around? 
> That sounds like a problem that has popped up every now and then. but never
> adequately explained.  Are you really sure you have previously received
> those price pairs, or did you simply not get any errors when retrieving
> prices?  Can you determine which equities/stocks use those entries (E000007
> E000025 E000044?)  Those are internal identifiers for KMM, and you shouldn't
> expect to fetch prices for them from the web.  What is needed is to figure
> out why KMM is trying to do so.

Yes, E000007 is the underlying equity for some mutual fund.  Internally, it is stored the same was as currencies, but KMM does know the difference.  Normally, you would have a price pair E000007 -> CAD, or the number of CAD (i.e., price) per share of the fund.  For some reason (and we have seen this before, but still do not have an explanation) the reverse price pair gets created.  If you go to Tools/Prices, you can sometimes delete such entries, but more likely not.  We do not yet understand why.

My suspicion (and it is no more than that yet) is that somehow, in entering a transaction for that mutual fund, the price got entered in some way that KMM thought it needed the number of E000007 per CAD, instead of the other way around.

However, as far as I can tell, the fact that it fails to retrieve that reverse price pair doesn't really matter, as long as the correct pair is retrieved successfully.

Created attachment 147023 [details]
Price Editor

Created attachment 147024 [details]
Update Stocks and Currency Prices

Jack,

I've tried deleting them in tools > prices, and it worked, but when I ran the online price updater , KMM re-added them!

-> Out of many mutual funds, I have 3 of them that I dont have online sources to update prices for (yahoo used to list them but stopped a while ago and Globe and Mail changed their website so the REGEXes are NO longer working)

-> These 3  funds (CIG2321, CIG2323 & CIG7710) are those causing this currency issue

See screenshots.

Other observations:

The date of the last price update changes randomly (probably more a symptom than a root-cause...).  Before I delete these prices they all had a last update date of 2022-01-29, now after deleting them and running the price updater, they have dates matching the last transaction dates of each commodities.

-> CIG7710's last REAL transaction date is 2021-12-17 and the last price update listed by the Price editor is 2021-12-17
-> CIG2321's last REAL transaction date is 2021-12-20 and the last price update listed by the Price editor is 2021-12-20
-> CIG2323's last REAL transaction date is 2021-12-13 and the last price update listed by the Price editor is 2021-09-01 (why different???)



(In reply to lp.allard.1 from comment #7)
> Jack,
> 
> I've tried deleting them in tools > prices, and it worked, but when I ran
> the online price updater , KMM re-added them!
> 
> -> Out of many mutual funds, I have 3 of them that I dont have online
> sources to update prices for (yahoo used to list them but stopped a while
> ago and Globe and Mail changed their website so the REGEXes are NO longer
> working)
> 
> -> These 3  funds (CIG2321, CIG2323 & CIG7710) are those causing this
> currency issue
> 
> See screenshots.

Is it possible for you to create a new kmy file, with one investment account which owns a few shares of one of those funds?  If the price update there fails in the same way, you can post the file, and maybe we can find what causes the problem.  It doesn't make sense for me for KMyMoney Currency to be the source of a price of a stock (I just found some examples in my own data file) or a mutual fund.

I think those date differences make sense.  On the last one, what type of transaction was it on 12/13?  That transaction would not have included a price update unless it was a buy or sell.  In the price tool, you can type the symbol in the search box, and then click the box to show all prices.  That will show you where each of those price entries came from.

The last investment that occurred on 2022-12-13 for "CIG2323" was a "Add shares" transaction (year-end dividend reinvestment).

I looked in the Price Editor for this investment, and indeed, the last price update is from 2022-01-21 (which was the previously reported update for the zombie pair commodity > CAD) and the other date before that is from 2021-09-01 which is now the reported update date for that zombie "pair"....

I am not sure to understand:  do you want me to create a brand new Kmy file and create the same investment and add shares from one of those 3 investments that fail to update and try to update prices?

If this is the case I already did, see attached. 

Just to clarify, these 3 investment (mutual funds) that do not update are not caused by a bug in KMM or my data file but because I dont have an online source at all to update them.....  Unless I misunderstood you, this is 100% normal I believe.

Created attachment 147028 [details]
kmy test file

Thanks, that sample kmy file is what I wanted.  I also found some more evidence of strange things in my own kmy file.
Yes, not updating a price if there is no source is fine - but inventing a price-pair that doesn't work and isn't what you entered is not ok.  We need to figure out why that happens, and why we can't remove some of those bogus price entries.  I suspect this may take a while.

Is there a way to run KMM in debug mode?  I tried running it from CLI to see if anything useful comes up, but other than a lot of verbose, nothing relevant ...

I'm not of much use in term of development but I can try a lot of twists and bends to try to see what can come up...

Maybe, you can provide some more information what we should see with the attached demo file. So far, I could successfully download price information for 0P000087HB.TO (Signature Global Income & Growth Fund Class A (CIG2111)) but not for the second one named  CIG7710.CF (Portfolio Series Balanced Fund (CIG7710)). The reason is, that the result is empty (symbol not found by Yahoo finance). So that one does not seem to be a problem of KMyMoney but of using the wrong symbol for whatever reason.

Git commit 31fa4f402441c364394d0a2b5d7ac46b5f944e42 by Thomas Baumgart.
Committed on 27/02/2022 at 17:48.
Pushed by tbaumgart into branch '5.1'.

Don't use invalid price pairs

For some reason price entries of the form "from currency to security"
are contained in the database (xml and sql). Such entries don't disturb
the price retrieval of KMyMoney as it can swap from and to if needed,
but they cause the online price update to be confused and failing.

The reason why these entries appear is yet unclear and needs to be
investigated further. To avoid these entries to cause problems during
online price update, they will not be listed in the price updater. This
change also removes those entries when writing the data file (or the
database for that matter) the next time. A respective debug message is
printed to the console.

This does not finally fix the problem reported but it cures the
symptoms.
FIXED-IN: 5.1.3

M  +4    -1    kmymoney/dialogs/kequitypriceupdatedlg.cpp
M  +11   -0    kmymoney/dialogs/kmymoneypricedlg.cpp
M  +5    -0    kmymoney/plugins/sql/mymoneystoragesql_p.h
M  +10   -3    kmymoney/plugins/xml/mymoneystoragexml.cpp

https://invent.kde.org/office/kmymoney/commit/31fa4f402441c364394d0a2b5d7ac46b5f944e42

Git commit 70d611778e96246d76d49f3981dd7eb9248a7779 by Thomas Baumgart.
Committed on 27/02/2022 at 18:16.
Pushed by tbaumgart into branch 'master'.

Don't use invalid price pairs

For some reason price entries of the form "from currency to security"
are contained in the database (xml and sql). Such entries don't disturb
the price retrieval of KMyMoney as it can swap from and to if needed,
but they cause the online price update to be confused and failing.

The reason why these entries appear is yet unclear and needs to be
investigated further. To avoid these entries to cause problems during
online price update, they will not be listed in the price updater. This
change also removes those entries when writing the data file (or the
database for that matter) the next time. A respective debug message is
printed to the console.

This does not finally fix the problem reported but it cures the
symptoms.

(cherry picked from commit 31fa4f402441c364394d0a2b5d7ac46b5f944e42)

M  +4    -1    kmymoney/dialogs/kequitypriceupdatedlg.cpp
M  +11   -0    kmymoney/dialogs/kmymoneypricedlg.cpp
M  +10   -0    kmymoney/plugins/sql/mymoneystoragesql_p.h
M  +16   -2    kmymoney/plugins/xml/mymoneystoragexml.cpp

https://invent.kde.org/office/kmymoney/commit/70d611778e96246d76d49f3981dd7eb9248a7779

Hello Thomas,

Sorry for not replying sooner I was away for a while.  To answer your questions:

1. Jack asked me to produce a test file with some of the mutual funds I own (CIG2321 & CIG7710).  I made it on purpose to have a fund that CAN be retrieved from Yahoo, the other, you are right, CANNOT be retrieved (not listed for whatever reason)....  I am trying to use Globe and Mail but the regexes are broken in KMM (currently trying to find the right regex but this is another topic)...

2. See attached screenshot.  I am sorry I thought it was clearer that each of the stock/equity that KMM cannot auto update from online sources had a pair of those "bogus price entries"...  Not being able to update the price from an online source seems to be the culprit for KMM creating these fake price pairs or whatever they're called.  I only have 3 funds that have no online sources and they are exactly those that KMM created a fake price pair for...

Created attachment 147189 [details]
prices

Quick question: for the commits and changes to the code (such as this one => 70d611778e96246d76d49f3981dd7eb9248a7779), do they usually make it right away to the AppImage ?  I saw the commit pushed to 5.1, then pushed to Master, but looking at the latest AppImage (2022-02-27) I dont see this commit implemented yet.

I am asking because I am "stuck" on 5.0.8 because the people upstream of Linux Mint have not updated the KMM package for many years now.  If need be I will uninstall 5.0.8 and run from AppImage from now on.  Is this the recommended way ?  I wouldnt want to report something that has already been debugged in a more recent release, this is why I always try to reproduce the bug with the latest AppImage before posting here...

70d611778e9 is the commit on master. AppImages are generated on a daily basis. All changes on the resp. branch that were commited until then are included. I am not sure what makes you think that it is not included. I pushed the changes yesterday around 7pm my time and the 5.1 AppImage 5.1-36 was build on 8:24 pm my time. So that is a gap of approx. 90 mins. The look at the console log of the build confirms that 8:24 was the start time, so the change should be included. The Help/About dialog of KMyMoney will show you the exact git commit that was used by the build. Same for master: commit around 7:20 pm and build started today 10 mins to 2pm (but fails since Feb 20th).

Using the stable AppImage is a good idea I think. Keep the downloads in case you have to go back at some point in time.

Let's see how the latest AppImage works for you.

I thought the commit was not included because I looked at the changes log and didn't see anything (the log was blank for some reasons).  You can ignore my reply about the changes.

I will use the latest AppImage from now on.

(In reply to Thomas Baumgart from comment #21)
> 70d611778e9 is the commit on master. AppImages are generated on a daily
> basis. All changes on the resp. branch that were commited until then are
> included. I am not sure what makes you think that it is not included. I
> pushed the changes yesterday around 7pm my time and the 5.1 AppImage 5.1-36
> was build on 8:24 pm my time. So that is a gap of approx. 90 mins. The look
> at the console log of the build confirms that 8:24 was the start time, so
> the change should be included. The Help/About dialog of KMyMoney will show
> you the exact git commit that was used by the build. Same for master: commit
> around 7:20 pm and build started today 10 mins to 2pm (but fails since Feb
> 20th).
> 
> Using the stable AppImage is a good idea I think. Keep the downloads in case
> you have to go back at some point in time.
> 
> Let's see how the latest AppImage works for you.

Thomas,  

kmymoney-5.1-36 AppImage doesnt ship with MySQL support?  The previous AppImages worked with MySQL connections.  This version gives me at startup:

WebConnect: Try to connect to WebConnect server
WebConnect: Connect to server failed
WebConnect: Running in server mode
Found html dir(s): ("/usr/share/kmymoney//html/")
Found an 'html' folder with CSS files: "/usr/share/kmymoney//html/"
System icon theme as reported by QT:  "Mint-Y"
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/budgetview.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/checkprinting.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/csvexporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/csvimporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/forecastview.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/gncimporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/icalendarexporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/kbanking.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/kcm_checkprinting.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/kcm_csvimporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/kcm_forecastview.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/kcm_icalendarexporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/kcm_qif.so" Validity false
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/kcm_reportsview.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/kcm_xmlstorage.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/konlinetasks_sepa.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/ofximporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/onlinejoboutboxview.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/qifexporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/qifimporter.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/reconciliationreport.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/reportsview.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/sqlstorage.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/woob.so" Validity true
Located plugin "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/xmlstorage.so" Validity true
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/budgetview.so"
Plugins: budgetview loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/checkprinting.so"
Plugins: checkprinting loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/csvexporter.so"
Plugins: csvexporter loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/csvimporter.so"
Plugins: csvimporter loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/forecastview.so"
Plugins: forecastview loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/reconciliationreport.so"
Plugins: reconciliation report loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/reportsview.so"
Plugins: reportsview loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/sqlstorage.so"
Plugins: sqlstorage loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/woob.so"
Python interpreter found: 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0]
Saved a copy of the embedded :/plugins/woob/kmymoneywoob.py script as "/tmp/kmymoneywicHYK.py"
Attempt to load the "kmymoneywicHYK" Python module from "/tmp"
The dependency 'sys' Python module loaded successfully
The 'path' Python variable loaded successfully
"kmymoneywicHYK" Python module failed to load
Traceback (most recent call last):
  File "/tmp/kmymoneywicHYK.py", line 9, in <module>
    from woob.core import Woob
ModuleNotFoundError: No module named 'woob'
kf.xmlgui: cannot find .rc file "woob.rc" for component "woob"
Plugins: woob loaded
Loading "/tmp/.mount_kmymonoYVQYU/usr/plugins/kmymoney/xmlstorage.so"
Plugins: xmlstorage loaded
Found html dir(s): ("/usr/share/kmymoney//html/")
Found an 'html' folder with CSS files: "/usr/share/kmymoney//html/"
QSqlDatabase: QMYSQL driver not loaded
QSqlDatabase: available drivers: QSQLITE QSQLITE3 QODBC QODBC3 QPSQL QPSQL7 QSQLCIPHER


then a popup saying

"Qt SQL driver QMYSQL is no longer installed on your system"

Please don't hijack a bug-entry for an unrelated issue next time. Thanks.

Here's the reason: https://invent.kde.org/packaging/craft-blueprints-kde/-/commit/188be44f6a3105d21392ad0da06a0a33fe0753cf Unfortunately no comment why :(

(In reply to Thomas Baumgart from comment #24)
> Please don't hijack a bug-entry for an unrelated issue next time. Thanks.
> 
> Here's the reason:
> https://invent.kde.org/packaging/craft-blueprints-kde/-/commit/
> 188be44f6a3105d21392ad0da06a0a33fe0753cf Unfortunately no comment why :(

This bugtracker thingy doesnt let users delete comments, so I cannot move to a new ticket.  Anyways, if I understand well, the MYSQL issue is NOT related to KMM itself but to one of its upstream components?  I agree lets keep this discussion on track with the original topic.

ON appimage Version 5.1.2-78fde521b the issue is resolved.  Since I also have no investments without a reliable online source for price update, I believe the issue is resolved (not sure if its the newer appimage or the fact that all investments now have sources)....