SUMMARY When I update stock/mutual fund prices using the online function within KMM it is taking ~50 seconds per each individual ID. The process is now taking about 20 minutes to complete which was previously about a 20-second process. The problem appeared to start when I upgraded my OS to Fedora 30 from 29. I'm also having a problem with mapping accounts that I feel is related to this issue. It is taking so long that no accounts are returned to select. Preventing the ability to map accounts. STEPS TO REPRODUCE 1. Use Fedora 30 OS 2. Choose the Tools → Update Stock and Currency Prices 3. Press Update All OBSERVED RESULT Stock price updating taking ~50 seconds per stock/fund EXPECTED RESULT I have observed previously the ability to update ~20 stocks/funds to update in less than 30 seconds SOFTWARE/OS VERSIONS kernel/Linux OS/Gnome: 5.0.13-300/Fedora 30/Gnome Version 3.32.2/ 5.0.13-300.fc30.x86_64 ADDITIONAL INFORMATION I'm not sure if this is an OS or KMM problem but what appreciate some guidance to what it might be and how to resolve the issue. I've also tested the issue in KMM AppImage 5.0.4-5824844-x86_64 & 5.0.80-67ad34c-x86_64 and the problem is there as well.
Most important: which online source are you using? We need to figure out where in the process the delay happens: is there a network delay actually getting a response from the source URL, or is it somehow taking a long time for KMM to process that result? If you run from command line, is there any output that might be interesting? I'd suggest trying a different source, but I've had enough problems getting any one source to work for me, I wouldn't want to have to find another. Do note testing might be easier if you request update of one price at a time, so it's not 20 minutes per test.
Do you have a lame DNS server in the loop? Just an idea, as this is not reproducible here. Tracing the network traffic with tcpdump/wireshark might also give some clue why it takes so long and where to look.
To expand on Thomas' question, you could try ping or traceroute to the source server, or try curl or wget to actually retrieve the full URL. Has anything else about your network connection changed recently?
Not sure it will help but I've had the same experience since upgrading to KMM5. I'm always running a fairly recent version compiled from either the 5.0 branch or the master branch and I've tested all of the released versions. I download my quotes from Yahoo. The strange part is that every once in a while after I installed a new version, stock quotes got back to the way they used to be. Now it takes 3 to 5 seconds per quote and when it's fast I get about 20 quotes in 2 to 3 seconds. The difference is huge. Unfortunately I have not run a fast version in at least 3 or 4 months so if it was something different in KMM it's not there any longer. I never figured out if it was just one of the branches or both that would get fast but I don't think it was a Yahoo problem because it would be fast on one computer and slow on the other. I pinged the Yahoo URL today and the response was fast. I'm using 1.1.1.1 as my DNS server but slow DNS should only affect the first quote, after that DNS should not have any affect. As I recall, the quotes would stay fast until I installed a newer version. All of this is from memory, I wish I had taken the time to keep track of the conditions when it was fast versus when it was slow.
When I updated stocks from Yahoo tonight the update was extremely fast. I tried it from both computers and they were both fast. Both are running KMM from the 5.0 branch from about 2 weeks ago and both were slow until tonight. Since nothing changed in KMM this appears to be a Yahoo (Verizon) problem.
Thanks for your input Brendan but my situation is something more than a Verizon/Yahoo issue. I'm still dealing with the 49 seconds per individual price update with Yahoo before it returns a price according to Wireshark. I tried the other online stock quote services and none of them work for me as I gave up after 120 seconds. I'll update the content here when and if I find the problem.
Michael - if Wireshark is saying that it's almost a minute before you get the response from Yahoo, then it really seems like a network related issue, although I know that's still broad target. How is just aiming a browser at Yahoo? Do ping and traceroute show any problems? Have you tried resetting your modem and/or router?
Jack - If I aim my browser at Yahoo, ie https://finance.yahoo.com/v7/finance/quote?fields=regularMarketPrice&symbols=AXP I get what I would define as an immediate response. When I do a traceroute in the terminal to finance.yahoo.com I see a stall right after the 5th gateway. I didn't actually measure the time but would guess around 10 seconds. Resetting the router resulted in no change in results.
Created attachment 120149 [details] KMM Yahoo Settings
What settings do you have for Yahoo? This changed so many times in the past or two that I'm not sure where this came from. My URL in KMM is: https://query1.finance.yahoo.com/v7/finance/quote?fields=regularMarketPrice&symbols=%1 The rest you can see int the attachment.
Yahoo quotes are still very fast for me today.
Brendan -- My URL was, https://finance.yahoo.com/v7/finance/quote?fields=regularMarketPrice&symbols=%1 I changed it to your https://query1.finance.yahoo.com/v7/finance/quote?fields=regularMarketPrice&symbols=%1 and it still is slow. If I use either URL in my web browser with a ticker symbol I get a very quick similar response.
Created attachment 120296 [details] attachment-13349-0.html Yahoo quotes have been slow all week. I have not changed anything on either computer. ---- Brendan Coupe On Fri, May 17, 2019 at 9:33 PM Michael Carpino <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=407494 > > --- Comment #12 from Michael Carpino <mfcarpino@gmail.com> --- > Brendan -- My URL was, > > https://finance.yahoo.com/v7/finance/quote?fields=regularMarketPrice&symbols=%1 > > I changed it to your > > https://query1.finance.yahoo.com/v7/finance/quote?fields=regularMarketPrice&symbols=%1 > and it still is slow. > > If I use either URL in my web browser with a ticker symbol I get a very > quick > similar response. > > -- > You are receiving this mail because: > You are the assignee for the bug.
What I can share at this point is based on the information I've been able to obtain thru the use of Wireshark and a Wireshark expert (Chris Greer) that's been willing to assist me with my issue. The problem for me seems to start in the communication from the client (my Computer). It sends out information telling the server that it supports the different version of Transport Layer Security (TLS). My computer (Fedora OS) tells Yahoo that it has available versions, 1.0, 1.1, 1.2 and 1.3. Yahoo comes back and tells my computer that it wants to use version 1.3. All of this happens extremely fast within the initial communication between them. It's less than a second and it's actually a small portion of a second. Here's where we think the problem is. Within that same (Hello) exchange when it tells my computer it wants to use version 1.3 it later says in the same message to change the cipher spec protocol to v 1.2 and the application data protocol to v 1.2. Now keep in mind this is all happening within a portion of a second. Then for some unknown reason my computer stalls (I believe within the OS) and it this point we think it's trying to figure out what to do with the conflicting information. Then at around 48-49 seconds later it sends what it needs back to Yahoo and boom I get the updated stock price. That's where things are right now as far as what we know about my issue.
What version of gnutls are you using? (or of whatever library you have providing the TLS stuff.) Can you upgrade/downgrade it? I have 3.6.7 (under Gentoo) but no other version to up/downgrade to.
Michael & Chris: impressive analysis. This is down on the TLS packets level and far away from any KMyMoney influence. Besides TLS being in between, there's also the KIO layer which we make use of here. This is the source code within KMyMoney that triggers the operation: const QUrl dest = QUrl::fromLocalFile(tmpFile); KIO::Scheduler::checkSlaveOnHold(true); KIO::Job *job = KIO::file_copy(url, dest, -1, KIO::HideProgressInfo); connect(job, SIGNAL(result(KJob*)), this, SLOT(downloadResult(KJob*))); The variable url contains the full URL of the price quote. Once the data is present, downloadResult is called and that seems to work as the price info is loaded into KMyMoney.
Jack - Version gnutls-3.6.7-1.fc30.x86_64 Thomas - Thanks! I found Chris on YouTube and he knows his stuff around Wireshark and has educated me a lot as this is a speaking "French" to me. I watched some of his videos as they a full of a wealth of information on Wireshark and the problems in the network. Thanks for sharing the source code as I'll pass it on to Chris so he can see it. Now, where is the problem? I'm just trying to narrow things down more so when I go back to the developers at Fedora I can point them in the right direction. It's bothersome and not a show stopper. I just want to resolve the issue.
Created attachment 120309 [details] attachment-25590-0.html What version of Fedora are you running? I'm running 29, my test system is on 30 but kmm decryption is failing (again - just like 29) so I have not tried kmm on it yet. --- Brendan On Sat, May 25, 2019, 9:06 AM Michael Carpino <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=407494 > > --- Comment #17 from Michael Carpino <mfcarpino@gmail.com> --- > Jack - Version gnutls-3.6.7-1.fc30.x86_64 > > Thomas - Thanks! I found Chris on YouTube and he knows his stuff around > Wireshark and has educated me a lot as this is a speaking "French" to me. I > watched some of his videos as they a full of a wealth of information on > Wireshark and the problems in the network. Thanks for sharing the source > code > as I'll pass it on to Chris so he can see it. > > Now, where is the problem? I'm just trying to narrow things down more so > when > I go back to the developers at Fedora I can point them in the right > direction. > > It's bothersome and not a show stopper. I just want to resolve the issue. > > -- > You are receiving this mail because: > You are the assignee for the bug.
Brendan - I'm using Fedora version 30. I'd also recommend that you try Wireshark for your problem. Turn off all other devices on your network, close all other programs on your computer, run wireshark and start your price updating within KMM. You'll see where your delay is.
FWIW: I am on the same gnutls version here on opensuse Leap 15 libgnutls30-3.6.7-lp150.9.1.x86_64 I yet have to trace, if any of my price service connections uses TLS 1.3. Maybe, this is also related to the Qt version. Which part of the software stack (from top to bottom (KMyMoney), KF5 (esp. KIO), Qt, GnuTLS, Kernel) is controlling any of this stuff. I put KMyMoney in parens as it is certainly not involved and does not play with any of the parameters.
I have tested downloading stock prices using other Linux OS which included OpenSuse and Ubuntu. Both of these OS performed the updating of stock prices within KMM as it should. I looked at the trace and it is performed with the use of TLS v2. Because this is not a KMM bug or problem I felt that it to be appropriate to close this issue. This appears to be specifcally within how Fedora handles the connection of TLS v3. I expect someday in the future Fedora/Redhat will provide a fix to efficient and proper handling of v3.