Bug 386462 - Yahoo Quote Updates quit working
Summary: Yahoo Quote Updates quit working
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Applications
Component: general (show other bugs)
Version: 4.8.1
Platform: Microsoft Windows Microsoft Windows
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
: 386465 386495 386505 386524 386617 388541 (view as bug list)
Depends on:
Blocks:
 
Reported: 2017-11-02 14:37 UTC by akoolwine
Modified: 2019-09-02 22:14 UTC (History)
18 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.8.2
ralf.habacker: Version_5+


Attachments
Quote Error Message (164.14 KB, image/jpeg)
2017-11-02 14:37 UTC, akoolwine
Details
attachment-1563-0.html (2.18 KB, text/html)
2017-11-03 01:34 UTC, Brendan
Details
example of a csv containing prices from alpha vantage (4.88 KB, text/csv)
2017-11-03 08:12 UTC, Robby Engelmann
Details
attachment-19597-0.html (1.43 KB, text/html)
2017-11-15 20:34 UTC, bilatino
Details
Patched as per comment #29 Yahoo Currency settings (43.26 KB, image/png)
2017-11-15 21:12 UTC, Pete
Details

Note You need to log in before you can comment on or make changes to this bug.
Description akoolwine 2017-11-02 14:37:14 UTC
Created attachment 108680 [details]
Quote Error Message

Quote updates using Yahoo have quit working
Comment 1 Robby Engelmann 2017-11-02 15:12:27 UTC
The same here using the KF5-based master version under openSUSE Tumbleweed. Yahoo is for me (ETFs) the only one that ever work reliably.
Comment 2 Robby Engelmann 2017-11-02 15:14:05 UTC
Did Yahoo change its API?
Comment 3 akoolwine 2017-11-02 21:58:44 UTC
New error message:

Fetching URL http://finance.yahoo.com/d/quotes.csv?s=ATD-B.TO&f=sl1d1...
Access denied to http://download.finance.yahoo.com/d/quotes.csv?s=ATD-B.TO&f=sl1d1.
Unable to update price for ATD-B.TO (empty quote data)

And from Yahoo:
It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com.
Comment 4 Pete 2017-11-02 23:23:16 UTC
For a bit more info and screenshots, please see:

https://bugs.launchpad.net/ubuntu/+source/kmymoney/+bug/1729709
Comment 5 Jack 2017-11-02 23:49:34 UTC
There is nothing KMM can do if Yahoo has decided to no longer provide such information in a useful way.  The same will happen with any version of KMM.  If someone can identify an alternative URL at Yahoo that will work, it can be included in KMM, although it seems unlikely that will happen before 5.0.  It may be necessary to choose one of the alternate sources for such information.
Comment 6 Brendan 2017-11-03 01:34:01 UTC
Created attachment 108690 [details]
attachment-1563-0.html

Keep in mind this is really Verizon. Surprised it took this long.



*----Brendan Coupe*

On Thu, Nov 2, 2017 at 5:49 PM, Jack <bugzilla_noreply@kde.org> wrote:

> https://bugs.kde.org/show_bug.cgi?id=386462
>
> Jack <ostroffjh@users.sourceforge.net> changed:
>
>            What    |Removed                     |Added
> ------------------------------------------------------------
> ----------------
>                  CC|
> |ostroffjh@users.sourceforge
>                    |                            |.net
>
> --- Comment #5 from Jack <ostroffjh@users.sourceforge.net> ---
> There is nothing KMM can do if Yahoo has decided to no longer provide such
> information in a useful way.  The same will happen with any version of
> KMM.  If
> someone can identify an alternative URL at Yahoo that will work, it can be
> included in KMM, although it seems unlikely that will happen before 5.0.
> It
> may be necessary to choose one of the alternate sources for such
> information.
>
> --
> You are receiving this mail because:
> You are the assignee for the bug.
>
Comment 7 Thomas Baumgart 2017-11-03 07:22:46 UTC
*** Bug 386465 has been marked as a duplicate of this bug. ***
Comment 8 Ralf Habacker 2017-11-03 07:53:23 UTC
If there is an updated url available please append it to this bug. People using the mentioned version can update to the new url in their local installation and it could be included in version 4.8.2 and git master.
Comment 9 Robby Engelmann 2017-11-03 08:11:32 UTC
Is there a good how-to to generate an own online quote entry? One could use this: https://www.alphavantage.co/documentation/
to get an csv of prices with a free api key.

Via an URL (https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=IUS7.DE&outputsize=compact&apikey=YOUR_FREE_API_KEY&datatype=csv) you get the csv which has a specific form (example attached). This is to me a quite reliable resource of historical prices in order to perform data analysis with R.
Comment 10 Robby Engelmann 2017-11-03 08:12:16 UTC
Created attachment 108692 [details]
example of a csv containing prices from alpha vantage
Comment 11 Robby Engelmann 2017-11-03 08:17:55 UTC
Is there a possibility to change the online quote resource for the currencies away from yahoo currency?
Comment 12 Thomas Baumgart 2017-11-03 14:40:12 UTC
*** Bug 386495 has been marked as a duplicate of this bug. ***
Comment 13 Bob Ewart 2017-11-03 16:45:29 UTC
Alpha Vantage works for stock prices and gives a lot more data than needed (100 data points.)

It does not appear to work for mutual funds.
Comment 14 Robby Engelmann 2017-11-03 18:50:20 UTC
Thats right, there is more data than needed. I only tested it for ETF's and that worked up to now.
Comment 15 Thomas Baumgart 2017-11-04 07:20:55 UTC
Please see https://forum.kde.org/viewtopic.php?f=69&t=142715&p=383426#p383426 for a possible workaround.
Comment 16 Joe Byers 2017-11-04 13:58:26 UTC
All, 
This was an issue with R quantmod.  They found a fix.  If you follow the link for details: https://github.com/joshuaulrich/quantmod/issues/157

other comments are here: 
https://stackoverflow.com/questions/44015838/cant-download-data-from-yahoo-finance-using-quantmod-in-r

Josh had to encode a cookie crumb to the URL 

i hope helps

Joe
Comment 17 Thomas Baumgart 2017-11-05 08:50:17 UTC
*** Bug 386524 has been marked as a duplicate of this bug. ***
Comment 18 Thomas Baumgart 2017-11-05 09:14:17 UTC
*** Bug 386505 has been marked as a duplicate of this bug. ***
Comment 19 shizuma 2017-11-05 21:47:11 UTC
I update BTC, XAU and XAG (bitcoins, gold and silver) daily and since yahoo stopped working, I got an API key from https://www.alphavantage.co as suggested by Robby Engelmann

I hardcoded CAD as currency of choice but I suppose you can put your own or use %2

Here are the settings to create a new online quote:

URL: https://www.alphavantage.co/query?function=CURRENCY_EXCHANGE_RATE&from_currency=%1&to_currency=CAD&apikey=your_api_key_goes_here

Symbol: From_Currency\sCode.*(\w+)

Price: Rate.*(\d+\.\d+)

Date: Refreshed.*(\d{4,4}-\d{2,2}-\d{2,2})

Date format: %y-%m-%d

Hope that helps.

Tested and everything's fine.
Comment 20 Pete 2017-11-06 22:13:53 UTC
(In reply to Jack from comment #5)
> There is nothing KMM can do if Yahoo has decided to no longer provide such
> information in a useful way.  The same will happen with any version of KMM. 
> If someone can identify an alternative URL at Yahoo that will work, it can
> be included in KMM, although it seems unlikely that will happen before 5.0. 
> It may be necessary to choose one of the alternate sources for such
> information.

The following URL appears to have all the info required, if you can parse the page.  It is from Microsoft:

Power Financial (PWF) from the Toronto Stock Exchange (TSE)
https://www.msn.com/en-ca/money/stockdetails/fi-127.1.PWF.TSE?symbol=PWF&form=PRFIEQ
Note the &form parameter can be dispensed with.

Amazon (AMZN) from the (NAS)daq stock exchange
https://www.msn.com/en-ca/money/stockdetails/fi-126.1.AMZN.NAS?symbol=AMZN&form=PRFIEQ
Note the &form parameter can be dispensed with.

International Business Machines Corp (IBM) from (NYS)E (ie: New York)
https://www.msn.com/en-ca/money/stockdetails/fi-126.1.IBM.NYS?symbol=IBM&form=PRFIEQ
Note the &form parameter can be dispensed with.

Rio Tinto PLC (RIO) on the (LON)don stock exchange
https://www.msn.com/en-ca/money/stockdetails/fi-151.1.RIO.LON?symbol=RIO&form=PRFIEQ
Note the &form parameter can be dispensed with.

BNP Parisbas (BNP) on the (PAR)is stock exchange
https://www.msn.com/en-ca/money/stockdetails/fi-160.1.BNP.PAR?symbol=BNP&form=PRFIEQ
Note the &form parameter can be dispensed with.

The quote search box to the right of the company name can be used to find your desired stock.

en-ca probably determined from my IP address in (ca)nada.

Hope this helps.
Comment 21 Ralf Habacker 2017-11-06 23:48:43 UTC
(In reply to Thomas Baumgart from comment #15)
> Please see
> https://forum.kde.org/viewtopic.php?f=69&t=142715&p=383426#p383426 for a
> possible workaround.
I'm going to add a fix using this workaround under source name 'Google Finance' and 'Yahoo Currency' as alias to support the online price update dialog if there are no objections.
Comment 22 Thomas Baumgart 2017-11-08 07:42:44 UTC
*** Bug 386617 has been marked as a duplicate of this bug. ***
Comment 23 Pete 2017-11-08 17:14:38 UTC
Yahoo quotes stopped working in the last week or so.

Globe & Mail and MSN.CA have never worked as feeds even though they provide quotes at different URL addresses.

Globe & Mail have quotes without the year part of the date:

https://www.theglobeandmail.com//globe-investor/markets/stocks/summary/?q=IBM-N

See comment 20 for MSN quotes.
Comment 24 Martin Schröder 2017-11-08 18:37:41 UTC
Just curious: Why is the bug's state still "UNCONFIRMED"?
Comment 25 Peter 2017-11-08 19:24:57 UTC
Chart Geany is using IEX as feed for their charting program.

The Site is at this address: https://iextrading.com/apps/stocks/#/

Contentwise is has the same stocks as Yahoo.
Could that be an alternative?
Comment 26 Jack 2017-11-08 19:41:23 UTC
(In reply to Martin Schröder from comment #24)
Martin - Don't pay too much attention to the state.  It simply means nobody has bothered to change it to anything else - it's a manual process.
Comment 27 Ralf Habacker 2017-11-14 15:57:34 UTC
Git commit 53550fbf3b0a85565c2ff47e38783115bf051357 by Ralf Habacker.
Committed on 14/11/2017 at 07:02.
Pushed by habacker into branch '4.8'.

Remove Yahoo quote sources

Removed the references to Yahoo quote sources so that they will not be
created anymore automatically.
Renamed the standard currency exchange rate source to 'KMyMoney
Currency' and assigned it to fx-rate.net until this is user selectable.

Patch recreated from commit 8559f9ec3a56c5c1526af79cd60f0c67966137fd
FIXED-IN:4.8.2
Differential Revision: https://phabricator.kde.org/D8715

M  +10   -44   kmymoney/converter/webpricequote.cpp
M  +1    -1    kmymoney/dialogs/kequitypriceupdatedlg.cpp

https://commits.kde.org/kmymoney/53550fbf3b0a85565c2ff47e38783115bf051357
Comment 28 Gorden 2017-11-14 23:10:50 UTC
I assume Comment 27 applies to program running under Ubuntu. Is there a similar fix for program running under windows?
Comment 29 Ralf Habacker 2017-11-15 04:20:09 UTC
(In reply to Gorden from comment #28)
> I assume Comment 27 applies to program running under Ubuntu. Is there a
> similar fix for program running under windows?

Yes, it will be in the next binary release or snapshot. Portable snapshots are available at https://software.opensuse.org/package/mingw64-kmymoney-portable?search_term=mingw64-kmymoney-portable. I triggered a rebuild update, which should be available soon - look for version 4.8.c829ec79.

In an installed kmymoney it could be changed with:  

1. Start KMyMoney
2. In "online price entry" tab of settings dialog change "Yahoo Currency" to: 

url: https://fx-rate.net/%1/%2
price: 1\s[^=]+\s=</span><br\s/>\s([^\s]+)
date: updated\s\d+:\d+:\d+\(\w+\)\s+(\d{1,2}/\d{2}/\d{4})
dateformat: %d/%m/%y
Comment 30 bilatino 2017-11-15 20:34:30 UTC
Created attachment 108887 [details]
attachment-19597-0.html

> 
>     Il 15 novembre 2017 alle 5.20 Ralf Habacker <bugzilla_noreply@kde.org> ha scritto:
> 
>     https://bugs.kde.org/show_bug.cgi?id=386462
> 
>     --- Comment #29 from Ralf Habacker <ralf.habacker@freenet.de> ---
>     (In reply to Gorden from comment #28)
> 
>         > > 
> >         I assume Comment 27 applies to program running under Ubuntu. Is there a
> >         similar fix for program running under windows?
> > 
> >     > 
>     Yes, it will be in the next binary release or snapshot. Portable snapshots are
>     available at
>     https://software.opensuse.org/package/mingw64-kmymoney-portable?search_term=mingw64-kmymoney-portable.
>     I triggered a rebuild update, which should be available soon - look for version
>     4.8.c829ec79.
> 
>     In an installed kmymoney it could be changed with:
> 
>        1. Start KMyMoney
>        2. In "online price entry" tab of settings dialog change "Yahoo Currency" to:
> 
>     url:
>     price: 1\s[^=]+\s=</span><br\s/>\s([^\s]+)
>     date: updated\s\d+:\d+:\d+\(\w+\)\s+(\d{1,2}/\d{2}/\d{4})
>     dateformat: %d/%m/%y
> 
>     --
>     You are receiving this mail because:
>     You are on the CC list for the bug.
>
Comment 31 Pete 2017-11-15 20:46:53 UTC
(In reply to Ralf Habacker from comment #27)
> Git commit 53550fbf3b0a85565c2ff47e38783115bf051357 by Ralf Habacker.
> Committed on 14/11/2017 at 07:02.
> Pushed by habacker into branch '4.8'.
> 
> Remove Yahoo quote sources
> 
> Removed the references to Yahoo quote sources so that they will not be
> created anymore automatically.
> Renamed the standard currency exchange rate source to 'KMyMoney
> Currency' and assigned it to fx-rate.net until this is user selectable.
> 
> Patch recreated from commit 8559f9ec3a56c5c1526af79cd60f0c67966137fd
> FIXED-IN:4.8.2
> Differential Revision: https://phabricator.kde.org/D8715
> 
> M  +10   -44   kmymoney/converter/webpricequote.cpp
> M  +1    -1    kmymoney/dialogs/kequitypriceupdatedlg.cpp
> 
> https://commits.kde.org/kmymoney/53550fbf3b0a85565c2ff47e38783115bf051357

By FIXED RESOLVED 4.8.2, do you mean the stock quote updates are also fixed over and above currency quotes?

Please clarify.  Thanks.
Comment 32 Pete 2017-11-15 21:12:38 UTC
Created attachment 108888 [details]
Patched as per comment #29 Yahoo Currency settings

Previous quote of November 10 2017 was manually updated.
Comment 33 Ralf Habacker 2017-11-18 10:22:46 UTC
(In reply to Ralf Habacker from comment #29)
> 1. Start KMyMoney
> 2. In "online price entry" tab of settings dialog change "Yahoo Currency"
> to: 
> 
> url: https://fx-rate.net/%1/%2
> price: 1\s[^=]+\s=</span><br\s/>\s([^\s]+)
> date: updated\s\d+:\d+:\d+\(\w+\)\s+(\d{1,2}/\d{2}/\d{4})
> dateformat: %d/%m/%y
skip html stripping need to be checked too
Comment 34 Ralf Habacker 2017-11-18 10:23:28 UTC
(In reply to Pete from comment #32)
> Created attachment 108888 [details]
> Patched as per comment #29 Yahoo Currency settings
> 
> Previous quote of November 10 2017 was manually updated.

see comment 33
Comment 35 Ralf Habacker 2017-11-18 10:25:19 UTC
> By FIXED RESOLVED 4.8.2, do you mean the stock quote updates are also fixed
> over and above currency quotes?
yes
Comment 36 Ralf Habacker 2018-01-04 21:01:47 UTC
*** Bug 388541 has been marked as a duplicate of this bug. ***
Comment 37 Ralf Habacker 2018-01-09 00:25:05 UTC
Git commit 1b615d6bc92e9cc808cd021f069b60f61b26e4ea by Ralf Habacker.
Committed on 09/01/2018 at 00:24.
Pushed by habacker into branch '4.8'.

Fixup of commit 53550fbf

Fixing the default online price source for ofx import is also required.
Related: bug 388698
CCMAIL:tbaumgart@kde.org

M  +1    -1    kmymoney/converter/mymoneystatementreader.cpp

https://commits.kde.org/kmymoney/1b615d6bc92e9cc808cd021f069b60f61b26e4ea
Comment 38 lp.allard.1 2018-02-03 23:50:57 UTC
4.7.2 here on Mint (I wish I could yupdate but the KMM website doesnt provide packages and the only Mint supported PPA (ppa:claydoh/kmymoney2-kde4) ceased to update in 2016 (no longer maintained?).

I have tried changing the settings of Yahoo Currency in the Settings to no avail, nothing works. Using the settings provided here by Ralf Habacker:

url: https://fx-rate.net/%1/%2
price: 1\s[^=]+\s=</span><br\s/>\s([^\s]+)
date: updated\s\d+:\d+:\d+\(\w+\)\s+(\d{1,2}/\d{2}/\d{4})
dateformat: %d/%m/%y

"Unable to update price for USD > CAD (no price or no date)"
Comment 39 Ralf Habacker 2018-02-10 22:50:46 UTC
The price regex seems to be changed in the meantime and needs to be changed to =</span><br */> *(\d+\.\d+)
Comment 40 Ralf Habacker 2018-02-11 16:47:11 UTC
Git commit 92d62efff36875a3dff5bb548ef6d1179858e101 by Ralf Habacker.
Committed on 11/02/2018 at 16:46.
Pushed by habacker into branch '4.8'.

Fix price regex for KMyMoney currency
FIXED-IN:4.8.2

M  +1    -1    kmymoney/converter/webpricequote.cpp

https://commits.kde.org/kmymoney/92d62efff36875a3dff5bb548ef6d1179858e101
Comment 41 Gorden 2018-02-12 02:44:15 UTC
Will KMyMoney 5.0 run under Windows 10.
Comment 42 Ralf Habacker 2018-02-23 08:05:12 UTC
Until 4.8.2 will be out there are snapshots for Windows available at 
https://software.opensuse.org/package/mingw32-kmymoney-portable
https://software.opensuse.org/package/mingw64-kmymoney-portable
https://software.opensuse.org/package/mingw32-kmymoney-installer
https://software.opensuse.org/package/mingw64-kmymoney-installer

Please open the rpmfile with 7zip and you will get the portable package or the installer
Comment 43 Ralf Habacker 2018-02-23 08:06:29 UTC
(In reply to Ralf Habacker from comment #42)
> Until 4.8.2 will be out there are snapshots for Windows

The snapshots are the packages with version 4.8.<hash>
Comment 44 Gorden 2018-06-14 02:27:07 UTC
Just updated to 4.8.2.  On line price updates for stocks does not work. Is there an additional step I need to take.
Comment 45 Ralf Habacker 2018-06-14 14:06:05 UTC
You did not wrote which online price service you are using: In case you are refering to the Yahoo service - it was shutted down some time ago.

After adding online quote check support to the online quotes settings dialog it turned out that currently only 1 from 18 services are working. The only service, which works out of the box, is 'KMymoney Currency'.
Comment 46 Ralf Habacker 2018-06-14 14:19:27 UTC
(In reply to Gorden from comment #41)
> Will KMyMoney 5.0 run under Windows 10.
You need to ask the KF5 development team.
Comment 47 Jack 2018-06-14 14:22:15 UTC
As has been said in many mailing list messages and on the forum, a version of KMM 5.0 has not yet been released for Windows.  There are some issues with creating the standalone installer which still need to be resolved.  Several people are working on this.  As soon as it is available, it will be announced on https://www.kmymoney.or and in the mailing lists.
Comment 48 Ralf Habacker 2018-06-15 10:59:36 UTC
Git commit e141d43c3c8b4d11ed7a8310f843d46002446fe8 by Ralf Habacker.
Committed on 14/06/2018 at 14:23.
Pushed by habacker into branch '4.8'.

Fix default online price source

In case an online quote source does not have a name, the unsupported
source name 'Yahoo' was used.

M  +1    -1    kmymoney/converter/webpricequote.cpp

https://commits.kde.org/kmymoney/e141d43c3c8b4d11ed7a8310f843d46002446fe8