Forums  > Software  > Yahoo Finance  
     
Page 1 of 1
Display using:  

EspressoLover


Total Posts: 221
Joined: Jan 2015
 
Posted: 2017-05-17 21:36
Anyone know what's up with their historical data CSV page? The original URL (icharts.yahoo.com) has been unresponsive this entire week. It's seemed to move to a new format (query1.finance.yahoo.com). However this now requires a GET param cookie (?...&crumb=[key]). Can't figure out how to get this working in curl. So...

1) Does anyone know if the icharts.yahoo.com is permanently discontinued? The page says "undergoing maintenance", but there's no other announcements from yahoo.

2) Anyone figure out a hack for the crumb= parameter to get it working outside the browser?

3) Any other alternatives for free historical interday data, which gets updated daily? I'd get a running CSI subscription, but I'm a cheap bastard.

xig


Total Posts: 2
Joined: May 2017
 
Posted: 2017-05-17 22:03
It doesn't seem too bad. Just did a quick test, seems like you just need to first scrape the crumb (with cookies on) then grab the data. For example (F#).


open System
open System.Net
open System.Text.RegularExpressions

type CookieAwareWebClient() =
inherit WebClient()
let cookies = new CookieContainer()
member x.Cookies = cookies
override x.GetWebRequest(uri:Uri) =
let req = base.GetWebRequest(uri)
let webReq = req :?> HttpWebRequest
if webReq null then
webReq.CookieContainer <- cookies
req

let wc = new CookieAwareWebClient()
let str = wc.DownloadString("https://ca.finance.yahoo.com/quote/^GSPC/history?p=^GSPC")

let m = Regex.Matches(str,"\"user\":{\"crumb\":\"(.*?)\"") |> Seq.cast |> Seq.head
let crumb = m.Groups.[1].Value

let csv = wc.DownloadString("https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1492458189&period2=1495050189&interval=1d&events=history&crumb=" + crumb)

xig


Total Posts: 2
Joined: May 2017
 
Posted: 2017-05-17 22:07
Looks like my pre tags got stripped despite the preview looking fine. Attached to preserve indenting.

Attached File: yahootest.zip

EspressoLover


Total Posts: 221
Joined: Jan 2015
 
Posted: 2017-05-17 23:46
Thanks! That worked for me.

Here's a shell script version in case anyone's interested

Yahoo Finance Bash Script

Edit Addendum: Heads up, the new CSV sheet appears to have reversed the time order from the previous one. Also some symbols now include rows with null values if you go back far enough. So make sure your parser handles these conditions.

kloc


Total Posts: 1
Joined: May 2017
 
Posted: 2017-05-18 19:21
Many thanks for this; just run into this in the morning myself. A minimal example for "pythonistas" attached below:

Attached File: snapy.zip

Basically, raquests.post(...) works like a charm here...

contango_and_cash


Total Posts: 66
Joined: Sep 2015
 
Posted: 2017-05-19 17:26
Saw this and decided to just start paying for some equity data.

I immediately switched to google (pandas wrapper) as a temporary fix.

Patrik
Founding Member

Total Posts: 1333
Joined: Mar 2004
 
Posted: 2017-05-21 22:57
Just in case people didn't remember to think about it - YFinance and Google Finance produce their equity data using different methodologies with regards to split adjustments, dividend adjustments etc. If it matters to you make sure you get it right :)

Capital Structure Demolition LLC Radiation

yanko


Total Posts: 59
Joined: Nov 2009
 
Posted: 2017-05-22 13:43
Hi,

the best way to get data from Yahoo is to use quantmod on R, the new URL has already been accounted for (see https://github.com/joshuaulrich/quantmod/issues/157).

However, as Patrik pointed out, the split/dividend adjustments are out of whack since the url change (see discussion in the above github issue, as well as https://github.com/joshuaulrich/quantmod/issues/160).

Also - Yahoo has seed it "stopped supporting" the API (see post by Yahoo-moderator Nixon here https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503/page/3). whatever they mean by that, since it has always been an unofficial "service/hack". Yahoo adding a cookie hash (the crum parameter, see github) does indicate that they would like to reduce API traffic by wagging off more inexperienced or less motivated users.

So, it's still working, but it's now even less reliable.

yanko

finanzmaster


Total Posts: 107
Joined: Feb 2011
 
Posted: 2017-05-25 23:30
YaWhore Dance with Yahoo Finance

We discuss the ways to circumvent the API change of Yahoo.Finance and alternatives to it, esp. Alpha Vantage.

www.yetanotherquant.com - Knowledge rather than Hope: A Book for Retail Investors and Mathematical Finance Students

BBands


Total Posts: 1
Joined: May 2017
 
Posted: 2017-05-26 01:49
Thanks Kloc,

I thought that you had to have a crumb with the new api, but that works perfectly.

Best,

John

ChadZ


Total Posts: 2
Joined: May 2017
 
Posted: 2017-05-28 00:48
Patrik, could you elaborate?

I'd like to make sure I'm not missing something because I did not see a big difference between Yahoo and Google's adjusted prices. Yahoo of course provides unadjusted prices while Google does not, but when comparing Yahoo's adjusted prices to Google's, it seemed to me that both adjusted for splits and not dividends, although the dividend amounts themselves were split adjusted for both. My memory is fuzzy regarding Yahoo's volume data, but I vaguely recall that from icharts at least it was not corrected for splits while Google's is. I think that was the extent of the differences I noticed, or am I missing something?

eeng


Total Posts: 15
Joined: Dec 2014
 
Posted: 2017-05-29 21:54
A few hours ago quantmod package on CRAN was updated to adapt to the new Yahoo service, seems to work OK but I got some missing values warnings:

https://cran.r-project.org/web/packages/quantmod/index.html

The fix for python-datareader seems to be stuck because of some failing tests:

https://github.com/pydata/pandas-datareader/pull/331

Now looking further, I can only see the Yahoo service going away, they didn't sell the company yet and they are already changing the API and removing dividend adjustments, I fully expect it to go black when they finally sell the company.

I tried using Google Finance but can see many missing symbols that were present on yahoo. I am thinking of trying Quandl, anyone has any color on it?

Patrik
Founding Member

Total Posts: 1333
Joined: Mar 2004
 
Posted: 2017-05-30 12:16
@ChadZ: I wouldn't bet my life on my memory being 100% accurate, and I don't know if anything has changed for the "new" yfinance API. From memory yfinance via icharts was providing split adjusted volume, split adjusted dividends, prices adjusted for both splits and dividends (and unadjusted data), while google finance produces only split adjusted prices (not dividends) and split adjusted volume.

Capital Structure Demolition LLC Radiation

ChadZ


Total Posts: 2
Joined: May 2017
 
Posted: 2017-06-01 05:00
@Patrik Just checked my code and see signs that I remembered wrong and you are right about ichart's volume being split adjusted. Aside from that we recall the same. Good to know I'm on the same page, as a mistake would really mess things up, thanks.
Previous Thread :: Next Thread 
Page 1 of 1