Forums  > Software  > Yahoo Finance  
     
Page 1 of 1
Display using:  

EspressoLover


Total Posts: 216
Joined: Jan 2015
 
Posted: 2017-05-17 21:36
Anyone know what's up with their historical data CSV page? The original URL (icharts.yahoo.com) has been unresponsive this entire week. It's seemed to move to a new format (query1.finance.yahoo.com). However this now requires a GET param cookie (?...&crumb=[key]). Can't figure out how to get this working in curl. So...

1) Does anyone know if the icharts.yahoo.com is permanently discontinued? The page says "undergoing maintenance", but there's no other announcements from yahoo.

2) Anyone figure out a hack for the crumb= parameter to get it working outside the browser?

3) Any other alternatives for free historical interday data, which gets updated daily? I'd get a running CSI subscription, but I'm a cheap bastard.

xig


Total Posts: 2
Joined: May 2017
 
Posted: 2017-05-17 22:03
It doesn't seem too bad. Just did a quick test, seems like you just need to first scrape the crumb (with cookies on) then grab the data. For example (F#).


open System
open System.Net
open System.Text.RegularExpressions

type CookieAwareWebClient() =
inherit WebClient()
let cookies = new CookieContainer()
member x.Cookies = cookies
override x.GetWebRequest(uri:Uri) =
let req = base.GetWebRequest(uri)
let webReq = req :?> HttpWebRequest
if webReq null then
webReq.CookieContainer <- cookies
req

let wc = new CookieAwareWebClient()
let str = wc.DownloadString("https://ca.finance.yahoo.com/quote/^GSPC/history?p=^GSPC")

let m = Regex.Matches(str,"\"user\":{\"crumb\":\"(.*?)\"") |> Seq.cast |> Seq.head
let crumb = m.Groups.[1].Value

let csv = wc.DownloadString("https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1492458189&period2=1495050189&interval=1d&events=history&crumb=" + crumb)

xig


Total Posts: 2
Joined: May 2017
 
Posted: 2017-05-17 22:07
Looks like my pre tags got stripped despite the preview looking fine. Attached to preserve indenting.

Attached File: yahootest.zip

EspressoLover


Total Posts: 216
Joined: Jan 2015
 
Posted: 2017-05-17 23:46
Thanks! That worked for me.

Here's a shell script version in case anyone's interested

Yahoo Finance Bash Script

Edit Addendum: Heads up, the new CSV sheet appears to have reversed the time order from the previous one. Also some symbols now include rows with null values if you go back far enough. So make sure your parser handles these conditions.

kloc


Total Posts: 1
Joined: May 2017
 
Posted: 2017-05-18 19:21
Many thanks for this; just run into this in the morning myself. A minimal example for "pythonistas" attached below:

Attached File: snapy.zip

Basically, raquests.post(...) works like a charm here...

contango_and_cash


Total Posts: 62
Joined: Sep 2015
 
Posted: 2017-05-19 17:26
Saw this and decided to just start paying for some equity data.

I immediately switched to google (pandas wrapper) as a temporary fix.

Patrik
Founding Member

Total Posts: 1331
Joined: Mar 2004
 
Posted: 2017-05-21 22:57
Just in case people didn't remember to think about it - YFinance and Google Finance produce their equity data using different methodologies with regards to split adjustments, dividend adjustments etc. If it matters to you make sure you get it right :)

Capital Structure Demolition LLC Radiation

yanko


Total Posts: 58
Joined: Nov 2009
 
Posted: 2017-05-22 13:43
Hi,

the best way to get data from Yahoo is to use quantmod on R, the new URL has already been accounted for (see https://github.com/joshuaulrich/quantmod/issues/157).

However, as Patrik pointed out, the split/dividend adjustments are out of whack since the url change (see discussion in the above github issue, as well as https://github.com/joshuaulrich/quantmod/issues/160).

Also - Yahoo has seed it "stopped supporting" the API (see post by Yahoo-moderator Nixon here https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503/page/3). whatever they mean by that, since it has always been an unofficial "service/hack". Yahoo adding a cookie hash (the crum parameter, see github) does indicate that they would like to reduce API traffic by wagging off more inexperienced or less motivated users.

So, it's still working, but it's now even less reliable.

yanko

finanzmaster


Total Posts: 100
Joined: Feb 2011
 
Posted: 2017-05-25 23:30
YaWhore Dance with Yahoo Finance

We discuss the ways to circumvent the API change of Yahoo.Finance and alternatives to it, esp. Alpha Vantage.

www.yetanotherquant.com - Knowledge rather than Hope: A Book for Retail Investors and Mathematical Finance Students

BBands


Total Posts: 1
Joined: May 2017
 
Posted: 2017-05-26 01:49
Thanks Kloc,

I thought that you had to have a crumb with the new api, but that works perfectly.

Best,

John

ChadZ


Total Posts: 1
Joined: May 2017
 
Posted: 2017-05-28 00:48
Patrik, could you elaborate?

I'd like to make sure I'm not missing something because I did not see a big difference between Yahoo and Google's adjusted prices. Yahoo of course provides unadjusted prices while Google does not, but when comparing Yahoo's adjusted prices to Google's, it seemed to me that both adjusted for splits and not dividends, although the dividend amounts themselves were split adjusted for both. My memory is fuzzy regarding Yahoo's volume data, but I vaguely recall that from icharts at least it was not corrected for splits while Google's is. I think that was the extent of the differences I noticed, or am I missing something?
Previous Thread :: Next Thread 
Page 1 of 1