Forums  > Software  > Auto-Retrieving CME Fixing Prices  
     
Page 1 of 1
Display using:  

TSWP


Total Posts: 448
Joined: May 2012
 
Posted: 2020-01-13 16:00
I spoke this morning with a CME rep at Global Command Center to find out if there was any way to automatically retrieve the Fixing Prices posted on this page on the various options expiry days.

He said they disseminate these values immediately onto this page OR to exchange members, no other solutions provided.

At the moment we are using Interactive Brokers to place our trades and it seems to me that the only Fixing Price tickers on IB TWS are:

ESF.FIX
NQF.FIX

I could not find any other Fixing Price ticker on IB TWS (please correct me if you think I am wrong). For example, no RTY.FIX for the Russell Futures, no ZYTL ticker for the 6E EUR/USD FX Futures, etc.

I was wondering if any of you is aware of other legal methods to retrieve these Fixing Prices, excluding webpage scraping (they will block it):

- any broker you know that offers the complete list of tickers?

- any other API/FIX/etc. method that would allow us to pull the required data?

Feel free to PM privately if you don't want to discuss this publicly.

Thanks.

nikol


Total Posts: 1172
Joined: Jun 2005
 
Posted: 2020-01-13 19:57
I am using this to scrap even ugliest pages.
https://docs.python.org/3/library/urllib.request.html

Quite straightforward.
Need help? Let me know.

Maggette


Total Posts: 1251
Joined: Jun 2007
 
Posted: 2020-01-13 20:32
With nikol here. Use the same library.

If I get you right, you want to download the new data once per day. I do have a very very hard time believing they implemented serious measures to block that.


Pretty sure nikol and I can come up with a script that maneuvers around every shit they have in place.


edit:
I mean, if you are realy serious about getting the data, you could set up phantomis or selenium

These things open the site in a browser, make a screenshot and then extract the data. And with a little trickery are virtual not distinguishable from human interaction.

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

nikol


Total Posts: 1172
Joined: Jun 2005
 
Posted: 2020-01-13 21:29
> These things open the site in a browser, make a screenshot and then extract the data

Very interesting. Thank you.


Regarding CME, I doubt, that they will be "deeply offended" with once a day request.

schmitty


Total Posts: 64
Joined: Jun 2006
 
Posted: 2020-01-14 00:22
> if you are realy serious about getting the data,
> you could set up phantomis or selenium

Absolute overkill. Data are available via simple XML endpoints like so:

https://www.cmegroup.com/CmeWS/md/FixingPrice/pricelist?&date=01/13/2020&count=150

You can easily pull via curl, wget, etc... Run on crontab. Not rate limited as far as I can tell. If you do run into pacing violations, run through local TOR socks5 proxies -- unlike, e.g. Bloomberg, CME does not block known TOR exit nodes.

Also, PhantomJS has not been updated in years, and is essentially deprecated. Use Chrome headless instead.

Maggette


Total Posts: 1251
Joined: Jun 2007
 
Posted: 2020-01-14 08:24
I think in this particular case you are absolutley right. They actually seem to have nothing going to prevent simple web scraping via shell script. THe sales rep talking to TSWP was either bluffing or clueless.

Just wanted to bring it up for the tougher cases though.

Forgot about the headless browser stuff. Thanks for that.

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

TSWP


Total Posts: 448
Joined: May 2012
 
Posted: 2020-01-14 08:37
Guys... you're AMAZIIIING!!! All of you!

Thank you so much for this, you just saved me a lot of headaches...

Schmitty: the XML solution seems to be the easiest to me, thank you.
Nikol and Maggette: I will check out also what you suggested, thank you.

In the meanwhile, for the sake of discussion:

I have found out that DTN/IQFeed does provide these tickers through their subscriptions packages, one can retrieve them via IQFeed API or via QCollector (the latter is simpler/quick fix: use tick data in portfolio, then write to csv file).

The DTN/IQFeed solution is harder to implement and cost some hundreds of $ per month in terms of subscription packages to DTN/IQFeed so it's less appealing in an alpha phase, maybe I will use that for beta or production but the XML path suggested by Schmitty would be great for now.

I guess there may be other data vendors out there that may offer these tickers but I have not investigated further.


TSWP


Total Posts: 448
Joined: May 2012
 
Posted: 2020-01-14 08:55
schmitty:

I was checking the tickers in this XML endpoint:
https://www.cmegroup.com/CmeWS/md/FixingPrice/pricelist?&date=01/13/2020&count=150

they have the "equities" tickers but they don't seem to contain the tickers from the "FX" tab/page on the CME Fixing Prices:
https://www.cmegroup.com/trading/fixing-price.html#fx

where did you find the syntax for the XML endpoint you've posted?
is there a different one for the FX fixing prices page?

schmitty


Total Posts: 64
Joined: Jun 2006
 
Posted: 2020-01-14 23:20
> is there a different one for the FX fixing prices page?

Try adding a "bracket" tag to the call:

https://www.cmegroup.com/CmeWS/md/FixingPrice/pricelist?&date=01/13/2020&bracket=09&count=150

FX futures appear to be bracket 09. Try other brackets sequentially to figure out which bracket applies to what futures.


> where did you find the syntax for the XML endpoint you've posted?

In the browser console. In Firefox, Tools/Web Developer/Browser Console. Then click (highlight) "XHR" while making sure all other choices are unhighlighted. Click on the trash can to clear the console. Reload the page in your main browser window. A list of XHR calls will appear in the console. Mouse over each in turn looking for likely XML or JSON calls. When you find one, right-click, copy, and test in another browser window/tab.

Using Chrome is similar. Also, for heavy duty use, nothing beats WireShark.

TSWP


Total Posts: 448
Joined: May 2012
 
Posted: 2020-01-15 08:53
Awesome, thanks again.
Previous Thread :: Next Thread 
Page 1 of 1