Forums  > Trading  > Docker  
     
Page 1 of 1
Display using:  

Praetorian


Total Posts: 224
Joined: Apr 2009
 
Posted: 2016-05-15 17:31
I am currently looking into docker, because I think it solves a problem of my trading infrastructure that really annoys me: Updates I deploy on the production machine sometimes break stuff, e.g. they are not compatible with some components I use in production. The second annoying part is, that if I install a new machine, I need about a week of to make all the different software parts to work together correctly (e.g. MySQL, R, Wine, Java, IQFeed, TickData, Backup stuff ...). Ok, all this are some luxury problems that do not occur very often. Nevertheless I strive for a perfect setup.

Now docker comes into play: I think about dockerizing everything. I hope that this way, I only have to write the Dockerfiles once and can move between machines without any hassle. Of course to dockerize every aspect of my infrastructure will be a lot of work (I think about half a year part time). The idea is to have update my production Ubuntu machine with all the security stuff and keep the docker components a stable a possible.

Has anybody of you experience with docker in a trading setup? Is my assumption about making my life easier correct? Has anybody first-hand experience about performance - my bottleneck would be parallel R (I do not want to have more than 5% overhead in a container).

Thank you for your opinions.

titanium77


Total Posts: 54
Joined: Sep 2012
 
Posted: 2016-05-16 01:58
Docker is great for development and production. See Docker is nearly identical to Native performance and faster than KVM in every category.

Once you get a handle of it the migration will actually be pretty quick and painless. You are going to want to look into some of the features around container communication such as: linking and volume mounting. It will also to some degree force you to think of your development design a bit different i.e. more service oriented/scalable. Docker Hub has plenty of useful images you can use or extend (there is a concept of inheritance of images). For example your going to have a container running mysql which will link to your R/Java app etc..From there the mysql container connection information will get injected into the environment of your application containers and you'll connect at runtime.

Scotty


Total Posts: 721
Joined: Jun 2004
 
Posted: 2016-05-16 02:53
Yep. We dockerize our environment. data, execution, reporting, trading algos.

“Whatever you do, or dream you can, begin it. Boldness has genius and power and magic in it.”

Patrik
Founding Member

Total Posts: 1338
Joined: Mar 2004
 
Posted: 2016-05-16 10:13
Yep, containers are helpful tools. We've used them since pre-docker time (pure LXC), and never looked back since.

It's not just prod deployment that is simplified, it also helps a lot with development and testing as we can get away from "well on my machine it works" - every setup is no longer a snowflake.

For quite a while there were unsolved questions around networking, service discovery etc where one had to roll ones own as a container user. Recently that's starting to change with projects like kubernetes and docker swarm. I'm in the process of throwing out my home rolled solutions and move deployment to Google Container Engine for a very smooth hosted kubernetes solution.

No input on performance from me - it's not important to me but anecdotally I've not seen noticeable degradation.

Capital Structure Demolition LLC Radiation

EspressoLover


Total Posts: 245
Joined: Jan 2015
 
Posted: 2016-05-16 20:39
Hitler's Opinion of Docker

Nonius
Founding Member
Nonius Unbound
Total Posts: 12699
Joined: Mar 2004
 
Posted: 2016-05-16 21:11
hehe, that is kinda funny.

Chiral is Tyler Durden

Patrik
Founding Member

Total Posts: 1338
Joined: Mar 2004
 
Posted: 2016-05-17 12:05
That's actually very good :) And reflects some of the "hype" pretty well I must say.


Capital Structure Demolition LLC Radiation

Praetorian


Total Posts: 224
Joined: Apr 2009
 
Posted: 2016-05-18 11:03
I started this architecture exercise by simply running a private registry container from my Java Scheduler. Seems to be a very simple task. Unfortunately I used the docker-java client, which is programmed by the worst computer scientists on the planet. This library needs 24!!! other java libraries to implement a simple task like starting a container. The Library is a complete mess, bloated, slow and totally over engineered. I nearly forgot that the java community often tends to produce such results.

Now I understand, why the Spotify java client is advertised by them as a "simple Docker client". I'll give them a shot now after wasting a whole day with docker-java.

TonyC
Nuclear Energy Trader

Total Posts: 1245
Joined: May 2004
 
Posted: 2016-05-18 23:05
>MySQL, R, Wine, Java, IQFeed, ...

You use IQFeed, is that the old PCQuote that was taken over by Reuters? ... Ive been thinking of switching my quotefeed. are you using it under wine, (i.e.using the WinDoze API) ... is there even a linux api?

flaneur/boulevardier/remittance man/energy trader

Praetorian


Total Posts: 224
Joined: Apr 2009
 
Posted: 2016-05-19 13:46
I don't know if IQFeed is the old PCQuote. But yes, there is no Linux implementation. Thats why you have to use wine. It works really good and stable, but is always a hassle if a new version comes out. You have to figure out which wine version to use and which windows dlls/libraries to install then. Fortunately I only have to update every two years or so. I have big hopes in dockerizing IQFeed.

P.S. IQFeed is a socket implementation. Just write a wrapper that connects to the socket and parse the data yourself.

svisstack


Total Posts: 303
Joined: Feb 2014
 
Posted: 2016-05-19 15:33
from my perspective having servers only on one operating system already is a choice to hassle especially when its linux and I dont think docker have something to fix that, not better having at least single windows server and not to hassle?

Time well wasted.

Patrik
Founding Member

Total Posts: 1338
Joined: Mar 2004
 
Posted: 2016-05-19 17:11
In this case he's talking about a single application that is non-linux. Creating one easily reproduced docker image (or just finding an already working one which he can do in this case) for running that one app via wine - compare that to setting up a separate odd-one-out box. To me it's the path of least resistance - especially when considering some of us are completely useless at administering windows boxes to begin with :)

Capital Structure Demolition LLC Radiation

Praetorian


Total Posts: 224
Joined: Apr 2009
 
Posted: 2016-05-19 20:37
Patrik, you put it better than I ever could.

goldorak


Total Posts: 1000
Joined: Nov 2004
 
Posted: 2016-05-19 22:14
> not better having at least single windows server and not to hassle?

You mean as a honeypot?

If you are not living on the edge you are taking up too much space.

svisstack


Total Posts: 303
Joined: Feb 2014
 
Posted: 2016-05-19 23:18
Patrik: thats right, everyone has different lowest resistance path, i didn't take into account that as i think windows server since few years beating linux, but old days it was different for a much longer period of time.

Time well wasted.

Tradenator


Total Posts: 1585
Joined: Sep 2006
 
Posted: 2016-05-19 23:34
Why isn't this under software? Why trading?

Praetorian


Total Posts: 224
Joined: Apr 2009
 
Posted: 2016-05-20 09:26
Yeah Tradenator. In retrospect, would have been the better decision.

Praetorian


Total Posts: 224
Joined: Apr 2009
 
Posted: 2016-09-18 13:49
Just to report back my experiences with Docker after three months of intensive tinkering on my infrastructure.

I am quite happy with Docker, I switched every program and service to a container now. I like very much that it is also possible to move stuff of the trading periphery (like duplicity backup) to a container. That way my "installation handbook" of the server was cut from 30 to 10 pages. Moving the production to another environment should be possible in about a day now.

Some of the caveats and solutions I encountered:
- Starting containers within a container to run on the host level (link the docker.socket to the starting container)
- User permissions are very frustrating. If you do not want to have everything as root, you have to do some voodoo with gosu
- If you encounter some specialities like with TickWrite (their application is 32-bit), then you have to dive really deep and build 32-bit containers
- All programs that need a gui to run or to administer can be containerized via vnc
- The Docker guys change their api a little bit too often
- I had to really brush up my Linux skills. Some stuff is easier done with another distribution than Ubuntu (the one I use most often). After switching from the main distro you hate yourself, because some config files reside in other places and other programs are used etc.

Containers built: Duplicity for Backup, IQFeed for live bid/ask data, Java container for all java programs, MySQL, OpenVPN as VPN server, TickWrite for historic market data, Registry as a Docker Registry, RStudio server for model calibration and development.

Hope that helps anybody.

P.S. Performance with Docker is a non-issue. If you have programs that communicate a lot via network together and run on the same machine, use --net=host. Otherwise I did not have to optimise anything.

EspressoLover


Total Posts: 245
Joined: Jan 2015
 
Posted: 2016-09-22 23:39
> That way my "installation handbook" of the server was cut from 30 to 10 pages. Moving the production to another environment should be possible in about a day now.

Good to hear. If you're happy with the results, I'd recommend considering an IaC, like Chef or Ansible, as the next step. You'll be able to ditch the installation manual completely, hit one button, and spin up a production server in ten minutes.

svisstack


Total Posts: 303
Joined: Feb 2014
 
Posted: 2017-02-27 17:59
https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/
https://thehftguy.com/2017/02/23/docker-in-production-an-update/

Time well wasted.

svisstack


Total Posts: 303
Joined: Feb 2014
 
Posted: 2017-02-27 17:59
https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/
https://thehftguy.com/2017/02/23/docker-in-production-an-update/

Time well wasted.

Praetorian


Total Posts: 224
Joined: Apr 2009
 
Posted: 2017-02-27 22:10
Thanks svisstack, interesting read! The guy is absolutely right in saying that the docker team just releases new stuff that always has some new bugs and is just not backward compatible. I experienced it first hand and it really sucked. Nevertheless, the article seems a little bit too pessimistic. I use Ubuntu 16 LTS, perhaps it is better than other distributions in supporting docker. I don't know.

ryankennedyio


Total Posts: 12
Joined: Nov 2016
 
Posted: 2017-02-28 00:29
I won't claim to have done anything close to what that guy has done, with regard to docker in production at *significant* scale, but there's a few things that are just completely false. I think he's kinda flamebaiting a bit, which hurts the credibility of some of the very valid points.

Interesting comment thread here too:
https://news.ycombinator.com/item?id=13713788

FWIW I've been using it in a large number of totally different projects, and have only run into something similar to the AUFS issue he mentioned, in that it's a total pain in the ass to work with the NodeJS package manager.

Actually we did also run into a weird problem with CentOS-on-Docker-using-CUDA-in-Mesos combination, where we essentially couldn't use any "rm" commands inside the Dockerfile.

I guess it's like most tech, the deeper you go, the more likely you are to run into "weird" bugs.

svisstack


Total Posts: 303
Joined: Feb 2014
 
Posted: 2017-02-28 11:42
yes, guy like ranting with a little bit of a noise, but important information's are extractable

Time well wasted.
Previous Thread :: Next Thread 
Page 1 of 1