Forums  > Trading  > Reverse engineering strategy using ML  
     
Page 1 of 1
Display using:  

Energetic
Forum Captain

Total Posts: 1488
Joined: Jun 2004
 
Posted: 2018-09-20 21:58
Someone scared me earlier by suggesting that my strategy can be easily reverse engineered from my signals given what I said about my inputs.

It so happened that I (for) now have access to a software package from an accredited vendor. So I thought why not give it a shot myself before someone else will.

So I loaded the input data and pressed the button and ... drum roll ... it didn't work, i.e. no performance out of sample. There is a whole bunch of configuration parameters, of course, so I spent some time pulling all strings that made sense, and then the rest of them too, for good measure. No dice. I even called the vendor consultant to make sure that I didn't miss anything (not) obvious. He gave me a few tips which didn't help either.

The likeliest explanation is that applied some data transformations that I didn't share with the software. As the consultant put it - that's why we will always need live analysts. (Good to know that AI is not taking over the world just yet.)

I used TreeNet, in case you're wondering.

For every complex problem there is an answer that is clear, simple and wrong. - H. L. Mencken

eeng


Total Posts: 21
Joined: Dec 2014
 
Posted: 2018-09-20 22:56
What about providing the same features (including any transformations applied) you use on your model, plus random features and see if the SW is able to find the signal within the noise? Then once you establish that, you can try to remove some of the transformations and see if the model can get over that.

Also, after googling the thing, this looks essentially like a nice GUI for random forests. How different is this from say sklearn?

quantmatters.wordpress.com

nikol


Total Posts: 520
Joined: Jun 2005
 
Posted: 2018-09-20 23:14
@Energetic

Thanks for confirming my guess. Having only trades (not even software) it is impossible (Am I too strong?) to recover the algo behind.

However, if broker notices an account at his disposal delivering consistent profit, he would be tempted to multiply trades of client with his trades...

ronin


Total Posts: 341
Joined: May 2006
 
Posted: 2018-09-21 10:54

This reminds me of the story of the early AI attempts at playing chess.

The bot was trained on grandmaster games. Makes sense, right? You want it to learn from the best.

Then the bot started playing. Things weren't quite right. After a few moves, the bot would always sacrifice its queen.

Why?

Well, when a GM sacrifices the queen, it is in 100% of cases a genius move that leads to a quick checkmate. And the bot never saw billions of games where queen sacrifice is a stupid thing that leads to a quick loss.


"There is a SIX am?" -- Arthur

doomanx


Total Posts: 14
Joined: Jul 2018
 
Posted: 2018-09-21 14:15
"This reminds me of the story of the early AI attempts at playing chess."

Also similar to the story about some military guys who tried to train a model to recognize tanks in the forest. They had amazing performance on their training/test/out of sample but they put it into the field and it didn't work at all.

Turned out in their data, all the images with tanks in the forest it was day time and all the images without tanks it was night time, so the algo learned to pick out the colour of the sky rather than the tanks from the forest.

nikol


Total Posts: 520
Joined: Jun 2005
 
Posted: 2018-09-21 14:23
"some military guys"

I believe, they were consultants.

Energetic
Forum Captain

Total Posts: 1488
Joined: Jun 2004
 
Posted: 2018-09-21 16:29
@ eeng

I could do what you are suggesting but I see no point wasting more time on it.

No, it's not a random forest, it is stochastic gradient boosting and it need not be a GUI although GUI makes it a lot easier to use. Don't know what sklearn is.

For every complex problem there is an answer that is clear, simple and wrong. - H. L. Mencken

eeng


Total Posts: 21
Joined: Dec 2014
 
Posted: 2018-09-21 16:56
sklearn is short for scikit-learn, a free software Python library for machine learning, it implements all the major methods: linear regression, knn, trees/forests, etc. There's also gradient boosted regression/classification, my query was what the commercial offered that was not available already in sklearn.

quantmatters.wordpress.com
Previous Thread :: Next Thread 
Page 1 of 1