Nezoo
|
|
| Total Posts: 4 |
| Joined: Nov 2017 |
| |
|
| Suppose you've developed a machine learning model for something. Presumably the underlying parameters should drift slowly and continuously over time. This problem is probably more of an issue in finance than other industries given the nature of things. How do you tend to deal with this? Just retrain on rolling time intervals of data? Train models on longer time frames and shorter time frames and perform some sort of model ensembling? |
|
|
|
 |
schmitty
|
|
| Total Posts: 64 |
| Joined: Jun 2006 |
| |
|
| Model your ML model parameters directly basis the center of your rolling training window, with relevant features and left-side (of your training window center) residuals as inputs. |
|
|
 |
|
You need an AI, which will optimally decide when to re-train you ML :)
Well, there is no universal recipe, often one does a periodical retraining, e.g. bi-weekly. Not because it is mathematically optimal but merely because it is simple.
Another approach may be to have two copies of your model: one in production and another, recalibarated overnight. Then you can estimate how they diverge and if it exceeds some threshold, you investigate what's going on and what to do further. |
www.yetanotherquant.com - Knowledge rather than Hope: A Book for Retail Investors and Mathematical Finance Students |
|
|
|
|
You need an AI, which will optimally decide when to re-train you ML :)
Well, there is no universal recipe, often one does a periodical retraining, e.g. bi-weekly. Not because it is mathematically optimal but merely because it is simple.
Another approach may be to have two copies of your model: one in production and another, recalibarated overnight. Then you can estimate how they diverge and if it exceeds some threshold, you investigate what's going on and what to do further. |
www.yetanotherquant.com - Knowledge rather than Hope: A Book for Retail Investors and Mathematical Finance Students |
|
|
ronin
|
|
| Total Posts: 708 |
| Joined: May 2006 |
| |
|
If your model needs training, then it obviously needs periodic re-training. Every few years or so, even the universe may be substantially different. So the parameters will be different as well.
But I wouldn't trust a model that you have to re-train more than every few years or so to accomodate the changing universe. Bi-weekly would be an immediate no, at least from me.
|
"There is a SIX am?" -- Arthur |
|
|
|
Maggette
|
|
| Total Posts: 1350 |
| Joined: Jun 2007 |
| |
|
There are applciations (outside of finance) where the "state" of the system changes regulary. You can use stacking here => a meta-model that decides which state you are in and which model to use. You train that once and not retrain it frequently.
I want to warn you that I only have seen this approach work in areas where the switch of the system is very obvious and often has a physical/logical explaination. Your signal to noise ratio has to be very good. Or else you gonna fit noise.
If you have to retrain your model very frequently you are most probably fitting noise. I mean think about it. What would you think are the effects you are learning there? |
Ich kam hierher und sah dich und deine Leute lächeln,
und sagte mir: Maggette, scheiss auf den small talk,
lass lieber deine Fäuste sprechen...
|
|
|
skAnkh
|
|
| Total Posts: 1 |
| Joined: Aug 2018 |
| |
|
Hi, you might be interested by this PhD thesis: https://ir.uiowa.edu/cgi/viewcontent.cgi?article=5248&context=etd
Especially chapter 4, which defines the notion of "concept drift", how to measure it and how to deal with it. Have a good read |
|
|
|
|
|
| In general I agree with @ronin. If your model is sensitive to small changes in your covariates' interaction with the target you're probably overfitting. OTOH if new data has covered previously unknown support your may have something of value to incorporate. Either way if you do end up retraining consider the meta-effect on overfitting via adjusting your model. |
|
|
|
