Forums  > Trading  > Kalman filters  
Page 1 of 1
Display using:  


Total Posts: 45
Joined: Feb 2019
Posted: 2019-03-24 18:53
Hello people,

Looking for your opinion on the matter of Kalman filters. Have you successfully used the technique for stock return forecasting? If not, what are the usual applications of Kalman filters within finance? For me to be able to implement it, I will need to invest some serious time to understand it first. This is the reason I am asking whether you guys see any value in it for this purpose.


Nous promettons selon nos espérances, et nous tenons selon nos craintes.


Total Posts: 204
Joined: Jul 2013
Posted: 2019-03-24 22:53
Should be in basics.
Kalman filter is a way to estimate the path of the hidden variables in a state space model.
Then the idea is that you need to understand the state space framework and then you apply the kf. Iteratively you can use the kalman filter the compute the conditional expectation of you hidden variable (markovian) the compute the likelihood and so if you have coefficients to estimate you can use ML or similar.
You can look at it from the bayesian perspective and you have priors there, that opens up the game to more complex MCMC in case your model is not in closed form.

Stock trading? Why not SS models have applications all over finance and are a good one to keep in your toolbox.

I know there are a lot of applications for trend following where things can get complicated with this stuff.

Keep it simple and robust, bad starting values or priors can cause instability

"amicus Plato sed magis amica Veritas"


Total Posts: 17
Joined: Oct 2008
Posted: 2019-03-25 16:22
The Kalman filter itself is the easy part. The hard part is formulating what goes in to your state space model (what hidden states you have, how they evolve over time, and how they give rise to the observed returns). Given you have a good (i.e. closely resembling reality) state space model, then applying the model with the Kalman filter will give you reasonably good forecasts. But again the big if is whether your state space model is good to begin with.


Total Posts: 45
Joined: Feb 2019
Posted: 2019-03-25 21:28
Yes that is indeed what I have been able to gather. But that also begs the question, if I knew the state space model, or I guess the state space variables, wouldn't that be very similar to knowing the x-variables in a normal linear regression? Wouldn't the only benefit of the Kalman filter in that case be to somewhat improve the accuracy of the model when forecasting?

Or am I missing something? My question is as follows: is there something fundamentally different in trying to determine state space variables compared to determining independent variables in linear regression? Of course this is all in the context of financial time series, I know that these models have been developed for a different purpose from the beginning.

Nous promettons selon nos espérances, et nous tenons selon nos craintes.


Total Posts: 45
Joined: Feb 2019
Posted: 2019-03-25 21:32
I'll move my future topics to basics until I've built on my knowledge a bit :)

Nous promettons selon nos espérances, et nous tenons selon nos craintes.


Total Posts: 204
Joined: Jul 2013
Posted: 2019-03-26 16:21
Missing a lot...
For reference I would go through this book from durbin and koopman

It will also be easy to implement following their algorithms.
A lot of the existing packages do not allow easy definitions of models so it is probably easier to follow those instructions to filter and estimate parameters.

The main applications are time varying parameter models, dynamic factor models and autoregressive models. To name a few.

"amicus Plato sed magis amica Veritas"


Total Posts: 45
Joined: Feb 2019
Posted: 2019-03-27 10:49
Thanks for the tip. I will look into it and see whether I can handle the maths.

Nous promettons selon nos espérances, et nous tenons selon nos craintes.


Total Posts: 405
Joined: Nov 2006
Posted: 2019-05-04 00:11
I am a bit puzzled by this thread. It makes Kalman filters look trivial.

Trivial? Perhaps in an ideal world in which God grants one a linear state space model and where all noise is additive, white and Gaussian. Who lives in that world?

Even in engineering, where masses and moments of inertia can be measured very accurately, tuning a Kalman filter is a difficult art. Of course, in engineering failure is quite obvious. In econometrics, however...

And the folks at NASA's Jet Propulsion Lab do not use the standard Kalman filters when sending things into space. Not sufficiently robust. They use more robust algorithms one can only find in exotic books that only a few graduate students have even heard of.


Total Posts: 1213
Joined: Jun 2007
Posted: 2019-05-04 01:39
Agree with rod.

There were exactly two occasions outside of finance (once profesionally with GPS data and once as a state estimation component of a toy project implementing PMC stuff) where I actually used Kalman Filters and new beforehand that it was the way to go.....but still it wasn't easy (or I am dumb, who knows)... and I had to dig deep and came up with some two layer Kalman Filter I stole from a paper.

I now have a very hard time envisioning scenario in finance where I would use Kalman trust the results...

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...


Total Posts: 28
Joined: Feb 2018
Posted: 2019-05-04 10:17
"I now have a very hard time envisioning scenario in finance where I would use Kalman trust the results... "

there is example of rather simple scenario in the cross-sectional equity approach:

if you have exposures of the stocks to "price" and then you force overall portfolio exposure to be 1 for the current moment and zero portfolio exposure for 1 year ago then when trying to solve this linear system of equations you can apply Kalman filter to get the weights. And the resulting signal will obviously resemble cross-sectional momentum. And you can generalize this approach easily.


Total Posts: 405
Joined: Nov 2006
Posted: 2019-05-05 00:40
If engineers are intimidated by Kalman filters and quants are not, then there must be a reason. I cannot see what the reason is. What am I missing?


Total Posts: 265
Joined: Dec 2012
Posted: 2019-05-05 18:59
I've used KFs for about a decade in HFT; more often than not they've provided marginal benefit over simpler algorithms like EWMAs and such, but I've appreciated the structure. I've used both EKFs and UKFs in my space days; can't comment on how, but the idea that KFs don't get used in space is crazy -- often they are literally the optimal solution.

I'd start here:

Where things get challenging is that kalman filters lack some very often desired properties (hard bounds, guaranteed numerical stability), there exist some pretty fancy solutions that target this, but in finance, I'd never bother, just have a meta component that ensures things aren't going wonky, and disable/reset in case you end up in unsafe gradient hell.

There are no surprising facts, only models that are surprised by facts


Total Posts: 405
Joined: Nov 2006
Posted: 2019-05-05 19:04
radikal, who mentioned that "KFs don't get used in space"?


Total Posts: 1206
Joined: Feb 2007
Posted: 2019-05-06 09:50
FWIIW my first consulting job was putting a Kalman filter in an early G3 phone with accelerometers. It wasn't so bad, though I had to band-aide it with some tricks to get it to do what I wanted it to (aka ignore gravity).

I know a guy who was successful with Kalman-like approaches in futures markets. I think he was doing something like what Zoho says. Or, like this:

I'd be curious about Rod's obscure NASA robust filters, but I think the reason it's trivial is measurement noise is kinda sorta zero, and varma is simpler than shooting rockets into space.

"Learning, n. The kind of ignorance distinguishing the studious."


Total Posts: 405
Joined: Jan 2015
Posted: 2019-05-06 19:44
EMAs are recoverable basis functions for any linear model. And that extends to any continuous non-linear model, if you feed them into a universal approximation like a neural network or kernel machine.

Therefore, I'd say your default should just be OLE on a whole bunch of EMAs spanning the full range of potentially relevant time horizons. OLE is still an unbiased linear estimator in the time-series domain. It's true you need time-series specific regressions if you want to rely on the p-values for coefficient signifigance or point estimation. But nowadays you're probably cross-validating anyways, and the computation cost of regression is basically free.

IMO Kalman filters, or any other state-space model, should probably only be used if you have very strong inductive priors or a small/high-variance dataset. Otherwise just get a shit-ton of data and throw it at a regression of EMAs.

Good questions outrank easy answers. -Paul Samuelson


Total Posts: 283
Joined: Oct 2004
Posted: 2019-05-07 10:26
Well economists have zero problem fitting kalman filters with 100 points of data so I don't know why you guys are so squeamish about it Wink

Et meme si ce n'est pas vrai, il faut croire en l'histoire ancienne


Total Posts: 993
Joined: Jun 2005
Posted: 2019-05-07 11:49
Kalman is ok for dynamic smile fitting, except when it jumps.


Total Posts: 204
Joined: Jul 2013
Posted: 2019-05-07 14:18
KF is not trivial at all but not even that difficult to implement for basic simple models.
Certain is that it is highly unstable if mispecified, which is often the case.
There is also a case of sensitivity to starting values, which in the bayesian world would be sensitivity to priors.
A sausage machine basically.
MCMC works best for state space more complex models, like the ones you can work on in trend following.
There too you need to be careful and model instability is often linked to almost non stationary mc samples.

On a more intuitive way you can figure this out easily, you are trying to estimate a hidden state that drives the observable Y, and this state is markov and has some behaviour given by the model of choice.
Still the state is not observable, so unless you are denoising, you need to know exactly what to expect from this behaviour or you are just guessing and coming out with a mispecified model.

"amicus Plato sed magis amica Veritas"


Total Posts: 64
Joined: Jul 2018
Posted: 2019-05-23 13:42
Bit late, but just to weigh in on state space models - I've used them a lot in HFT (for example pressure in related assets). There are some areas where non-linear dynamics come up and particle filtering is very useful (stochastic-vol is the most prominant example I can think of where noise is multiplicative). Some people will do EKF/unscented (I think this is hard to make work in practise). I like Bayesian stats so I use sequential MC for this kind of thing (to make it work in practise you need a resampling step or your importance weights will be crap, but this can be scary with non-stationarity). The hardest part as already mentioned by others are the classic noisy signal processing issues. Choice of prior, GIGO and all that. Filter, filter and then filter again (hint: one can combine the above two approaches via Bayesian anomaly detection).

did you use VWAP or triple-reinforced GAN execution?


Total Posts: 4
Joined: Nov 2017
Posted: 2019-06-06 04:52
This is really interesting. I've always done the old regression (OLS, which I presume is the same thing as OLE?) on a bunch fo EMAs, and figured that was a crude hack that I came up with that seemed to get the job done. Didn't know that it was a standard technique or that there was a theoretical basis for it. What do you mean by a recoverable basis function? Quick googling didn't yield any hits.


Total Posts: 204
Joined: Jul 2013
Posted: 2019-06-11 13:45
This is what most people do, although I prefer arma models that do pretty much the same thing, you can then make assumptions on the conditional variance of the error (but you can control that effect with regression too).
State space too ends up being the same thing, although the biggest advantage is in estimating in one step, which is much better than doing in more steps and with different estimators.

"amicus Plato sed magis amica Veritas"


Total Posts: 4
Joined: Nov 2017
Posted: 2019-06-13 12:05
Why is the one step estimation that useful? Also, my problem with ARMA comes when I want to include other possibly nonlinear parameters in the time series model. With the EMA approach, you can pretty easily fit these by just tossing everything (EMA + factors) into a random forest or something. Also the conditional variance estimation approach only really works for normal errors, which don't come up much in my experience, right?


Total Posts: 204
Joined: Jul 2013
Posted: 2019-06-17 14:30
Correct, obviously it is easy to overfit that way so it needs care.
For conditional variance wrong as garch fits very well daily returns distributions and captures fat tails and more depending on the flavour you use.
You can even go multivariate which has obvious implications for portfolio construction.

"amicus Plato sed magis amica Veritas"

Nuclear Energy Trader

Total Posts: 1330
Joined: May 2004
Posted: 2019-10-09 17:55
EsspressoLover wrote:
" Therefore, I'd say your default should just be OLE on a whole bunch of EMAs spanning the full range of potentially relevant time horizons. "
And later wrote:
"Otherwise just get a shit-ton of data and throw it at a regression of EMAs."

EspressoLover, can you point me toward some literature that decribes how to impliment/use this technique?

flaneur/boulevardier/remittance man/energy trader


Total Posts: 73
Joined: Apr 2018
Posted: 2019-10-10 15:19
What he's saying is that all linear models are special cases of OLS:

y = c_t x_t + c_(t-1) x_(t-1) + ...

In turn, the x_t's can be written as a linear combination of their EMA's:

EMA_(t, h) = x_t + h * x_(t-1) + h^2 x_(t-2) + ...

So you could just choose a bunch of h's and regress y against EMA(t,h1), EMA(t,h2),...
Previous Thread :: Next Thread 
Page 1 of 1