Forums  > Trading  > Shannon's Demon  
     
Page 3 of 4Goto to page: 1, 2, [3], 4 Prev Next
Display using:  

NIP247


Total Posts: 540
Joined: Feb 2005
 
Posted: 2006-04-27 23:34

kr, I like your scepticism and when it comes to would be market beating backtested strategies it's often warranted. Nevertheless, I have a strong feeling that given Jim Simons background and him saying something to the effect of " in the beginning we did very little modelling (of prices as in forecasts I presume), Cover's dynamic rebalancing algorithm is somewhere in the same space.

If you take a few volatile commodities/futures and the low cost of execution then maybe we have something.

You did a good effort to debunk the stuff but I guess it was too technical for me. I quite like these three papers that I mentioned before:

Can we learn to beat the best stock ( they play their algorithm in reverse time as well)

Universal portfolios w and w/o transaction costs

and the one mentioned before:

kernel regression for sequential investments that relaxes some of the assumptions that we know nothing about the distribution...

I'm wondering, wouldn't the universal portfolio approach be quite interesting as a way of forming a portfolio of exotic new instruments that we know nothing about? I'm  specifically thinking about CO2, Electricity (where thinking that you have a model will put you out 4-5 std devs...), Sulfur, Palm Oil, Weather contracts etc... What's your view? (Exotic is just Toxic in french?)

I also found this dissertation. If any of you could provide me with a copy, I would be much obliged.

 


On your straddle, done on the puts, working the calls...

lexx


Total Posts: 60
Joined: Apr 2006
 
Posted: 2006-04-28 16:04
Note though that Cover's approach has nothing to do with Borodin & et.al.- "can we learn to beat the stock market"
Boroding's approach is based on autocorrelations (and cross-) and seems to exploit mean-reversion.

For universal (Cover) algorithm to work, looks like it is preferable that stocks are not highly correlated (positively) with each other and they have strong autocorrelation

Reagarding applying this concept to exotic undelyings - may make sense, since they are not probably highly correlated with each other, but - low liquidity/high transaction costs !!!! Universal portfolios if I'm not mistaken, sometimes have tendency for high turnover.Should your algorithm try to do that with low liquid/high tcost assets, it will eat up all the benefits

kr
Founding Member
NP Raider
Total Posts: 3561
Joined: Apr 2004
 
Posted: 2006-04-29 10:41
247 I appreciate the comeback, will have a look over the three-day and think some more.  My intuition on this is poor - that's why I'm interested.  Really I much prefer illiquid stuff where the trade price can be pretty far from what people might consider 'fair value'.  That said, I am typically not thinking about daily rebalancing, and my variables are likely to have some autocorr or other strange persistent behavior.  If one then switches to relatively high-frequency things, it can be distracting - did these guys just ride the wave in an overly complicated fashion?  And that is a common story in finance as well - incidental market timing (property wave, internet wave, etc.) 

Qualitatively, the rebal algorithm in continuous time looks like a simple change of variables and nothing more.  It's not path-dependent.  So my 'intuitive lemma' is that the distribution of the long-term price movements may be different, but it's a martingale issue like any other, you can't improve the mean return this way unless the underlying has a more complicated behavior than you've assumed.  Which is what I assumed above, fine, but this is a roundabout way of getting trading gains from complicated behavior.

my bank got pwnd

LongTheta
The Snowman

Total Posts: 3117
Joined: Mar 2004
 
Posted: 2006-04-29 12:22

Why would size of investment have any impact

Well, in principle, it doesn't, but in practice, you need to faithfully cover the market + survive transaction costs and slippage. I saw some later work by Cover and co discussing these issues.

They have a number of papers describing optimal ways to implement the algorithm. They also take other factors into account: the price of oil, interest rates, etc. So, they know that the pure algorithm doesn't work and needs help.


Time is on my side.

kr
Founding Member
NP Raider
Total Posts: 3561
Joined: Apr 2004
 
Posted: 2006-04-29 16:09
Ok, so I went through the U-portfolios thing and see that it is not this fixed-rebal story.  Cover actually makes it quite clear in the paper how it works.  First, the weighting scheme works, because the average exp growth of the various assets is essentially the exp growth of the best one.  This is simply b/c the exp function grows so rapidly.  Secondly, you need a universe of assets that contains at least one moon-shooter to win.  If you only have boring stocks like Coke and IBM, you will wind up betting on the best of those, getting the obvious reinvested results.  Finally, you need volatile assets, because such assets are more likely to show a big dispersion of sample means during the test period. 

In fact I think the best thing to do is NOT run it for asymptotics, because most moonshooters come back down to earth, not being able to replicate super-GDP growth for a long time.  There are some interesting ideas but I think it's not exactly obvious how good the performance will be, because "exp space" is used throughout.  If this was done on a returns basis I would be more convinced.  I do think that the convergence-to-winners should be pretty rapid - this is a function of exp space, if you outperform then it will be hard to miss.  But I think you have to have some kind of burnout function to get off the bandwagon early and in this case, systematically.  That's a practical point and probably not a theoretical point. 

At the end of the day, I think this looks like 'momentum', but doesn't talk much about convergence rate speed on the one hand, and what I'd call 'decay of local stationarity' on the other.  It would be interesting to watch the portfolio weightings change, and see if you could summarize what's in fashion and what's not. 

I am cautious but a little bit intrigued - there is a theory here. 

my bank got pwnd

quantie


Total Posts: 883
Joined: Jun 2004
 
Posted: 2006-05-09 04:28
ok..here is a dumb Q I haven't really read the stuff carefully...

If you take the algorithm and find the best crp today. And then do the optimization again in a month wouldn't your best crp change with new information?. Also is the covariance information being used somewhere or is that not material as we are not looking at single period optim?. Also is it possible to add constraints??

If someone can point me to a book or other reference for this to start with would be much appreciated..

goldorak


Total Posts: 1001
Joined: Nov 2004
 
Posted: 2006-05-09 14:07

Some of you might like this article :

 

http://arxiv.org/abs/cond-mat/9801240

 


If you are not living on the edge you are taking up too much space.

Johnny
Founding Member

Total Posts: 4333
Joined: May 2004
 
Posted: 2006-05-09 14:58

I'm interested in the effects of parameter mis-estimation on these models. Perhaps one of the many NP Kelly fans could comment on the effects of mis-estimating returns?

 


Stab Art Radiation CSD LLC

filthy


Total Posts: 1258
Joined: Jun 2004
 
Posted: 2006-05-09 15:49

i haven't done much with UPs but with kelly mis-estimating returns is a big problem. over-estimating win rate or win size leads to over-betting which leads to bankruptcy. in fact, as you do as well by betting half kelly as twice kelly most people err heavily on the side of caution. quarter kelly betting seems to be close to standard.

there are formulae for these things but you are better of throwing together a spreadsheet and running a few simulations.


"Game's the same, just got more fierce"

lexx


Total Posts: 60
Joined: Apr 2006
 
Posted: 2006-05-09 16:01
If I'm not mistaken that's the beauty of UP : there are NO parameters,
all one needs is a price history of stocks, and that's it !
Possibly a dumb question : to speed up convergence of UP to optimal
growth crp, would it help if we use shorten the time scale- instead of daily returns use, I don't know, 15-min returns and optimize on shorter time interval. Some authors complain - yeah, you can arrive at this optimal growth portfolio, but it takes long period to learn. Therefore using more data but on a shorter time-scale would it help ?

Johnny
Founding Member

Total Posts: 4333
Joined: May 2004
 
Posted: 2006-05-09 16:34
Thanks Filthy, that's rather what I suspected. It all seems rather arbitrary (1/2 Kelly, 1/4 Kelly, 1/pi Kelly ... ) when you get down to it.

Stab Art Radiation CSD LLC

filthy


Total Posts: 1258
Joined: Jun 2004
 
Posted: 2006-05-09 16:59

it is arbitrary because you are choosing a utility function. still i think it is important to know what happens to your portfolio when you make this arbitrary choice. just because you may not "like" kelly doesn't mean your trade sizing choice is irrelevant. but for some reason people tend to get very polarized over this issue. arguments both for and against tend to become heated and emotional. i've actually been called psychotic for "believing in that kelly crap".

somewhat related to the UP stuff... i recently went through my PA to see what allocation the constrained kelly criterion would suggest. so i worked out all the correlations, plugged in my best estimates of returns and vols, then went through the lagrange multiplier stuff to get the optimal weights. it said my portfolio (currently 24 stocks) should consist of 98% of one stock and 2% of another. THAT would be psychotic.


"Game's the same, just got more fierce"

mib


Total Posts: 354
Joined: Aug 2004
 
Posted: 2006-05-09 17:58
lexx, if there are no parameters, it does not mean that that the method can not be misspecified. Things can be genuinely non-stationary or it may be just a sampling error - either way something fitted to past history may behave not that great in the future. The question is how sensitive is the UP to small changes in data process.

Head of Mortality Management, Capital Structure Demolition LLC

Johnny
Founding Member

Total Posts: 4333
Joined: May 2004
 
Posted: 2006-05-10 09:51

Filthy, yes, for some reason this can become rather ideological. I don't know why. Actually, my objection to Kelly isn't an objection to the principle but an empirical difficulty.

As you said some people object to specifying a utility function. But actually, all position sizing strategies imply a utility function, so it's better to make it explicit rather than leaving it buried at the bottom of a pile of assumptions. Some people don't mind specifying a utility function, but think that the ln function is too aggressive. I have some sympathy with this view, but it doesn't bother me too much.

The difficulty I face is that Kelly seems rather sensitive to mis-specification of win/lose rates and win/lose sizes. As a systems trader I am in a better position than most to be able to assess my payoff distribution. However, I know that the distribution of payoffs that I experience in trading can differ for long periods from the inferred distribution of payoffs in backtesting. And that the backtested results are subject to the usual inference problems. I therefore have two sources of error when specifying win/lose rates and win/lose sizes. And, as you remarked, Kelly is sensitive to this.

This isn't a random attack on Kelly. If anyone has constructive comments on how to control this problem I'd be interested to hear.

 


Stab Art Radiation CSD LLC

FDAXHunter
Founding Member

Total Posts: 8356
Joined: Mar 2004
 
Posted: 2006-05-10 10:13

Johnny is spot on with his illumination of the problem.

I'd like to quote one of my friends on the subject: He put it a bit  more tongue-in-cheek:

"Kelly/Optimal-f gives you the optimal length of the rope.... which you can then hang yourself with"

The reality of distributions is, as Johnny pointed out, a bit too unstable to concern oneself with fractional betting schemes that are optimized on Terminal Wealth alone.


The Figs Protocol.

opmtrader
Founding Member

Total Posts: 1333
Joined: Mar 2004
 
Posted: 2006-05-12 14:20

Guys, I'm just getting into the required reading here.  Interesting stuff.  The challenge for me is getting it down to a basic enough level where it could be implemented.  I was wondering if anyone can clear up some basic concepts for me.

1)  Does the UP implement an equal amount of capital to every possible portfolio at inception and *never* change it?  Or do you try to constantly hone in on the best weighted portfolio?  I think it is the first instance.  In that case, are we to believe that the extreme winning nature of the few winning portfolios outpaces all the losing portfolios by a longshot?

2)  When we say *all possible portfolios*, what is the practical granularity for dividing the weights?  Has anyone studied the effect of trying to minimize the possible portfolios to speed computation time?

 


kr
Founding Member
NP Raider
Total Posts: 3561
Joined: Apr 2004
 
Posted: 2006-05-13 12:13

No - the optimal portfolio U* assumes that the weighting is constant, and then the theorem is that the updating procedure converges to the optimal at an exponential rate.  The difference between the growth rate of the optimal and the growth rate of the updated portfolio can be estimated, and I think he would say that it is a function of the entropy of the data generating process for the returns - maybe not in this paper.  Let's not get into that, what was not discussed was the volatility story.

The optimization is just an integration over the weight space against the gain/loss outcomes, not at the level of returns but nominals.  Because he does it this way there is a telescoping property of the calculation which allows you to understand the convergence story. 

In terms of granularity I think it's probably not all that important, and one should start with a fairly rough integral.  One could probably say something about convergence in this case as well.  One could even do a pure yes/no analysis - i.e. just include a stock, or not.  I think it's important to see that you can put bounds on the weights through the integration step as well.  The big problem is dimensionality, which I also think is not really discussed much.

What I don't really like about the paper is that it seems engineered to avoid not just the volatility issue but the nonstationarity issue.  Tacitly it is argued that the weights SHOULD converge - that is, the outperforming universal portfolio is static.  That is false.  Hence we are back to the main discussion - overconvergence and disaster avoidance. 

What I do wonder is how the portfolio outperformance can be paired up with the weight dynamics.  That is, you assume instead that there will be some amount of nonstationarity, and the objective is to understand something statistical about that - i.e. is there any first-order assumption that we don't intuitively know yet (b/c in this discussion I don't think anybody has mentioned any, other than the fact that as soon as you've reached 30% of your target, the market ALWAYS goes against your weights and you get stopped out).  Is it a question of timescales and spectral analysis, half-lives of trends vs. limits you put on the weights, etc.  If I have any kind of intuitive guess, it would be that trends burn out, so if you risk-adjusted your weight convergence, you would want to ride the trend until it looks too risky, and then go somewhere else.  Following that hunch, I'd say you're still at risk because the broader market itself can trend as well, so you might think you are 'switching to a fresh trend' when in fact you are just in a different seat on the same rollercoaster.  What would be first/second order then would be to understand whether trend decay at the index level is different than at the level of the underlyings. 

All very vague of course - need to run some numbers to really make better conjectures.  It just appears that this is a sophisticated moving-averages kind of thing, and a technical person would want to say something about crossings... I am wary of the cointegration thing as well, maybe somebody has some relevant insight on the interplay.


my bank got pwnd

opmtrader
Founding Member

Total Posts: 1333
Joined: Mar 2004
 
Posted: 2006-05-13 20:10
kr, thank you for taking the time to clarify.  I will have to return to the papers for some rereading.

saffron


Total Posts: 181
Joined: May 2006
 
Posted: 2006-06-01 22:05

Hi,

As an IT guy (that's Information Theory, not Information Technology...Smiley) I would be very interested to know if you guys are aware of any other worthwhile applications of IT to trading, specifically, in developing alpha generating strategies (aside from the UP material discussed in this thread). Any references to relevant papers will be much obliged.

 

Thanks.

 


Chuck


Total Posts: 328
Joined: May 2006
 
Posted: 2006-07-08 06:15

what about pairs trading if one of the securities is exactly 2x the volatility...buy more volatile stock and short less volatile on down days or as spread widens.

thoughts?


Speculator

Mars


Total Posts: 336
Joined: Aug 2005
 
Posted: 2006-07-08 08:50
I think this question was already asked in another thread.

NÄ ES TÖ. TÖ ES NÄ. Nikito Nipongo.

Baltazar


Total Posts: 1764
Joined: Jul 2004
 
Posted: 2006-09-04 16:08
I looked at the site mentionned by NIP (The phrenchy that work on that)

He has a good presentation that may get some ligth on the topic.

The exemple he describes for the "Volatility Pump" does NOT involve a stock doing +100% or -50% with equal prob (as tonyc pointed out).
The exemple goes as follows the stock can do +100% or -66% with equal probability.
This time (compared to the +100/-50) you cannot just go long and wait until a lucky string of +100, because the value is really decreasing.

Of course expectation is positive (1/6)

Using the kelly criterion, one compute k= (1/6) / (1*2/3) =25% (gain is +1 and loss is -2/3)
Rebalancing so that every day you have 25% of your wealth invested in that stock gives an average daily gain of 2% (according to Herlemont paper)

So you make some money on a stock even if you are always long in it and it is going down.

Why isn't the technique used so much that the excess vol is sucked away?
It seem to be very demanding computationaly speaking.

regarding the fraction on the kelly to use:
He relates the fraction of the kelly to use to the max drawdown we are ready to suffer:

He says (not clear if it is a general assertion or just for his problem)
Kelly implies a probablity(drawdown bigger then 50%) of approx 50%
for kelly/4 probablity(drawdown bigger then 50%) of approx .1%
and probablity(drawdown bigger then 10%) of approx 50%

Regarding convergence
He says that one can use the 1/sharpe^2 relation to have an estimate time before witch the algorithm should converge.
In the exmple he proposes, sharpe is about 3 wich gives a very fast "convergence".

He is very ligth on details as one can except though.

Qui fait le malin tombe dans le ravin

TonyC
Nuclear Energy Trader

Total Posts: 1245
Joined: May 2004
 
Posted: 2006-09-07 18:21
> regarding the fraction on the kelly to use:
> He relates the fraction of the kelly to use to the max drawdown we are ready to suffer:

> He says (not clear if it is a general assertion or just for his problem)
Kelly implies a probablity
> (drawdown bigger then 50%) of approx 50% for kelly/4 probablity(drawdown
> bigger then 50%) of approx .1% and probablity(drawdown bigger then 10%) of approx 50%

Baltazar,

gee, i didnt see any of that

was all that in the phrench papers i couldnt read, or am i so old and senile i just missed it while reading the english papers?

could you provide the paper's title?

TonyC

flaneur/boulevardier/remittance man/energy trader

kr
Founding Member
NP Raider
Total Posts: 3561
Joined: Apr 2004
 
Posted: 2007-03-10 09:59

I wanted to bump up this thread again, because I hadn't done a proper study the first time round and somehow the issue has popped to the top of my subconscious stack.  I had a reread of the initial Cover paper, and had some more nontechnical thoughts about it - that were addressed below but I didn't fully appreciate them before.

This is also connected with my reading of Swensen's Uncommon Profits, and bonus season.  I wanted to do more p.a. investing, and do it more systematically.  I will admit that I don't have the time to do proper analysis of everything I buy, and at the same time I don't think I'd be satisfied with outright indexing the way Swensen suggests.  It is the memory of the investment results of my very first bonus - I put a bunch in the big indices, and didn't do so well. 

As a starting point, I do take Swensen's point that one should be cautious about frictions, so my initial risk set will be ETFs and/or currencies.  The ETFs idea is obvious; the currencies point is that since I've moved to London I'm not so sure what my 'measurement currency' ought to be, so I might as well just try to do the best of all ccys. 

In Cover's UP paper, there is a lot of effort devoted to getting the asymptotics analysed, on a completely nonparametric basis.  But I think this obscures the main substance of the paper, which is to achieve the performance of the best-growing constant-weighted portfolio.  This is a definition of the challenge and not a lemma, proposition or theorem.  You'll see that there is some funny language about this point in the paper which 'justifies' it. 

You have to appreciate that this assumption isn't all that close to practical life.  Most people rebal all the time (to their disadvantage, Swensen would argue) - so 'constant weighting' is a strong assumption.  Secondly, most people don't like drawdowns or volatility - so the best-growing assumption is pretty far off of most peoples' utility functions.  Both of these points are addressed separately in the thread below. 

If you take the assumption out of it, then UP is an algorithm but it's certainly not the only one, and even though Cover says it is nonparametric, there are some tacit assumptions about the process that sneak into the results.  On the first point, you could just search at every stage for the best constant-proportion portfolio.  Using his integration process is nice and yes it converges, but you could really just search and at the end you'd get the right result.  You could also use a different weighting scheme, like only averaging over relatively nearby states, and you'd also get the right answer even though the convergence might be a good deal slower.  On the second point, the thing is really that the growth in the shares is supposed to be of an exponential structure, which means that all his inequalities can be proven.  That is a parametric assumption, which is not unrealistic of course, but is substantial. 

Apart from ETFs and things, I want to run some scenarios on specific processes.  There is already one I can be assured will fail:  Let x_i be a Petersburg-type process with growth lambda_i.  Think credit derivatives here:  Your collateral grows from 1 to 1 + lambda_i . dt most of the time, except when it goes to 0 with probability lambda_i . dt.  Intuitively, UP will search out the credit-riskiest asset, which will look good until you lose almost all of your capital.  Maybe Kin Ark didn't behave this way, but tech stocks did (except GOOG!)  and I do wonder how UP would have fared over the techs in 1999-2003.  That is an experiment I can run.

Here's another:  Cyclicality.  Let x_t = exp(sin(lambda_i . t + alpha_i)), for lambdas and alphas distributed uniformly in some interval.  By construction, this mess is going nowhere.  Now we are very dependent on the UP updating scheme structure.  The best-performing constant-proportion portfolio will be flat; does UP do any better?  My point here is more on the constant-proportion bit... for myself, I really hope to capture cyclicality through intelligent rotation.  At the very least I'd hope to get the 'dollar-cost-averaging', which is a bit like UP but in a way I don't fully comprehend. 

Of course anybody who knows Lebesgue integration might think ok, you can do it for constant-proportion, then isn't the best portfolio (apart from risk) the sup over all portfolios that are 'piecewise constant-proportion'?  I think the risk part is a bit easier, but the optimisation is tougher.  If instead we look at a utility function which is a function of the investment path (i.e. so it can capture volatility, max drawdown etc), then I don't at all see how Cover's algo can be extended.  Ok, let me restate that:  I don't see that it would necessarily converge without a whole bunch more assumptions.  Just as a full-search might work (as I proposed above), some weighted-average type 'Cover' search might work, but now the whole thing depends on the structure of the utility pretty significantly. 

Anyhow, I am really going to try this out w/ real money at some point but I need to get comfortable.


my bank got pwnd

kr
Founding Member
NP Raider
Total Posts: 3561
Joined: Apr 2004
 
Posted: 2007-03-12 22:33

Now I've done some first-cut foundations work which are worth copying out here.

My list of ETFs is SHY, TLT, GLD, USO, IYR, DIA, SPY, QQQQ, MDY, IWN, IWO, EFA, EEM.  USO and GLD I may drop for lack of data.  We are looking at monthly performance as well, since in reality I can't justify more frequent rebalancing. 

I think this is a decent list to start from.  Average pairwise corr is only 38% - suprised me a bit how low it was, means that the Markowitz-optimal should be somewhat interesting. 

The first thing I noticed was that even with a decent-sized piece of capital for the p.a., it will be hard to do more than binary allocation.  For instance you will need something like USD120k to get a round lot of all of the above.  The implication for the UP algorithm is pretty serious - we're all worrying how long it will take to compute the multidim integral, when in fact we only need to do a handful of points in any given dimension.

The next thing I reminded myself of was the truly crap performance of QQQQ in the wake of the tech collapse (2002).  If you used a constant-rebal strategy (which you'd even need to do with Markowitz, as long as you didn't update your means and variances), you'd be adding cash to a sinking strategy in a bad way.  I suspect the drawdown during this phase must be significant.  Just as UP picks the best-returning stock, it may take a strong exposure to the worst-trending stock.  On that basis I suspect UP is significantly inconsistent with a drawdown-resistant strategy, which is an interesting thought.

Another thing goes to the example of the stock that alternates between +50% and -50%, where the UP strategy makes money.  It doesn't take much thought to see that UP wins here precisely b/c the stock performance has a strong mean-reversion component to it.  So it may be that again UP wins partly b/c it bets on mean-reversion during times of significant market stress.  That's a strategy, but it seems to be a tough one to play without options. 

With these ETFs I want to really find the answer here - do we see that the best performers all have the biggest drawdowns or integrated-realized-vols?  To be contrarian, can you choose the weights so that the portfolio vol remains nearly constant throughout?


my bank got pwnd
Previous Thread :: Next Thread 
Page 3 of 4Goto to page: 1, 2, [3], 4 Prev Next