haakhtar


Total Posts: 3 
Joined: Feb 2020 


So I've been working on a model using WLS and one of the problems I run in to is unstable estimates of the regression parameters. I was wondering if anyone had any idea about ways to mitigate this? 



gaj


Total Posts: 112 
Joined: Apr 2018 


A few possibilities off the top of my head, 1) there's not enough data, 2) the features are bad, 3) you have highly correlated features.
1 and 2 are kind of related. If you have bad features but you have a lot of data, the regression should give zero weights to those features. You can check the pvalues and confidence intervals of the regression parameters to diagnose this problem. You should get rid of features that have no predictive power. You can also use shrinkage methods like ridge regression to make the coefficients more conservative and hence more stable.
For 3, if you have several features that are very highly correlated, the coefficients will blow up. The ideal solution is to construct the features to be uncorrelated in the first place. Sometimes this is not possible, so you could try some dimension reduction techniques like PCA. 


bullero


Total Posts: 70 
Joined: Feb 2018 


Most likely, you have a near singular correlation matrix.
notes: http://www.stat.cmu.edu/~larry/=stat401/lecture17.pdf 



haakhtar


Total Posts: 3 
Joined: Feb 2020 


Thanks for the info bullero & gaj, I'll do some more reading according to your suggestions. 

