hi pj

I am interested the same topic (specifically related to interest rates). I think a lot of it is acquired knowledge bordering on folklore.

I don't know if you will find any single reference although the usual books (Brigo & Mercurio, Rebonato ) have some discussion. I particularly like the discussion chapter at the end of B&M which takes the form of a question-and-answer with an imaginary trader.

There are one or two big expensive books on IR modelling in the last few years  (possibly by Gatarek and / or Andreasen?) which I haven't read but may be worth a look.

Regarding the subject of calibration, there are several steps that have to be carried out.

1. You have to have a well defined objective function. In case of Stochastic volatility LMM, for example, this could be sum of squared difference between model and market swaption prices in the swaption cube. This part is really very model specific and usually has to rely on closed form formulas that could be easily calculated very quickly.

2. Minimization of objective function is the second part and this has to rely on some kind of optimization technique. Though same optimization technique could work for various model defined objective functions, some optimization techniques work better for some problems and vice versa.

But I think when we usually talk about calibration it is usually more about choosing a good optimization algorithm. 

I have worked with several optimization algorithms ranging from those employing fancy mathematics to other very simple and common sense methods. I have found that simple and faster common sense methods usually work better than other methods using advanced mathematics though NOT ALWAYS.

One important thing that I learnt from experience in large scale optimizations involving for example several hundred parameters in case of SVOL LMM and may be 9-10 parameters in case of simple SVJD models is that one has to resist the urge to take large steps when moving towards minima. Very good optimization methods have a tendency to get into local minima and other problems which can many times be avoided by restricting the step size with step size uniquely chosen for particular problem by experimentation. If you think you have a very good minimization algorithm but it continues to get trapped into local minima, just try to restrict the step size and there is a good chance that it will result in a far more robust algorithm.

I have done extensive work on calibration oflarge scale  SVOL LMMs, SV and SVJD models and if you want more information you can email me which should be in my profile.

Vanilla instruments obviously should be the ones that you use for hedging the exotics. However there are other dynamics like correlations which cannot be fully translated into the model when you calibrate only to the instruments you use to hedge and that may require you to add additional instruments into calibration.

 If you only use hedge instruments in calibration, there can be several ways a model like LMM with many parameters, can be calibrated to market so you mayhave to add other instruments to pin down free variables. On the other hand short rate models may not be a perfect fit even to hedge instruments due to lack of free parameters. So you should ask yourself whether calibrated model fits to prices of hedge instruments and captures other market dynamics like correlations and skew, smile etc.

Choosing a good model is a study of fitting model using implied calibration and repeating over historic data and then you can see which model fares better.

Obviously if the purpose is only to fit market prices, you can just ask some brokers instead of getting into model development.

"but the issue is hedging performance. with MC your risk will be poor"

This is not true for all exotic instruments but for some it is true. But really you may not be able to find the risk of these exotic instruments by PDE either due to complexity that can only be tackled by MC.

"Add to that the daily calibration error of LMM and esp LMM heston"

I believe properly calibrated Heston LMM models have less calibration error than short rate models. Do you fit to market smile when pricing exotics with HW. Heston LMM does that quite nicely.