Backtesting

Backtesting is a term used in oceanography, meteorology and the financial industry to refer to testing a predictive model using existing historic data. Backtesting is a kind of retrodiction, and a special type of cross-validation applied to time series data.

Financial analysis

In a trading strategy, investment strategy or risk modeling, backtesting seeks to estimate the performance of a strategy or model if it had been employed during a past period. This requires simulating past conditions with sufficient detail, making one limitation of backtesting the need for detailed historical data. A second limitation is the inability to model strategies that would affect historic prices. Finally, backtesting, like other modeling, is limited by potential overfitting. That is, it is often possible to find a strategy that would have worked well in the past, but will not work well in the future.[1] Despite these limitations, backtesting provides information not available when models and strategies are tested on synthetic data.

Backtesting has historically only been performed by large institutions and professional money managers due to the expense of obtaining and using detailed datasets. However, backtrading is increasingly used on a wider basis, and independent web-based backtesting platforms have emerged. Although the technique is widely used, it is prone to weaknesses.[2] Basel financial regulations require large financial institutions to backtest certain risk models.

Hindcast

Temporal representation of hindcasting.[3]

In oceanography[4] and meteorology,[5] backtesting is also known as hindcasting: a hindcast is a way of testing a mathematical model; known or closely estimated inputs for past events are entered into the model to see how well the output matches the known results.

Hindcasting usually refers to a numerical model integration of a historical period where no observations have been assimilated. This distinguishes a hindcast run from a reanalysis. Oceanographic observations of salinity and temperature as well as observations of surface wave parameters such as the significant wave height are much scarcer than meteorological observations, making hindcasting more common in oceanography than in meteorology. Also, since surface waves represent a forced system where the wind is the only generating force, wave hindcasting is often considered adequate for generating a reasonable representation of the wave climate with little need for a full reanalysis. Hindcasting is also used in hydrology for model stream flows.[6]

An example of hindcasting would be entering climate forcings (events that force change) into a climate model. If the hindcast showed reasonably accurate climate response, the model would be considered successful.

The ECMWF re-analysis is an example of a combined atmospheric reanalysis coupled with a wave model integration where no wave parameters were assimilated, making the wave part a hindcast run.

In 2003, Dake Chen and his colleagues “trained” a computer using the data of the surface temperature of the oceans from the last 20 years.[7] Then, using data that had been collected on the surface temperature of the oceans for the period 1857 to 2003, they went through a hindcasting exercise and discovered that their simulation not only accurately predicted every El Niño event for the last 148 years, it also identified the (up to 2 years) looming foreshadow of every single one of those El Niño events.[8]

See also

References

  1. ↑ BacktestBroker. "Does Backtesting Really Work?".
  2. ↑ FinancialTrading. "Issues related to back testing".
  3. ↑ Taken from p.145 of Yeates, L.B., Thought Experimentation: A Cognitive Approach, Graduate Diploma in Arts (By Research) dissertation, University of New South Wales, 2004.
  4. ↑ "Hindcast approach". OceanWeather Inc. Retrieved 22 January 2013.
  5. ↑ Huijnen, V.; J. Flemming; J. W. Kaiser; A. Inness; J. Leitão; A. Heil; H. J. Eskes; M. G. Schultz; A. Benedetti; J. Hadji-Lazaro; G. Dufour; M. Eremenko (2012). "Hindcast experiments of tropospheric composition during the summer 2010 fires over western Russia". Atmos. Chem. Phys. 12: 4341–4364. doi:10.5194/acp-12-4341-2012. Retrieved 22 January 2013.
  6. ↑ "Guidance on Conducting Streamflow Hindcasting in CHPS" (PDF). NOAA. Retrieved 22 January 2013.
  7. ↑ Chen, D., Cane, M.A., Kaplan, A., Zebiak, S.E. & Huang, D., "Predictability of El Niño Over the Past 148 Years", Nature, Vol.428, No.6984, (15 April 2004), pp.733-736; Anderson, D., "Testing Time for El Niño", Nature, Vol.428, No.6984, (15 April 2004), pp.709, 711.
  8. ↑ Not only did their hindcasting demonstrate that the computerized simulation models could predict the onset of El Niño climatic events from changes in the temperature of the ocean's surface temperature that occur up to two years earlier — meaning that there was now, potentially, at least 2 years' lead time — but the results also implied that El Niño events seemed to be the effects of some causal regularity; and, therefore, were not due to simple chance, or to some other “chaotic” event.
This article is issued from Wikipedia - version of the 11/8/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.