The Hostile Environment of Financial Markets
Applying machine learning to quantitative trading is an endeavor fundamentally different from its application in other fields. While domains like image recognition benefit from stable patterns and high signal-to-noise ratios, financial markets are characterized by a low signal-to-noise ratio, non-stationarity, and adversarial dynamics.
The core assumptions that grant power to many algorithms—stationarity, independence, and normality—are systematically violated. Acknowledging these violations, known as “stylized facts,” is the first and most critical step in moving from academically elegant but practically useless models to ones that have a chance of being robust in live markets.
Violation 1: The Illusion of Stationarity
Stationarity implies that a time series' statistical properties (mean, variance, autocorrelation) are constant over time. Asset prices are almost never stationary. They exhibit trends, cycles, and structural breaks driven by evolving macroeconomic conditions, technological changes, and shifts in market sentiment.
The primary model for this behavior is the random walk, where the next price is the current price plus an unpredictable shock:
Unit Root Process
This is a process with a unit root, meaning that shocks have a permanent effect on the price level. Models that assume stationarity will fail because they learn relationships that are specific to a particular regime and do not generalize.
Practical Solution
The practical solution is to work with asset returns, which are typically closer to stationary:
Violation 2: Volatility Clustering & Heteroskedasticity
While asset returns themselves show little serial correlation, their volatility is highly persistent. This phenomenon, known as volatility clustering, means that large price changes are often followed by more large changes, and small changes are followed by more small changes.
ARCH Model Framework
This behavior is formally captured by Autoregressive Conditional Heteroskedasticity (ARCH) models, which model the variance at time t (σ²t) as a function of past squared error terms:
Violation 3: Non-Normality (Fat Tails)
Many statistical models assume that data, or at least the error terms, are normally distributed. Financial returns, however, exhibit leptokurtosis, meaning they have “fat tails.” Extreme events (market crashes, massive rallies) occur far more frequently than predicted by a normal distribution.
The kurtosis of a distribution measures this “tailedness.” For a normal distribution, kurtosis is 3. Financial returns often have a kurtosis significantly greater than 3.
Kurtosis Formula
Risk Warning
Models assuming normality will drastically underestimate the probability of extreme losses. This is a primary reason why risk management systems based on Gaussian assumptions often fail during market crises.
Assumptions of Tree-Based Models
Random Forest & Gradient Boosting
Tree-based models are popular in quantitative finance due to their ability to capture non-linear interactions in tabular data. They are non-parametric, meaning they don't assume a specific functional form for relationships.
Implicit Assumption: Stationarity of Relationships
While a tree model doesn't assume input features are stationary, it implicitly assumes that the relationships it learns between features and the target are stable over time.
Limitation: Inability to Extrapolate
A decision tree's prediction is always the average of target values within the terminal leaf. It can never predict a value outside the range seen in training data.
Assumptions of Deep Learning Models
RNN & LSTM Networks
Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are explicitly designed for sequence data. Their core assumption is that the order of data matters and that past information is relevant for predicting the future.
Challenge: Overfitting in Financial Data
Their complexity and flexibility make them highly prone to overfitting the noise in financial data. Without very large datasets, careful regularization, and robust validation techniques, an LSTM can easily memorize spurious patterns.
Comparative Framework: Model Trade-offs
| Model Type | Primary Assumption | Handles Non-Stationarity? | Key Strength | Primary Risk |
|---|---|---|---|---|
| MLP (Feedforward) | Static, non-linear relationships. Assumes IID data. | No. Requires stationary features. | Universal function approximator for complex patterns. | Ignores time series dynamics; prone to overfitting. |
| Tree-Based (RF, GBM) | Hierarchical, non-linear relationships. No functional form assumed. | Implicitly. Assumes learned rules are stable. | Excellent with tabular data, robust to outliers. | Alpha decay; cannot extrapolate beyond training range. |
| RNN / LSTM | Data is a sequence where order matters and past informs future. | Partially. Can model trends but assumes stable dynamics. | Explicitly designed for time series and temporal dependencies. | Extremely prone to overfitting noise; computationally expensive. |
Strategic Recommendations
For Practitioners
- •Always test for stationarity using ADF tests
- •Work with returns, not prices, when possible
- •Include volatility regime features in models
- •Use robust loss functions that handle outliers
- •Implement walk-forward validation techniques
Model Selection Guidelines
- •Tree models: Best for cross-sectional factor analysis
- •LSTM: Use only with very large datasets and careful validation
- •Ensemble methods: Combine multiple model types for robustness
- •Regular retraining: Account for regime changes
- •Out-of-sample testing: Essential for realistic performance estimates