I. System Architecture
The Hybrid Cloud Topology: Optimizing for both Latency and Capacity.
The Hybrid Physical Topology
We cannot run everything in the Cloud due to latency, nor everything On-Prem due to cost. The architecture is split into two distinct zones connected by a secure leased line.
Zone A: Co-location (On-Prem)
NY4 / NJ2 Data Centers
Directly cross-connected to exchanges. C++ 20, FPGA, < 5µs Latency.
Zone B: The Cloud (AWS/GCP)
Elastic Compute
For research, massive data storage, and non-latency sensitive tasks. Python, Kubernetes, Petabytes.
Logical Microservices
The monolith is dead. The system is composed of specialized, single-responsibility engines communicating via high-performance messaging buses (Aeron / ZeroMQ).
Ticker Plant
- Protocol Normalization (SBE/FIX → Structs).
- Book Building (L3 → L2 → L1).
- Multicast Distribution to Strategy Engines.
Alpha Engine
- Receives normalized market data.
- Computes features & runs inference.
- Emits "Target Portfolios" (Not orders!).
Risk Sidecar
- Pre-trade checks (< 5µs).
- Fat-finger limits.
- Kill switch functionality locally.
Smart Router (SOR)
- Takes target portfolio diffs.
- Slices orders (TWAP/VWAP/POV).
- Routes to specific venues (Dark vs Lit).
The Network Stack (Latency War)
In the co-location zone, the OS kernel is the enemy. Context switches cost microseconds. We bypass the kernel entirely using Solarflare/Mellanox (mapping NIC memory directly to user-space) and isolate specific CPU cores, disabling interrupts to prevent jitter.
II. The Data Foundation
The 3-Tier Storage Model optimizes for access patterns. Trading engines need nanoseconds; researchers need terabytes.
Tier 1: Hot (kdb+ / Redis)
In-memory. Last 24 hours of ticks. Used for live signals and real-time dashboards.
Tier 2: Warm (Parquet / Delta)
NVMe SSDs. Recent history (5 years). Used for daily retraining of models.
Tier 3: Cold (S3 Glacier)
Object Store. Full history (20+ years). Used for deep-dive research.
Point-in-Time Correctness (Bitemporality)
Crucial for fundamental data. If a company restates earnings, the database changes, but your backtest must know what the value was on the trading day, not what it is now. We store both the "Valid From" (event time) and "Transaction Time" (knowledge time).
Level 3 Order Book (Microstructure)
We don't just use price (L1). We reconstruct the full book (L3) from raw multicast messages (Add/Modify/Delete) to see the queue position of every order.
III. Machine Learning Design
Modern quants use a hybrid approach, combining the interpretability of tree-based models with the feature-extraction power of deep learning.
TabNet & ResNets
Neural networks adapted for tabular data. Attention mechanisms select relevant features instance-wise.
Graph Neural Networks (GNNs)
Used for supply chains. Stocks are nodes, relationships are edges. Shocks propagate through the graph.
Transformer Encoders
Applied to time-series (replacing LSTMs). Uses "Time2Vec" encodings to capture microstructure periodicities.
Meta-Labeling
One model predicts the Side (Long/Short), while a secondary meta-model predicts the Probability of Success (Bet Size) to filter false positives.
Differentiable Sharpe Ratio Loss
Standard loss functions optimize for accuracy, but we care about risk-adjusted returns. We implement differentiable versions of financial metrics directly into the neural network's backpropagation.
Sharpe Optimization Objective
IV. Backtesting & Simulation
Retail backtests often use "Vectorized" calculations, which are fast but prone to look-ahead bias. Professional backtests use an Event-Driven loop that processes data one tick at a time, exactly mimicking the live execution environment. Your backtester code should share the same core logic as your live execution code.
Transaction Costs (Implementation Shortfall)
The difference between "Paper Returns" and "Live Returns" is cost. We model this using the Square-Root Law of market impact.
Total Cost Model
The 3 Sins of Simulation (Bias Detector)
Survivorship Bias
Testing only on stocks that exist today, missing failed companies like Enron. Fix: Use 'Delisted' datasets.
Look-Ahead Bias
Using data not available at the time of trade (e.g., trading at Open using the day's High). Fix: Lag all data.
Restatement Bias
Using current Macro data for past dates despite revisions. Fix: Point-in-Time (PIT) databases.
V. Risk & Convex Optimization
We don't pick stocks; we pick weights. The optimizer solves for the optimal weights $w$ that maximize expected return minus a risk penalty, subject to constraints.
Mean-Variance Objective
Constraints
Leverage Limits
Gross Leverage: Sum of absolute weights. Typical Limit: 2.0x to 4.0x NAV. Limits blowout risk.
Turnover Constraint
Transaction costs eat alpha. Limits absolute weight changes between days to throttle trading speed.
Neutrality
Dollar Neutral (Longs=Shorts), Beta Neutral, and Sector Neutral limits macro exposure.
Tail Risk: VaR vs. CVaR
Expected Shortfall
While Value at Risk (VaR) only tells you the 95th percentile worst outcome, Expected Shortfall (CVaR) averages the losses in the tail. We optimize for the "average day," but we survive based on CVaR. Furthermore, CVaR is mathematically convex, allowing it to be natively embedded in our optimization solver.