The problem
Sports prediction APIs fail in predictable ways. Model quality degrades silently — accuracy drops before any infrastructure alert fires, and by the time users notice, the damage to trust is done. The harder failure mode: a model that performs well in a notebook becomes unreliable under live traffic when feature distributions shift mid-season or inference latency creeps above the window where predictions are still useful.
SabiScore needed to solve both problems: deliver accurate predictions during live match windows, and detect model degradation before users experience it.
What was built
An end-to-end ML platform with three distinct layers:
Inference layer: FastAPI serves ensemble gradient-boosted predictions (XGBoost, LightGBM, CatBoost) with Redis Pub/Sub for concurrent sessions. A stale-while-revalidate strategy is keyed by model version so predictions from an old model cannot survive a retrain.
Observability layer: Prometheus tracks Brier score, PSI (Population Stability Index), prediction latency, and cache behavior per model version. The alerting path compares quality signals with an established baseline — a model performance signal, not only a system-health check.
Dashboard: Next.js 15 interface surfacing predictions with legible confidence signals and a model health view. The product communicates trust before it explains capability.
Key decision: FastAPI + Redis Pub/Sub over synchronous REST with polling
Chosen: FastAPI + Redis Pub/Sub for inference serving
Over: Synchronous REST endpoints with database polling
Because: Synchronous polling creates a thundering herd against the database as concurrent sessions request the same update. Redis Pub/Sub fans out from one published event, while the reconnect path gives temporarily disconnected subscribers a defined recovery boundary.
Key decision: Time-based train/test splits
Chosen: 80th-percentile date as the train/test boundary
Over: Random shuffle cross-validation
Because: Sports outcomes are non-stationary. Team form, injury status, and competition stage change over time. Random shuffling allows future match statistics to leak into training data — the model learns from information it would not have had in production. A model trained with random shuffle will report better CV accuracy and underperform in production. Time-based splits produce a model that generalises forward, which is the only direction that matters.
Key decision: PSI-triggered unscheduled retraining
Chosen: PSI threshold monitoring with unscheduled retraining runs
Over: Fixed weekly retraining cadence only
Because: Tournament windows produce feature distributions that look nothing like regular-season distributions. A weekly-only cadence would tolerate weeks of degraded predictions during exactly the periods when user demand is highest. PSI drift detection triggers an unscheduled run when the incoming feature distribution crosses the threshold — false positives cost compute time, false negatives cost prediction quality at peak visibility. The asymmetry favours the conservative threshold.
Constraint
Ensemble inference must remain useful under concurrent load and through model transitions. The Redis cache layer must invalidate by model version so stale predictions do not survive a retrain.
The cache key strategy handles this: keys are scoped to match_id + model_version. On retrain, only keys containing the old model_version expire. Predictions for unchanged matches remain cached. The retrain does not flush the entire cache.
Evidence record
- Public source — the repository exposes the system structure and implementation history
- Model-version boundary — cache keys make retraining invalidation inspectable in code
- Degraded behavior — dependency failure returns a labelled lower-confidence baseline rather than stale output
- Private measurements — uptime, latency, and detection reports are not presented as public proof without their underlying artifacts
Lessons
The most valuable production pattern was the fallback path. When Redis is unavailable or the feature store is slow, the system degrades deliberately: it serves a lightweight single-model prediction from a pre-computed baseline, with a visible confidence reduction in the response payload. Users see a lower-confidence score rather than an error.
Reliability is usually decided in the fallback design, not in the training notebook.
Status
The public demo is available at sabiscore.scardubu.dev. The source repository and architecture record are the public evidence paths; operational measurement artifacts are not published here.