Intro

I built an automated retraining loop for a spam classifier behind an API. Watch the inputs for drift, train a challenger, score it against the champion on the same labelled rows, promote it if it is clearly better by moving an alias.

Every real bug I hit had the same property. Nothing raised errors and nothing logged an error.

1. Accuracy on an imbalanced problem

The data is about 5 800 messages, 83.4% non-spam (ham). I ran the model on deliberately corrupted inputs to see how far it fell. Accuracy went 0.98 to 0.87, which reads like a model that mostly still works.

In reality, it caught 4 spam messages in 100. Accuracy stayed high because predicting "ham" for everything scores 0.834 on this data. The model collapsed onto the majority class and accuracy rewarded it.

The same trap one level down: MLflow's autolog reports F1 with average="weighted", which read 0.77 on that collapsed model. Macro F1 read 0.49 on the same predictions.

2. A drift score that could not see the drift

I first wrote the detector by hand. It used seven numeric descriptors for each message. For each descriptor, I calculated the Population Stability Index (PSI). I created the bins from quantiles of the reference data.

On known-corrupted inputs, the detector reported PSI 0.000 for uppercase share. This was a problem. The reference data had zero uppercase letters in 6.5 million characters. The corruption added capital letters everywhere. Uppercase share should have been the strongest signal.

The problem was the quantile bins. When a column has a constant value, all quantile bins collapse to one edge. All values fall into the same bucket in both datasets. The distributions look identical, so PSI is zero.

The metric could not see the change in that column. It reported the same 0.000 for both corrupted data and genuinely stable data.

Evidently scored the corrupted data 4.417.

Running clean data against clean data helped set the threshold. It made the threshold a measurement instead of a guess.

3. Two models, two different test sets

The first comparison used 290 rows for the champion and 2,893 rows for the challenger. It then printed the difference as an improvement.

Both numbers were correct. The subtraction was not meaningful because the models used different test sets.

I changed the gate to raise an error when the two metric dictionaries have different support.

4. Leakage through a concat

Retraining uses both original and corrupted text. A model trained only on corrupted text forgets how to read plain messages.

My first version joined both copies and then split the data into train and test.

This caused data leakage. Every test row had a matching row in the training set. The corruption leaves many unique tokens unchanged, so the vectorizer could recognise them.

The challenger's score was higher than it should have been. I could not measure how much higher. The champion did not get the same boost. The minimum required improvement was 0.02.

Split the data first. Then corrupt each side separately.

5. Drift that never goes away

The drift check compares live inputs with a stored reference. The reference describes the data used to train the champion.

After a promotion, the new champion was trained on corrupted data. But the reference still described the original data. The drift check then reported drift forever.

The calculation was correct but the system would retrain forever.

The fix is to move the reference when the champion changes. I store the descriptor data as an artifact of the run that produced each model version. Then I find it through the @champion alias.

This removes the need for a separate reset step that someone might forget.

I still got this wrong the first time. I stored the wrong data.

The challenger trains on original and corrupted text. I stored a reference describing that mix.

But live traffic is entirely corrupted. A mixed reference still reports drift.

The reference must describe what live traffic looks like, not what the model was trained on.

Drift detection is the cheap part

Drift detection needs no labels. That is why it exists. The detector is only a few descriptors and a library call.

The expensive parts are:

• Getting labelled examples from the new data.
• Deciding what "clearly better" means.
• Keeping the reference in sync with the champion.