features›transaction_features.py
transaction_features.py
Hints
Imagine a recipe developed in a test kitchen. The chef weighs every ingredient on a digital scale, times each step to the second, tastes and adjusts before writing anything down. The finished recipe gets typed up and sent to a restaurant.
At the restaurant, during a Friday dinner rush, nobody has time to weigh flour on a scale. The line cook eyeballs a scoop, glances at a timer instead of watching it, and moves on to the next ticket. They're following the same recipe. On paper, it's identical. In practice, it isn't the same dish.
That's what happens when a machine learning model's inputs, called features, get computed one way while the model is being built, and a different way once the model is actually answering real requests. Same feature name. Same intention. Different arithmetic underneath.
"Average transaction amount, last 30 days" sounds like one well-defined number. But define what happens when a customer has no transactions yet, and you've quietly defined two different features that just happen to share a name.
training: avg_amount = df['amount'].fillna(df['amount'].mean()) serving: avg_amount = transaction_cache.get(user_id, default=0.0)
Same column name. Same intention. A missing value quietly becomes the dataset's average in training, and exactly zero the moment the model runs in production.
So far, that's the pipeline working as intended: the same features feed the model whether it's learning or answering. This concept starts one step later, at the moment "the same features" turns out to mean two different pieces of code.
A feature is a name plus a computation, not a name plus a meaning. The model never sees "average transaction amount." It sees whatever number lands in that slot. It has no way to check that the number was produced the same way it was during training. If two different computations produce the number under the same label, the model can't tell the difference, because there isn't one from where it's sitting.
Training and serving are naturally different code paths. Training runs offline, in a batch job, with patient access to the full historical dataset and nobody watching the clock. Serving runs online, has to answer in milliseconds, often can't afford a slow database join, and is frequently written by a different team, sometimes in a different language entirely. Nothing forces those two pipelines to agree on what "missing" means, or how to fill it in.
The divergence is invisible to normal testing. The training pipeline gets tested against training data. The serving pipeline gets tested against serving inputs. Both pass. Neither test ever runs both pipelines side by side on the same real-world input to check whether they agree, because that's not what either test was written to do.
The trap: the model trains fine, validates fine, and looks great on every training-set metric, because training never runs the serving code at all. Skew doesn't show up as a bad accuracy number during development. It shows up later, quietly, on requests that never touched the training pipeline in the first place.
Don't take our word for it. Watch the same customer get two different numbers.
Pick a customer. Both pipelines compute the same named feature from the same underlying record. Watch whether they agree.
Select Customer B and see the two pipelines disagree on the exact same feature to unlock the problem stage.