Live Training: Models That Keep Learning From Their Own Traffic

How It Works

Live training is a per-pipeline switch in the admin. With it on, every prediction the pipeline serves is appended to each step's dataset, labeled with the step's own output, and after retrainEvery new rows the step rebuilds itself from the complete dataset. The default interval comes from the config and each pipeline can set its own, so a busy pipeline can retrain every 500 rows while a quiet one retrains every 20.

Everything stays visible along the way. The appended rows land in the same jsonL dataset as hand-added ones, readable in the admin table and on disk, so you can always audit exactly what the model has been learning from, prune rows you want gone, and retrain by hand whenever you feel like it. Predictions stay fast too: serving the answer is the priority, and a live retrain that hits a snag simply reports a notice while the prediction returns as normal.

Where Live Training Shines

The switch fits jobs where the data drifts and there is no single right answer to curate:

• Feedback clustering. A K-Means pipeline over incoming reviews keeps its groups current as topics rise and fade, so this month's clusters describe this month's feedback.
• Anomaly baselines. Traffic grows, patterns shift, and an anomaly detector that keeps learning treats the new normal as normal, alerting on genuinely strange days instead of last quarter's definition of busy.
• Engagement and behavior models. Audience behavior moves with seasons, campaigns, and product changes, and a self-feeding model moves with it.

This is the self-learning AI idea in its most concrete form: a system that gets better every day it runs, on your own server, with the training data in plain sight.

When to Leave It Off

Keep the switch off when you want full control of the training data, and the engine makes curated training just as smooth. Classifiers with exact right answers, spam or not, billing or technical, do their best learning from confirmed labels, so the strongest pattern there is correction-driven: your app calls dataset/add with the verified label when the truth is known, a ticket lands in its final queue, a human confirms the spam report, and retrains on that gold standard data. Both patterns use the same dataset machinery, live training just automates the feed.

Tuning the Interval

The retrainEvery value is a rhythm knob. Small values keep the model maximally fresh and suit low-traffic pipelines where every row carries signal. Larger values suit high-traffic pipelines, retraining once per batch of new experience. Training always rebuilds from the complete dataset, so whatever the interval, the model after each retrain is exactly what the visible dataset says it should be.

Watching It Learn

The tools for supervising a live pipeline are the same ones you already use: the dataset table shows what traffic has been teaching it, the test box shows what it currently thinks, and the pipelines endpoint reports training status to outside systems. A pleasant weekly habit is one look at a live clusterer's groupings, five minutes that often surfaces a new theme before any human has read enough messages to spot it, and pairing the pipeline with our Workflow Chain Engine can turn that check into an automated report.