AI System Lifecycle: From Problem Framing to Inference and Monitoring

An AI system is not one training event

An AI-system lifecycle begins before model selection and continues after deployment. A useful foundational sequence is: frame the problem and objective; define data and inputs; develop or select components; evaluate before deployment; deploy and perform inference; then monitor, review, update, or retire as needed. Real work includes loops: evaluation may send us back to data or problem framing. The important point is not to jump from “we have data” to “deploy the model” without defining success and limits.

The NIST AI Risk Management Framework treats AI risk management as spanning design, development, deployment, and use rather than as one final test. We use that principle in simplified form: verification, evaluation, and monitoring are continuing responsibilities whose details vary by stage and risk.

Stage | Core question Problem framing | What decision or output is needed and who is affected? Data and inputs | Is the data suitable, permitted, and representative of the intended setting? Development/selection | What is the simplest suitable approach and what are its assumptions? Evaluation | Does it meet required measures and what errors appear? Deployment/inference | How does input reach the system and how is output used? Monitoring | Have data, performance, or use changed, and what happens when they do?

Training, inference, and monitoring: three functions

Training changes model parameters using data and a learning algorithm. Inference uses the current model version to produce a new output. Monitoring collects post-deployment signals: output quality, error patterns, data shifts, user behavior, incidents or complaints, and where relevant latency or cost. Monitoring does not automatically mean retraining. Findings may lead to a threshold change, interface change, additional human review, data update, retraining, or suspension of use.

If retraining is proposed, we need to define the new data, examine its quality and permitted use, preserve suitable evaluation data, compare the new version with the old one, and define deployment and rollback conditions. If the cause is a confusing interface, changing the interface may be better than retraining a model whose core performance has not changed.

What if we use a pretrained model?

Using a pretrained model or external service changes what we build internally, but it does not remove the lifecycle. We still need to frame the task, define inputs and what data may be sent, test outputs on cases representative of our use, establish privacy and security controls, define human review where needed, and monitor performance and changes. An external provider’s model update may make reevaluation more important because it changes a component we did not train ourselves.