AI agents: what to watch after the agent is already working for you

AI agents is what this article is about, even when the technical tool has another name. For a renovation contractor the day looks like this: quotes in WhatsApp, photos in a camera roll, and job status in the foreman’s head. The agent sorts incoming messages, drafts a reply and marks what is urgent. It does not send anything to a customer before you approve it. The message lands in the information system, the agent suggests a plain reply, and you approve or correct it in one pass. The agent sits on the information system and on the WhatsApp bot. Without the system it answers in a vacuum. Without the bot the customer stays in a separate chat. The agent sees only the files of the role you approved, not the whole business. The agent does not close a deal, change a price or delete a record. A person keeps those. The rest of the article is how a software house builds that without leaving the team with a demo.

Machine learning in production is operations under uncertainty. Offline metrics prove the model can perform in the past; production proves it can keep performing when users, vendors, seasonality, and logging pipelines change. MLOps is the discipline that prevents “winner notebooks” from silently rotting once traffic arrives.

Start from ownership: who is accountable for model quality after deploy? Without a named owner, inference pods become pets nobody feeds. Product, data science, and platform engineering share responsibilities-clarify who monitors drift, who approves rollbacks, and who communicates customer impact.

Baseline and regime detection matter more than chasing a single accuracy number. Track business KPIs aligned to the model’s purpose (conversion lift, support deflection, fraud dollars) alongside traditional ML metrics. If business KPI diverges while offline accuracy looks fine, your label or attribution may be wrong, not the model.

Data drift and concept drift require different responses. Input drift may be benign (new marketing channel) or harmful (broken sensor). Concept drift means the relationship between features and outcome changed-often needing retraining, policy updates, or feature retirement. Monitoring feature distributions and prediction confidence can provide early signal.

Version everything that defines behavior: training code, preprocessing, feature definitions, hyperparameters, random seeds, and data snapshots. Artifact registries store models; metadata ties each artifact to evaluation evidence. When an incident hits, you need to diff models like you diff code.

Separate training from serving. Training stacks invite heavy dependencies; serving should be lean, predictable, and horizontally scalable. Containerized microservices behind autoscaling groups or Kubernetes deployments are common; serverless suits bursty, moderate-latency workloads when cold starts are tolerable.

Latency budgets shape architecture: batching GPU inference, quantization, distillation, and caching identical inputs help. For personalization, consider approximate nearest neighbors or two-stage retrieval-ranking to avoid scoring entire catalogs per request.

Online experimentation reduces regret. Shadow deployments score incoming traffic without affecting responses; canary releases expose a slice of users; champion-challenger loops compare candidates under real load. Always guard experiments with automated rollback on guardrail metrics.

Feature stores and feature pipelines align training with online inference. Inconsistent features are a top cause of silent degradation-when offline training sees aggregates updated hourly but online scoring uses stale caches. Document freshness SLAs and enforce lineage.

Fairness and misuse risk grow with impact. Before scaling automated decisions in hiring, credit, pricing, or safety contexts, invest in review with legal and ethics stakeholders, subgroup evaluation, and human escalation paths. Not every model warrants the same bar-match rigor to consequence.

Security extends to ML: model theft, adversarial inputs, and data poisoning are real threat models for exposed APIs. Rate limiting, input validation, and monitoring anomalous query patterns complement traditional appsec.

Cost governance belongs in MLOps. GPU hours, embedding recomputation, and log-heavy debugging can explode spend. Chargeback or showback by team, budgets on training clusters, and automated idle shutdowns keep science productive without surprise bills.

In summary: production ML is a living system-monitor reality, version artifacts, isolate serving, experiment carefully, and scale governance with impact. Reliability comes from boring rituals: dashboards that people read, rollbacks that are rehearsed, and diffs that explain change.

You can come back from here to a plain conversation: Bring ten real messages from this week, without names if you must, and the reply you would have sent yourself. In the morning you see a queue of drafts, not an endless inbox. What you approved yesterday already went out. The rest is waiting. AI agents, an information system and a WhatsApp bot stay the same offer, with human approval, from an official Meta provider.