Machine Learning and AI Engineering

The demo works. Is there a product here?

An interviewer gives you a plausible assignment:

A customer-support organization wants an AI assistant that drafts replies for
agents. The goal is to reduce handle time, but the system must not send
unsupported, unsafe, or policy-violating answers. Design the first production
version.

The tempting answer begins with a model, a vector database, and an API. Those components can produce an impressive demonstration while leaving the important questions untouched. Which tickets qualify? What evidence may the assistant use? How will anyone distinguish a retrieval failure from a generation failure? Who can stop the launch? What happens when a model upgrade makes the service faster and the answers worse?

Machine-learning and AI interviews concentrate senior judgment in this gap between a capable model and a dependable product. The behavior is statistical, the evidence changes, and a successful request can still return a harmful answer. The candidate therefore has to make uncertainty operable rather than pretend to remove it.

For the support assistant, the first useful decision is a restraint: it drafts for an agent; it does not send to the customer. That boundary creates a real first release. It also establishes what later evidence would have to justify a larger action.

Decide what may change for the user

“Reduce handle time” is a business hope, not yet a model task. Clarify the workflow before choosing the mechanism. Agents currently search approved help content, inspect account context, write a reply, and decide whether the case needs escalation. The first version may shorten the search-and-draft portion, but the agent remains responsible for the customer-facing action.

Now the outcomes can be named. Median handle time and time to first response matter, but neither is sufficient. A fast draft that agents rewrite completely has moved cost rather than removed it. A plausible draft that cites the wrong refund policy may reduce handle time by creating customer harm.

The launch therefore needs several kinds of evidence:

• whether the retrieved sources support the proposed answer;
• whether the draft follows current policy and avoids sensitive disclosure;
• whether agents accept, edit, reject, or escalate it;
• whether the workflow becomes faster without worsening reopen rate, customer satisfaction, complaints, or serious safety events;
• which ticket categories, languages, products, or customer groups carry the errors.

The exact measures depend on the organization. What matters in the interview is refusing to collapse them into one “AI quality” number. Quality, harm, latency, cost, and adoption answer different questions and may move in opposite directions.

A non-model baseline belongs here. Better search, approved response templates, or rules that route common tickets may capture much of the benefit with easier verification. Shipping one of those first is not a retreat from ML. It creates a comparison, improves the workflow immediately, and may collect cleaner examples for whatever learns next.

Build the evidence path before the serving path

The assistant can only be as current and authorized as its knowledge path. Start with a corpus inventory, not “embed everything.” Approved help-center articles and product documentation may be broadly usable. Internal policy pages need owners and effective dates. Resolved tickets can contain mistakes, customer secrets, or outdated exceptions; using them requires a deliberate selection and scrubbing policy.

Each source needs an owner, access class, freshness expectation, and removal path. A document that an agent may read is not automatically a document every agent, model, or analytics job may retrieve. Indexing must preserve tenant, region, product, and role boundaries where they exist. Deletion from the source must eventually remove the corresponding chunks and cached results as well.

Chunking and indexing are engineering choices with visible failure modes. Chunks that are too small may lose the qualification that changes a policy; chunks that are too large may bury the relevant sentence and waste context. Metadata filters can improve precision, but stale or missing metadata can hide the right answer. An index rebuild needs a version, validation, and a way to keep serving the last trusted version if the new one is defective.

Before connecting a generator, assemble a modeled evaluation set from real question shapes without copying private customer content into an uncontrolled fixture. For every case, record the expected source or the fact that no approved answer exists. Include routine cases, ambiguous wording, policy exceptions, stale-document traps, conflicting sources, attempts to override instructions, sensitive-data requests, and questions that should escalate.

Evaluate retrieval on its own. Did the approved evidence appear near enough to be used? Did access filters exclude material the agent should not see? Did the retriever return an obsolete or poisoned source? A generator cannot repair evidence it never received, and a fluent answer can conceal the miss.

Then evaluate generation with the retrieved evidence held visible. Review whether the draft is supported, complete enough for the workflow, correctly qualified, safe, and candid when evidence is absent. A useful rubric gives reviewers concrete error categories and an escalation rule. It does not ask them to compress judgment into a decorative score.

Finally evaluate the whole workflow. Retrieval and generation can each look acceptable while the combined product interrupts agents, adds verification work, or encourages them to trust polished mistakes. End-to-end evidence comes from the interaction between the assistant, the agent, the ticket, and the eventual customer outcome.

Follow one draft through production

A request arrives with ticket text, product and locale, agent identity, and only the account context the workflow is authorized to use. An input boundary classifies the ticket, minimizes sensitive fields, checks size, and decides whether this category is eligible. Unsupported categories take the ordinary search or escalation path.

The retriever applies access and product filters, searches the versioned approved corpus, and returns document identifiers, revisions, and passages. The orchestration layer constructs the model input from the ticket, approved instructions, and retrieved evidence. Retrieved text is untrusted content: a document can contain instructions that should never acquire the authority of the system policy. The design must keep those roles distinct and constrain the actions available to the model.

The model produces a draft or declines because evidence is insufficient. The agent sees the relevant citations beside the text, not hidden in a debugging view. They can accept, edit, reject, or escalate. No acceptance signal should silently become a claim that the whole draft was correct; agents may click through under time pressure or edit for reasons the telemetry cannot infer.

The trace for this request should make later investigation possible. Record the prompt and policy version, model and routing decision, corpus and index version, retrieved document identifiers, filter outcomes, latency, resource or provider cost, safety outcome, and agent action. Keep raw ticket content and customer identifiers out of broad analytics unless there is a justified, restricted need. Observability is not permission to copy sensitive data.

Failures need an intentional degradation order. If retrieval times out, the system should not ask the model to improvise a policy answer. If the preferred model is unavailable, a smaller model is safe only if it has passed the same task-specific release gate. If the entire assistant is unhealthy, agents return to search and approved templates. The fallback is part of the product, so its latency and correctness deserve rehearsal.

This trace covers an AI application, but the same discipline applies to a classical model. For fraud scoring, replace retrieved passages with versioned features and replace the draft with a risk score plus a threshold policy. The unit of prediction, label definition, feature availability, false-positive cost, manual-review path, and delayed outcome are all product decisions. A calibrated score does not decide who gets blocked until policy turns it into an action.

Training and inference must meet at the same contract

A notebook can use a cleaned dataset assembled after the outcome. Production cannot. For a trained ranking, forecasting, classification, or anomaly model, state what is known at prediction time and enforce that time boundary during dataset construction. Leakage often enters through a convenient join, a label computed from the future, or a feature whose offline backfill is fresher than its online value.

Version the training data definition, feature or preprocessing code, label policy, model artifact, and evaluation result together. Reproducibility does not require freezing the world; it requires being able to explain which world produced the candidate and to compare it with the one now being served.

Training and serving should share transformations where practical and compare their outputs where they cannot. Monitor missingness, ranges, categories, freshness, and representative feature values at the serving boundary. Holdout data should follow the training window in time when the product will face a changing future. A random split can hide leakage, recurring users, duplicated items, or seasonality.

Retraining is not a remedy by itself. A schedule or drift alert may nominate a new candidate, but promotion still needs data validation, slice evaluation, comparison with the deployed baseline, resource checks, and a reversible rollout. Some drift changes input frequency without harming the decision; some concept change damages the product before a generic distance metric looks alarming. Monitor the user outcome and the decision policy, not only the feature distribution.

Compute choices buy a particular product promise

An interview answer does not become senior by announcing GPUs. Begin with the quality, latency, throughput, privacy, availability, and cost the workflow can justify.

A hosted model may be the right first release when iteration speed matters and the data boundary, provider behavior, rate limits, observability, and fallback are acceptable. Self-hosting buys control only by adding model loading, capacity, upgrades, scheduling, isolation, and on-call responsibility. Fine- tuning earns its cost when evaluation shows a stable task gap that prompting, retrieval, workflow changes, or a smaller specialized model cannot close.

For GPU serving, model size and numeric representation determine memory pressure; traffic and sequence length determine work and queueing. Batching can improve throughput and utilization by letting requests share execution, but waiting to form a batch consumes latency budget. More concurrent model instances can raise throughput until memory or compute contention makes the tail worse. Quantization or distillation may lower memory, latency, and cost, but the resulting artifact needs evaluation on the actual task and hard slices, not only a generic benchmark.

Measure queue time separately from model execution and from the rest of the request. Watch p95 or p99 latency, pending work, timeouts, batch sizes, memory headroom, utilization, cold starts, tokens or examples processed, and cost per useful outcome. A cheaper generated draft is not an improvement if agents reject it more often. Model routing, caching, batching, and smaller models are product policies because each changes quality or freshness as well as spend.

Launch to learn without making users absorb the uncertainty

Offline evaluation determines whether a candidate deserves contact with the workflow. It does not prove that agents will use it well or that customers will benefit. Begin with a narrow ticket category, trained agents, visible evidence, and a limited cohort. Keep a concurrent control or other credible comparison, and decide the ramp, pause, and rollback conditions before enthusiasm changes the standard.

The first online questions are modest. Does the assistant save time after verification? Which error categories survive? Do agents over-trust drafts or avoid the tool? Does benefit differ by language, product, or experience? Do latency and provider limits alter behavior? Novelty can temporarily increase use, and one agent’s treatment of a case can affect later labels, so an A/B result still needs interpretation.

Human review is a control with capacity limits. Reviewers need a defined task, an escalation path, calibration examples, and relief from repetitive low-value decisions. Measure disagreement and review delay. If the design relies on humans to catch nearly every defect, the real system may be an expensive manual process wearing an AI interface.

Responsible launch judgment is concrete. Limit sensitive inputs, examine performance and harms across relevant groups and cases, test misuse and instruction-conflict paths, preserve audit evidence, assign incident owners, and state which actions remain forbidden. “Human in the loop” and “responsible AI” mean little until the human’s authority, workload, and response are specified.

When the service stays green and the answers decay

Two weeks after a model and index update, handle time improves but agents begin rejecting more billing drafts. The endpoint returns 200, GPU utilization looks healthy, and aggregate acceptance barely moves because billing is a small slice. This is an ML incident even if no availability alert fired.

First contain the decision risk. Disable billing drafts or route them to the last trusted configuration while ordinary search remains available. Preserve the affected model, prompt, corpus, index, and policy versions. Tell support leadership which categories are affected and what fallback agents should use.

Then localize the first broken boundary. Did the billing policy change? Was it indexed and did its access metadata survive? Does the evaluation set contain the new exception? Did retrieval return the right revision? Did the prompt retain its qualifications? Did model routing send long billing cases to a smaller model? Did a latency timeout truncate context or trigger a degraded path?

Suppose the index rebuild copied the policy text but dropped its product-region metadata. The global search now retrieves a similar policy from another region. Reverting the model would not fix this retrieval defect. The repair is to restore the metadata contract, rebuild and validate the affected index, replay the billing evaluation set, and ramp that category again. A source-to- index parity check and region-conflict cases strengthen the release gate.

The incident also changes the monitoring design. Add acceptance and rejection by ticket category, region, corpus version, and fallback path; sample grounded quality where labels arrive; and alert on changes that have an owner and a containment action. Aggregate uptime was true but answered the wrong question.

This is the shape of a strong debugging answer: define the user-visible symptom, inspect the release boundary, compare expected and actual evidence, separate data, retrieval, model, policy, and serving failures, contain before guessing, and leave a cause-specific guard behind.

What the interview loop can reveal

The specialty changes the material, not the need for ordinary engineering discipline.

In a practical coding round, you may transform messy examples, implement a metric, rank retrieval results, build an evaluation harness, batch requests, or debug orchestration. State the invariant before the syntax: no training example may use future information; access filters apply before evidence leaves retrieval; each evaluation case records its corpus and policy version; a retry cannot duplicate a customer-facing action.

In system design, follow one decision from input evidence through data or retrieval, model execution, policy, user action, feedback, monitoring, and rollback. Interviewers can then perturb latency, label delay, privacy, traffic, cost, autonomy, or quality without forcing you to redraw an unrelated system.

For a project deep dive, prepare the moment after launch. Explain which baseline you used, how the data was produced, where evaluation changed the design, how the release was controlled, what users revealed, and which durable artifact remained: an evaluation suite, dataset contract, model registry, retrieval audit, launch gate, incident playbook, or cost policy. A project that ends at deployment conceals the specialty’s most important evidence.

Behavioral prompts often place uncertainty under organizational pressure. A leader wants autonomous replies before the evidence supports them. A slice performs poorly but the aggregate wins. Privacy review removes useful data. A provider change threatens the launch date. Strong answers name the disputed decision, show the evidence and its limits, reduce or phase scope, assign the risk owner, and give the team a safe next experiment. Jargon is not a substitute for a recommendation.

Answers that close the question too early

Interrupt these habits when you hear them in practice:

• “Use the largest model.” Quality is task-specific and must repay latency, capacity, operational, privacy, and cost consequences.
• “Accuracy improved.” Name the decision, error costs, calibration or ranking behavior, important slices, and comparison with the baseline.
• “RAG prevents hallucinations.” Retrieval can be missing, stale, unauthorized, poisoned, or ignored; the action still needs a limit.
• “We monitor drift.” Name the distribution, user outcome, threshold, owner, and response. Drift is evidence to interpret, not an automatic cause.
• “We will retrain.” Validate the new data and candidate, compare it with the deployed system, and earn promotion through the release gate.
• “A human reviews it.” State which human, what they see, how much they can review, what authority they hold, and what happens under overload.
• “Batching makes inference faster.” It can improve throughput while adding queue delay; measure the latency distribution under realistic traffic.
• “The request succeeded.” Availability does not establish usefulness, fairness, safety, groundedness, or affordability.

Each correction returns the answer to a user decision and the evidence needed to own it.

Practice by changing one constraint

Use the support-assistant prompt until the design stops being memorized:

1. Define the first allowed action, the fallback, and the evidence required before the assistant may act more autonomously.
2. Build twelve evaluation cases. Include missing evidence, conflicting policy, stale content, a regional exception, sensitive data, an instruction hidden in retrieved text, and a case that must escalate.
3. Separate retrieval, generation, and end-to-end failures. Give each one an owner, a release gate, and a containment action.
4. Add a strict two-second latency target. Decide what may be cached, routed, batched, shortened, or declined without hiding a quality loss.
5. Cut serving cost in half. Compare a smaller model, quantization, batching, routing, caching, and narrower eligibility by cost per accepted draft.
6. Let one region change its refund policy. Trace source approval, indexing, evaluation, rollout, monitoring, rollback, and deletion of stale material.
7. Remove human review for one low-risk category. State the evidence threshold, action boundary, audit trail, stop condition, and residual-risk owner.

Then take one project of your own and rehearse it without tool names. If you cannot explain the product decision, baseline, data boundary, evaluation, release, failure, and learning without naming a platform, the platform is probably carrying too much of the answer.

A compact ML and AI answer frame

When a prompt sprawls, make these lines specific:

User decision and first allowed action:
Simple or non-model baseline:
Task, unit of prediction, label, or evidence boundary:
Training data or retrieval corpus, including time and access rules:
Evaluation before launch, hard slices, and known blind spots:
Inference path, latency budget, capacity, privacy, and fallback:
Experiment, ramp, pause, and rollback conditions:
Monitoring across quality, safety, drift, latency, cost, and user outcome:
Forbidden action, evidence needed to permit it, and accountable owner:

You are ready when uncertainty no longer makes the answer vague. You can say what the system knows, what action that evidence permits, how the product will learn, and where it must stop. Then a different model, retrieval method, GPU, or vendor becomes a change inside an owned system rather than the system’s entire identity.

Related links

• Data Engineering
• Search and Retrieval Systems
• Observability, Performance, and Cost
• Security, Abuse, and Privacy
• AI-Native Systems