Model Selection: Rules, Classical ML, Deep Learning, Foundation Models, and Hybrids

Five Extractors, One Payment Risk

An accounts-payable team tests five ways to extract supplier, invoice number, date, currency, totals, tax, and line items. Template rules are nearly perfect for its three largest suppliers and fail on unfamiliar layouts. Optical character recognition plus a small classifier is cheap, but loses relationships between labels and values. A specialized document model handles known suppliers consistently. A multimodal foundation model reads unusual layouts best, yet occasionally supplies a plausible normalized value that is not on the page. A hybrid catches the difficult layouts and checks the output against source coordinates and payment rules.

If the team ranks candidates by average field accuracy, the foundation model wins. That result does not answer the payment question. A wrong remittance account is more serious than a missing description; documents must remain in an approved region; every extracted field needs traceable evidence; low-confidence work must fit the review queue; and the service must survive a provider or model change.

Those constraints change the winner. The team ships deterministic controls around a specialized extractor, using the larger model only as a bounded fallback. It has not chosen the most capable model in isolation. It has chosen the least complex system that satisfies the whole production contract with adequate margin.

Model selection is an elimination argument. Begin with the job and its hard boundaries, reject candidates that cannot meet them, then require measured benefit for every layer of complexity that remains.

Write the Contract Before the Candidate List

A candidate becomes difficult to reject once a team has built a demo around it. Write the decision contract first, in terms that do not favor a model family.

Start with the output and the action it enables. Is the system producing a category, score, forecast, ranking, extraction, answer, or action? Will that output inform a person, route work, block a transaction, approve a claim, generate content, or execute a change? The same error can be tolerable in a search hint and unacceptable in a payment instruction.

Then state the acceptance evidence: thresholds and severe errors, important segments, available training and evaluation data, and the uncertainty that the decision owner will accept. Add the conditions the service must honor—latency tails, throughput, availability, device or regional limits, privacy and residency boundaries, explanation and contestability, update cadence, human-review capacity, rollback, and exit cost.

Separate hard constraints from preferences. A model that sends restricted invoices to an unapproved region is ineligible; lower infrastructure cost cannot compensate for that failure. Among eligible candidates, cost and accuracy may be traded, but the trade should remain visible rather than disappear inside a weighted score.

For the invoice system, the contract names evidence coordinates, payment-field error limits, regional processing, batch latency, review capacity, and cost at peak volume. Only after writing those conditions does the team decide what to test.

Start Below the Model

The first candidate is the current job, including its frustrations. Manual entry, queue design, clearer forms, search, or a repaired upstream process may solve enough of the problem. Without that baseline, a team can automate waste and call the resulting speed an improvement.

Next try the simplest explicit behavior that could satisfy the contract. Rules suit stable eligibility boundaries, formats, known high-precision signals, and safety blocks. They are fast and inspectable. They also become brittle when exceptions interact or the input surface changes faster than maintainers can describe it. The invoice team keeps rules for supplier identity, duplicate detection, arithmetic, and payment controls, but template rules alone cannot cover the supplier mix.

Classical statistical and machine-learning methods deserve a real attempt, especially for structured data. Linear models, trees, boosted ensembles, clustering, and time-series methods can produce calibrated scores, rankings, forecasts, and classifications with modest serving demands. On OCR-derived invoice features, a small classifier provides a useful confidence baseline. It cannot reliably recover the spatial relationships needed for unfamiliar documents, so the evidence—not fashion—moves the team onward.

Specialized deep models earn their cost when perceptual or sequential structure in images, audio, text, or other high-dimensional signals is central to the job. Their learned representations can capture relationships that hand-built features miss. The price includes training evidence, robustness work, serving hardware, and harder explanations. A document model is the strongest routine extractor in the invoice evaluation, and its narrow job makes its failures easier to characterize.

Foundation models become plausible when broad language, code, vision, audio, or multimodal capability closes a measured gap that narrower candidates leave open. Their generality is useful for novel invoice layouts. It is not proof of task reliability. The team evaluates the exact model version with the exact prompt, image processing, schema, and tool configuration it intends to deploy. The unsupported normalized values prevent the model from owning payment-sensitive fields, but its unusual-layout coverage makes a bounded fallback worth testing.

The ladder is a search order, not a ranking of sophistication. A rule can remain the final design. A foundation model can be the simplest adequate choice for a broad language task. “Simplest” means least total system burden for the contract, not the fewest parameters or components.

Compare the System That Would Actually Ship

An offline metric describes only part of a candidate. Evaluate task quality overall and on critical segments, then inspect severe errors rather than allowing averages to hide them. Test shift, missing input, edge cases, and attacks. If scores drive thresholds or escalation, test whether their confidence is calibrated in the intended operating region.

Run the deployed path under realistic load. Median latency can look excellent while tail latency breaks the workflow. Estimate lifecycle and marginal cost at expected and peak volume, including review labor, evaluation, incident response, and replacement—not just inference price.

Control is evidence too. Can the team reproduce the configuration, constrain permissions, inspect failures, roll back, and operate a fallback? Does the deployment respect data boundaries? Are explanations stable and useful to the person who must act on or contest the result? Who will monitor, patch, support, and eventually retire every dependency?

Report uncertainty and non-dominated choices. If one candidate is cheaper and faster while another catches more rare layouts, decision owners should see that frontier. A single composite score often conceals the assumptions that made the decision.

Use the same outcome measures for at least three baselines when they make sense: the current human or system workflow, a rule or search approach, and the simplest credible learned model. Baselines do more than restrain ambition. They become fallbacks, monitoring references, and estimates of what the added complexity actually buys.

Deployment and Ownership Are Part of the Model Choice

“Open” and “proprietary” do not describe an operating arrangement precisely enough. Record the exact artifact, version, license, provider, and deployment mode.

For a hosted service, examine how the provider uses and retains data, where processing occurs, how model changes are announced, whether rate limits fit peak demand, what logs and evaluations are available, how safety-policy changes affect behavior, and how the team exports or replaces the dependency.

For downloadable or open-weight models, examine license restrictions, provenance disclosures, artifact integrity, security updates, hardware, inference optimization, patching, safety controls, and the skills required to operate them. Self-hosting can increase control while transferring more responsibility to the team. Hosting can remove infrastructure work while adding dependency and opacity. Neither arrangement wins before the contract is known.

Small specialized models deserve the same scrutiny. They can be excellent when the task is narrow, volume is high, offline operation matters, sensitive data should remain local, and strong labeled evidence exists. They still need dataset rights, segment evaluation, monitoring, security, versioning, and retirement. Their interface should route inputs outside the supported boundary instead of forcing a prediction.

Build Hybrids to Contain Failure

A hybrid earns its interfaces when each component has a bounded job. The invoice path makes authority explicit:

deterministic supplier and payment checks
-> approved-region document extraction
-> source-coordinate and schema validation
-> bounded fallback for unusual layouts
-> calibrated routing to auto-complete or human review
-> component and end-to-end monitoring

The generative component may propose structured output; it may not invent a payment destination or override a duplicate-invoice block. If extraction fails, validation does not silently bless a guess. If the fallback is unavailable, the item enters review. Each boundary has an interface contract, version owner, evaluation, and observable failure behavior.

A pile of models is not a design. Two components that can disagree need a named authority or escalation path. Retrieval failure must not cause a generator to substitute memory. Added interfaces must justify their latency, monitoring, security, and version dependencies with measured improvement.

The invoice hybrid ships because it expands layout coverage without granting the least predictable component authority over the most consequential fields. The decision also includes a subtraction rule: if routine extraction improves or fallback volume falls below a recorded threshold, remove the larger model rather than preserving complexity by default.

Model Selection Decision Record

1. Decision and owner: system, task, date, and accountable approver.
2. Acceptance contract: outcomes, segments, severe errors, latency, cost, rights, security, explanation, and operations.
3. Baselines: current workflow, non-AI approach, and simplest credible learned model.
4. Candidates: exact model, version, license, provider, deployment mode, configuration, and dependencies.
5. Evidence: datasets, sampling, metrics, error analysis, stress tests, uncertainty, load tests, and operating estimates.
6. Eliminations and trade-offs: hard failures, non-dominated choices, assumptions, and residual risks.
7. Architecture: component authority, validation, permissions, human review, and fallback behavior.
8. Decision: selected configuration and the measured benefit that earns its complexity.
9. Release conditions: thresholds, approvals, monitoring, runbook, and rollback.
10. Revisit triggers: data shift, version change, cost, latency, incident, policy, review capacity, or a stronger baseline.

Try the Argument on Another Task

A legal-search team proposes a foundation-model assistant because users ask questions in ordinary language. Before choosing it, write the contract: which sources are authoritative, how permissions and jurisdiction apply, what citations must show, when the system must abstain, how current the corpus must be, and what an answer is allowed to influence.

Now compare the existing search workflow, metadata and keyword search, semantic retrieval, a reranker, and generation over retrieved passages. Eliminate candidates that cannot enforce access or provenance. For every remaining layer, state the failure it fixes and the new failure it creates. Stop when the contract is met; the exercise has no requirement that generation survive.

That is the model-selection habit worth carrying forward: make complexity prove its place in the deployed system. Once the class and architecture are chosen, the next question is which behavior layer—task specification, retrieval, workflow, fine-tuning, or training—should change when measured performance falls short.

Source Notes

• Google Rules of Machine Learning, engineering guidance on simple baselines, solid pipelines, interpretable first models, and choosing heuristics when they perform better; verified 2026-07-10.
• NIST AI RMF 1.0, system-context and lifecycle framing for mapping, measuring, and managing risk; verified 2026-07-10.
• Hidden Technical Debt in Machine Learning Systems, production risks from entanglement, glue code, undeclared consumers, and system-level dependencies; verified 2026-07-10.