Chapter 9: Choosing & Evaluating Models

Capability Is Table Stakes - Selection Is Strategy

By 2026 the frontier models are all remarkable out of the box. Claude Opus 5.5, OpenAI's GPT-6 Astra, Google's Gemini 4 Argon, and SpaceXAI's Grok 4.7 will each one-shot work that would have been a research project two years ago. That is exactly why picking a model is no longer about finding "the best one." It is about knowing which model earns its keep on which job, and running a deliberate multi-model strategy so that no single vendor's price hike or bad release ever holds your roadmap hostage. Multi-model fluency is a leading indicator of a healthy AI-native team, tracked alongside delivery and quality metrics.

The 2026 Model Classes

Frontier generalists. Claude Opus 5.5, OpenAI GPT-6 Astra, Google Gemini 4 Argon, and SpaceXAI Grok 4.7 - the last priced aggressively below the rest. Deepest reasoning, strongest agentic coding, largest context. Use them for architecture, complex multi-file work, and ambiguous problems where a wrong turn is expensive.

The tier above the flagship (new in 2026). Anthropic's Claude 5 family introduced something the market had not seen before: a class above the flagship. Claude Fable 5.1, the current head of the new Mythos-class tier, sits above Opus in capability - built for the longest autonomous runs and the hardest reasoning, at a price to match. The pattern matters more than the product: vendors now segment by depth of judgment, not just speed and cost. Reserve this tier for the work where a wrong decision compounds - architecture calls, deep multi-system debugging, agentic runs that span hours - and let the flagship and fast tiers carry everything else. And when a new tier lands, re-run your bake-off, because it resets the ranking.

Capability that is deliberately gated (also new in 2026). The other new tier is one you mostly cannot buy. OpenAI publicly paused parts of its next frontier model over cyber-capability thresholds and ships offensive-security capability only through a vetted-access program; Google restricts its cyber fine-tune to governments and trusted partners; Anthropic's full Mythos-class model is limited to approved organizations, with Fable 5.1 as its publicly hardened form. Frontier capability and open availability have formally decoupled. For a buyer this means two things: the public leaderboard no longer shows the actual frontier, and access programs are becoming part of procurement for security-sensitive work.

Fast, cost-optimized models. Claude Haiku 5.5, OpenAI's smaller GPT-6 tiers (Sol and Luna), and Google's fast-shipping Gemini Flash line (currently Gemini 3.8 Flash). Lower latency and far lower cost. Use them for high-volume, well-specified work: classification, routing, simple edits, summarization, and the inner loop of agent tool calls. Watch the generation gap in this tier, because it does not reliably track the flagship. Anthropic closed its gap in October 2026: Haiku 5.5 shares the current generation's 1M window and knowledge cutoff, where Haiku 4.5 - now legacy, and still pinned in plenty of routing configs - sat generations behind at 200K. Google's gap opened the other way: the flagship moved to Gemini 4 while Flash is still a 3.x model. A routing rule that assumes the cheap model shares the expensive model's window and recency will quietly truncate its input or answer from staler knowledge - and one that assumes the cheap model stays cheap at any length will miss that some fast tiers now price long prompts at a higher rate. Check the spec sheet of the exact model ID you pinned, not the family's headline numbers.

Reasoning models. Extended-thinking modes that spend more tokens and time to crack harder problems. Use them when correctness on a gnarly problem matters more than speed - tricky debugging, algorithm design, planning a large refactor.

Open-weight models. Qwen and Llama-class models served on your own infrastructure (for example on vLLM). Capable, controllable, and behind your own security boundary. Use them for regulated data, cost control at scale, and workloads where a managed API is a non-starter.

Match the Model to the Job

Task                              | Recommended class
----------------------------------|------------------------------
Architecture, large refactor      | Frontier generalist
Ambiguous debugging, planning     | Reasoning model
Bulk mechanical edits, migrations | Fast / cost-optimized
Classification, routing, triage   | Fast / cost-optimized
Regulated data, on-prem only      | Open-weight (self-hosted)
High-stakes one-shot correctness  | Frontier or reasoning

Run a Bake-Off, Not a Vibe

Do not pick a model from a leaderboard. Build a small eval set from your own representative tasks - twenty to fifty real problems with known-good outcomes - and run each candidate against it. Score on what actually matters: outcome attainment, constraint adherence, cost-per-outcome, and latency. This is the same discipline as agent evals (Chapter 21): a model that wins on a public benchmark but loses on your intent evals is not shipping. Re-run the bake-off when major model versions land, because the ranking changes month to month.

Model Routing

You do not have to choose one model for everything. Route by task: try a fast, cheap model first and escalate to a frontier model on failure or detected complexity. Most tools support per-task model selection - Copilot's model picker, Claude Code's model setting, and the agent SDKs all let you bind a model to a step. Routing keeps the bill down without sacrificing quality on the hard 20% of work that needs it.

Local and Self-Hosted: From Ollama to vLLM

Two situations push you off the managed APIs entirely: experimentation and regulated data. For local experimentation, Ollama runs open-weight models on a laptop or a shared box with one command - perfect for trying a model, prototyping a feature, or working offline without sending anything to a vendor. It is how a skeptical engineer kicks the tires with zero procurement and zero data-exposure risk.

For production at scale, the serving layer changes. On a regulated platform handling protected health information, I served an open-weight model (a Qwen-class model) on self-hosted vLLM so that inference stayed inside the compliance boundary - protected data never left for an external endpoint. That single architectural decision is often what makes an agentic feature shippable in healthcare or finance at all: not which model is smartest, but which deployment keeps the data where the auditor needs it. Ollama gets you exploring today; vLLM and a self-hosted open-weight model get you to production behind your own boundary (see Chapter 22).

Avoiding Single-Vendor Lock

Keep your stack portable. Reach models through model-agnostic SDKs and the protocol layer (MCP) rather than hard-wiring one vendor's API throughout your code. Track price-performance across providers as an ongoing discipline, not a one-time decision. The goal is that swapping a model is a configuration change, not a rewrite - so the team always runs whatever earns its keep this quarter.

2026 made this concrete. The most popular AI IDE was acquired by a model vendor. A widely used free CLI agent was retired and replaced by a closed-source successor with a fraction of the free tier. Subscription access through third-party harnesses was shut off by terms-of-service enforcement. And nearly every vendor moved to usage-based pricing. None of these were predictable a year out, and every one of them punished teams whose workflow was welded to a single vendor's tool. Portability stopped being a hypothetical hedge; it is the difference between a config change and a migration project when your tool's owner changes strategy.

The back half of the year sharpened the point. A model vendor moved to cut off supply to a rival-owned IDE, so the models available inside a tool turned out to depend on who owns the tool. A major vendor's original agent API reached end of life on schedule with no automated migration path, so teams that had built against it rewrote rather than upgraded. Included allowances on agent subscriptions were repriced mid-year after promotional periods lapsed, so the budget you planned around was a starting offer. Read together, these are not four unlucky events - they are the normal weather of a market this young. Assume every vendor relationship in your stack is temporary and design so that being right about that costs you a day, not a quarter.