Field Report · Pier 48, San Francisco · April 28–29, 2026

Notes from
AI Dev 26
in San
Francisco.

A condensed, opinionated read on two days of talks — the coding agents, memory layers, defect-severity frameworks, and team-shape questions that should actually change how we work next quarter.

3,000+Attendees
62Countries
2Days · 1 Pier
17Sessions Covered
DeepLearning.AI conference stickers and lanyard
SWAG · PIER 4826 · 04
At AI Dev 26 SF step-and-repeat backdrop
AI DEV 26 · SFDAY 01
On the AI Dev 26 SF main stage
MAIN STAGECONF · SF

I didn't take all these notes alone. Two collaborators made this report possible.

Synthesis & Drafting
Read every transcript, pulled the threads that connect across sessions, drafted the section narratives, and built this page.
Live Transcription
Captured each talk in real time so I could put the laptop down, look at the speakers, and trust I'd have the words later.
01

What the whole conference was secretly about.

7 themes
17 sessions
2 days
Read this if nothing else

Across keynotes, panels, and breakouts, the same seven ideas surfaced again and again. The detailed session notes below are just evidence for these claims.

01 / Bottleneck
Coding is no longer the bottleneck. Deciding what to build is.
Multiple speakers (Cursor, JetBrains, AMD, Andrew Ng) converged on the same point. Code quality, product judgment, taste, and intent specification matter more than typing speed now.
02 / Adoption
80% AI-assisted is the worst place to be.
Andrew Ng's strongest claim: at 80% AI, humans become the review bottleneck. At 100% AI workflows, teams hit a step-function in throughput. Pick a side.
03 / Data quality
AI exposes every data sin you've ever committed.
Quote of the panel from Joe Reis. The most concrete near-term risk for anyone shipping agents: bad columns, drift, undocumented joins all surface immediately when agents start querying.
04 / Memory
Agent memory is the layer most teams haven't built.
Without it, agents re-derive context every session — paying token costs and hallucinating along the way. Short-term, procedural, and episodic memory are now distinct things to design.
05 / Defect rate
Reliability — not capability — is the real constraint.
AWS framed it as a 2×2: defect frequency × severity. Most agents today are "high frequency, low severity" — fine for slop, not for anything that touches production. Classify before you ship.
06 / Token economics
Token efficiency is now a cost-and-latency lever.
OpenAI explicitly framed GPT-5.5 around fewer thinking tokens and shorter outputs. Audit your prompts and harnesses as you would query plans.
07 / Team shape
Teams are collapsing into generalists with domain depth.
Multiple speakers reported teams of 5–10 generating tens of millions in ARR. PM-to-engineer ratios flattened to 1:1 or merged entirely. The "AI-native" profile is leverage + systems thinking + product sense.
08 / Safety
Containment goes outside the model, not inside.
Docker's argument: don't rely on the model to refuse bad behavior. Sandbox the agent first, apply policy at the boundary, then trust capability. Model-driven safety is layer four, not layer one.
Day One
April 28 · Foundations, Frameworks, Memory
02

Intent velocity — the new engineering metric.

Session 02
09:10–09:25
15 min
09:10 — 09:25 AM
AMD
Anush Elangovan
VP, Software

Impact of AI on Software

Anush's argument: the bottleneck has moved from writing code to guiding AI and deciding what to build. He calls the new metric Intent Velocity — the time from idea to production.

He framed engineers' future as a "K-shaped" split. Going up the curve: systems thinking, judgment, problem framing. Going down: language-specific syntax skills and manual implementation. The transformation timeline has compressed from decade cycles to months — sometimes weeks.

The real-world examples are striking. A GPU instruction translation problem that historically took 4–5 years and ~300 engineers was prototyped by agents in 48 hours. AMD's tokenizer — about 200k lines of code that speeds up LLM inference — was written by AI. The point isn't the headcount math; it's that parallelism is now the default: run many agents simultaneously rather than queueing work through one human.

⟡ Why this matters for us
  • Reframe KPIs. Lines of code, PRs shipped, and tickets closed all measure the wrong thing. Intent velocity — idea to production — is the metric to track.
  • Audit for parallelism. Where in our pipeline are we still serializing work that several agents could do at once?
  • Concretely: data prep, eval sweeps, model-architecture searches, and content drafts are exactly the kind of embarrassingly-parallel work that suits this pattern.
03

The sorcerer's apprentice problem.

Session 03
09:25–09:40
15 min
09:25 — 09:40 AM
AWS
Marc Brooker
VP & Distinguished Engineer

Why agent safety lives outside the agent

Marc's frame: the future of agentic AI isn't constrained by raw capability — it's constrained by defect rate. And defect rate has two independent axes: frequency × severity.

Marc Brooker presenting the agent defect matrix — How Frequent? × How Bad? — with quadrants labeled Everybody, Slop, Experts Only, Nobody?
Marc Brooker · AWSThe defect 2×2 — frequency × severity

He drew the 2×2: high-frequency / high-impact defects make a system unusable; high-frequency / low-impact gives you "slop" (summaries, low-stakes drafts); low-frequency / high-impact is the dangerous category — powerful but with hidden failure modes; low-frequency / low-impact is the target state for mass adoption. Most teams are improving frequency but not severity.

His symbolic-AI investments are notable: Hydro (Rust framework for correct distributed systems), Cedar (policy language for authorization), Kiro (spec-driven coding agent), and auto-formalization work that converts natural-language policies into mathematically precise specifications. The argument: agents need formal scaffolding, not just better models.

"We need to take our worst days as seriously as our best ones." — Marc Brooker, AWS
⟡ Why this matters for us
  • Build a defect-severity matrix for any agent we ship. Frequency alone — "it's right 90% of the time" — is meaningless without severity.
  • Failure severity ≫ failure density. Benchmarks like pass@10 hide what matters. A wrong number in a financial report is fundamentally different from a wrong sentence in a summary — and the same model can produce both.
  • Spec-driven > prompt-driven for anything in production. Treat the spec as the agent's persistent context.
04

The future of software engineering — five practitioners disagree productively.

Session 04
09:40–10:35
55 min
09:40 — 10:35 AM
The Panel
Catasta · Maloney · Alake · Reis
Replit · LandingAI · Oracle · Practical Data Media · Mod: Marina Mogilko

Software engineering isn't dying — it's being redefined

Universal panel consensus, with productive nuance: coding is being commoditized; thinking, taste, and judgment are not. The entire SDLC — writing, testing, deploying — is being automated. The interesting questions are about who does the judging.

Future of Software Engineering panel on stage: Marina Mogilko moderating with Michele Catasta (Replit), Dan Maloney (LandingAI), Richmond Alake (Oracle), and Joe Reis (Practical Data Media)
The panel · moderated by Marina MogilkoCatasta · Maloney · Alake · Reis

The most striking number from the panel: the population of "builders" expanding from ~50M developers to potentially 1–2 billion knowledge workers who can now produce working software. Junior engineers, paradoxically, now get exposed to full system complexity immediately — a steeper curve but faster growth.

Their critical-skill stack, in order of importance: (1) critical thinking — to debug AI outputs; (2) taste — to judge good vs bad; (3) truth-seeking — and crucially, "AI becomes dangerous when it's 90% right"; (4) continuous learning; (5) communication. The winning profile is generalist + deep domain expertise. The "manager" title may disappear — everyone manages agents, everyone owns outcomes.

"AI shines a light on past data sins." — Joe Reis, Practical Data Media
⟡ The takeaway — and one quote in particular
  • Data quality is the #1 hidden risk. The moment an agent queries internal datasets, every undocumented join, every silent NULL, every column-with-three-different-meanings gets surfaced. A data-quality audit on the inputs we feed into AI pipelines is cheap insurance.
  • "90% right" is the dangerous regime. A 90%-correct ranked list is worse than a 60%-correct one, because the team will trust the former. The dangerous middle deserves more attention than either extreme.
  • Lean into generalist + domain depth. Whoever can ship the agent, evaluate the agent, and explain it to the business is the profile that compounds.
05

Why 80% AI is worse than 100% AI.

Session 05
11:05–11:25
20 min
11:05 — 11:25 AM
DeepLearning.AI
Andrew Ng
Founder & Host

Software as composable building blocks — assembled by agents

Andrew's mental model: software is Lego bricks — AI + non-AI (APIs, DBs, auth, UI). More bricks → exponentially more combinations. The job of AI agents is to assemble them.

His most-debated claim: teams stuck at ~80% AI-assisted have humans as the review bottleneck. Teams that go to ~100% AI hit a step-function in speed. He freely admits this isn't a universal — if you're writing flight software, write it by hand — but for typical work, the last 20% of human review is where speed dies.

A second-order claim, equally important: once coding speeds up 10–100×, new bottlenecks appear — product management, design, legal/compliance, marketing, sales. The old "1 PM per 5–8 engineers" ratio is collapsing toward 1:1 or fully merged roles. Team size is collapsing too — 2–10 cross-functional generalists.

The "AI-native engineer profile": AI leverage + systems awareness + product thinking. And — relevant to anyone working with code — "fresh context" is now critical infrastructure. Agents will happily call deprecated APIs they were trained on. Without injected fresh context, you get confidently wrong code.

⟡ Why this matters for us
  • Pick a workflow to push to ~100% AI. Probably something low-risk like notebook scaffolding, dataset documentation, or eval harness generation. Measure the speed gain.
  • Watch for the bottleneck shift. If we accelerate the modeling work 5×, our blockers move to data access, legal review, and stakeholder alignment. Plan for that.
  • Build a fresh-context pipeline. The latest internal API docs, the current data dictionary, the current model registry — all retrievable on demand so the agent never falls back on stale training data.
06

The model and the harness are the same product.

Session 06
11:30–12:00
30 min
11:30 AM — 12:00 PM
OpenAI
Marc Manara
Startups Team, OpenAI

Fireside · what's working, what's brittle

Marc's team partners directly with companies like Cursor, Augment, and Cognition before launches — because the model and the prompt harness around it must be tuned together. They behave like one product.

Marc Manara of OpenAI in conversation with Jonathan Heyne of DeepLearning.AI on the main stage
Fireside · Marc Manara × Jonathan HeyneMain Stage · Pier 48

GPT-5.5 was a deliberate bet on token efficiency: fewer thinking tokens, shorter outputs, same or better accuracy. This isn't a model-team vanity metric — it's directly a cost and latency lever for anyone serving the model in production.

Two more behaviors the team explicitly tuned for. First, narration during tool-call sequences — when an agent is making 20–30 calls in a row, narrating builds developer trust even when it doesn't change correctness. Second, a new "tool search" primitive in the Responses API: instead of stuffing every tool into context, the model dynamically selects which to load at inference time.

What's still brittle: ambiguous intent from non-technical users (vibe coding), and tool selection when the catalog of tools is large. The next unlock he flagged: multi-hour autonomous runs with no human interruption.

On the startup side, the data is striking — teams of 5–10 reaching tens of millions in ARR. He used the phrase "swarm recruiting": a few exceptional people outperform volume hires. Engineering, product, and design lanes are blurring. Fastest-growing sectors: legal (Harvey) and healthcare (Abridge, Ambience) — both characterized by unstructured data + expensive talent.

⟡ Why this matters for us
  • Audit prompt harnesses for token bloat. Long system prompts that worked in 2024 are now an active cost line. Treat them like SQL query plans — look for the unused joins.
  • Test the Responses API tool-search feature on any multi-tool agent we run. Particularly relevant if we wire up 10+ data sources or internal APIs.
  • Healthcare and legal patterns transfer. Unstructured data + expert review is the same pattern we have with clinical/research/financial documents.
07

Agents without memory are paying for amnesia.

Session 07
14:10–15:15
65 min
02:10 — 03:15 PM
Oracle
Eli Schilling
Cloud Architect

Context & memory engineering with Oracle AI Database

Eli's argument: without a memory layer, agents rebuild context from scratch every session. That means hallucinations, repeated mistakes, and token costs that accumulate silently.

He laid out a clean three-part taxonomy: short-term memory (within a session, semantic-hashed for fast lookup); long-term procedural memory (solutions to recurring problems); and long-term episodic/semantic memory (past conversations, knowledge bases, tool logs). Each plays a different role and needs different storage.

The data-layer challenge for agents is particularly worth flagging: a single agent typically needs relational, vector, graph, spatial, JSON, and time-series data. Naively that's 4–6 databases stitched together — operational overhead nightmare. Oracle's pitch is a converged engine with on-the-fly vector embeddings, but the architectural point holds regardless of vendor: fewer, more capable data stores.

The mental model he pushed — "the Three Musketeers": Context Engineering (purpose-driven prompts, avoid bloat), Memory Engineering (durable retrieval across sessions, teams, orgs), and the Agent Loop (assemble context → invoke LLM → iterate → extract learnings → write back to memory).

Glossary — memory primitives, quickly
Short-term (working) memory
State held within a single agent session. Semantic hashes accelerate retrieval. Think: the variables in your notebook kernel.
Procedural memory
"How I solved problem X last time." Reusable solution patterns. Think: a saved feature-engineering pipeline.
Episodic / semantic memory
Specific past interactions and the general knowledge derived from them. Think: a vector store of past findings tagged by source.
Agent flywheel
The pattern where repeated answers in short-term memory get promoted to procedural memory over time. The agent gets faster at things it does often.
⟡ Why this matters for us
  • If we're running agents without a memory layer, we're paying token costs on every restart. Worth measuring before adding to the architecture.
  • Evaluate a converged data store. If we're stitching together Postgres + Pinecone + S3 + Neo4j for one agent, that's a tell.
  • Free DeepLearning.AI course on agent memory exists — worth assigning. A 4-hour Bay Area deep-dive is also coming up.
08

Multimodal as the default, not the feature.

Session 08
10:50–11:05
15 min
10:50 — 11:05 AM
Google DeepMind
Paige Bailey
Engineering Lead, Developer Relations

Research to reality — Gemini, Gemma, and the model ladder

Paige's framing: Gemini 3 is natively multimodal — video, images, audio, text, code in and out. The 3.1 Live model can hold a conversation, share screen, and accept video input, which matters for robotics and AR more than for typical SaaS work.

Paige Bailey on stage with a slide showing AR glasses overlaying navigation and weather info on a city street
Paige Bailey · Google DeepMindMultimodal in, multimodal out

The model ladder is now wider than it was a year ago. Pro for largest tasks. Flash as the everyday workhorse (used internally across Google products). 3.1 Flash Lite for performance-critical paths. Nano on-device. The new Gemma 4 open family ships in four sizes — 2B / 4B / 26B MoE / 31B dense — under an Apache license. The 2B fits on a phone; the 4B runs on a laptop.

The most interesting piece in this section was Gemma 4's performance-per-size curve. Gemma punches above its weight, which means the right question is no longer "frontier or nothing" — it's "where does a fine-tuned small open model beat a generic frontier API at a tenth the cost?"

⟡ Why this matters for us
  • Run a Gemma 4 evaluation pass. Specifically against tasks where we currently call a frontier model: classification, extraction, structured-output generation. The Apache license clears legal hurdles for our use case.
  • Multimodal input as input, not novelty. Documents, scanned forms, charts in screenshots — we have all of these and we still treat them as text-only.
  • The on-device story (Nano, Gemma 2B) opens experiments we previously dismissed on latency or privacy grounds.

Pier 48 — the conference, mapped.

Three stages, ~50 exhibitors, four food stations, and a sponsor lounge spread across one of San Francisco's old waterfront sheds. The 401–403 zone (Oracle, Actian, AMD) sat directly across from Stage 1 — convenient if you wanted to chat with a speaker right after their talk.

The exhibitor list reads like a map of where the agent stack is being assembled: data layers (Chroma, Qdrant, Neo4j, Redis, Snowflake), reliability (Datadog, CircleCI, Temporal), agent frameworks (LangChain, CopilotKit, CrewAI), code tools (Cursor, JetBrains, CodeRabbit, Sonar), and a long tail of newer names worth watching.

AI Dev 26 SF venue floor plan showing Stage 1, Stage 2, Stage 3, exhibitor areas 101–122 and 301–308 and 401–403, sponsor lounge, food stations, registration, and the full exhibitor guide
Pier 48 floor plan3 stages · ~50 exhibitors
Day Two
April 29 · Agents in Practice, Quality, Reliability
09

The third era of software development.

Session 09
09:00–09:40
40 min
09:00 — 09:40 AM
Cursor
Amrita Venkatraman (Anastasia, Field Eng.)
Field Engineering, Cursor

From tab completion to cloud-native agent factories

The Cursor team frames AI coding as three eras. Tab completion (next-event prediction over your shoulder). Agentic coding (AI in the driver's seat, you steer). Agent systems (teams of agents kicked off in the background, doing increasingly autonomous tasks). We're transitioning from era two to era three right now.

The enterprise data they shared: over 60% of code lines in Cursor's enterprise deployments are changed by AI in some form (as of March). That number reframes the code-review conversation entirely — you can't manually review what AI is producing at that volume.

Cursor's internal workflow is a planner agent that delegates to worker agents, which then push to git. Different from a feature-branch team, similar to one in spirit. The headline anecdote: they ran a system of agents uninterrupted for one week — it generated ~3 million lines of code and built a working browser from scratch.

"There is no barrier to ideas anymore." — Anastasia, Cursor (quoting an Uber driver who'd just started shipping code)
⟡ Why this matters for us
  • The job is shifting to specification, review, and orchestration — not typing.
  • Review tooling needs to catch up with generation throughput. AI code review (CodeRabbit, Graphite) is now a stack we should evaluate, not a curiosity.
  • The planner-+-workers pattern transfers directly to our DS pipelines: one orchestrator agent, several specialist agents for cleaning, feature gen, modeling, eval.
10

The coding agent multiverse.

Session 10
09:45–10:25
40 min
09:45 — 10:25 AM
Databricks
Ankit Mathur
AI Infrastructure, Databricks

Coding agents at enterprise scale

Ankit's framing is honest: code is cheaper to build than ever, but infrastructure is expensive to run. Every enterprise is now building several flavors of agent — customer-facing support, internal copilots, custom workflows on top of platforms like Databricks — and the operational question is harder than the model question.

The "multiverse" framing came from cataloguing the variants: agents built on frameworks (LangChain, LlamaIndex), agents built on platforms, agents that are the platform. Each has different deployment, observability, and cost characteristics. Treating "agent" as one thing in our planning is going to mislead us.

⟡ Why this matters for us
  • Cost modeling for agents ≠ cost modeling for models. Tokens are one line item. Storage, retrieval, eval harness, observability, and failure-replay are bigger ones at scale.
  • When we scope a new agent project, name which variant we mean — framework, platform-extension, or standalone service — and price each differently.
  • Lean on platforms we already pay for before building a custom stack.
11

Containment first, model-driven safety later.

Session 11
10:30–11:10
40 min
10:30 — 11:10 AM
Docker
Tushar Jain
Docker, Agent Platforms

Shipping agents safely — boundaries that actually work

Tushar's opening was an honest admission: post-December he and his team got "addicted" to running agents constantly. Productive. But also: they were running them with permissive permissions, on laptops full of credentials, just to skip the safety friction.

The argument: model-driven safety is a layer, but it shouldn't be the first layer. Cloud's auto-mode is a step forward — but it's still asking the model to refuse. Docker's stance is to put containment at the root: a new micro-VM layer that runs across Mac, Linux, Windows. The agent runs inside the VM boundary, policy applies outside, and even potential prompt injections happen outside the trust boundary.

The dev experience matters: they showed terminal use (a new spx command that wraps any agent), and editor integration (Zed + Codex via ACP, all sandboxed). The principle: don't make safety a friction tax that engineers route around.

⟡ Why this matters for us
  • If we're running coding agents on developer laptops with full creds, we have a problem. Worth piloting a sandboxed runtime before the first incident.
  • The principle is layered defense: containment → policy → model-level guardrails, in that order.
  • Sandboxed agents make experimentation safer — engineers will try more things if they're not afraid of rm -rf.
12

Vibe coding — the new unit of work is the prompt.

Session 12
13:00–13:40
40 min
01:00 — 01:40 PM
Replit
Brandon Middleton
Replit · Vibe Coding Master Class

Vibe coding master class

Brandon's frame: when he learned to code, the unit of work was the line of code and expertise lived in syntax. Today the unit of work is the prompt. Expertise lives in three places — specification, review, and orchestration.

Specification means knowing what you're actually asking the agent to do and being able to articulate it clearly. Review is reading code you didn't write, looking for security and correctness issues, getting comfortable with that. Orchestration is stitching working systems together that survive contact with reality, not just the ideal academic environment.

His teaching method is project-first ("learning by project") and the Replit team is leaning into a buildathon format — students ship working software in an afternoon. Things that took him nine months at the start of his career now take 48 hours.

⟡ Why this matters for us
  • Train the team on specification, not on agent-frameworks. The frameworks change every quarter; the ability to clearly describe what you want doesn't.
  • Internal buildathons are a real format — not a gimmick. Pair them with a quality gate (see CodeRabbit / Sonar below) and you've trained both speed and judgment.
  • Three skills to evaluate ourselves on: spec clarity, review fluency, orchestration competence.
13

Every change is a hypothesis until proven.

Session 13
13:45–14:25
40 min
01:45 — 02:25 PM
CodeRabbit
Erik Thorelli
CodeRabbit

Deploying AI code review at scale — turning velocity into a quality gate

CodeRabbit's pitch is the structural mirror of Cursor's. If AI is now writing 60%+ of enterprise code, then AI also needs to be doing first-pass review — because human reviewers can't keep up at that throughput, and human reviewers fatigue.

The framing that resonated: "Every change is a hypothesis until proven." That's a deeply scientific frame for software — and it transfers cleanly to anything experiment-shaped. Every feature added, every model retrained, every threshold changed is a hypothesis about the world.

"Every change is a hypothesis until proven." — Erik Thorelli, CodeRabbit
⟡ The takeaway
  • AI-generated code needs AI-led first-pass review. Then human review on the residual.
  • The "hypothesis until proven" frame should apply to model PRs and product PRs alike. What's the test? What's the rollback? What's the success metric?
  • Eval coverage is to ML what test coverage is to code. Underrated and worth investing in.
14

Can LLMs generate enterprise-quality code?

Session 14
14:30–15:10
40 min
02:30 — 03:10 PM
Sonar
Tom Howlett
Sonar (SonarQube)

The enterprise quality gap

Tom's opening analogy: the current LLM moment looks like the 90s web. Anyone can spin up a website that works. Then you try to deploy it in the enterprise and the whole SDLC isn't ready. The tooling, the processes, the infrastructure are catching up.

The Carnegie Mellon study he cited is sobering: when teams adopt Cursor and similar tools, productivity does spike 3–5× initially. But static-analysis findings rise. Code complexity rises. And by around month three, the team is slowing down again — fixing bugs in code they don't fully understand. The output compounds; the problems compound faster.

His distinction matters: LLMs naturally produce code at a certain quality level — fine for prototypes and internal tooling, not yet fine for long-lived mission-critical applications. If you're shipping a 50k-LOC internal app you'll maintain for six months, current tools are great. If you're shipping a system that'll run for years against adversarial users, the SDLC has to wrap the model with stricter quality gates.

⟡ Why this matters for us
  • Match tool to lifespan. Throwaway analysis notebooks → full vibe coding. Production model serving → much stricter quality gates.
  • Don't celebrate week-one velocity without measuring month-three velocity.
  • Static analysis, type discipline, and tests aren't outdated — they're how we afford to ship faster.
15

Code, staff, quality — pick which problem you're actually solving.

Session 15
15:30–16:10
40 min
03:30 — 04:10 PM
JetBrains
Paul Everitt
JetBrains, Developer Advocate

The shift to agentic engineering

Paul's session was the conference's most direct pushback on hype. His thesis: coding was never the bottleneck. Productivity is the product of five things, and speeding up coding alone gets you maybe 10% of the win — the DX study and others bear this out.

Paul Everitt on stage in front of a slide reading 'Nobody knows anything. Everybody knows everything. This is fine.'
Paul Everitt · JetBrainsNobody knows anything. This is fine.

He cited a 2025 study that "95% of companies got nothing out of their AI initiatives." Not because the models are bad — because organizational capability lags model capability. And he flagged quality as the bigger emerging problem: a 50% defect rate on agent-shipped code is structurally bad, and Simon Willison's "Challenger disaster" framing — fully autonomous agents shoving things into production without human gates — is the failure mode to worry about.

His core point on quality: we don't yet have a good shared definition of "good." Pass-at-N benchmarks measure correctness on isolated tasks but say nothing about long-term maintainability, security posture, or operational behavior. Agentic engineering as a discipline is partly about building that definition.

"Coding has never been the bottleneck. Code quality is the bigger issue. We need a good way to define 'good.'" — Paul Everitt, JetBrains
⟡ Why this matters for us
  • Pick the bottleneck before picking the tool. If our slow path is stakeholder alignment, faster code generation won't help.
  • Define "good" for the outputs we ship beyond accuracy. Calibration, fairness, robustness, latency, explainability — each is a quality dimension that needs its own threshold and its own owner.
  • Spec-driven development as a practice — the spec is the contract the agent and the reviewer both check against.
16

The next 100 agents — and the office they live in.

Session 16
16:15–16:55
40 min
04:15 — 04:55 PM
Datadog
Diamond Bishop
Datadog

Building the agent-native office

Diamond's frame is operational: most companies are currently running one or two agents in production. The interesting question is what happens when that number is 100 — agents doing finance reconciliations, support triage, deployment gating, security review, data quality, expense reporting.

Diamond Bishop on stage with the title slide 'Your Next Hundred Agents — and how to prepare for the next thousand'
Diamond Bishop · DatadogFrom one agent to one hundred

That world needs the same things software always needed at scale: observability, tracing, evaluation, and graceful failure. But the primitives are different. Agent traces span tool calls, model calls, and downstream side-effects. Errors are often semantic rather than syntactic ("the agent did the wrong thing correctly"). Evaluation has to be continuous, not just at release.

⟡ Why this matters for us
  • Build observability before you build the second agent. Adding tracing later costs 10× more.
  • Semantic errors need semantic monitors. "Tool call returned 200 OK" doesn't mean the agent did the right thing.
  • The agent-native office turns dashboards into control panels — humans review exceptions, not the median case.
17

Multi-model pipelines — better results for less.

Session 17
17:00–17:20
20 min
05:00 — 05:20 PM
Zencoder
Andrew Filev
Zencoder

Multi-model pipelines — how to get better AI results for less

Andrew's closing-day session was the practical capstone: route different parts of a task to different models. A cheap fast model for triage and routing. A frontier model for the actual hard reasoning step. Often a smaller, specialized model for the last-mile cleanup.

The math is straightforward once you stop treating "the model" as one decision. A pipeline that uses Haiku-class for 80% of the work and Opus-class for 20% can match the quality of an Opus-everywhere approach at a fraction of the cost. The engineering challenge isn't the routing — it's having good evals to verify the cheaper paths haven't degraded quality.

⟡ Why this matters for us
  • Stop defaulting to one model. Every step in our pipeline is its own model decision.
  • Routing logic needs eval coverage. Otherwise you're saving money on something that quietly got worse.
  • Smaller open models (Gemma 4, etc.) become routable destinations once we have the eval harness in place.
Emma McGrattan, CTO of Actian, on stage with a slide titled 'AI Is Entering Its Production Era' covering regulatory pressure, latency reality, and data residency
Emma McGrattan · ActianThe production-era turn

"The first wave was about what's possible. The second wave is about what works reliably, securely, at scale."

Emma McGrattan, CTO of Actian, framed it most directly — but it was the through-line of the entire two days. Every keynote, panel, and breakout was about the same shift: from can it work? to can it work every time, at our scale, under our constraints?

Regulatory pressure, latency reality, and data residency are no longer edge-case concerns — they're shaping where the agent stack lives (cloud vs. on-prem vs. edge) and how it's built. The action items below all flow from that same shift.

18

Eight things to actually do this quarter.

Action plan
Q3 / 26
Owners TBD
01

Data quality audit on AI pipeline inputs.

Before any agent queries our datasets, find the silent NULLs, the ambiguous columns, and the joins nobody documented. Joe Reis was right.

Owner: TBD
02

Defect-severity matrix for every production agent.

Frequency × severity, per AWS framing. We classify agent tasks by what happens when they fail — not just by accuracy.

Owner: TBD
03

Pilot one workflow fully AI end-to-end.

Per Andrew Ng — push past the 80% ceiling on one low-risk workflow (probably eval-harness generation or dataset docs). Measure the speed gain.

Owner: TBD
04

Evaluate an agent memory architecture.

Short-term + procedural + episodic. Whether Oracle, a vector store + Postgres, or something simpler — the goal is to stop paying token costs on every restart.

Owner: TBD
05

Test the OpenAI Responses API tool-search.

For any multi-tool agent we run, see if dynamic tool selection reduces context size and improves selection quality.

Owner: TBD
06

Sandbox runtime for coding agents.

Per Docker. Stop running agents with full developer creds on bare metal. Pilot a containerized runtime before the first incident, not after.

Owner: TBD
07

Multi-model routing audit.

Per Zencoder. Inventory the model calls we're making. Identify the steps where a smaller, cheaper model would do the job. Build the eval to prove it.

Owner: TBD
08

Reframe team KPIs around intent velocity.

Per AMD. Idea-to-production, not PRs-per-week. Pick three workflows where we can measure this honestly over a quarter.

Owner: TBD