Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit

The Pragmatic Engineer 29min 4 min #67
Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit
Watch on YouTube

Summary

  • Context and framing: This is a keynote by Laura Tacho, CTO, at The Pragmatic Summit, drawing an extended analogy between the space race and the current AI era. The core argument is that AI is an accelerator and multiplier, not a silver bullet, and that organizations win with AI only when they apply it to real systems-level problems rather than treating it as a surface-level coding tool. The talk blends new industry benchmark data, case studies, and lessons from a retreat with Martin Fowler, Kent Beck, and others.

New industry benchmarks (Q1 2026)

  • Adoption is near-universal but uneven in impact:
    • 92.6% of developers across 121,000 developers at 450+ companies use an AI coding assistant at least once a month; about 75% use one at least weekly.
    • Self-reported time savings hover around 4 hours per week per developer, roughly a 10% productivity signal, consistent with recent Google research.
  • AI-authored code in production is rising fast:
    • Across ~42,600 developers, 26.9% of code merged into production was AI-authored (no significant human intervention), up from 22% last quarter.
    • Daily AI users have crossed the 30% threshold for AI-authored code reaching production.
  • Onboarding has been cut in half:
    • Time to a developer’s 10th PR has roughly halved from Q1 2024 to Q4 2025.
    • A separate Microsoft study (Brian Hulkait, co-author of the SPACE framework) found that early onboarding performance correlates with productivity for the engineer’s first two years at a company.
    • AI is effective not just for new hires but also for engineers moving across projects and non-engineers onboarding into technical work.

Why averages mislead and impact is uneven

  • Averages hide divergence: Industry-wide numbers stay stable while individual organizations move in opposite directions.
  • AI amplifies existing organizational health:
    • Some organizations see 50% fewer customer-facing incidents, higher code maintainability, and higher change confidence.
    • Others, especially those already dysfunctional, see twice as many customer-facing incidents and are “dysfunctional faster.”
  • High adoption, low transformation:
    • A July 2025 MIT study (“Gen AI Divide”) of 152 organizations found ~90% adoption but low transformation.
    • Most AI is deployed at the level of individual coding tasks, which has a low ceiling for productivity gain.
    • Real economic impact requires organizational change management, not just tool licenses.

Agents and agentic workflows

  • Agentic use is growing but still early in measurement:
    • Sample of ~3,000 developers at 6 companies that are ahead of the curve in instrumenting agentic workflows.
    • ~80% of developers use agentic workflows at least weekly; over 50% use them daily.
  • Codex signals:
    • Codex desktop app launched February 2, 2026, with over 1 million downloads and 60% user growth in the last week.
    • OpenAI reports 95% of its internal developers use Codex to ship, and Codex users ship ~60% more PRs per week than users of other AI tools.

Case studies

  • Haven Headache and Migraine Center (healthcare startup):
    • Uses agentic workflows (e.g., Ralph loops) to rapidly prototype patient workflows from Linear and Figma artifacts, producing high-quality prototypes with strong documentation and tests.
    • Trains a HIPAA-compliant model on hundreds of thousands of symptom logs to route patients to medication refills or follow-up appointments.
    • Results: 3x industry-average customer satisfaction, fewer headache days per month, and reduced headache severity for patients.
  • Cisco:
    • 18,000 engineers use Codex daily for complex migrations and code review, achieving a 50% reduction in code review time.
  • JPMorgan Chase:
    • Built a multi-agent framework for annotation (MAFA) where specialized agents annotate interactions, and other agents rerank, calibrate, and validate outputs.
    • Uses consensus algorithms among agents; Laura identifies agent consensus as a major problem to solve in 2026.

What winning organizations do

  • 1. Set goals and measure progress:
    • “Spray and pray” (giving everyone licenses and hoping for the best) does not work.
    • Winning organizations tie AI experimentation to concrete problems and measure whether goals are being met.
    • Laura co-authored an AI measurement framework with Abby Nott (CEO of DX) that tracks not just AI usage/adoption but also impact on speed, developer experience, quality, and innovation ratio, alongside cost.
  • 2. Treat developer experience (DevX) as foundational:
    • Fast CI, clear documentation, well-defined services, and strong testing are critical for agentic workflows.
    • AI time savings alone do not compensate for bad meeting culture, constant interruptions, unplanned work, or slow build/test cycles.
    • Winning organizations use AI to attack those systemic problems (e.g., reducing meeting frequency, improving CI wait time, reducing dev environment toil) rather than only accelerating individual coding tasks.
  • 3. Experiment by solving real customer problems:
    • Unbounded experimentation (e.g., Gas Town-style projects) is fun and expands possibility, but sustainable organizational results come from focusing experimentation on real customer problems.

AI readiness and organizational models

  • DORA AI Capabilities Model:
    • An AI readiness framework correlating organizational practices with good AI outcomes.
    • Having a clear, communicated AI stance is one factor linked to better organizational results.
  • Thoughtworks Forest Framework:
    • Another well-researched AI readiness model, useful for internal audits or convincing leadership to invest in foundational practices.

Key tensions and conclusions

  • AI does not fix organizational systems problems by itself:
    • From a retreat with Martin Fowler, Kent Beck, Steve Yegge, and others: AI can be applied to system problems, but organizations must first acknowledge and address human and systems-level constraints.
    • Without that, organizations “take their problems to space with them.”
  • The space race analogy as a guide:
    • The point of going to the moon was not to live on the moon; it was to improve life on Earth.
    • Similarly, the value of AI and agents is not in unbounded experimentation for its own sake, but in applying that expanded capability to real organizational and customer problems.
  • Closing message: Stay grounded, stay skeptical, stay human, and stay pragmatic while embracing the sense of possibility that AI and agents create.
Back to The Pragmatic Engineer