TempoInstrument the baseline before interpreting change
Record normal delivery behavior before a rollout, enablement program, or policy change. Use a window with enough completed pull requests to describe the typical case and the slow tail, and note major changes in team composition or work mix.
The baseline should cover adoption, flow, quality constraints, and cost where available. Without it, a post-rollout number may look impressive while describing a seasonal shift or a different class of work.
- PR cycle time p50 and p90, split into draft, pickup, review, and deploy stages.
- PR throughput and size distribution.
- No-review, rework, rollback, or failure signals available to the team.
- AI tool availability, active use, and attribution coverage.
- Qualitative concerns about review burden, confidence, and code ownership.
Measure adoption from evidence with known coverage
Vendor telemetry, editor events, surveys, commit metadata, and local session matching measure different things. Report the method beside the rate so a change in instrumentation is not mistaken for a change in behavior.
Tempo focuses on positive evidence attached to commits and pull requests. That connects adoption to shipped artifacts while preserving an explicit unclassified category for work without a supported signal.
| Signal | What it establishes | Main limitation |
|---|---|---|
| License assignment | A developer had access | Does not establish active use or shipped work |
| Vendor activity | The assistant received or produced interactions | Tool-specific and difficult to compare across vendors |
| Survey response | A developer reports use or perceived impact | Subject to recall and perception bias |
| Git metadata | A commit or PR contains an explicit positive signal | Coverage depends on retained metadata |
| Local file matching | Assistant-touched files overlap a commit | Requires local instrumentation and supported formats |
Connect adoption to flow and balancing signals
Compare AI-assisted and unclassified work across similar repositories, work types, and time windows. Start with medians and distributions, then inspect the pull requests behind a visible difference.
| Dimension | Useful metrics | Risk of reading it alone |
|---|---|---|
| Adoption | AI commit rate, AI PR rate, active tools | More use does not prove more value |
| Flow | Cycle time, pickup, review, deploy time, throughput | Faster work may shift cost downstream |
| Review constraint | PR size, reviewer load, no-review rate, review rounds | More comments can mean rigor or confusion |
| Quality constraint | Rework, failed deployments, incidents, rollback signals | Rare events require longer windows |
| Economics | License and usage cost, retained time, delivery outcome | Time saved is not automatically business value |
Run a bounded measurement experiment
Choose a specific intervention, such as enabling one agent for two comparable repositories, training reviewers, or changing batch-size guidance. Record the hypothesis and the signals expected to improve or deteriorate.
A before-and-after comparison is evidence, not proof of causality. Team learning, project phase, staffing, and tooling can move at the same time. Use matched groups or staged rollouts when practical and state the remaining uncertainty.
- Hypothesis: the expected mechanism, not just the desired number.
- Population: teams, repositories, and work types included or excluded.
- Window: long enough to contain representative completed work.
- Outcome and balancing signals: what should improve and what must not degrade.
- Decision rule: what evidence would justify expanding, changing, or stopping the rollout.
Use the result to change the system
If adoption rises while review time and PR size worsen, the answer may be reviewer capacity, smaller batches, or better agent instructions—not removing the tool. If adoption stays low, investigate workflow fit and enablement before evaluating ROI.
Share findings as system learning. Avoid league tables of individual users, document uncertainty, and repeat the measurement as tools and working practices evolve.
Frequently asked questions
How do you measure AI coding productivity?
Measure adoption separately, then compare AI-assisted delivery artifacts with similar work across cycle time, throughput, review load, quality constraints, developer experience, and cost. No single output metric is sufficient.
Is AI acceptance rate a productivity metric?
It is a vendor-usage signal. Acceptance rate does not show whether accepted code shipped, reduced delivery time, improved outcomes, or created more review and maintenance work.
Can a before-and-after comparison prove AI caused the improvement?
No. It can support a hypothesis, but staffing, work mix, project phase, process changes, and learning effects may also explain the difference. Staged rollouts and matched comparisons reduce some uncertainty.
What should engineering leaders watch as AI adoption increases?
Watch whether PR volume or size grows faster than review capacity, whether the slow tail changes, whether no-review or failure signals increase, and whether teams remain confident owning AI-assisted code.
Sources and further reading
- DORA 2025 research reportResearch on AI-assisted software development and the organizational systems around delivery performance.
- SPACE framework for developer productivityWhy productivity measurement should span several dimensions rather than one output number.
- Tempo AI coding detection methodologyHow Tempo creates evidence-based AI attribution and reports its limitations.