|
 |
👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here.
Deepdive: How 10 tech companies choose the next generation of dev tools
Tech businesses from seed-stage startups to publicly-listed companies reveal how they select and roll out next-generation IDEs, CLIs and code review tools. And how they learn which ones work… and whic
|
| |||||||||||||||
Right now, it seems like almost every tech company is changing its developer tooling stack, which is a big shift from eighteen months ago when the answer to “what to use for AI-assisted coding?” was simple: buy a GitHub Copilot license and boot up ChatGPT. In our AI tooling survey in 2024, those two tools racked up more mentions than all the others combined.
But no more. Today, a plethora of tools outpace Copilot in various ways, like Cursor, Claude Code, Codex, and Gemini CLI, and there’s also AI code review tools like CodeRabbit, Graphite, and Greptile, not to mention all the MCP integrations which plug into agentic tools.
So, for this deepdive I asked 10 tech companies which tools their engineers use and, crucially, how they made their choices from among all the options. These businesses range from a 5-person seed-stage startup, to one that employs 1,500 people and is publicly listed. All are anonymous, except for Wealthsimple and WeTravel. WeTravel has also kindly shared the most detailed measurement framework I’ve yet seen.
We cover:
Speed, trust, & show-and-tell: how small teams select tools: At places with fewer than ~60 engineers, tooling decisions are fast and informal: developers try them for a couple of weeks and those which “stick” win.
How mid-to-large companies choose: bureaucracy, security, and vendor lock-ins. At companies with ~150 engineers, adoption is considerably slowed down by security reviews, compliance requirements, and executive-level budgetary considerations.
Measurement problem: metrics are needed but none work. Every workplace struggles to prove its AI tools work, and common metrics like lines-of-code-generated are distrusted by engineers.
How Wealthsimple measured and decided. The flagship Canadian consumer fintech ran a 2-month selection process to choose an AI code review tool. Rolling out Claude Code to all engineers was a decision made by the CTO, backed with a mix of personal conviction, validated by usage data from Jellyfish.
How one company accurately measures code review usefulness. WeTravel built a structured -3 to +3 scoring system across five dimensions, with five engineers evaluating ~100 comments. They found no AI code reviewer suitable for their codebase.
Comparative measurements at a large fintech. A team ran Copilot, Claude, and Cursor simultaneously across ~50 PRs, scoring ~450 comments. They found Cursor reviews the most precise, Claude the most balanced, and Copilot the most quality-focused.
Common patterns. Developer trust drives adoption more than mandates, the Copilot → Cursor → Claude Code migration path is well trodden, and nobody has cracked productivity measurement yet.
The goal of this article is to showcase what tech companies of different sizes are doing, and to offer a few pointers on measuring and comparing the tools. It’s hard to do, but not impossible, as two in-depth case studies illustrate, below.
Don’t forget, what matters is to find tools that work for your team. During this research, I found vendors which are beloved by one company and loathed in other workplaces. There’s no single vendor that’s universally rated by every team in all contexts.
As always, I have no affiliation with any vendor mentioned in this article, and was not paid to mention any of them. I used to be an investor in Graphite, but no longer am. For more details, see my ethics statement.
The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.
1. Speed, trust, & show-and-tell: how small teams select tools
Decisions are informal and made quickly at the smallest businesses in our survey, with the decisive factor being how people feel about the tools. Trial periods are short, at around two weeks, and individual developers have outsized influence on whether a tool is adopted or binned, with the decisions spreading organically. Below are two examples:
Seed-stage logistics startup (20 people, 5 engineers)
The head of engineering at this startup describes their approach as high-trust and developer-led:
“We agreed to try new tools for 2 weeks and see how everyone felt. We didn’t use any hard-and-fast measurement. TLDR: I trust our devs and their opinion is a big part of this”.
Developers there suggest which tools to try and decide whether to keep using them or to seek alternatives. For AI code reviews, the team first tried Korbit for around a week but the tool felt “off”, so they roadtested CodeRabbit which “stuck” within a few days:
“Within a few days of using CodeRabbit I could tell the devs just liked it and were embracing the suggestions, unlike with Korbit which they ignored when they’d lost trust [in it].”
And that was that: decision made. As a small team, it’s easy to switch to something better and it only takes a single engineer to suggest it.
The broader tooling stack of this startup has evolved quickly over the last year:
Figma for designs, which works nicely with Linear. The company has 5 devs and one UX designer.
Linear for ticketing and collaborating across UX and development. The UX person creates Linear tickets alongside her Figma designs.
Claude Code and Cursor for development, connected to Linear via MCP.
Claude Code writes tickets: a recent change which is working nicely with CodeRabbit, as more context is passed downstream for AI code review.
“Show and tells” – where team members show colleagues their tooling setups during weekly team meetings and demos – are used by this startup to identify which tools do or don’t work:
“Our show-and-tell process greatly helps. There are so many new tools, skills, IDEs, etc, that it can be overwhelming. We all learn from seeing what others in our team are doing.”
The team makes a clear distinction between company-wide tools like Claude and CodeRabbit that everyone is expected to use, and devs’ personal environments (IDE choice, terminal setup), over which individuals have full autonomy.