How Langston Ships With AI

Three engineers in an eighteen-person company. AI writes the large majority of our production code, merges deploy themselves, and code review is the last human gate standing. Here is how that actually works, what it costs, and what broke.

Written by Claude, from an interview with and refinements by Neil Poulin, VP of Engineering. This is one person's view of where our tooling stands today, not a formal engineering specification.

Some context for scale: we're eighteen people, three of whom are engineers. Most of the company is non-engineering, and we don't have a dedicated product manager — product direction happens across the team rather than through one person's roadmap. Almost everything below is shaped by that.

Two things have mattered more than any particular model: making AI-built tools shareable, and getting agents to reason across our whole system instead of one repository at a time. Most people can get an LLM to build something great for themselves; the hard part is making it work for everyone else. Almost everything below exists because I forced that constraint from day one.

For a fuller account of how our engineering practice changed, Alex Greco, a senior engineer on our team, wrote it up in two posts — the before state and the after: 1% AI and 99% AI.

The short version of his arc: in early 2025 AI wrote maybe 1% of our production code and mostly answered Stack Overflow-style questions. By late 2025, after we studied how experienced developers were actually using these tools and rebuilt our workflow around agents, it was writing the large majority of it. The work didn't disappear — it moved upstream into planning, design, and review.

Cross-repository context

I'd guess your company doesn't run off a single repository either. Ours is seven — the API, the front end, database migrations, a few services, and our infrastructure — but they're one system, not seven projects. To reason about anything real, an agent needs to read the front-end code, the API code, and the database schema together.

Almost all the guidance and tooling out there still treats a single repository as the unit of work. Claude handles crossing that boundary reasonably well now in some cases, but it's still not a first-class use case anywhere.

So from the beginning I set our agents up a level above the repos: everything is checked out as siblings under a parent folder, and I run agent sessions from that parent. The shared instructions live there and describe the whole structure — what each codebase does, how they interact, how to work across them. That change alone improved the quality of what agents produce more than anything else I've done. Alex describes the same setup in his second post as the foundation everything else rests on: the benefits of a monorepo without the cost of actually building one.

If parts of your system are genuinely isolated and you'd rather an agent only see one of them, fine. That's not us, and I'd guess it's not most teams.

Two habits follow from this:

• Configure cross-repository review context. Greptile supports it, and we set it per repository — reviewing a change in the API repo also pulls in the merge service and the database migrations so it can catch bugs across the seam.

• Cross-link related pull requests. When a change spans a database migration, an API change, and a front-end change, each PR links the others and states the merge order. With that, Greptile reads the companion PRs before reviewing and stops flagging calls to endpoints that "don't exist" because they only exist in the open PR next door.

Code review — the one real gate

The part that carries the most weight

Any employee can open a pull request now, through Claude in Slack or Buzz. And I've stripped the human steps out of our release pipeline: once a pull request merges to main, it ships to production automatically, database changes included. That was a big change for us.

I think human-in-the-loop oversight has to live somewhere, and for now what we've settled on is code review.

What I'm less sure about is whether that gate stays human. The alternative is trading review for recovery: no human step, but monitoring good enough to catch problems fast and a pipeline that can fix forward in minutes. I don't think we're there, and I'd want much better tests and observability before trying. But I don't assume code review is the permanent answer — it's the answer while the cost of being wrong is still high.

Greptile does the automated pass and has gotten steadily better. It reviews every pull request on open and gives it a confidence score. It builds a knowledge base of our product as it reviews, and I've refined the rules we give it — they live in the repo alongside the code — so its feedback matches our actual code preferences instead of generic best practices. It catches real bugs, not just style nits. Alex puts its useful-suggestion rate around 85%, which matches my experience.

We deliberately don't run Greptile on every commit — too noisy. Instead our agents know how to request a fresh review and incorporate the feedback, iterating until the score clears our bar. We evaluated the alternatives (CodeRabbit, Devin's review product, a couple of others); Greptile suited us best, though I've heard good things about CodeRabbit from others. GitHub Copilot also reviews — less impressive, but improving, and it occasionally catches something Greptile misses. Cheap enough to keep both.

Two things back that gate up:

• Tests. We went from almost none — a few unit tests on simple functions — to meaningful coverage, including a harness that spins up a real database next to the tests so we can exercise an endpoint end to end. That part is still early, more experiment than backbone, but it's the foundation for covering our most sensitive flows.

• Automated deploy-order checks. A change can declare which database migrations it depends on, and both the pull request and the release pipeline verify those have actually shipped. That used to be a developer remembering to deploy in the right order. Now it's checked, and if it's wrong the release fails instead of breaking production.

Scrutiny follows the data

We don't review everything equally. Scrutiny scales with distance from the data model. Anything touching database schema, persistence, or how we think about storing data gets the most attention, because everything else flows from it. The analysis engine — the code that computes our statistics — gets nearly as much, because an error there is silent and still ends up in every report. A React component gets much less — it's low risk, it's far from the data, and if it's wrong it's easy to fix. That seems to be where a lot of other people have landed too, and it matches how I'd triage a human's pull request.

The rest of the answer is speed of recovery rather than prevention. Our philosophy is that we need to be confident we can get in and fix things quickly. Worst case, merge to running in production is about five minutes across the full stack; sometimes it's closer to one. We have error reporting and logging, and — this part matters — developers' own agent tools can read logs from our hosted environment, so diagnosing an incident is something you can do inside the same session where you're writing the fix. A lot of the work I did to shorten the deploy path was in service of that: find it, fix it, patch production fast.

We also try to bake blast radius into how agents work. Is this greenfield, or is it touching something core that other parts of the system depend on? If I open a pull request and everything lives in a new folder, leans on existing services, and follows our established patterns and permissions libraries, that's close to a green light as long as it solves the problem. If it reaches into something sensitive, it needs real scrutiny — and ideally the agentic system flags that itself.

I don't have a clean answer for how to do this correctly, and I suspect there isn't one. My approach has been to ask what I'd do as a human reviewer — what questions would I ask, where would I go for the answers, what would I do about them — and then build that into the system, or at least give both the agents and the humans the tools to answer those questions themselves.

Some of that lives in our internal wiki and our architectural decision records. We stamp decision logs on the larger calls as we make them, along with where we're trying to go. That matters because we often ship a small slice of something that doesn't obviously make sense on its own, but does in the context of the system we're building toward. Giving agents that broader view has helped with blast radius more than any single rule.

A shared control-plane repository

One internal repository holds the shared agent instructions, our skill files (task-specific playbooks an agent loads on demand), and our database MCP server. MCP is the standard way to give an agent a new tool. Because it's a repo, improvements are reviewed, versioned, and immediately available to the whole team — nobody maintains a private prompt collection.

What's in it:

• A database MCP server that lets an engineer's agent query our local, staging, and production-replica databases. Read-only is enforced in the server and again by the replica itself.

• Full-stack development guidance — how to spin up an isolated environment for a feature branch spanning multiple repos, plus the tooling to manage it, including a local dashboard showing every active feature environment, its services, and its open pull requests, with links out to the Linear issue it belongs to.

• Skill files documenting how and when to use each of these.

Alex's post covers how these get used in practice: a planning agent that knows the project and talks to Linear, then a building agent that knows our style and architecture, then review.

Agentic issue triage

The biggest lever of the last few months

Anyone on the team files an issue in Linear. From there an agent picks it up and runs it forward: it researches the issue against our codebase, comes back to the requester with a question when the ask is underspecified, and moves the issue into implementation once it's confident it has what it needs. At that point it can write across any of our repositories. It can spin up a preview environment and post that link back to whoever asked, so they can try the change and sign off before an engineer is involved. It ends by opening a pull request that it iterates against Greptile until the score clears our bar — only then does it ask a human to look.

How it's built, since the architecture is the interesting part:

• Linear is the source of truth, not our codebases. Requirements, decisions, conversation, and each agent's progress all live on the issue. A fresh agent with no context can pick up where the last one stopped and lose nothing, and we don't litter the repositories with planning files.

• Our own API is the control plane. A webhook fires when an issue changes, our API claims it, and dispatches a run to a GitHub Actions runner with its own environment — one that can query the database securely, pull more context from Linear, clone repositories, and open pull requests. The run reports back a structured result: what it completed, what's still open, what status the issue should move to next. We persist that onto the Linear issue as a comment.

• It's a state machine, deliberately modeled on how I'd move a piece of work through planning, triage, and building myself — including explicit points where a human should be pulled in. Those handoffs are meant to be obvious and stable, so work can be put down and picked back up.

• Scheduled sweeps re-examine issues in each state: has someone left a question, has the pull request moved, did a reviewer leave feedback worth incorporating? If so, it wakes another session to carry the work as far forward as it can.

Getting here was iterative and cost me real time. We went through several approaches to tracking issue and decision context first — a separate repository of markdown files, a tool called Beads, agents auto-generating their own issues, others. Linear is where it landed and I'm happy with it.

It works. Issues go from filed to reviewable pull request fast, and there's no question it's shortened the path from someone reporting a bug or asking for a feature to that thing actually existing.

Making it cheap to build moved the bottleneck to review. We have a standing queue of well over a hundred open pull requests against a team of three.

Even merging a dozen a day, roughly that many get opened across our various pipelines. It's genuinely overwhelming, and the work in front of me is tuning the system to produce less and better rather than simply more. There are real rough edges left. But the only way I could have found them was by building it and running it.

Data access

Two AI surfaces reach our data, and they reach it differently:

• Langston Bot, the Slack bot I built in-house, writes read-only SQL against the production Postgres database.

• Claude in Slack, Anthropic's Slack integration, queries BigQuery — the analytics copy of that same data — through a dedicated read-only service account scoped to a fixed set of datasets.

Separately, engineers reach the production replica as themselves. Access is granted by membership in a Google Group and enforced by Google Cloud IAM, so adding or revoking someone is a group membership change rather than a credential rotation. That plumbing took real effort. The result is that when someone wants a report, they can usually just ask rather than filing a request with engineering.

Personal assistant and internal wiki

I built a hosted dashboard, Almanac, that each person's Claude account writes into. Scheduled routines run in their own cloud Claude account, using their own connected tools — so we never hold anyone's credentials — and push a daily digest into their own personal dashboard, refreshing a few times through the day. They pull from Slack, Gmail, Calendar, and Linear, plus optional sources like our error tracker and our data warehouse. Because the routines are cloud-scheduled rather than running on a laptop, they run whether or not anyone's machine is open. It's a web app that installs on a phone.

People check items off, snooze them, leave themselves notes, and give the assistant feedback ("you're surfacing too much of this"), which it honors on its next pass.

It's tied into an internal wiki built on the pattern Karpathy described as an LLM wiki: a folder tree of markdown files an agent helps organize and cross-link, which other agents then read from. That's also where our architectural decision records live. Still early, but it's the same idea as Linear-as-source-of-truth applied to knowledge rather than work.

Slack: Langston Bot and Claude in Slack

I built Langston Bot in December 2025. Anyone can tag it in Slack and ask a question. It writes its own read-only SQL against the production database, and it searches a vector index covering our internal knowledge-base articles, our process documentation, our survey definitions, and the source of three of our core repositories. It also has direct tooling over our crosstab analysis output. So it chains semantic search with live queries and answers anything from "what is this conversion rate" to "what are the top insights from this project" to "how does this feature work" — the last one by searching an index of our source code.

It changed how people work more than anything else, for relatively little effort. The unlock was coaching, not capability. When someone asks me how something works, I'll often tag Langston Bot publicly, let it answer, then react in the thread — "yeah, that's right," or "that's not quite it, let me ask differently," or "I'm going to give it access to more data so it can do this better." That shows people what good prompting looks like, models evaluating the output instead of trusting it, and makes clear I'm still their partner on the problem rather than pushing them off onto a robot.

I haven't touched it much since. Claude in Slack is what I reach for more often now — it's more fluid. It reaches different data than Langston Bot: it queries BigQuery, reads our repositories through the Claude GitHub App, and reads our Linear issues. It can also write code and open pull requests.

It's also easier to steer than our own bot. It has a dedicated repository of its own holding its skill files and system instructions, and it can open pull requests against that repository itself when it runs across something it thinks could be improved. That hasn't happened often, but the fact that it's possible is the point.

Prompt by voice

Smallest change, biggest return

Worth passing along to anyone on your team. Most of my prompting is now spoken rather than typed.

It feels ridiculous for the first few days. But it removes writer's block entirely — you speak spontaneously, and the model doesn't care that you repeated yourself, stuttered, or circled back. You think out loud and get it all down. Distilling a large pile of text into what you actually meant is one of the things LLMs are genuinely good at. For getting an idea out of my head and into the model, nothing else comes close on bandwidth.

Claude and Codex have transcription built in. I use a third-party desktop app called Handy — keyboard shortcut to start, speak, stop, and it pastes the transcript into whatever chat box I'm in, with your choice of transcription model behind it. Worth trying for a week before deciding.

Letting a non-engineer ship product

Once an agent in Slack can write code and open pull requests, an obvious question follows: does the person driving it need to be an engineer? I ran an experiment to find out.

Tom, one of our co-founders, leads research. He's sharp, very AI-forward, and of anyone here he has the best read on what our users actually need, since he's a user himself. I gave him Claude in Slack and coached him on using it to build directly in our product.

The timing wasn't a coincidence: his first sprint was the week after I got preview environments working. That's what gave me permission, in my own head, to turn him loose — I finally had a way to say "before we merge this, here's a link where you can see your full-stack change running." He'd go from a prototype to real changes in the actual system, then open pull requests for me to approve, amend, or finish. I instructed the agent that I'm the final decision maker on technical questions — stop and ping me rather than guess.

It worked. Over a week or two and something like a hundred pull requests, we closed a large backlog of papercuts — the small gaps and friction points that had sat around because they were never worth prioritizing individually. Alongside that, a real facelift so the product matches how users actually expect to interact with it, plus a few larger swings that had never been well scoped and that Tom is now able to carry forward himself.

It also caused problems — the same volume problem the triage pipeline creates. A small team gets buried, and moving that fast broke a few things. But it's the closest our roadmap has ever tracked what users actually want, and it exposed a concrete list of gaps to close for both his experience and the reviewer's.

Preview environments are still the weak link. They aren't spun up or maintained as automatically or reliably as I'd like, so in practice they get used less than they should. When I'm there helping, they're hugely valuable.

I've since extended this to a few more people, partly through Buzz, Block's open-source tool for multiplayer interaction with agents. Results are mixed. The others are less iterative and less willing to just try things than Tom, so the payoff has been smaller though still useful. Buzz itself is early software and rough enough to be clunky.

Who reviews all this

Ownership works the way it always has: the developer who owns that feature reviews it. On a team of three that's usually obvious, either from the product area or because they started the work stream.

The bot-opened pull requests with no clear owner, I've taken on myself. Partly to absorb the load, but mostly because reading them is how I learn where the system is doing well, where it stumbles, and where it needs better guidance. Refining that system is a large part of my job now, and the pull requests are the evidence.

Cost

Without getting into our actual numbers, here are the broad strokes. We track AI spend company-wide as one line: the automated pipelines above, the AI features inside our own product (we ship an insights assistant and other AI-assisted tooling to customers), every developer's Claude Code and Codex usage, and the Slack bots. It rose sharply — several times over — in the months the agentic triage pipeline came online, tracking pull request volume almost exactly.

How much of that is the agentic workflow specifically, I couldn't tell you off the top of my head. Everything is scoped to separate service accounts and API keys, so it's separable in principle, and our ledger already splits it four ways: employee usage, API and system calls, developer tooling, and everything else. But that's spend by category, not by outcome. I'm building a system now to attribute it to specific features and products.

The goal is being able to say what a given feature cost in tokens across the full stack. At first that's a backward-looking reporting tool: are we spending more on internal developer tooling than we meant to, and is it good enough that we should stop? Should we slow a workstream down? Or — the case I'd like to be able to make — are the outcomes good enough per dollar that we should simply pour more in? Longer term I think scoping work shifts from estimating time to estimating cost. That project is early and may not pan out, but that's the direction.

What hasn't worked

A challenge I saw across my own team and in other orgs, and wanted to tackle early: anything that only works on one person's machine.

A local dashboard or script only you can run fails four ways. It goes stale when you stop maintaining it. Nobody else benefits. It creates divergent workflows. And it becomes a maintenance burden nobody else can pick up. If a tool is good, the instinct should be to get it in front of the team, not to keep it as personal tooling.

The concrete example is Beads, the issue-tracking system I tried before Linear. Nominally git-based, but effectively local, storing everything in JSON files no human could read. There was no way to report on it externally — I tried building dashboards over it and never got anywhere. That's what pushed us to Linear.

Also abandoned: various plugins and third-party add-ons to our coding harnesses, and API-based billing for Claude, which we moved to subscriptions to control cost.

None of these were wasted. Even Buzz, which I'd call mixed at best, forced me to build tooling and permissions that other things now depend on. Most of what "didn't work" taught me what does.

Where I'd start

There isn't a clean ordered list — it depends on your organization. But roughly:

• Get your hands dirty first. Use the tools, try the different coding harnesses (Claude Code, Codex), and learn what levers exist. Most of what I've built came from things I couldn't have predicted before starting.

• Turn repeated prompts into shared files. As soon as you're re-typing the same context, put it in a file the team shares, so everyone gets the same process.

• Store those centrally. The real unlock is that an engineering or product leader can update a shared procedure once and have it reach everyone immediately.

• Give the agents context and tools. Your issue tracker, bug reports, customer service requests, backlog, product vision — anything that makes a decision better-informed. Then write the guidance for when and how to use each.

• Get people excited. Show them what's possible, give them examples, let them be creative. The Slack bot changed our company more through coaching than through capability.

What the numbers showed

Pull request volume went from 212 merged in June to 430 in July — the month the agentic triage pipeline came online — and has stayed near that level since. More issues were opened than closed in every single month of the year, and the backlog grew by roughly 700 issues across those eight months. It never had a month of shrinking. Making it cheap to file and cheap to build did not make it cheap to decide.

The typical pull request merges within hours; the median has fallen from eighteen hours in January to about one now. But that only counts the ones that get merged. There are 117 open in our main repositories right now, median age sixteen days, and about three-quarters of them were opened by the agent. Merging isn't the constraint. Getting to them is.

Distinct pull request authors went from three to eight on a three-engineer team — the new ones are agents and non-engineers. Average churn per merged pull request roughly halved while volume rose, so the work got sliced into smaller pieces. That's the only reason the volume is reviewable at all.

From mid-July the pipeline began opening pull requests under its own account rather than a person's. About one in five merged pull requests now comes from an agent.

A couple of caveats. More pull requests isn't more value — plenty of these were papercuts. And this stretch included a product facelift and a backlog purge, so I wouldn't treat the last few months as our new baseline.

Resources

A course I signed up for but haven't taken yet: AI Hero by Matt Pocock. I've followed his TypeScript work for years and like how he approaches teaching, so I expect it to be good.

On social media, the advice I'd give first is to not follow forty accounts. Here's a starter set that gives you a usable timeline without drowning in AI Twitter:

• @bcherny: Claude Code. The primary signal for agentic coding in the terminal.
• @cursor_ai and @amanrsanger: the IDE most teams compare against. The founder account is lower volume and more substantive.
• @opencode and @thdxr: open-source coding agent. A useful counterweight to vendor marketing.
• @OpenAIDevs: Codex and OpenAI platform updates without the CEO-account noise.
• @AnthropicAI: Claude model and product releases.
• @OfficialLoganK: Gemini and the Google AI developer surface.
• @simonw: the best public notebook of what these tools actually do when tested. "Did this really work?"
• @karpathy: sparse, but the highest-signal technical intuition available. The wiki pattern above is his.
• @emollick: evidence on AI at work. The strongest account here for a non-engineer.
• @swyx: the ecosystem map. Helps place a new tool in a category.
• @chipro: production AI systems and evaluation tradeoffs.

Add if you're specifically doing coding agents in production: @hwchase17 (agent orchestration), @dexhorthy (agent control — "can we let this touch real systems?"), @zeeg (MCP and permissions friction), @clairevo (AI-augmented PM workflow), @FactoryAI and @aidenybai (competitor and code-review-bot comparisons).

Separately, these are the individuals I keep citing in my own research notes — a good proxy for who's actually producing durable ideas rather than takes:

• @mattpocockuk — TypeScript, now AI engineering. Clear and practical.

• @GergelyOrosz — The Pragmatic Engineer. The best source on how real teams are adapting, including a good survey on how teams decide which pull requests still get human review.

• @poteto — React Compiler, now at Cursor. Open-sourced the skill set she uses daily; the most concrete thinking I've seen on making agents prove their work.

• @VictorTaelin and @dzhng — the "review the decisions, not the diff" argument, which is the single most useful reframe I've adopted.

• @kentcdodds — building in the open on personal AI assistants and MCP.