My Checklist Before I Let a New AI Coding Tool Anywhere Near a Client Project

There’s a new AI coding tool announced practically every week at this point — a new agent, a new IDE fork, a new CLI that promises to be the thing that finally makes the others obsolete. I used to try most of them, out of genuine curiosity and a nagging fear of missing whatever the actually-better tool turned out to be. That habit cost me more time than it saved, because switching tools mid-project has a real cost that doesn’t show up until later: a different mental model for how context gets built, different conventions for where config lives, and a workflow that doesn’t transfer cleanly. So I built an actual checklist I run before I let something new touch a client codebase, instead of adopting on vibes because a demo video looked impressive.

First: can it read my existing conventions file without me rewriting it for the new tool? Every project I maintain has some version of a CLAUDE.md or equivalent — architecture notes, naming conventions, the things a new contributor (human or agent) needs to not immediately break the codebase’s patterns. If a new tool needs its own bespoke config format to get equivalent context, that’s not automatically disqualifying, but it’s a real cost I weigh against whatever the tool claims to do better. I’ve skipped adopting tools purely because the migration cost of restating everything in a new format wasn’t worth what I’d gain.

Second: I test it on a throwaway branch of a real project, never a toy repo. Demo videos and quickstart tutorials are built on codebases with no legacy weirdness, no half-migrated patterns, no fifty small decisions accumulated over two years. My actual projects have all of that. A tool that looks brilliant generating a fresh Next.js app from scratch tells me almost nothing about how it’ll behave when asked to add a feature to an existing 40,000-line codebase with its own quirks. So the test is always: pick a real, moderately annoying ticket from an actual project, give it to the new tool on a branch nobody’s depending on, and see what happens when the codebase fights back a little.

Third: what happens to my data, specifically, not what the privacy page says in general. For client work this isn’t optional — some of my contracts have explicit clauses about where code can be sent, and “we use industry-standard security practices” on a landing page doesn’t answer whether a specific client’s proprietary logic is being used for further model training, or retained past the session, or processed in a jurisdiction that matters for their compliance requirements. I ask this directly, in writing, before code from a client repo ever touches a new tool. More than once the honest answer changed what data I actually fed it — sanitized fixtures instead of the real thing — even when I ended up adopting the tool for other work.

Fourth: does it degrade gracefully or fail silently? This is the one I care about most and that demo videos never show, because demos don’t include the failure case. I deliberately give a new tool a task slightly outside what it’s obviously good at — an ambiguous requirement, a file with an unusual pattern — and watch what it does when it’s uncertain. Some tools ask a clarifying question or flag their own uncertainty. Others confidently produce something plausible-looking and wrong, with no signal that they were guessing. The second category is the one that costs me hours later, because confident-wrong output is exactly the kind of bug that slips past a quick review — I’ve written before about how that’s the actual risk with agent-generated code, more than the obviously bad output ever was.

Fifth: can I actually roll back if it doesn’t work out? Some tools want to own more of the workflow than I’m comfortable handing over on day one — their own branch management, their own commit conventions, their own way of structuring a PR description. I check what happens if I use the tool for two weeks and decide to walk away: is my repo in a state any other tool or a plain editor can pick up cleanly, or has the tool encoded assumptions into the project that only it understands? A tool that’s genuinely good doesn’t need to lock me in to prove it — the ones that make walking away deliberately annoying are telling me something about their confidence in their own value.

Sixth: cost, measured against what I’m already paying, not against zero. Every new tool’s pricing page compares itself favorably to doing the work by hand, which isn’t the comparison that matters to me — I’m not choosing between this tool and no AI assistance, I’m choosing between this tool and the one I already use and have already built workflow habits around. The bar for switching isn’t “better than nothing,” it’s “better enough than what I already have to justify relearning my own habits.” That bar is higher than most launch announcements want you to think, and it’s the single biggest reason I’ve stuck with the same core toolset for months while trying plenty of others on the side.

What this checklist has actually saved me from: two tools this year that looked genuinely impressive in their own demos and fell apart on step two — real codebase, real mess — within an hour of actual use. Neither failure would have shown up if I’d judged them the way most people judge a new tool, which is by watching someone else’s polished five-minute demo and extrapolating. The demo is the tool at its best, on a codebase built to make it look good. My checklist is just an attempt to see it closer to its worst, on a codebase that looks like my actual work, before I bet real client hours on it.

None of this makes me anti-new-tools — I still try most of what gets released, because the field moves fast enough that sitting still has its own cost. It just means trying isn’t the same as adopting, and the gap between those two used to cost me more time than I noticed until I started tracking it deliberately.