Every professional services firm we talk to has been pitched by at least three "AI automation" agencies in the last year. Most of those pitches sound identical: a demo of a chatbot, a promise of "40 hours saved per week," and a proposal that reads like it was generated by the same tool it's trying to sell you. Some of these agencies are excellent. A lot of them are two people with a ChatGPT API key and a Canva subscription.
We've been on both sides of this evaluation — as the agency being vetted, and as the team helping clients replace an agency that overpromised and under-delivered. Here's the checklist we'd want a prospective client to run us through, and the questions that actually separate a firm that can build durable systems from one that can only build demos.
1. Ask to see something that's been running in production for six months
Anyone can build an impressive-looking automation in a sales call. The real test is whether it's still running, unattended, half a year later, without someone quietly patching it by hand every week.
When we talk about our own work, we point to specific systems, not case study slides. Airmax Heating & A/C, an HVAC contractor in Stamford, CT, runs on a custom CRM we built with ten operational dashboards — pipeline, scheduling, estimates, invoicing, customer records, reporting, follow-ups, job costs, quotes, and outreach — all on Cloudflare Workers and KV, wired into Stripe for payment links and deposits, the Google Ads API for Local Services Ads lead pulls, and Resend for outreach engagement tracking. That system has been in daily use processing real jobs and real payments, not sitting in a demo environment.
Ask the agency: which of your systems is still running today, unmodified by you in the last month, and can I talk to the person who uses it daily? If they can't name one, or if every example they show you requires "we'd need to check with the client," that's a signal their work needs constant hand-holding to survive.
2. Understand what happens when their AI doesn't know the answer
This is the single most important technical question you can ask, and almost no agency volunteers the answer unprompted.
AI systems fail in two very different ways. They can fail loudly — refuse to act, flag the gap, and ask a human — or they can fail quietly, by guessing and presenting the guess with the same confidence as a verified fact. The second failure mode is the dangerous one, because it looks like success until someone downstream discovers the AI invented a number, a name, or a status.
We built a document-parsing system for a multi-location medical practice that extracts patient demographics, attorney information, and insurance carrier details from incoming records — a stack of PDFs where the same field ("date of injury," "DOI," "date of loss") gets labeled three different ways depending on who typed the form. The parser normalizes those label variants into a single schema. But the design decision that matters more than the normalization logic is this: when a required field is genuinely missing or illegible, the system flags it as missing. It does not guess. A wrong guess on a date of injury can affect a legal filing deadline; an honest "not found" gets routed to a human in under a minute. Ask any agency proposing document automation, claim processing, or intake parsing: what does your system do when it isn't sure? If the honest answer is "it does its best," push back — "best guess presented as fact" is exactly the failure mode that erodes trust in these systems.
We apply the same standard to internal reporting. When one of our own automated systems can't verify a piece of data — a broken API, a stale cache, an integration that silently stopped syncing — the report should say so explicitly rather than quietly filling the gap with an old number. That's a small thing to ask about in a sales conversation, but it's the difference between a system you can trust unsupervised and one you have to double-check forever.
3. Ask where the approval boundary sits, and get it in writing
Full autonomy is a marketing word, not an operating model. Every credible automation system draws a line between "the agent can do this on its own" and "the agent proposes, a human approves." The interesting question isn't whether that line exists — it's where the agency drew it, and whether they can explain their reasoning.
For the prior-authorization workflow we built for a medical practice, the AI classifies incoming procedure requests, maps them to the correct provider based on procedure type, applies clinical threshold rules to flag anything that needs escalation, and drafts the routing decision — but it does not submit anything to a payer or make a final clinical determination without a human in the loop. The system's job is to take the repetitive 90% of the classification work off a person's desk and hand them a clean decision to confirm, not to remove them from the decision.
Get specific here. Ask: what dollar amount, what data type, or what action requires a human click before it happens in your system? If an agency says "our AI handles everything end-to-end," ask what happens on day one when it's wrong — because it will be wrong eventually, and the design of the approval boundary is what determines whether that mistake costs you a few minutes or a real problem.
4. Check whether their systems self-heal or just self-report
A dashboard that tells you something broke is table stakes. A system that notices something broke and fixes the parts of the problem it's allowed to fix, without waiting for a person to see a Slack message, is a different tier of engineering.
An ops-monitoring system we run watches ad spend pacing and flags budget anomalies before they become a wasted week of spend. It also monitors its own infrastructure: when a page returns a 404 that shouldn't, it rebuilds the page — with a strict rule against fabricating content it can't verify from the original source. That last clause matters. Self-healing without a no-fabrication rule just means the system is confidently wrong faster.
Ask what happens in this agency's systems when something breaks at 2 a.m. Does a human have to notice a Slack message and act, or does the system correct what it safely can and only escalate the part that genuinely needs judgment? Both are legitimate answers depending on the risk level of the task — but you want an agency that's thought about the distinction, not one that's never considered it.
5. Look for delta-aware state, not "run this script every day"
A lot of "automation" is really just a script on a timer that redoes the same work every day, whether anything changed or not. That's fine for a low-stakes task. It's expensive and error-prone for anything customer-facing, because it either spams people with duplicate notifications or silently misses updates.
The systems worth paying for track state and act only on what changed. A provider billing-followup system we built tracks per-provider status deltas — new charges, aging thresholds, timely-filing deadlines — and only sends a reminder when something in that provider's specific situation actually shifted, with portal access details included so the follow-up is self-contained. That's a meaningfully different design than "email everyone with an open balance every Monday," and it's the difference between a system staff actually read and one they've learned to ignore.
Ask an agency to walk you through how their system decides when to act, not just what it does when it acts.
6. Verify their claims the way they should verify data
If an agency's case study leans on a stat — "40 hours saved per week," "3.8x pipeline growth," "62% lower cost per lead" — ask them to show you the underlying source, not just the number. A firm confident in its work will pull up the dashboard, the CRM record, or the client's own reporting. A firm that gets vague or defensive about sourcing its own results is telling you how it will behave when you ask it to source a decision six months into your engagement.
This cuts both ways: it's also the standard we hold ourselves to internally. When we report a number to a client, it should trace back to a live query against the actual system, not a cached figure from three weeks ago presented as current. If we can't verify something, we say so — and any agency you're evaluating should be willing to do the same.
7. A short scorecard
Before you sign, you should be able to answer yes to most of these:
- They showed you a system in production for 6+ months, not just a demo.
- They can explain exactly what their AI does when it's uncertain, and it isn't "guesses."
- The human-approval boundary is specific (a dollar amount, an action type, a data category), not "we review things sometimes."
- Their systems distinguish between what they can safely self-correct and what needs a human, and they can give you an example of each.
- They track state and act on changes, not on a fixed schedule regardless of what happened.
- Every performance claim they've made to you traces back to a source they can show you live.
- They talk about specific client systems by name and detail, not composite "a client in your industry" anecdotes.
None of this is exotic. It's the same diligence you'd apply to hiring a controller or an operations lead — except most firms don't apply it to AI vendors because the sales process is newer and the technical claims are harder to independently verify. That gap is exactly why so many automation engagements end in disappointment: not because AI doesn't work, but because the buyer never asked the questions that would have surfaced whether this particular build would work.
If you want a second opinion on a proposal you've already received, or you'd rather start with a firm that will walk you through exactly this kind of detail before you sign anything, get in touch with our enterprise team. We're happy to review someone else's SOW with you, no strings attached — we'd rather you go into an automation project with clear eyes than get burned by a pitch deck.