Frame
I’ve been building integrations for three and a half years and haven’t put an LLM inside one until now.
The systems I build are workflow plumbing: automations firing, data moving between platforms, records getting created and updated, the usual machinery of operations work. Everything I’ve built needed a consistent outcome for a given input, not reasoning. The big-picture reasoning was done beforehand, on paper, during analysis and design.
Where consistency matters, code outperforms any model – models are slower, more costly, harder to debug, and inconsistent at a detail level.
So the question I keep coming back to – and I don’t think I’m alone — is: where does an LLM actually earn its place inside systems like these? Not theoretical or in a demo, but inside a system that’s already running, already reliable, already costing the right amount of money.
This week I’m adding one. This is the before post laying out what it will do, why I judged this was the spot, and what I’m betting will happen when it goes live.
My Decision
The workflow step where I’ve chosen to implement an LLM is small. When a contact comes into the system, instead of writing straight to Notion as before, the contact data first goes to the Anthropic API for a research pass. The API returns structured research about who this person is and what’s worth knowing about their company before reaching out. That research goes to a separate Notion database, not directly onto the contact page – more on the sandbox in the next section.
That’s it. One API call, one new database, no agent loops, no autonomous behavior. The LLM does one thing, and the existing integration carries on as it always has.
Why this step specifically?
Every systematic workflow I’ve designed has had two kinds of work in it: coded automations and judgment calls. Code handles anything that needs a consistent outcome for a given input. Judgment calls have always been handed off to a human or omitted.
Researching a new contact is a judgment call. Trying to code a function that says “here’s what’s worth knowing about this person before we email them” is a poor approach because the answer is different for every contact and shifts depending on what you find. So historically, this step has been skipped for low-priority contacts or done manually for high-priority contacts, which could take several hours depending on the contact.
Why an LLM, not something else?
I could have used data enrichment services. The ROI math killed it:
( per-contact service cost + build-and-test integration cost ) × a mostly low-value contact pool = a yield not worth pursuing
Scraping services I ruled out faster: the output is too unstructured to drop cleanly into a CRM record without significant cleanup work. The status quo sensibly became – prioritize contacts manually, have a human research the high-value ones, and input findings. That works for the top of the list, but the long tail is left ignored.
LLMs change the math in two ways. First, per-contact cost is low enough that I can potentially research every contact, not just the prioritized ones – the long tail comes into reach, and a vast pool of “poor” contacts can be trawled for diamonds in the rough who got misplaced. Second, and more important: the LLM doesn’t just return data; it reasons about what else might be worth knowing about a particular contact. That’s something neither enrichment nor scraping could do. The output is not just a populated field set but a structured description shaped by judgment.
The capability trajectory matters too. LLMs are improving fast enough that the work I do today – prompt design, output schemas, evaluation patterns – compounds. But that’s a reason for later, not the reason for today. Today’s reason is the first two points: the long tail comes into reach, and the model can reason as well as gather and learn.
What makes an LLM the right tool here?
The work this step does has three properties that make it fit an LLM call cleanly:
- The judgment is bounded. The output isn’t an action – it’s a structured description of the contact. The model isn’t deciding what to act on, just what to find and return, then stop.
- The failure mode is recoverable. If the research is wrong, thin, or weird, nothing downstream depends on it being right. The contact still exists in Notion; the integration still runs. The worst case is a useless research blob, not a broken system.
- The cost is small and trackable. Per-contact LLM cost is apparently in the low cents. With my cost tracking plan that I’ll describe in the next section, every LLM call is logged, and the per-contact cost is visible to the client in USD.
That third one matters a lot. Most “should we use AI here” decisions stall on cost because no one’s put a number to it. If you can’t tell the client how much this costs per operation, they can’t reason about it, and reasonable clients won’t sign off on it. Making cost legible from day one isn’t a nice-to-have; it’s part of why the LLM gets to be in the system at all.
Implementation
This LLM step will be new. Everything around it exists already. My goal is to add capability to the current system without touching the reliability of what’s already running – which means being deliberate about where the new thing sits and how much it’s trusted before it earns more rope.
The sandbox.
Research output doesn’t go directly onto the contact page. It goes to a separate Notion database first. The existing automation – the part that creates the contact record – runs exactly as it always has. The LLM step sits alongside it, not inside it.
Promotion from the sandbox to the contact page happens only after quality is validated by a human. What that validation looks like in practice is still being decided – likely a status property that fires another automation – but the principle is fixed: the LLM output lives in isolation until I trust it. The sandbox may eventually become a permanent audit log. For now, it’s a quarantine.
This matters because the failure mode for a bad integration isn’t usually dramatic. It’s quiet corruption – records that look fine but aren’t, data that’s slightly off in ways that compound. Keeping LLM output separate until it’s trustworthy is the right way to roll this out.
Cost tracking.
The third reason an LLM belongs here – that cost is small and trackable – only holds if cost is actually tracked. So the tracking is built in from day one, not bolted on later.
Every API call is logged to the sandbox Notion database against the research output: timestamp, model, input tokens, output tokens, the rates in effect at the time of the call, and cost calculated by the code at the moment it runs. Recording the rates at the time of the call means the log is historically accurate regardless of future pricing changes.
A Notion formula sits alongside the logged cost and recalculates independently from the raw token data. At the time of logging, both figures should match – if they don’t, that’s a bug signal. But the formula also has a second use: update the rates it uses, and every historical call gets restated at current pricing. The same token data, expressed in today’s money. Useful for knowing whether the economics of the step are getting better or worse over time.
The client thinks in dollars, not LLM tokens. So the reporting surface isn’t a token count – it’s cost per contact, total contacts processed, total spend for the period. A number they can reason about and hold against the value of having the research done.
Structured outputs and completion judgment.
The LLM will return JSON in a specific schema that already exists. The code receives the JSON, validates it, and writes it to the sandbox. The model doesn’t decide when it’s done its job, it doesn’t loop, and it doesn’t retry on its own judgment. One call in, one structured response out, code handles the rest.
This isn’t the only way to build an LLM step. It’s the right way to build a first one inside an existing integration. The model is the new variable. Everything else – the job queue, the webhook, the Notion writes, the error handling – is known. Keeping the model’s role tightly scoped means that if something goes wrong, the blast radius is small and the cause is easy to find.
My Bets
Some of these will look obvious in hindsight. I’m writing them down now so the “after” post has something real to grade against.
Cost per contact will come in well under $0.50.
That’s my ceiling estimate, not my expectation. Token consumption will vary – some contacts will pull more context, some less – but I expect the average to land meaningfully lower. At the full database scale of 35,000 contacts, even an average of $0.10 per contact amounts to $3,500 to enrich the entire pool. That’s a number worth having a conversation with the client about, and one they can weigh against the value of having every contact researched. We’re starting with the highest-scoring contacts – in the hundreds, not thousands – so the cost conversation happens at a small scale first, with real per-contact figures, before anyone commits to the long tail.
The prompt will need more iteration than I expect.
My experience with LLMs so far is that I tend to underspecify context upfront and discover the gaps when responses come back thin or misdirected. In a chat context that’s recoverable – you see the gap and add context in the next message. In an integration, it isn’t. The prompt is fixed; the model runs against every contact with whatever context it was given. The same gap repeats across every contact until the prompt is updated. So the first version of this prompt will almost certainly produce output that reveals what I failed to specify, and the first round of prompt development will be filling those gaps.
The prompt is also doing more work than a simple “research this person” instruction. Contacts are business owners and operators from across the US – sometimes international – and key data lives in state databases that each handle it differently: different formats, different levels of detail, different gaps. Think of the scene in The Social Network where Zuckerberg is scraping Harvard House databases to build Facemash – approaching each House differently, pivoting when the data isn’t there. When he hits Quincy, he gets nothing: “Quincy has no online Facebook, what a sham. Nothing I can do about that.” The model needs to handle the same reality: situational approaches depending on the state, and clear instructions on what to do when the data doesn’t exist, because the wrong answer to a data gap is an invented one.
After my prototype is built, the client’s research team is handling prompt development; their working version of the prompt lives in a Notion page, fetched and cached each morning for the day’s work. That architecture means prompt iteration doesn’t require a code deploy – the prompt is a living document, not a hardcoded string.
The sandbox will evolve but won’t disappear.
Right now, the sandbox is a hard gate: nothing writes to the contacts database without explicit human approval. That’s the right posture while trust is being established. But I don’t expect it to stay that way permanently. Once the research quality is consistent enough, the model will flip to default-write – research goes directly to the contact page – with an easy undo option for the cases where something looks wrong. The hold point becomes an escape hatch rather than a gate. The LLM earns more rope as it earns more trust. How long that takes depends entirely on what the first few hundred contacts look like.
Kill Criteria
Three things would make me pull this step.
The first is a prompt dead end. If iteration consistently fails to fill data gaps or prevent quiet hallucinations – not early struggles, but a genuine ceiling where more effort stops producing better output – the step comes out. A research blob that’s confidently wrong is worse than no research at all.
The second is cost. I don’t have a fixed threshold yet because I haven’t priced the alternative – what a research firm would charge per contact for the same work. Once the first batch runs and I have a real per-contact figure, I’ll put both numbers in front of the client. If the LLM cost doesn’t beat outsourcing on value, the case for keeping it weakens – though time and consistency have value, the invoice doesn’t capture. The cost tracking built into the system exists precisely so that conversation can happen with real numbers, not estimates.
The third is accuracy by state. If the state-by-state variation makes the model consistently unreliable for certain states, the first response isn’t to kill the step – it’s to route those states to their own LLM calls with tailored prompts. That’s a hurdle to clear, not a reason to stop. But if routing doesn’t fix it and accuracy stays unacceptable, that’s a kill.
What won’t kill it: early prompt struggles, thin output on the first batch, a sandbox that takes longer than expected to earn promotion. Those are expected. The bar is whether the step produces research that’s useful often enough to justify its place in the system – and whether the client agrees the cost is worth it.
Three and a half years of integrations without an LLM. This is the first one. I’ve laid out my reasons, my implementation decisions, and what I’m expecting to happen – some of which will be wrong, and that’s the point of writing it down before shipping. The only way to find out is to run it. I’ll report back in the after post.