SAP updated its API policy to block third-party AI agents from interacting with its systems. Salesforce made it generally available for developers to build and deploy agents directly from the command line. Two large companies, moving in opposite directions, betting differently on the same question: how should AI agents actually talk to enterprise software?

The answer to that question determines what’s automatable in your organization over the next few years. It’s worth understanding.

What an Agent Actually Does

An AI agent is a language model that takes actions, not just answers. It can query a database, open a file, send an email, update a record. The difference between a chatbot and an agent is a set of tools and the autonomy to decide when to use them.

In an enterprise context, the tasks agents are most valuable for tend to span multiple systems. Reconciling invoices against the GL touches an AP inbox, an ERP, and maybe a shared drive. An agent has to reach across all of them, which means connection is the foundational problem.

The Two Dominant Approaches: MCP and CLI

MCP — Model Context Protocol

MCP is an open standard, originally created by Anthropic and now stewarded by the Linux Foundation, that gives AI agents a structured way to interact with external systems. An MCP server sits in front of a tool or data source and exposes a menu of callable functions along with descriptions of what each one does and what parameters it takes. The agent reads the menu, decides what it needs, and makes calls accordingly.

The appeal is standardization. In the same way that APIs let SaaS products plug into each other, MCP lets agents treat external systems as interoperable building blocks. NetSuite ships an AI Connector Service built on MCP. Salesforce supports MCP. The ecosystem is growing.

CLI — Command Line Interface

A CLI is a program you run from a terminal. It accepts commands, executes them, and returns output. It’s not a new idea; it’s how developers have interacted with systems for decades but it’s becoming a first-class integration surface for agents.

Salesforce’s move to ship a dedicated agents plugin for its CLI is the clearest recent example. Developers can now build, configure, test, and deploy Agentforce workflows using the same terminal-based toolchain they’ve always used for code. Agents can do the same.

The CLI’s advantage is composability. The output of one command becomes the input of the next. Data can be filtered, reshaped, and piped through a chain of operations without ever loading it fully into memory. That property, mundane in a developer workflow, turns out to matter enormously for ERP tasks.

When There’s No API

A significant share of enterprise systems — especially on-premise deployments, legacy platforms, and older ERP configurations — don’t have usable APIs. Or the API exists but covers a fraction of the data model, requires expensive licensing, or is so poorly documented as to be effectively inaccessible.

In those cases, the integration surface is whatever the system already exposes.

Email remains one of the most durable integration channels in enterprise software. A large portion of the operational data that flows through a business still moves through inboxes. An agent that reads and writes email participates in those flows without touching a database.

SFTP is how older systems exchange bulk data at scale. Many SAP, Oracle, and legacy MRP environments produce nightly or weekly file drops An agent that reads from an SFTP folder has access to a rich, structured data stream, no API required.

Screen automation, aka RPA is the last resort, and increasingly a viable one. It’s fragile when interfaces change, but for systems that expose no other programmatic surface, it works. AI-powered document parsing has made the unstructured outputs of these interfaces considerably more tractable.

The through-line: the system doesn’t need to change. The agent adapts to whatever surface is already there.

The Three Things an Agent Needs

Reducing this to first principles, an agent automating work in an ERP or system of record needs three things:

1. Data

The information relevant to the task: transaction records, invoice data, vendor master, chart of accounts, whatever the workflow touches. It might come from the ERP directly, from a file share, from email, or from a warehouse. Getting it to the agent cleanly and completely is the first constraint.

2. A Code Environment

ERP tasks are really a chain of transformations and lookups. "Do these invoice line items match the PO within tolerance?" is not answerable by reading data. It requires joining two datasets, computing variances, applying business logic, and producing a structured output. An agent with access to a code environment can do this properly: write a script, run it, inspect the result, adjust, and iterate. Without one, the agent is limited to reasoning in natural language about data it has already read into memory.

3. A Way to Push and Pull from Systems

Once the agent has reasoned about the data and reached a conclusion, it needs to either pull more or push something back. This is where MCP and CLI come in. The integration layer is what makes the agent's output operational rather than advisory.

Why CLI Wins for ERP Work

MCP's value proposition is tool discovery: a structured, typed interface where the agent can ask "what can this system do?" and get a documented answer back. For exploratory tasks or unfamiliar APIs, that's genuinely useful.

But ERP data has a characteristic that breaks the core assumption of most MCP deployments: volume.

A material master in SAP can span millions of rows across multiple views. When an MCP server loads this into the agent’s context window, the architecture becomes bounded by what fits (or rather what doesn’t). Language models have finite context windows, and production ERP datasets will exceed them.

CLI sidesteps this because the code environment handles the data, not the context window. The agent writes a query or script to extract exactly the records it needs. That data lands on disk. A second script transforms and filters it. The output is what reaches the model. The model reads a few hundred rows instead of a few million.

The data is also persistent between steps. An agent working through a multi-step task doesn’t need to hold everything in memory simultaneously. It fetches, stores, computes, fetches again. CLI composability, i.e piping output from one command into the next, is exactly the right tool for this pattern.

That doesn’t make MCP useless. For structured write operations where typed validation before anything touches the system of record is valuable, for multi-agent setups where a central orchestrator coordinates across many downstream tools, MCP earns its overhead. The pattern that tends to work in our experience: CLI for data work, MCP at the point of write.

What the SAP Policy Actually Means

SAP’s new API policy prohibits the use of its APIs for “interaction or integration with semi-autonomous or generative AI systems that plan, select, or execute sequences of API calls.” It also bans “systematic or large-scale data extraction.”

The practical effect is that third-party agents (anything not running through SAP’s own AI layer, Joule) are now in technical violation if they’re making API calls autonomously. The policy says enforcement rights include throttling, suspension, and termination. It explicitly prohibits circumvention through proxies, custom code, or intermediary services.

The DSAG, representing tens of thousands of SAP customers across Germany, Austria, and Switzerland, has publicly criticized the policy for creating uncertainty and potentially blocking innovation. They’ve noted that the policy doesn’t clearly specify which APIs are affected, leaving customers unable to determine what’s compliant. SAP has reserved the right to change the approved API list unilaterally.

SAP markets this as a system stability measure. The commercial effect is that it steers AI workflows toward Joule and SAP’s own ecosystem. These are not mutually exclusive positions.

The workarounds for teams who need to operate outside that ecosystem are, again, the non-API surfaces: SFTP exports for bulk reads, email-based flows for exception handling, and human-in-the-loop checkpoints before anything writes back. Not the architecture anyone would design from scratch, but functional, auditable, and fully outside the policy’s scope.

The Bottom Line

Agents can already connect to your ERP. Through APIs where they're open, through file drops and inboxes where they're not, and through screen interfaces as a last resort. The connection question is largely solved. What separates successful automation from nice Proof-of-Concepts is the combination of a code environment capable of handling real data volumes and an architecture that keeps humans in the loop at the right points.

Thoughts, pushback, or things working differently in your environment: hit reply.

Last week at basically the same time Anthropic & OpenAI announced joint ventures with major PE firms.

Anthropic: ~$1.5B. Blackstone ($300M), Hellman & Friedman ($300M), Goldman Sachs ($150M), Anthropic ($300M), plus Apollo, General Atlantic, Leonard Green, GIC, & Sequoia.

OpenAI: $10B valuation. $4B from 19 investors anchored by TPG Brookfield, Bain Capital, Advent, Dragoneer, SoftBank, Palantir, Goldman Sachs, Snowflake.

Both adopt the trendy Palantir forward deployed engineer model. Small technical teams embedded inside portfolio companies to rewire workflows. The combined investor base controls thousands of mid-market companies.

Why this is happening

The firms involved hold direct equity in the model companies themselves.

Blackstone holds ~$1B in direct Anthropic equity across multiple rounds. TPG sits in Anthropic’s cap table and anchors OpenAI’s DeployCo. Goldman Sachs is in Anthropic’s Series G, on OpenAI’s revolving credit facility, & committed $150M to the Anthropic join venture. Dragoneer coled Anthropic’s Series G & invested in DeployCo.

The firms benefit at all levels of the transaction but not where they are supposed to:

1. Direct equity in the model company: appreciation tied to revenue growth & IPO valuation

2. Equity in the deployment vehicle: implementation fees and guaranteed returns

3. Control of the portfolio companies being told to adopt the product

The PE firm mandates adoption at the portco. The portco pays the model company and their own owner. Implementation revenue accrues to the model company. Model company revenue inflates the PE firm’s direct equity ahead of IPOs both companies are targeting in 2026. Every dollar of portco spend is serving multiple positions on the same balance sheet.

The position generates returns for the owners but none depend on whether the implementation creates operational value at the portfolio company. It seems pretty early in a new tech hype cycle to already see this level of rent seeking as returns are extracted from positions without a focus on you know… successfully implementing AI.

The old SV playbook

Silicon Valley ran a slimmed down version of this for years. Big VCs encouraged portfolio companies to buy each other’s software inflating ARR ahead of fundraises & IPOs. The practice drew criticism, but the exposure was a lot more limited. The 2026 version is at a much more damaging scale.

The old VC cross-sell inflated metrics inside a smaller system of mostly consenting participants and had the added value of helping each other as design partners. Branching outside of software companies selling to software companies, there are bigger implications to the mid market economy.

17.5% Guarantee

OpenAI contractual returns made the nature of this deal even more explicit if it wasn’t already. DeployCo guarantees PE investors 17.5% annual return for five years. If DeployCo underperforms, OpenAI covers the gap. 17.5% on $4B = up to $700M/year in payout.

It’s pretty clearly purchased customer acquisition structured as a financial instrument. The return is contractual and not performance based so it doesn’t matter whether a single implementation actually creates value for the underlying portfolio.

Who pays the bill

Mid market Portcos. The forward deployed engineers arrive to automate without a clear alignment on goals and outcomes. The owner of these initiatives sits one level down from the pushers. When a project doesn’t have good mutual incentives the outcomes are almost always worse.

Application layer startups. Every vertical AI company selling into mid market PE backed companies is now fighting an uphill battle to sell into this space. You can’t win a competitive evaluation that never happens. The deployment companies and associated circular deals are a moat against the entire application layer of new companies.

Public market investors at IPO. The biggest exposure. Both IPOs will be priced on enterprise revenue manufactured through captive distribution. When the PE hold periods end & the mandates fade, what happens to retention? Index funds, retail investors, & pension funds hold that paper.

What this means for AI

The best AI implementations I've seen start with a specific operational problem and work backward to the implementation. These start with a capital structure and work backward to justify it. The mid market companies powering the economy and writing the checks for AI agent deployments deserve better than being a line item in the AI capital arms race.

Goldman Sachs is now testing whether that equation can be broken. The bank has entered a six-month partnership with Anthropic, embedding the AI company’s engineers directly inside Goldman to build autonomous agents targeting accounting and compliance workflows. The scope is specific, the investment is real, and the implications extend well beyond one bank.

The Specifics of Goldman’s Bet

Goldman’s CIO Marco Argenti framed these systems as “digital co-workers,” a term that obscures the ambition underneath. What Goldman is building are autonomous agents capable of processing documents at volume, applying internal policy logic, flagging exceptions, escalating to humans where needed, and generating audit-ready output.

The bank chose two initial domains. The first is trade and transaction accounting: the capture, matching, reconciliation, and documentation cycle that runs continuously across every desk. The second is client vetting and onboarding, covering KYC, due diligence, and internal policy compliance. Both domains share the same characteristics that make them attractive for automation: high volume, dense rule sets, and a long tail of exceptions that currently require manual triage.

The stated goal is not incremental efficiency. Goldman wants to compress these workflows dramatically, absorbing business growth without proportional headcount increases.

From Code to Operations

Goldman’s path to this point is instructive. The bank’s initial AI deployment was in software engineering, where it rolled out an autonomous coding tool called Devin across its developer teams. That was a conventional application of large language models.

What caught Goldman’s attention was that Claude’s core strength turned out to be reasoning through rule-dense, ambiguous problems, not just generating code. That capability maps directly onto operational finance work.

The difficulty in accounting and compliance has never been the mechanics. Debits and credits are straightforward. The difficulty is that source systems disagree, data arrives on inconsistent timelines, documentation is partial, internal policies sometimes conflict with each other, and no two exceptions look exactly alike.

Take a specific scenario. A fixed-income desk executes a block trade settling across three counterparties in two currencies. The trade capture system books the notional in USD. The risk platform converts to EUR using a rate snapshot from 4 p.m. The general ledger applies the prior day’s closing rate. Reconciliation surfaces the discrepancy. Today, a human analyst traces the rate sources, determines whether the variance reflects a timing difference or an actual booking error, checks the resolution against internal policy, and writes up the finding for audit. One exception, 45 minutes.

An agent that can pull the trade record, compare rate sources, classify the root cause, and draft a resolution memo citing the relevant policy does not eliminate the need for human oversight. It does turn a 45-minute task into a seconds-long triage for routine cases, freeing analysts to focus on the exceptions that genuinely require judgment.

Most firms saw AI’s coding ability and filed it under “developer tools.” Goldman saw the underlying reasoning capability and recognized its application to the operational core of the business. That leap is the part most organizations have not yet made.

The Architecture That Matters

LlamaIndex recently published work on what they call “long-horizon document agents,” and the framing is useful for understanding what Goldman is actually constructing. Document-processing AI has historically operated in two limited modes: batch extraction (load a PDF, pull fields, output to a spreadsheet) and conversational retrieval (ask a question, get an answer, lose the context). Neither mode reflects how operational teams work in practice.

Operational document work persists across days and weeks. It involves batching, escalation, approval chains, and revision cycles. An effective agent in this environment needs to activate on events (a new filing arrives, a reconciliation breaks), maintain task state across sessions, and present its work through interfaces designed for queue management rather than conversation.

Functioning AI in finance requires three layers operating together. First, a structured data layer that normalizes inconsistent inputs into a form the agent can reason over. Second, a reasoning layer that handles ambiguity, applies policy logic, and classifies exceptions. Third, a deterministic workflow layer that enforces auditability, manages throughput, and governs human escalation. Goldman appears to be investing across all three layers, which distinguishes this effort from the chatbot experiments that most firms have run so far.

The Mid-Market Problem

Goldman can afford to embed an AI company’s engineering team for half a year and build custom infrastructure from scratch. The question this raises for every other firm is straightforward: how do you get to the same destination without those resources?

The operational problems are identical regardless of firm size. Invoice matching at a regional distributor involves the same structural challenge as trade reconciliation at a global bank: messy inputs, rule-based processing, exception handling, and documentation requirements. The gap is entirely in implementation capacity.

Three paths are available. The first is inaction, which is a bet that the competitive landscape will not shift quickly. Goldman’s investment suggests otherwise. The second is internal development, which for most firms will stall after handling the easy cases because the hard engineering sits in the data normalization and workflow layers, not in the AI model itself. The third is adopting products that have already built the full stack, including the data infrastructure, the reasoning layer, and the workflow orchestration. The challenge with this path is evaluation: knowing whether a vendor has solved the architecture problem or simply wrapped a language model in a user interface.

The distinguishing features of a production-ready system are a data architecture that handles document variability at scale, workflow interfaces built for review and exception management rather than chat, complete audit trails covering every agent decision, and human-in-the-loop patterns that function under real operational pressure.

Where This Leaves the Profession

Goldman is conducting a controlled experiment on how much procedural work in accounting and compliance can be handled by autonomous agents. The outcome will not be binary. Regulatory accountability will continue to require human responsibility. Client relationships will continue to require judgment. Risk ownership will remain with people.

What is changing is where human effort concentrates. The value is migrating from executing procedural steps to designing the processes agents follow, defining the rules that govern exception handling, reviewing agent output for errors, and defending decisions under regulatory scrutiny.

That represents a meaningful shift in the skill set the profession rewards. And it raises a secondary question that few institutions are addressing yet: if the entry-level procedural work that has historically trained junior professionals gets absorbed by agents, how does the next generation develop the judgment that the senior roles require?

Goldman is pressure-testing what can be automated. The rest of the industry needs to think carefully about what comes after.

This works. We’ve built a finance platform around it. But running it in production revealed what the thesis leaves out: coding agents alone can’t handle high-volume, regulated work. Two additional layers made it work:

- Infrastructure to process unstructured data at scale

- Workflow and review interfaces to run processes repeatably

Use case-specific agent operating systems can’t be individual sandboxes like Claude Code. Without this scaffolding, the filesystem pattern fails under real-world constraints.

---

Why We Built This Way

We started building accounting automation in 2024. The problem we were solving forced a particular architecture: unstructured documents coming in (PDFs, emails, bank feeds), structured data going out (ERPs, general ledgers). Standard finance operations, but messy in practice.

The system needed to be composable because the inputs were unpredictable. Every customer had different document formats, different GL schemas, different vendor naming conventions. We couldn’t hardcode the logic.

In the last month, we integrated a coding agent with a harness and filesystem. We moved core infrastructure to a project-based filesystem that controls what context the agent can access. The fit was natural. The agent could read ingested documents, write code to transform them, and persist its reasoning. It worked because of the infrastructure we’d already built.

That infrastructure turned out to be weight-bearing. Without it, the coding agent couldn’t operate at scale. With it, the agent became genuinely powerful.

---

The Three Layers

Layer 1: Structured Data Infrastructure

Here’s the problem with “just give the agent a filesystem”: if you’re processing 20,000 invoices per month, you can’t dump them all into a directory structure and expect an agent to work with them efficiently.

The raw volume breaks the context window. The variety breaks pattern matching. If your agent is reading raw documents just to find basic fields, you’re burning expensive inference on trivial work.

You need infrastructure that uses LLMs to enforce structured outputs on unstructured inputs. This is extraction, not reasoning. The LLM doesn’t decide what the data means, it just forces chaos into schema: turning PDFs, emails, and spreadsheets into normalized, queryable tables.

This separates extraction from reasoning. The extraction layer handles repetitive work at scale. It produces clean, structured data that the coding agent can query flexibly without drowning in noise.

What breaks without this layer: The agent’s context fills with raw documents. It spends tokens on parsing instead of reasoning. Latency becomes unacceptable. Cost spirals.

---

Layer 2: The Coding Agent

The coding agent operates on top of structured data. It’s good at three things:

1. Building the environment.

When a new client onboards, the agent scaffolds their operational setup: Standard Operating Procedures (SOPs), vendor categorization rules, reconciliation logic. It reads existing documentation, asks clarifying questions, writes configuration files, and generates the initial ruleset.

This is a huge unlock. Traditional enterprise software requires a human consultant to do this setup, which takes months and costs tens of thousands of dollars. The agent does it in an afternoon by working collaboratively with the process expert to encode their knowledge into the file system.

2. Handling exceptions.

Structured workflows (more on this in Layer 3) handle the 80% case. The coding agent handles the 20%: unusual document formats, one-off data migrations, edge cases that don’t fit the standard process. It writes custom scripts, tests them, and integrates data back into the system.

Without this flexibility, you’d need to escalate every exception to engineering or reject the work entirely. The agent turns exceptions into solvable problems.

3. Ad-hoc analysis.

Once project-specific structured data exists, the agent becomes a natural language interface to it. If you need to create a custom report, export data in a specific format, or investigate an anomaly, the agent writes SQL or Python to pull exactly what you need without waiting for someone to build a feature.

This is where the “coding agent as OS” thesis really shines. You’re not constrained by the software vendor’s prebuilt reports and workflows. You have direct, flexible access to your own data.

What breaks without this layer: The system becomes brittle. Every edge case requires developer customization. Every new customer need requires a product roadmap discussion. You lose the adaptability that makes AI-native software different.

---

Layer 3: Deterministic Workflows

Here’s what the “coding agent for everything” framing misses: you don’t want an agent improvising when you’re booking $2M to a general ledger across thousands of transactions.

Daily operations like invoice matching, approval routing, and posting to the ERP happen in purpose-built interfaces with event-driven workflows. The process is deterministic, auditable, and includes human checkpoints where they matter.

This is load-bearing architecture rather than limitation.

In domains like accounting, the process by which a result is derived matters as much as the result. If an auditor asks why a transaction was categorized a certain way, you need to show them a rule, not a sampling of LLM output. You need to prove the logic is consistent, not just that it’s usually correct.

A coding agent generating novel logic for each transaction creates variability. Variability breaks auditability. Auditability is non-negotiable in accounting, healthcare, legal, or any other domain where mistakes have regulatory consequences.

The structured workflows are also where throughput happens. An agent handling 500 invoices per day by writing fresh Python for each one would be slow, expensive, and unpredictable. Event-driven pipelines with clear rules handle the volume efficiently. The agent helped design the flow, but the flow itself runs without the agent’s ongoing involvement.

What breaks without this layer: Cost and latency make high-volume operations impractical. Auditability vanishes. Human reviewers can’t keep up because every transaction looks different. Trust erodes.

---

The Human as a Tool

Most agent hype focuses on removing humans from processes. In consequential domains, this is backwards.

The right pattern is to treat the human as a tool the agent can invoke. When the reconciliation logic hits a fuzzy vendor match that’s not in its instructions, the agent doesn’t crash or guess. It pauses, surfaces the ambiguity, and asks:

“This looks like Amazon Web Services, but the address is Dublin and the amount is 15% higher than usual. Should I proceed?”

The human confirms. The agent resumes.

This is different from traditional “human in the loop” design, where humans are exception handlers. Here, invoking the human is part of the agent’s normal operation. The agent routes judgment calls to the human the same way it would query a database or call an API.

The human provides judgment where judgment matters: validating assumptions, catching anomalies, approving anything consequential. The agent provides speed and consistency where those matter: processing volume, applying rules, surfacing what needs review.

This isn’t a fourth layer—it’s orthogonal to the architecture. It’s about control flow. The agent can invoke the human from any layer.

---

Why This Architecture is Hard to Retrofit

If this pattern is effective, why isn’t everyone doing it?

Legacy software vendors and AI-first companies face architectural lock-in.

Their AI stack is built for retrieve → answer → forget. Moving to stateful work loops and filesystems means rewriting the data model, the API layer, and the UX.

They also sold customers on “read-only” AI. Giving an agent write access to the general ledger requires rethinking the security model and permissions system from scratch.

Horizontal agent companies face the application layer problem.

A general-purpose coding agent can learn domain knowledge. You can give it GAAP documentation, explain three-way matching, teach it intercompany transfers. The knowledge can live in the filesystem.

But knowledge alone doesn’t create a usable application. The agent needs structured workflows to express that knowledge in a way teams can actually use to complete work. It needs purpose-built interfaces for invoice approval, reconciliation review, and GL posting, not just a chat window where users describe what they want done.

Domain expertise without workflow infrastructure is just an expensive consultant with no tools.

---

What This Means for Vertical AI

The winning AI Application companies won’t be chatbots over databases or coding agents with filesystem access. They’ll be operating systems with three weight-bearing layers:

1. Structured data infrastructure to handle volume, normalize inputs, and update context.

2. Coding agents to provide flexibility, handle exceptions, and adapt to customer-specific needs.

3. Deterministic workflows and purpose-built interfaces to deliver throughput, audit-ability, and consistency where they’re non-negotiable.

The coding agent is the architect and analyst. The structured workflows are the assembly line. The human is the supervisor with final authority.

Get all three right, and you have something that actually works in production.

Skimp on any one, and you’re either too rigid (traditional software), too expensive (agents all the way down), or too untrustworthy (LLM output without guardrails).

Subscribe now

Let me just tell you that working on this for a while made me appreciate that supply chains work at all.

One of the ugliest recurring problems is reconciling records when there isn’t a clean join key.

In the real world:

• An invoice references a PO that’s missing in the bank transfer.

• Names don’t match exactly (“Google” vs “Alphabet”).

• Dates drift (invoice date vs payment date vs settlement date).

• Amounts drift (fees, FX, partial payments).

• Remittance info is missing, truncated, or inconsistent.

• Mix and match all of the above

A meaningful fraction of cases can’t be auto-reconciled with a deterministic rule like (amount, date, counterparty) and must be investigated using circumstantial evidence.

What ended up working best

We tried a couple of approaches to solve this problem. We tried to see if dumping a couple smaller Excel files into Gemini would work (it didn’t). We also tested if an Assistant could come up with a rule set (it could for the easy cases, but failed on edge cases and didn’t create any rules that were out of the dataset).

What worked really well was an Agent with internal MCP access to the underlying data stored in a relational database.

For anyone not following MCP: it’s an open standard for connecting an AI client to external tools/data sources in a structured way. It’s basically a standard port that lets a model discover and call tools rather than scraping everything through natural language.

A bit surprisingly gpt-5-mini worked as well on this problem as the more powerful models, but that’s not the main thread of this article.

We made the tool surface area tiny on purpose

Our data access MCP server only exposes three capabilities:

1. List available tables

2. Describe a table (schema/columns; optionally comments)

3. Read-only querying (with Limits)

It’s a quite boring design, but by limiting the available paths while exposing dynamic functionality we implicitly tell the model how to do the task.

In a finance environment, write tools are a trust-killer anyway: what users say they want (“automate it”) often diverges from what they accept emotionally (“show me, let me approve”).

Also, from a safety perspective: the MCP ecosystem has already had very real issues with misconfigured servers exposed to the internet and prompt injection, so you want to assume every tool needs strict access control, auditing, and least privilege.

The one caveat: semantics matter a lot

This only works if table & column names are semantically meaningful and (ideally) columns/tables have descriptions or comments that match domain language

If the schema looks like T_014, col_7, x_value (hello SAP), the agent would need to solve internal archaeology first.

In practice, we found the database surface becomes the context layer. The LLM learns how to think about the task through names and structure. In other words: The schema is the prompt.

How the agent actually reconciles

The agent’s behavior looks surprisingly like a decent analyst:

1. Try perfect matches first: Match based on exact amount, exact reference / invoice number and strict date windows

2. Then relax constraints: fuzzy name matches, date tolerance (settlement vs invoice vs posting), grouping/splitting (one payment covering multiple invoices). Start pulling data windows instead of single rows.

3. Accumulate evidence until it can decide: match, partial match, dispute / needs review, everything is messed up.

If you want a math analogy: a chunk of reconciliation is basically an assignment problem and, in many-to-one or one-to-many cases, it starts to resemble subset-sum / knapsack (“which set of invoices best explains this payment?”). Soft constraints like names and dates acting as additional signals.

The model is not literally running the Hungarian algorithm, but it does perform a kind of iterative constrained search: strict -> relaxed -> stop when confidence is high enough.

Results: it handled our recurring hard cases

We tested this against ~20 frequent edge cases on production-level data (things that had historically required humans). The agent was able to recognize and resolve all of them in the way our ops team would expect.

Auditable reasoning trails matter as much as end results in Finance Ops. The Agent is also outputting decision making explanations like

• “Amount matches exactly”

• “Counterparty name differs but resolves via known alias pattern”

• “Payment date aligns with settlement date range”

• “Only one candidate fits all constraints; next best candidate breaks constraint Y”

It’s immensely helpful for users to verify results or to understand what has already been tried.

(We keep humans-in-the-loop when it’s ambiguous; we’re not claiming full autonomy in a high-stakes workflow.)

Where it fails

The main failure mode isn’t hallucination. Despite the noise in socials, hallucinations haven’t been an issue in tasks that are operating on underlying data, be it a database or a code repository.

A missing necessary signal in the data that the Agent can see is an issue though.

Example: Invoice states amount X. Then after bank fees, X − 50 arrives but also remittance info is missing and multiple invoices around that value exist.

If you don’t have a fees table, or a reliable expected fee model, the agent may overfit to circumstantial fields (date/name proximity) and end up with an imperfect match even though no clean knapsack-like solution exists.

The obvious fix is add the missing data (fees, remittance enrichment, counterparty identifiers). Most of the time it doesn’t work like that, simply because your dependency on getting the data in is another person.

So, usually we have to admit it’s ambiguous and make it simple for the operator to execute their internal playbook for that case, for example by forwarding the task to another agent that requests info from the customer. But that’s a topic for another post.

Observability: What to log

Understanding how the agent decides is useful.

We collect the user prompt, the sequence of MCP calls (tables described + queries executed) and the intermediate outputs that lead to the final decision

This is basically a replayable trace of the investigation, which is extremely helpful in debugging and building evals.

(And for the obvious governance point: log retention + access controls matter a lot here.)

Prompting is a weak lever and “Hydra prompts” are real

This was another surprising lesson.

Keeping system prompts minimal and generic usually helped. When we tightened prompts to fix a failure, we often got a Hydra effect: edge case A improved, edge cases B, C and D got worse.

In practice, LLMs overweigh what you tell them right now (prompt salience / instruction dominance). So adding more instructions often causes over-indexing and reduces generality.

With that in mind, the strongest way we found to improve behavior was improving the semantic quality of the MCP + data layer over working on the prompts.

This shifted our work from prompt context engineering to data access context engineering.

This pattern matches what we see in coding agents

The only agentic use case at scale right now is coding (I’m a skeptic of the actual vs. marketed scale of AI SDRs).

And what made coding agents work wasn’t a perfect prompt. It was the ecosystem of searchable codebases, clear file/module names, fast feedback loops (tests, linters) and tools that give agents discoverability (mostly grep).

That’s the same story here: make the right context and affordances available, and the agent will often figure out the procedure without you teaching it step-by-step.

It also matches broader “AI-in-production” survey data: teams report heavy use of database access and growing use of MCP/tool calling, plus persistent pain around evals and governance (Theory Ventures AI in Practice Survey 2025).

Closing takeaways

1. Agents can already do surprisingly high-level white-collar investigation work (at least in scoped, tool-grounded settings).

2. Prompting is useful, but it’s not a reliable primary control knob; it’s easy to overfit and regress.

3. The biggest leverage is building a high-quality semantic data surface (schema, views, descriptions, access patterns) and restricting tools to what you can govern.

If you’ve shipped something similar (reconciliation, audit, fraud ops, compliance investigations, etc.), I’d love to compare notes, especially around eval design and how you handle true ambiguity.

• In physical-goods industries, most of the cost isn’t in running the ERP. It’s in getting reality into the ERP and reconciling exceptions.

• AI-native ERPs can work well for digital-first companies, but they rarely touch the real complexity that humans work through in these industries.

• The opportunity is not replacing SAP, but surrounding it with AI that handles documents, reconciliations, and exceptions.

AI accounting tools are red hot and raising big money ($400m+ in 2025), but when I ask finance leaders in traditional businesses like manufacturing and consumer goods about the new wave of “AI-native” ledgers, the response is skeptical.

The technology isn’t bad; the issue is who the tools are built for and what workflows they assume.

When you look at early customer lists, most of the adoption is in high growth software companies. The founders come from that world, so they design for that world.

For a SaaS company with clean data, one revenue stream, and API friendly tools, that’s fine. For a food producer, a distributor, or a manufacturer shipping containers through ports and customs, it is not.

Features don’t match the job

Because the tools are built for tech finance teams, they reflect that reality. Instead of logging into an interface to send an invoice reminder, an agent takes care of it and alerts you of the progress in an internal messaging app. It’s a smooth experience for lightweight tasks.

In document-heavy traditional finance environments, that’s not how the work happens.

Entire teams spend their days on reconciliation and exceptions. They live across SAP, Excel, email, carrier portals, and folders full of PDFs. Their job is not simple approvals in Slack or Teams. They figure out what actually happened across 15 different systems and documents.

The contract vs the manifest

In physical-goods businesses, transactions are not as simple as contract and invoice. They represent a sequence of events in the real world.

A single international shipment for a CPG company might involve:

• A purchase order & confirmation

• A packing list & bill of lading

• Customs documents & export declarations

• Insurance & inspection certificates

• Proofs of delivery

• Supplier invoices, freight invoices, & credit notes
  

Each document is a snapshot in time from a different party. Quantities change, delivery dates slip, and claims appear after the fact. Every company has its own way of stitching this together, so you cannot just ship one “intake template” and assume it fits. The variability and custom logic is why Excel is still the dominant tool.

Trying to hide this complexity inside a single AI-native ERP quickly hits a wall. Finance teams need to control the logic and can’t trust a black box.

The Swivel-Chair API

Getting this data into the ERP is still very manual. You can’t force every customer, customs broker, and vendor to integrate with your system. They send PDFs, CSVs, and emails when they are ready.

So teams fall back on a swivel chair API, a real human to do the brutal work.

You have operators with the ERP on one screen and a PDF on the other. They are comparing lines, spotting mismatches in quantity or price, investigating why the mismatch exists, and calling partners to clarify.

Exceptions transition the job from data entry into investigation.

In most finance teams, a majority of time is spent on this reconciliation work. The big cost is in stitching together a coherent story from messy, conflicting evidence and finding a resolution.

Replacing the core feels wrong

The pain is around the system, not inside it. Swapping one set of tables (SAP) for another (Modern ERP) doesn’t address where the money is actually lost.

The real costs sit upstream: in document handling, in errors that slip through, and in overpayments that only show up in recovery audits. ERP replacement is a multi-year undertaking, and for these teams, the risk is simply not worth the benefit.

What a better approach looks like

If the ERP is entrenched and the pain is upstream, the goal should be different: Do not replace the core. Build smart systems that surround it and operate with the same context a human uses.

Keep the legacy ERP as the system of record, but replace the toil around operating it.

Concretely, you want AI that can ingest messy files (PDFs, Emails, EDI), extract and normalize the data, and match it against your POs and receipts while handling the edge cases.

AI becomes the reconciliation and compliance layer around the ERP, with infinite time to investigate issues and solve problems humans miss.

AI agents change that.

Autonomous agents are changing the fundamentals of how software works. They read messy documents, reason across systems, and execute workflows the way an experienced analyst would.

In finance operations, that is a huge unlock.

In this post I will:

• Outline why finance workflows are such a brutal testbed

• Show where AI agents are genuinely strong, and where they are fragile

• Sketch what an actual Finance Agent Platform needs to look like

• Walk through a reconciliation case that a human analyst would dread and an agent can actually handle‍

The target audience here: Finance Managers, VPs of Finance and CFOs who care less about model architectures and more about close quality, cycle times and headcount constraints.‍

Why finance workflows are so hard for generic AI Agents and workflow tools

A lot of back office AI decks quietly assume the inputs are nice, structured tables. That is not necessarily the reality.

Typical finance workflows look like this:

• Unstructured data

  • PDFs, scans, emails, screenshots

  • Supplier statements with creative layouts

  • Payment provider exports that change format without warning

• Large, ugly datasets

  • Payment Provider exports with hundreds of thousands of semi-structured rows

  • Invoice summaries with hundreds of line items

  • GL dumps with years of history

• Non standard documents

  • Contracts with custom fee ladders and carve outs

  • One off side letters that materially change economics

  • Marketplace payout reports that do not match invoice logic one to one

• Indirect, circumstantial matching

  • No single ID that ties invoice, PSP transaction and settlement together

  • Need to triangulate by amounts, dates, currencies, users, SKUs

  • Many situations where you can only reach a high confidence hypothesis, not a perfect match

• Multiple data sources involved

  • ERP, PSP dashboards, bank statements, app store reports, contract repositories, ticketing systems

  • None of them designed to talk to each other natively

These are not simple problems. They are multi step, investigative tasks and that is exactly where AI agents start to look interesting.

Where AI agents are strong vs where they are fragile

At their core, today’s best AI models are extremely capable reasoning engines with limited working memory and no native integrations.

Understanding that tradeoff is key to designing something useful for finance.‍

Strengths

1. Handling non standard inputs

Agents are very good at reading weird stuff:

• Invoices that do not match the template

• Contracts written by lawyers who have never seen your chart of accounts

• Payment Service Provider reports that mix English labels, local language fees and internal codes

Pattern recognition and layout understanding are generationally better than what legacy OCR or rule based extraction can do.

2. Deductive reasoning

Agents can follow multi step instructions such as:

For each invoice, try to match against PSP exports. If no direct match exists, search by user, by amount within tolerance, and by date proximity. If still unresolved, propose the most likely explanation and a concrete next action.

This is not simple classification, it is a chain of hypotheses:

• “If X does not match, try Y”

• “If Y and Z conflict, flag an exception rather than forcing a match”

• “If data is missing, recommend which system to query next”

Pretty much that’s how an Analyst would think about that.

3. Understanding finance tasks and procedures

With the right prompting agents can internalise:

• Basic accounting logic (debits, credits, accruals, prepaids)

• Policy rules (approval thresholds, materiality, risk tiers)

• Domain specific workflows (PSP reconciliation, vendor statement recs, revenue share calculations)

This makes it realistic to encode your playbooks as agent policies instead of if/else decision trees.

Weaknesses

1. Limited context for large datasets

Models have a finite “context window”: The amount of data they can actively hold in working memory in one go.

Finance blows through that quickly:

• You can’t dump a 200k row CSV and a 300 page PDF into the model and say “reconcile this”.

• Naively splitting the data to make it fit loses context and produces inconsistent reasoning.

2. Instruction overload

If you ask an agent to:

Read six documents, understand five policies, reconcile three systems, draft a report, and redesign the chart of accounts

in a single run, quality collapses.

Technically, this is instruction overload: the task surface is too broad for the model to maintain a coherent reasoning process. You get shallow, generic answers rather than deep, investigative work.

3. Being detached from real systems

Out of the box, agents:

• Do not see your Inbox

• Do not connect to your ERP

• Do not push entries into your GL

• Do not post into MS Teams or Slack to collaborate with your team

Without a platform around them, they are powerful but isolated analysts who can only ever say: “Here is what I would do, if I could touch the systems.”

What an optimal Finance AI Agent platform must look like

To make agents actually useful for finance teams, you need more than a model and a chat UI. You need three layers that work together:

1. Data Capture Layer

2. Operations Layer

3. Reporting and Integration Layer

Think of it as designing an operating model around a very smart analyst: you define how work enters, how they process it step by step, and how the outcomes flow back into your systems and reports.

1) Data Capture Layer: giving agents structured eyes

The job of this layer is to normalise messy reality into something agents can query and reason over.

Capture and extract

Agents handle the intake of:

• Invoices

• Purchase orders

• Contracts and side letters

• Bank statements

• PSP exports and app store reports

• Vendor statements

They:

• Read documents in whatever format they arrive

• Extract key fields and line items

• Link them to entities such as suppliers, customers, SKUs, accounts, cost centers

Routing and classification

Routing agents then:

• Decide where each document or record should go

• Tag them by workflow (payables, receivables, revenue share, disputes, etc.)

• Trigger the right operational agents in the next layer

This layer is about data fidelity. If you do not get this right, everything downstream is built on sand.

2) Operations Layer: where actual finance work happens

This is where the platform moves to an agent that actually does finance tasks.

Matching and reconciliation

Agents here perform tasks such as:

• Matching invoices to POs, GRNs and contracts

• Reconciling PSP settlement data with your invoices and your bank statements

• Checking whether refunds recorded in your system were actually processed at the PSP

• Surfacing missing documents or unexplained balances

They operate against the structured data produced by the capture layer.

Workflow and conditional logic

These agents also orchestrate next steps:

• Apply conditional logic such as:

  • “If all checks pass and amount < 10k, move to auto approval”

  • “If contract and invoice disagree by more than 3 percent, route to finance review”

  • “If refund mismatches are detected, create a ticket for payments ops and attach evidence”

• Collect additional evidence autonomously:

  • Fetching additional exports for specific dates

  • Pulling fee schedules from contracts

  • Checking historical patterns for similar exceptions

The important property:

Agents take instructions and procedures and interact with the Data Capture Layer autonomously, exploring deeper where their reasoning says “this does not add up”.

3) Reporting and Integration Layer: closing the loop

Once work is done, results must leave the Agent world and enter the finance world.

That means:

• ERP and ledger pushes

  • Posting suggested journal entries

  • Updating invoice and payment statuses

  • Tagging disputes and credits

• Reports and spreadsheets

  • Generating reconciliation reports that a controller can review and sign off

  • Exporting exception lists to Excel or Google Sheets when needed

  • Producing management views for close and audit

• Collaboration hooks

  • Summaries and links posted into MS Teams, Slack or email

  • Human in the loop approvals where required by policy

Case study: reconciliation that a human analyst hates and an agent can handle

Consider a very typical but painful situation.

You run a consumer app. Revenue flows through multiple payment service providers (PSPs). For a given period, you want confidence that:

• Every invoice is backed by a real transaction

• Fees match the contract

• Refunds are real and reflected correctly

On the table:

• An invoice summary with hundreds of line items

• PSP exports with thousands of rows across multiple days

• Several contracts with different fee structures and payout timings

A human analyst will:

• Open five systems

• Fight with CSV filters

• Keep a mental model of amounts, users and dates in their head

• Spend hours on edge cases that do not quite match

Here is how an agent inside the platform would tackle it.‍

Step 1: Build a coherent picture

The agent:

• Pulls the invoice summary and normalises the line items

• Loads the relevant PSP exports for the period plus a window around it

• Parses the contracts and derives fee rules and expected net payouts

• Aligns currencies, timestamps and customer or order identifiers

Now it has a structured graph to reason over.

Step 2: Attempt deterministic matching

For each invoice line, the agent tries the obvious matches:

• Same order id in the PSP export

• Same amount, same currency, within the expected date window

• Same user or account id where available

Where this works, it confirms:

• Gross amount

• Fees according to contract vs actual fees

• Settlement timing vs expectations

Step 3: Investigate exceptions with deeper reasoning

The interesting part is what happens when matching fails or produces contradictions. Examples of insights the agent can surface:

• Missing PSP transaction

  • “Invoice A for user 0144 is for EUR 7.99 and is expected to be processed by Adyen, but there is no corresponding Adyen transaction. There is a Stripe transaction for EUR 7.99 that does not reference our app order id. This suggests a routing or mapping issue between our order system and PSP configuration.”

• Refund mismatch

  • “Invoice B is marked as refunded in our system, but the Adyen export only shows a captured payment of EUR 3.56 with no refund record. Amounts match, but the refund flow is missing. Recommended follow up: check Adyen refund transactions and settlement batches for this transaction id, and if no refund exists, escalate to payments ops to reconcile or issue the refund.”

• Invoice vs app store status conflict

  • “Invoice C is marked as refunded in the ledger, but the corresponding App Store transaction is still captured. The invoice total matches the App Store gross amount and the PSP fee matches the contract. Recommended action: verify whether a refund was actually initiated at the store. If not, either process the refund or update the invoice status to ‘paid’ to reflect reality.”

These are structured hypotheses about what went wrong and what to do next.

Step 4: Produce auditor friendly output

Finally, the agent produces:

• A reconciliation report:

  • Matched items, mismatches, missing transactions, suspected mapping issues

  • For each exception, a suggested explanation and a concrete next step

• System updates:

  • Suggested journal entries, status updates for invoices, and tagged exceptions

A finance manager can review, spot check, and sign off.

The delta vs today is not that the task suddenly requires zero effort. It is that the bulk of the pattern recognition, cross referencing and explanation drafting is now done by an always on, perfectly consistent analyst.‍

Why this is a meaningful tech unlock for finance leaders‍

From a CFO or VP Finance perspective, a Finance Agent Platform is about fundamentally changing the constraints of your operating model.

‍1. Complexity no longer scales linearly with headcount

Today, adding new geographies, PSPs or product lines often means more spreadsheets and more analysts.
Agents let you grow complexity without a one to one growth in manual reconciliation effort.

2. Quality becomes systematic

Right now, many reconciliations and investigations depend on a few people who “know how things really work”.
Agents, if designed correctly, encode that reasoning into procedures that are:

• Repeatable

• Inspectable

• Auditable

3. You get a live view of risk

Tight integration into your systems means agents can run continuously:

• Surfacing mismatches before month end

• Flagging fee leakage against contracts in near real time

• Highlighting process failures that would otherwise only appear in an audit

4. Human expertise is used where it matters

You still need humans:

• To define policies and materiality thresholds

• To handle exceptions with commercial or legal nuance

• To design controls and own the results

But instead of going through data work your team is reviewing agent results and focusing on judgment calls.‍

Where do you start?‍

If you are a finance leader looking at this, the path is not “replace the team with agents”. The path is:

1. Pick one painful, investigative workflow. PSP reconciliation, vendor statement recs or refund audits are perfect candidates.

2. Map the data sources, the playbook your best analyst already follows, and the failure modes.

3. Deploy agents in the three layer model:

   • Capture and normalise all inputs

   • Encode the investigative steps in an operations agent

   • Integrate outputs directly into your ERP, reports and collaboration tools

The tech is now good enough that this is not speculative. What matters is platform design: respecting where AI is strong, compensating for where it is weak, and integrating it deeply into your finance stack.

The finance teams that get this right will close faster, with fewer surprises, while operating more complex businesses than their peers can support.

That is the real unlock.

This blog is also on our website.

‍

150 years of research, engineering breakthroughs and hundreds of IDP products later we were finally able to scan a receipt and have the fields be filled out - if it looked nice and friendly enough to the OCR model. Heureka.

Unfortunately for Tesseract, Abbyy and co. they suffer from the complication that documents are written by humans. And humans love to do things like:

• Stamp over the most critical data because it feels like the right spot

• Organize data in tables with four nested of levels of groupings

• Disagree on standards of data, thus abandoning any approach of standardization and simply sending their very own format around

• Add handwritten comments in their own language

• Create documents that need at least a college degree in their field to correctly understand.

This meant OCR models were basically just helpers for data scientists, handling cleanups, routings, and post-validations to get something only vaguely close to real automation at work.

Multimodal LLMs enter the scene

With the launch of Multimodal LLMs, most prominently Gemini-Flash-2.0, many AI fields with decades long track records of active research became obsolete. Image Classification? Solved. Answering questions over images, text and tables? Auf Wiedersehen. And OCR? Exactly.

What gives LLMs the power to declare victory in dozens of areas that were previously considered their own domain comes down to two characteristics of the Transformer architecture and its training:

1. The model’s architecture allows it to have global context of everything that is in the input. I.e. it can for example understand that a table row belongs to headers printed 5 pages prior.

2. The model is trained on the entire internet and can understand concepts like stamps, order forms with handwriting over them and that invoices can have as many layouts as there’s people on this planet.

With that intuition in mind it’s not surprising why they are able to easily solve problems that OCRs gave up on. Instead of just transforming pixel patterns into words they can look at the whole document at once and then even reach for their corpus of all human knowledge to make sense of it on a conceptual level. Not only that, they can even return information about an embedded Image with barely any text at all! Think of a technical drawing for example. You will be able to pull data from that where OCR would give you a transcript of the labels if you’re lucky.

Where OCRs got an ace up their sleeve - for now

Remember the part where I said that Language Models are awesome with their huge training set and global context?

Makes them a bit expensive unfortunately. A PDF with a few hundred pages and embedded images will cost you a couple Dollars, depending on the model. In document processing we also suffer from the relatively small output context windows. 64 thousand tokens may be enough to write a small codebase in Cursor but will blow up when trying to extract just a couple dozen pages of tables from docs. There’s also something reassuring about clear, explainable rules over the black-box nature of LLMs.

With all that said, I still believe that processing documents will be a solved problem in a couple years time. It feels like we’re 95% there. Models will get cheaper and more efficient with longer context windows. The focus will be on automating away the flow from document to System of Record, and AI Agents are already starting to being helpful here too.

My company cloudsquid works on this problem. If you’re interested in working with us or simply wanna chat about this post contact me through our website, filip@cloudsquid.io or on LinkedIn.

Yes, that's right, another in-depth technical blog showcasing an array of bar charts with cloudsquid predictably ranked as the number one vendor in the space with the "most accurate AI available ever developed."

"How is that possible?" you may ask. "How are there 1000 document AI vendors who all claim to have the most accurate AI on the planet?" "People wouldn't just go on the internet and lie, would they?"

Our benchmark approach

Alright, let's drop the pretense. We ran a benchmark, and we did pretty well. Here's the thing: we're not reinventing the wheel with something brand new in AI. We offer different AI pipelines optimized for different use cases, and we wanted to objectively evaluate how they perform on a challenging open-source document dataset.

Methodology

We used the Contract Understanding Atticus Dataset (CUAD), a collection of 500 legal contracts with over 13,000 annotations across 41 different legal provisions. For our evaluation:

• We tested our two main document AI pipelines (Advanced and Flash)

• Each document was processed through our unmodified production pipelines

• We extracted the same data points defined in the CUAD ground truth

• We used identical prompts across all models with zero customization for this dataset

• Accuracy was measured as the percentage of correctly extracted data points compared to the ground truth

Findings

Several interesting patterns emerged from our evaluation:

• OCR remains critical - Even in a legal contract setting with primarily text content, adding an OCR step to the pipeline instead of relying solely on an LLM-based approach improved accuracy by approximately 10%.
  

• Our Advanced pipeline achieved 98.5%1 extraction accuracy across the 500 documents and thousands of data points.
  

• Gemini 2.0 Flash, despite the substantial OCR-killing hype, still lags in standalone accuracy. It remains a solid option for simpler use cases, but the performance gap is noticeable in complex document understanding tasks.

Document markdown performance is a commodity

Let's get to the heart of the matter. Cloudsquid isn't doing anything particularly novel on the model and OCR side. Our pipeline is straightforward: best-in-class OCR + best-in-class model. We ran this benchmark with zero additional optimizations tuned for this dataset.

This leads to several observations about the current state of document AI:

1. Marginal performance differences - Different approaches in AI pipeline setup are currently separated by tiny increments in performance. While these incremental gains might still matter in certain contexts, they're rapidly eroding as foundation models improve.

2. Technical moat illusion - Startups that emphasize fine-tuning, preprocessing, training their own OCR, or any of another half dozen esoteric techniques you’ve seen discussed in hacker news threads are often engaged in marketing exercises designed to make document AI sound more complicated than it is. They want to create an illusion of a technical moat so that developers won’t try to build it themselves.

3. Benchmark skepticism - In our experience, customers don't particularly trust accuracy studies. They generally assume everyone is using similar technology and base their purchasing decisions on other factors like price, integration capabilities, user experience, and scalability.

Where the Real Innovation Lies

What's really interesting about the CUAD benchmark is the diverse and numerous data extractions required and the quality of the prompts used to test against the ground truth. It provides an efficient way to evaluate models.

Achieving nearly 99% accuracy with zero tweaks to our pipeline or prompt adjustments feels impressive. However, the more compelling area for innovation lies in building systems around the model to:

1. Create high-quality prompts tuned to the document

2. Iterate and evaluate results quickly

3. Adapt to different document types automatically

4. Tightly integrate outputs into workflows and business logic

The majority of inaccuracies in data extraction still stem from shortcomings at the prompting and evaluation level rather than issues in converting documents into machine-readable formats. This is where engineering efforts should be focused, not on marginal OCR improvements or custom model training. Creating intelligent systems that optimize the interaction between documents, models, and actual enterprise workflows and their corresponding systems.

Conclusion

While we could have produced yet another "we're #1" benchmark post, we believe greater value comes from transparency about where the real challenges and opportunities in document AI truly lie. The performance ceiling of current technologies is very high, but consistent real-world implementation remains challenging and limited for LLM based vendors.

For the pragmatic and experienced evaluator, the question shouldn’t be "which vendor has the best accuracy on paper?" but rather "which solution can consistently deliver high accuracy across my specific document types with minimal configuration overhead?"

We're betting that the future belongs to systems that can intelligently adapt to document variations and integrate with business workflows and logic rather than those claiming a few extra fractions of percentage points on controlled benchmarks.

If you’re interested in how we ran this, happy to share the specific points. We can also walk you through exactly how we set up our AI pipelines in the product, it’s no secret sauce.

When working with language models, receiving data in a well-defined format (like JSON) ensures that downstream systems can process the results automatically. Instead of dealing with paragraphs of text, you get a fixed set of fields that map directly to your use case. This approach:

• Helps integrate the model’s responses into business workflows without cleanup

• Prepares and transforms the data for direct use in an application

• Reduces ambiguity and hallucinations in output

• Makes validation easier

Who This Guide is For

Anyone who needs to reliably extract specific information from text using an LLM and wants to receive that information in a structured format. Most likely you are a technical person setting up a data extraction pipeline from scratch, or a user of a platform like Cloudsquid that abstracts the code-based steps to allow easy batch data processing or pipeline setup.

Examples include:

• Retrieving critical legal terms from customer contracts

• Pulling product details from technical manuals

• Getting line items from an invoice or purchase order

• Processing ID documents or other information for KYC checks

This guide is not intended specifically for cloudsquid customers but as a general-purpose guide for structured data extraction prompting. Many online guides focus primarily on technical implementation and schema structure rather than deeply exploring how to write comprehensive, performant prompts. For cloudsquid-specific guidance, see our prompting guide in our docs page.

Theoretical Considerations

How to Think About Prompts

Think of prompts as instructions you'd give to someone unfamiliar with the task. Ask yourself: "Could someone perform this extraction accurately using only these instructions?" Clearly specify:

• What you want (e.g., "Extract the price for installation labor.")

• How you want it formatted (e.g., "Output as a decimal value.")

• Any constraints (e.g., "If multiple values appear, return null.")

Be explicit—vagueness often results in hallucinated outputs. Providing constraints and fallbacks limits this issue.

What LLMs Are Not Good At Doing

• Complex Math & Logic: If your task requires calculations, averages, or complicated logic, LLMs might make mistakes. It’s better to have the LLM extract relevant components of data, then perform complex calculations externally or with specialized tools. One thing to note, reasoning models are much better at math but you will most likely be using a more basic model with structured outputs or JSON mode for data extraction tasks.

• Resolving Contradictory Data: Clearly instruct how to handle contradictory information to avoid random guesses.

Setting Up a System Prompt

Purpose of a System Prompt

A system prompt sets foundational rules for data extraction, independent of individual prompts.

Example System Prompt (Legal context example)

"You are a legal clerk assisting in data extraction. Always follow these rules:

1. If extracting exact legal terms, retain original language and wording.

2. Only extract numeric values with an exact document match.

3. Return null or an empty string if conflicting values exist."

Individual Prompts

How to Describe a Field You Want to Extract

• Name: Specify the exact field name (e.g., "title" vs. "product_title").

• Prompt: Aim for precision with added details and context to help the model fully understand the task. For example you could include examples, locations in the document, or words that are descriptive of the task like “Gross Salary” vs. “Salary”

• Format or Data Type: If it’s a date, say something like "date": "YYYY-MM-DD". For numbers, specify if you need integers or floats.

• Fallbacks: If data for that field doesn’t exist, specify what to do. For example, “If no date is found, return null.”

Writing Prompts, Bad vs. Good

Bad Prompt

Extract all the details about products that you can find from the quote

Why it’s bad: It’s too vague, doesn’t specify structure, doesn’t explain the required fields, and doesn’t handle missing data.

Good Prompt (including schema for structured output mode)

Extract all product line items from the provided quote. Each product line item includes:

1. Product Description: A textual description of the product.

2. Product SKU: The unique identifier or stock-keeping unit (SKU) for the product.

3. Quantity: The number of units of the product (as a number).

4. Price Per Item: The price for one unit of the product (as a decimal).

Adhere strictly to the following schema:

{

"type": "object",

"properties": {

"product_line_items": {

"type": "array",

"items": {

"type": "object",

"properties": {

"product_description": { "type": "string" },

"product_sku": { "type": "string" },

"quantity": { "type": "number" },

"price_per_item": { "type": "number" }

},

"required": [

"product_description",

"product_sku",

"quantity",

"price_per_item"

]

}

}

},

"required": ["product_line_items"]

}

Note: The above example is the full input including the schema which needs to be passed to the model directly with the prompt, the model then ensures the output conforms to the schema. This might vary depending on the foundational model, this example is specific to OpenAI.

Accepted Ranges and Validations

In some cases, you may want to limit or validate numeric fields. For example:

• Price Range: “If the price is below 0 or above 999.99, return null for price.”

• Rating: “Ratings should be between 1 and 5. If the extracted rating is out of range, set it to 5 if it’s above 5, or to 1 if it’s below 1.”

By including such rules, you decrease the chance of invalid data.

Default Values for Extractions

Real-world data can be messy, and you may not always find a particular field. Provide defaults:

• Strings: Use an empty string ("").

• Numbers: Use 0 or null, depending on your system’s preference.

• Arrays: Use an empty array ([]).

Example instruction:

If you cannot find the manufacturer in the text, set "manufacturer": "".

Extracting Lists or Arrays

For fields where multiple entries might appear, instruct the model to provide a list:

"tags": ["tag1", "tag2", ...]

If there’s a maximum:

Include only the first 5 tags you find. If none are found, return an empty array.

This helps keep the output standardized.

Using Categories to Force a Specific Output

Sometimes data must fit one of a few predefined options (e.g., "Low", "Medium", or "High" for a risk rating). The LLM might invent new categories like "Very High". To avoid this:

• Explicitly restrict it: “Possible values for risk_level are only: Low, Medium, High. If no relevant data is found, set risk_level to Low.”

• Validate on your side too. If the LLM tries returning something else, you can revert to your default or handle the error.

Summarization & Generation Tasks

While structured extraction is your main goal, you might also want short summaries or creative text under fixed keys:

"summary": "A single sentence summary of the text."

"headline": "A catchy 5-10 word headline describing the document."

Be clear with length limits:

“Summaries should be under 30 words. Truncate if longer.”

Additional Tips & Topics

Nested Objects vs. Flat Data

If your data is hierarchical (e.g., an address with street, city, zip), sometimes it’s clearer to keep those fields nested:

"location": {

"street": "123 Main St.",

"city": "Example City",

"zip": "12345"

}

Otherwise, a flat structure may be easier for quick lookups:

"street": "123 Main St.",

"city": "Example City",

"zip": "12345"

Choose based on how your system will use the data.

Testing & Iterating Prompts

• Test with examples that span a range of real-world data for each use case.

• Check if the model’s output adheres to the constraints (e.g., does it really truncate the description? Does it respect max array sizes?).

• Adjust your prompt until you consistently get the structure you want.

Handling Large Inputs

If your input text is too long (e.g., multiple pages of instructions or a large PDF), consider chunking the text and running multiple extractions. You can then merge or reconcile the results afterward.

Conclusion

Key Takeaways

1. Be Explicit: LLMs rely on clear, direct instructions to produce consistent JSON or other structured outputs.

2. Provide Fallbacks: Ensure you specify default or null values for missing data.

3. Validate: Even with a good prompt, always validate the output to handle unexpected results against a large enough sample size of data. Consider using a tool to run evals if you need to test on an ongoing basis.

Receive PDF → extract data → validate outputs → manually input to systems fields.

This inefficient workflow, using humans as APIs, spans industries, job roles, and regions. In the U.S. alone, the white-collar labor market—worth $9.1 trillion—is bogged down by low-value, data-heavy tasks. It’s more than just wasted time; it’s lost potential.

Legacy AI solutions have limitations

Existing OCR based document AI solutions are built to handle simple, repetitive, and transactional tasks. They require extensive data labelling and training, and can process text or documents that follow rigid, predictable formats. They don’t understand the context of the document; they categorized. A small deviation—an extra field, a misaligned table—can derail the entire process. They also perform poorly on complex, mixed-content documents that combine text, tables, and images.

These tools are still useful for automating specific back-office workflows, but are too rigid to handle real world variability like humans can.

LLMs change the data extraction and transformation Paradigm

Unlike their OCR predecessors, LLMs don’t just categorize unstructured data—they understand it. They excel in complex environments, capable of handling diverse content like documents that mix text, images, and tables. 

Prompting combined with rule-based guardrails surpasses human accuracy, even in scenarios where variability is high. You no longer have to rely on rigid templates or constant retraining—LLMs adapt to the document, not the other way around.

Going past demo and into production

The promise of LLMs to solve unstructured data challenges is clear. We spoke with many product teams rushing to build Generative AI functionalities for data extraction in-house. From vertical software startups to large enterprises, they all struggled with critical last-mile issues around accuracy and reliability, preventing them from moving to production.

Creating an AI pipeline from scratch to handle complex documents at scale, designing interfaces to refine and test prompts, and implementing features like observability to monitor performance is an impractical challenge. Even a basic version would take a dedicated engineering team months to build, so they don’t.

Why we’re building cloudsquid

Our vision is to solve these data challenges at the infrastructure level for technical teams. Developers and data teams should be focused on building valuable end-user experiences from newly accessible data streams—not on rebuilding essential data infrastructure from the ground up. Unstructured data ETL will become foundational for AI feature development, and we’re building it to enable teams to harness Generative AI with the speed, security, accuracy, and scalability needed for production.

We’re thrilled to have HTGF and Backbone Ventures join us in this journey, along with experienced AI product leaders Reetu Kainulainen and Udi Miron. With their support, we’re poised to scale our engineering team and core platform. Expect much more in the coming months as we expand cloudsquid’s capabilities.