In Part 2, I shared the functional architecture of our agentic refund loop. It works. But in operations, “working” is only the baseline. For financial transactions, “safe,” “auditable,” and “resilient” are the real requirements.
When we reviewed the MVP with stakeholders, one question came up repeatedly: “Why build this level of rigor for refunds under $50? Isn’t this overkill?”
The answer is no, but not because $50 matters financially. It matters strategically. The client chose sub-$50 refunds as the proof-of-concept precisely because the financial risk was contained. If something went wrong, it wouldn’t bankrupt the company. But if the governance failed, if there were no audit trails, no duplicate prevention, no fraud checks, the process could never be trusted with larger amounts. This wasn’t about protecting $50; it was about proving that the system could be managed, stopped, or scaled at a moment’s notice without introducing unacceptable risk.
Before this system could touch real money, even small amounts, we had to answer 25 critical questions spanning financial integrity, AI governance, operational resilience, and compliance. These aren’t edge cases; they’re the foundation of any production-grade automation. And they’re non-negotiable regardless of transaction size, because governance isn’t proportional to dollar value. A $50 refund processed without an audit trail is still a compliance failure. A duplicate refund caused by missing idempotency keys is still a control breakdown. The safeguards don’t scale down just because the amount does.
This final installment details those unanswered questions, the risks they address, and the concrete controls we implemented to transform a fragile prototype into a trustworthy ops partner. This is where the real engineering begins, and where trust is earned, one controlled step at a time.
Pillar 1: Financial Integrity & Auditability
If you cannot reconstruct every transaction or guarantee single execution, you do not have automation, you have liability.
- Audit Logging & Traceability: Airtable stores current state but not history. We implemented an append-only Event Log recording timestamp, previous/new state, actor (AI/Human/Workflow), workflow version, prompt version, API responses, and notes. Accounting can now reconstruct any refund six months later.
- Idempotency (Duplicate Prevention): Stripe refunds succeeded but network timeouts triggered retries, causing double refunds. We now generate a unique idempotency key per request and reuse it on every retry, stored in Airtable and passed to Stripe on every API call.
- Race Conditions & Concurrency: Two workers could read the same VERIFIED record simultaneously. We introduced a PROCESSING state that locks the record before execution, preventing concurrent processing.
- Human Editing Controls: Managers could accidentally edit refund amounts. We restricted editable fields to Status only; Amount is read-only and validated on all human changes.
- State Machine Completeness: Our original states missed critical transitions. We documented every legal state transition, adding PROCESSING, WAITING_CUSTOMER, WAITING_FINANCE, DUPLICATE, PARTIAL_REFUND, FRAUD_REVIEW, CANCELLED, EXPIRED, and FAILED_PERMANENTLY.
- Retry Strategy: Retrying every 15 minutes forever creates unnecessary load. We implemented exponential backoff: 15m → 30m → 1h → 2h → 4h → 8h, then escalate.
- Dead Letter Queue: Some failures never resolve automatically. We introduced FAILED_PERMANENTLY after max retries, routing these to a dedicated queue for manual investigation.
- Data Reconciliation: Stripe said refunded but Airtable didn’t match. We scheduled daily reconciliation between Stripe and Airtable, flagging discrepancies within 24 hours.
Pillar 2: AI Governance & Reliability
LLMs are probabilistic; financial systems must be deterministic. These controls bridge that gap.
- AI Confidence Validation: GPT may incorrectly extract Order ID or Amount. We store an AI Confidence Score and automatically route low-confidence extractions to Human Review.
- Prompt & Model Versioning: GPT behavior changes over time. We record Prompt Version, Model Name, Temperature, and Timestamp for every extraction, ensuring reproducibility.
- Deterministic LLM Output: Free-form text breaks workflows. We always require structured JSON output with Temperature set to 0.
- Duplicate Email Detection: Customers resend emails. We detect duplicates by matching Order ID, Email Thread, Message-ID, and Customer before creating a new request.
- Privacy & PII Protection: Customer emails contain names, addresses, and payment details. We mask sensitive fields and send only required data to the LLM.
- AI Cost Controls: Unchecked API calls create budget surprises. We implement usage limits: max email size, max attachment size, monthly AI budget cap, and max retries per request.
- Attachment Handling: Customers send PDFs, screenshots, receipts. We defined handling rules: ignore non-relevant files, use OCR/Vision Model for receipts, route complex attachments to Human Review.
- Fraud Detection: Refunds exceeding thresholds or from high-risk customers need scrutiny. We created pre-execution fraud rules checking refund amount, monthly frequency, chargeback history, VIP status, and partial refund patterns.
Pillar 3: Infrastructure & Scalability
MVP tools break at scale. These are the non-negotiable foundations for production resilience.
- Security: API keys stored in plaintext are a breach waiting to happen. We use a secure secrets manager with encryption at rest and automated key rotation.
- Workflow Versioning: We couldn’t trace which workflow processed a refund. We version every deployment and enable safe rollbacks.
- Airtable Scalability: Excellent for MVP, but concurrency and complex queries fail at volume. We planned migration to PostgreSQL/Azure SQL with proper indexing and transaction support.
- Disaster Recovery: What if Airtable, n8n, OpenAI, or Stripe goes down? We documented recovery procedures for each dependency, including fallback states and communication protocols.
- Testing Strategy: We couldn’t safely test refunds. We built a full sandbox environment using Stripe Test Mode and mock CRM APIs, with replay capability for workflow validation.
- Architectural Extensibility: The refund-specific design wouldn’t scale to other workflows. We generalized the platform to support returns, cancellations, account updates, subscription changes, onboarding, escalations, and compliance reviews.
Pillar 4: Operational Visibility & Performance
You cannot manage what you cannot measure. Separate accounting reports from ops health metrics.
- Operational Monitoring: Accounting dashboards don’t show system health. We built a separate Ops Dashboard tracking queue depth, failure reasons, average processing time, refund volume, AI cost, and human intervention rate.
- SLA Tracking: We capture timestamps for Request Received, Verification Started/Completed, Refund Initiated/Completed. Metrics include Average Time to Refund, Average Verification Time, and Average Human Review Time.
- Reporting Separation: Accounting gets financial summaries; Ops gets performance diagnostics. We decoupled these reports to prevent noise from obscuring actionable insights.
The Takeaway for Ops Leaders
Building an agentic loop is only 20% of the work. The remaining 80% is answering these 25 questions before deployment. This isn’t bureaucracy, it’s the difference between a demo that impresses in a boardroom and a system that survives Monday morning at 9 AM when volume spikes and APIs timeout.
Your team doesn’t need another chatbot. They need a reliable partner that handles ambiguity without introducing new risks. That reliability is built in the gaps between the happy path.
Let’s Discuss
Which of these 25 controls surprised you as non-negotiable? Or did we miss a critical question your team faced in production?
Drop a comment below, I’d love to hear what safety rails you’ve had to build.