A Series-B startup's agent-driven CI/CD pipeline.

Why Ledgerline rebuilt rather than scaled the existing pipeline

The decision to rebuild rather than scale was made at a Q4 2025 engineering offsite in Penang. Lim presented the on-call rotation data to Ledgerline's CTO, Priya Kannan: the previous quarter had produced 47 production incidents across the payments rails service, the customer-facing API, and the merchant onboarding flow. Of those 47 incidents, 23 had triggered a manual rollback. The mean engineer-hours-to-recovery sat at 4.2, which on a team of 31 engineers was creating an unsustainable interruption load. Kannan and Lim agreed that the scaling answer — hire two more SREs, expand the on-call rotation — was a treadmill that would not produce a different long-term trajectory. The rebuild proposal was framed not as automation theatre but as an attempt to compress the operational surface area to a level the existing team could actually own.

The architectural reference Lim brought to the offsite was a writeup from a Stripe internal engineering blog (republished in November 2025 on the public engineering site) that described a partial agent-driven deployment pipeline running inside Stripe's North America acquiring infrastructure. The Stripe writeup did not provide a full implementation reference, but it documented the failure-recovery loops in enough detail to give Lim's team a starting position. Lim's adaptation diverged from Stripe's in three structural ways. First, Stripe's pipeline runs in a custom orchestration runtime; Ledgerline's runs in GitHub Actions with LangGraph as the orchestration layer above it. Second, Stripe's pipeline uses a custom model gateway; Ledgerline's calls Claude Code directly with no intermediate routing layer. Third, Stripe's pipeline is governed by an internal policy engine that runs against a proprietary risk schema; Ledgerline's runs against a policy file checked into the same repository the pipeline deploys, with the policy enforced through Claude Code hooks at the pre-tool-call boundary.

The team's first week of design work surfaced a question the Stripe writeup did not answer: how would the pipeline handle deployment-blocking situations that required human judgement? Lim's resolution was to define three escalation tiers. Tier 0 was full agent autonomy: routine deployments, dependency updates, and rollback decisions on incidents with clear telemetry signatures. Tier 1 was agent-recommends-human-confirms: deployments that touched the payments rails service in production, schema migrations on the merchant database, or any change that the agent flagged as carrying a confidence interval below a defined threshold. Tier 2 was human-only: changes touching the regulatory-reporting subsystem, anything that altered the cryptographic key management layer, and any incident whose root-cause analysis crossed a defined complexity boundary. The tier structure was documented in a 12-page architectural decision record that Lim circulated to the engineering team in early January 2026 for review.

The review surfaced two concerns that shaped the implementation. The first concern, raised by the senior backend engineer Marisol Pereira, was that the tier definitions used qualitative language — "clear telemetry signatures", "defined complexity boundary" — that would require operational interpretation each time the pipeline ran. Lim's resolution was to translate every qualitative tier criterion into a quantitative threshold encoded in the policy file: confidence intervals as numerical probabilities, complexity boundaries as graph distances in the dependency mapping, telemetry signatures as named alert patterns from the existing Datadog rule set. The second concern, raised by the SRE-on-call lead Rajat Nair, was that the rollback automation needed an explicit human-pull lever that an on-call engineer could trigger from a phone at 3 AM. Lim's resolution was to add a PagerDuty-integrated kill switch that any on-call engineer could trigger by acknowledging a specific incident type, which would freeze the pipeline immediately and notify the leadership team. The kill switch was used twice in the four months following deployment, both times by Nair, and the post-incident review concluded that both uses were appropriate.

The architecture: Claude Code, LangGraph, LangSmith, the policy file

The runtime architecture is conceptually simple and operationally precise. A push to the main branch of any Ledgerline service repository triggers a GitHub Actions workflow that boots a Claude Code instance with the repository checked out and the service-specific policy file loaded. Claude Code reads the diff, the recent deployment history, the active alerts in Datadog, and the on-call calendar. The agent then constructs a deployment plan that lists the change set, the affected downstream services, the rollback procedure, and the tier classification. The plan is submitted to a LangGraph workflow that orchestrates the sequential steps: pre-deployment checks, canary deployment to 5 per cent of traffic, validation against a named set of acceptance criteria, ramp to 25 per cent, validation, ramp to 100 per cent, post-deployment monitoring window, sign-off. Each step's outcome is captured in LangSmith as a structured trace, with the reasoning the agent used at each branching decision preserved in the trace.

The policy file is the substantive control surface. Each service repository contains a policy file at .ledgerline/policy.yaml that defines the tier classification for the service, the named alert patterns that constitute a clear telemetry signature, the named services that require Tier 1 or Tier 2 treatment, the rollback playbook reference, the on-call escalation path, and the confidence threshold below which the agent must escalate. The policy file is enforced through Claude Code hooks that run at the pre-tool-call boundary: before the agent invokes a deployment action, the hook reads the policy, evaluates the proposed action against the policy constraints, and either allows the action, requires a human approval, or blocks the action with a structured rejection message. The hooks themselves are versioned in the repository, reviewed in code review like any other code change, and audited quarterly by the engineering leadership team.

The LangGraph orchestration layer handles the workflow's stateful logic. The workflow is defined as a directed graph with nodes for each deployment step and edges that capture the conditional logic — a failed canary returns to a rollback subgraph; a successful canary advances; an ambiguous canary triggers a Tier 1 escalation. The LangGraph definition is checked into a separate ledgerline-pipelines repository that the platform team maintains and that all service repositories reference. Changes to the workflow definition go through the same code review process as application code, with platform team approval required. Lim's team explicitly chose to keep the workflow definition declarative and the agent logic procedural — the graph captures the structure of the deployment process and the agent supplies the situated reasoning at each node. The separation has held under operational pressure, and Lim has described it during internal architecture reviews as the single most important design decision the team made.

LangSmith provides the observability layer that the platform team relies on to debug the agent's reasoning when an incident occurs. Every Claude Code session that the pipeline triggers is logged as a LangSmith trace with full prompt, response, tool-call, and intermediate-state capture. The traces are retained for 90 days under the standard retention policy and indexed by service, deployment ID, and tier classification. When an incident occurs, the SRE-on-call lead can pull the LangSmith trace for the deployment that preceded the incident and read the agent's reasoning at each decision point. The trace capture has been the single most valuable diagnostic capability the rebuild produced; the SRE team's post-incident reviews now reference the agent's reasoning trace as a primary diagnostic artifact rather than reconstructing the reasoning from logs alone. The implication for the team's incident response cadence has been substantial — the median post-incident review time has fallen from 5.5 hours to 1.8 hours since the trace capture went live.

The failure-recovery loops and the named SRE interactions

The failure-recovery loops are the part of the architecture that the engineering team treats as the most consequential and the most fragile. A deployment that fails a canary validation triggers the rollback subgraph, which the agent executes with a defined sequence: drain the canary, revert the deployment manifest, restore the previous version, validate the rollback against the same acceptance criteria the deployment was attempting to satisfy, and write an incident summary to the on-call channel. The agent's autonomy in the rollback loop is bounded — it can execute the documented rollback procedure for any Tier 0 deployment, must request human confirmation for Tier 1 deployments, and must request human execution for Tier 2 deployments. The boundary has held in operational practice; the team has not yet encountered an incident where the agent executed a rollback that the on-call engineer would have judged inappropriate.

The named SRE interactions that the team documents in the post-trial review are illustrative. In late January 2026, Rajat Nair was the SRE-on-call when the payments rails service began producing a 0.4 per cent error rate on a specific merchant onboarding endpoint. The agent detected the rate increase against the Datadog alert pattern that the policy file flagged as a clear telemetry signature, classified the incident as a Tier 0 rollback, executed the rollback automatically, and posted the incident summary to the on-call Slack channel within 90 seconds of the alert firing. Nair received the PagerDuty notification, read the agent's reasoning trace in LangSmith, confirmed the rollback was appropriate, and closed the incident within 11 minutes. The previous-quarter equivalent incident — a similar error rate on the same endpoint in October 2025 — had taken 47 minutes to resolve and required Nair to manually execute the rollback after diagnosing the cause. The compression of the operational loop is the productivity gain that the rebuild was designed to produce.

A more instructive named SRE interaction occurred in early March 2026, during a planned deployment of a schema migration to the merchant database. The agent classified the migration as Tier 1 and requested human confirmation. Marisol Pereira, who was on-call at the time, reviewed the agent's deployment plan and the reasoning trace, and approved the migration. The migration executed cleanly through the canary phase but produced an unexpected latency spike on a downstream reporting service during the 25 per cent ramp. The agent detected the latency spike but did not classify it as a rollback trigger because the latency was within the documented acceptance range. Pereira disagreed with the agent's classification and manually paused the deployment using the PagerDuty kill switch. The subsequent investigation found that the latency spike was the leading indicator of a more severe issue that would have surfaced at 100 per cent ramp, and Pereira's judgement saved the team a production incident that would have required a full rollback. The post-incident review classified the agent's behaviour as correct-by-policy but incorrect-in-practice, and the resolution was a policy file update that tightened the latency acceptance range for the affected service.

The third named interaction is the one Lim references when explaining the architecture to engineering leaders at other startups. In late February 2026, a third-party dependency that Ledgerline uses in its merchant onboarding flow released a security patch. The agent detected the patch, generated a pull request to update the dependency, ran the test suite against the update, and submitted the PR for code review. The PR was reviewed by a senior backend engineer, approved, merged, and deployed through the same automated pipeline. The total elapsed time from upstream patch release to Ledgerline production deployment was 4 hours and 17 minutes — a compression from the team's previous-quarter average of 6 days for similar dependency updates. The interaction illustrates the productivity gain the architecture produces at the routine end of the deployment workload, and the team's reading is that this routine-end compression is what produced the larger share of the headline velocity numbers. The exceptional cases — the Pereira intervention, the Nair rollback — get the attention. The routine cases produce the throughput.

What did not work, and what the team had to redesign

The failure modes the team encountered during the rebuild are the most useful part of the field study, because they are the modes that the publicly available reference architectures tend to understate. The first failure mode was a context-window exhaustion problem that surfaced in early February 2026. The Claude Code agent operates against the full repository context for each deployment, which on Ledgerline's largest service — the payments rails service, with approximately 180,000 lines of code — pushed against the model's context window. The agent's behaviour at the boundary was to silently truncate context rather than to surface the truncation as an error, and the truncation produced deployment plans that omitted downstream services the agent had not seen. Lim's team's resolution was to add a context budget check at the pre-deployment boundary that fails the deployment if the budget would be exceeded, and to refactor the largest services into modules with explicit dependency manifests that the agent could load selectively. The refactor took six engineering weeks and was the single largest operational cost of the rebuild.

The second failure mode was a tool-call loop that the agent entered when it encountered a flaky integration test. The agent's default behaviour on a test failure was to investigate the cause, and when the test was flaky the investigation produced an inconclusive result that the agent's policy interpretation treated as grounds for further investigation. Several deployments in mid-February ran for over 40 minutes in tool-call loops before timing out. Lim's resolution was a deterministic loop detector encoded in the LangGraph workflow: the workflow tracks the number of tool calls per investigation phase, and after 12 calls without a deterministic conclusion it escalates to a human review rather than continuing to investigate. The detector has been triggered nine times in the two months since deployment, and in each case the human reviewer was able to resolve the underlying issue — flaky test, network blip, dependency cache miss — in under 15 minutes. The team's reading is that the loop detector preserves the agent's autonomy on the cases where investigation produces a deterministic result while bounding the cost when it does not.

The third failure mode was more interpretive. In late February the engineering team began noticing that the agent's deployment plans were becoming more conservative — recommending Tier 1 escalations for changes that the policy file did not require them on. The conservative drift was traced to a feedback loop in the LangSmith trace data: the agent was reading recent traces as part of its context, and the traces included the policy-tightening updates the team had made after the Pereira incident. The agent interpreted the recent updates as evidence of a more conservative operational posture and biased its own recommendations accordingly. Lim's resolution was to remove the recent trace data from the agent's context and to provide a stable policy summary as the input instead. The drift behaviour stopped within three deployments. The team's reading is that the agent's context sensitivity is a feature when the team wants it to learn from recent operational reality, but the feedback loop has to be designed deliberately rather than allowed to emerge.

The fourth failure mode is the one the team has not fully resolved. The Claude Code agent's reasoning on cross-service incidents — incidents that originate in one service and cascade to others — has been weaker than its reasoning on single-service incidents. The agent tends to focus on the service where the alert fired and to underweight the upstream service that may have caused the cascade. The team has worked around this by adding explicit cross-service dependency mapping into the policy file and by instructing the agent in the deployment plan template to consider upstream services for any incident that touches the payments rails or merchant database services. The workaround has reduced the frequency of cross-service-misdiagnosis incidents from approximately one per week in February to one per month in April, but the team's view is that the underlying capability gap will require either a model improvement from Anthropic or a more sophisticated multi-agent architecture that runs separate agents for each service domain. Lim has flagged the multi-agent option as a 2026 H2 investigation. The capability gap has not blocked the rebuild's overall velocity gain, but it is the most significant outstanding risk to the architecture's continued operational success.

The numbers that justified the rebuild

The headline metrics are the metrics the engineering leadership team monitors monthly at the operational review. Deployment frequency moved from 7 per week in December 2025 to 31 per week in April 2026 — a 4.4× increase. Mean-time-to-recovery on production incidents fell from 73 minutes to 19 minutes — a 74 per cent reduction. The change failure rate, measured as the percentage of deployments that triggered a rollback or required a follow-up correction within 48 hours, fell from 17 per cent to 6 per cent. The on-call hours per engineer per month fell from 24 to 11, which has produced a measurable improvement in the engineering team's recovery scores on the quarterly engineering satisfaction survey. The DORA metrics suite as a whole moved from a profile the engineering leadership classified as "medium performer" in December to a profile that approaches "elite performer" in April, by the DORA team's published benchmarks. The improvement is the operational backdrop the architectural choices have to be evaluated against.

The cost numbers complicate the productivity story in ways the headline metrics obscure. The monthly Anthropic API spend on the agent-driven pipeline ran at $42,000 in April 2026, compared with the previous-pipeline DevOps tooling spend of $28,000 per month in December 2025. The headline cost increase is $14,000 per month. The cost saving from reduced SRE hires the team would otherwise have made is more difficult to compute precisely, but the engineering leadership team's working estimate is that the rebuild avoided 2.5 SRE hires that the previous trajectory would have required by Q2 2026. At Ledgerline's Singapore base salary for a senior SRE — approximately $14,500 per month including employer-side costs — the avoided-hire saving is approximately $36,000 per month. The net economic benefit at the current operational level is approximately $22,000 per month. The economic case improves as the pipeline scales: the marginal cost of additional deployments through the agent-driven pipeline is meaningfully lower than the marginal cost would be on the previous-architecture pipeline. The CFO's projection at the April 2026 board meeting was that the pipeline would deliver $1.4M in cumulative economic benefit over the 18 months following deployment.

The cost projection assumes a stable Anthropic API pricing trajectory, which is the projection's most significant uncertainty. Lim and Kannan have both flagged that a meaningful pricing increase from Anthropic — or an alternative model provider producing a Claude-Code-equivalent product at a meaningfully different price point — would change the architecture's economics. The team has built the architecture to be reasonably model-agnostic at the LangGraph layer: the Claude Code agent could in principle be swapped for an alternative agent runtime if the economics required it. The substitution would not be costless — the policy file enforcement, the trace observability, and the named workflow integrations have Claude-Code-specific implementations — but the architecture is not so deeply Claude-Code-coupled that a substitution would require a rebuild. The team's reading is that this optionality is the structural insurance against model-provider lock-in that the architecture's economics depend on.

The qualitative outcomes are the outcomes the engineering team weights most heavily when explaining the rebuild's value to other engineering leaders. The on-call rotation no longer produces the weekly burnout reports that triggered the rebuild in the first place. The engineering team's quarterly satisfaction survey moved from a 6.2 average on the question "I feel that my on-call work is sustainable" to a 8.4 average over the four months following deployment. The leadership team's read on the qualitative outcomes is that they matter as much as the headline DORA metrics, because they are the leading indicators of retention and the leading indicators of the engineering team's ability to take on more product surface in the next twelve months. The architectural rebuild is, in the leadership team's framing, an investment in engineering capacity that the cost numbers underweight. The capacity is not yet fully consumed by new product work, but the leadership team's plan for the second half of 2026 anticipates a meaningful increase in product surface that the pre-rebuild engineering organisation could not have supported.

What to watch

Ledgerline's pipeline has been in operational use for four months at the time of this writeup, and the architecture's first major stress test will arrive in Q3 2026 when the company plans to expand its regulatory footprint into two new APAC jurisdictions. The expansion will introduce new policy constraints, new compliance reviewers in the pipeline's escalation path, and a substantially larger surface area of changes that touch the regulatory-reporting subsystem.

• Whether the policy file enforcement scales to the new jurisdictional constraints without becoming a maintenance burden that erodes the team's velocity gain; the present policy files run to approximately 380 lines per service and the team has not yet measured the maintenance overhead at the next scale tier.
• Whether the cross-service incident diagnostic gap that the team has worked around with policy-level dependency mapping is closed by an Anthropic model release or whether Lim's team will need to invest in the multi-agent architecture that has been flagged as the alternative resolution.
• Whether the Anthropic API pricing trajectory remains stable enough for the architecture's economic case to hold over the next eighteen months, or whether a pricing event triggers the model-substitution exercise that the team has architected for but not executed.
• Whether the engineering team's qualitative satisfaction metrics — the on-call sustainability survey, the engineering recovery scores — hold up under the additional load that the Q3 expansion will introduce; the architectural rebuild has produced capacity, and the question for the leadership team is whether the capacity is preserved or consumed as the product surface grows.
• Whether other Series B and Series C startups in the APAC payments segment adopt the architecture pattern Ledgerline has documented; Lim has presented the architecture at two regional engineering conferences in April 2026, and the engineering leadership teams at three competing startups have circulated preliminary plans for similar rebuilds.

Frequently asked

How does the tier classification system work, and what stops the agent from misclassifying a change?

The tier classification is defined in a policy file checked into each service repository, with quantitative thresholds for every classification criterion: confidence intervals as numerical probabilities, complexity boundaries as graph distances in the dependency mapping, telemetry signatures as named alert patterns from Datadog. The agent reads the policy at the start of each deployment and the classification logic runs through a Claude Code hook at the pre-tool-call boundary. The hook is versioned in the repository and reviewed in code review like any other change. The on-call engineer can override the agent's classification at any time using the PagerDuty kill switch, which freezes the pipeline immediately and notifies the leadership team. The override has been triggered twice in four months, both times by the SRE-on-call lead Rajat Nair, and both overrides were judged appropriate in the post-incident review.

Why did Ledgerline choose Claude Code rather than a custom agent runtime or an alternative provider?

The decision was driven by the integration surface, not the model quality. Claude Code provides hooks, a tool-call protocol, and a session model that fit cleanly into a CI/CD pipeline; alternative agents at the time of the rebuild required substantially more custom integration work. The team explicitly designed the architecture to be model-agnostic at the LangGraph layer so that a model substitution would be feasible if the economics required it. The agent could in principle be swapped for an alternative runtime, though the substitution would not be costless because the policy file enforcement, trace observability, and named workflow integrations have Claude-Code-specific implementations. The optionality is the structural insurance against model-provider lock-in.

What is the cost-benefit calculation, and how does it hold up under different assumptions?

The April 2026 monthly Anthropic API spend was $42,000 versus $28,000 in previous-pipeline DevOps tooling spend in December 2025 — a headline increase of $14,000 per month. The avoided-hire saving from 2.5 SRE positions the previous trajectory would have required is approximately $36,000 per month. The net monthly benefit at the current operational level is approximately $22,000. The CFO's projection at the April 2026 board meeting was $1.4M in cumulative economic benefit over 18 months. The projection's most significant uncertainty is the Anthropic API pricing trajectory; a meaningful price increase would compress the net benefit and accelerate the model-substitution exercise the team has architected for but not executed.

What is the context-window exhaustion problem, and how did the team resolve it?

The Claude Code agent operates against the full repository context for each deployment. On Ledgerline's largest service — the payments rails service at approximately 180,000 lines of code — the context approached the model's window limit, and the agent's truncation behaviour produced deployment plans that omitted downstream services. The team added a context budget check at the pre-deployment boundary that fails the deployment if the budget would be exceeded, and refactored the largest services into modules with explicit dependency manifests that the agent could load selectively. The refactor took six engineering weeks and was the largest operational cost of the rebuild. The resolution has held in operational practice; no production deployment in the four months since has hit a context exhaustion failure.

What is the cross-service incident diagnostic gap, and why has the team not fully resolved it?

The Claude Code agent's reasoning on cross-service incidents — incidents that originate in one service and cascade to others — has been weaker than its reasoning on single-service incidents. The agent tends to focus on the service where the alert fired and to underweight the upstream service that may have caused the cascade. The team has worked around this by adding explicit cross-service dependency mapping into the policy file and instructing the agent in the deployment plan template to consider upstream services for incidents that touch the payments rails or merchant database services. The workaround has reduced cross-service-misdiagnosis incidents from approximately one per week in February to one per month in April. The underlying capability gap will require either a model improvement from Anthropic or a multi-agent architecture that runs separate agents per service domain. Lim has flagged the multi-agent option as a 2026 H2 investigation.

How does Ledgerline's pipeline compare to the Stripe internal reference the team started from?

Stripe's pipeline runs in a custom orchestration runtime; Ledgerline's runs in GitHub Actions with LangGraph above it. Stripe uses a custom model gateway; Ledgerline calls Claude Code directly. Stripe's pipeline is governed by an internal policy engine running against a proprietary risk schema; Ledgerline's runs against a policy file checked into the same repository the pipeline deploys, with enforcement through Claude Code hooks at the pre-tool-call boundary. The structural differences reflect the scale and maturity differential between Stripe's engineering organisation and a Series B startup. Stripe's investment in custom infrastructure is rational at Stripe's scale; Ledgerline's choice to use commodity tooling (GitHub Actions, LangGraph, LangSmith) with thin custom integration is rational at Ledgerline's scale. The architectural pattern transfers between the two; the implementation does not.