Distributed transactions are one of those problems that look simple until you split your monolith into services. In a single database, a transaction gives you ACID guarantees for free — either everything commits or nothing does. The moment your order service writes to its own PostgreSQL instance while the payment service talks to a separate database, those guarantees evaporate. You can’t BEGIN a transaction that spans two connection pools.
The Saga pattern is the most widely adopted solution. Instead of one big transaction, you break the operation into a sequence of local transactions — each scoped to a single service. Every step has a corresponding compensation: if step three fails, you undo steps one and two in reverse order. It’s eventually consistent instead of immediately consistent, and that tradeoff shapes every design decision you’ll make.
Let’s walk through how sagas work, the two coordination strategies, and what it takes to implement them correctly in Go.
The Core Problem: Why Two-Phase Commit Doesn’t Scale
Two-phase commit (2PC) is the textbook answer to distributed transactions. A coordinator asks all participants to prepare, then asks them all to commit. It works — but it holds locks across databases for the entire duration, which means latency, blocked writes, and catastrophic stalls if any participant becomes unresponsive. In microservices running on Kubernetes where pods restart and networks partition, 2PC is a liability.
Sagas abandon distributed locking entirely. Each step is a short local transaction. If the saga fails midway, compensating transactions roll back the work that already committed. The system reaches consistency eventually, not atomically.
Two Ways to Coordinate: Orchestration vs Choreography
Every saga needs someone to decide what happens next. There are two approaches, and the choice has real architectural consequences.
Choreography has no central coordinator. Each service publishes events and reacts to events from others. It’s loosely coupled and works well for simple flows — two or three steps with no branching logic. But as the number of steps grows, the flow becomes implicit and hard to trace. Debugging a failed saga means reconstructing the chain of events across multiple service logs.
Orchestration uses a dedicated component — the orchestrator — that calls each service and handles failures explicitly. The saga’s logic lives in one place, which makes it readable, testable, and observable. The tradeoff is tighter coupling: the orchestrator knows about every service involved. For most real-world flows with conditional logic, parallel steps, or complex compensation, orchestration is the pragmatic choice.
Building an Orchestrated Saga in Go
Let’s model an e-commerce checkout saga with three steps: reserve inventory, process payment, and create the shipment. Each step needs a compensating action.
package saga
import (
"context"
"fmt"
)
// Step represents one local transaction with its compensation.
type Step struct {
Name string
Execute func(ctx context.Context) error
Compensate func(ctx context.Context) error
}
// Orchestrator runs a sequence of steps and compensates on failure.
type Orchestrator struct {
steps []Step
}
func New(steps ...Step) *Orchestrator {
return &Orchestrator{steps: steps}
}
// Run executes steps in order. If a step fails, it compensates
// all completed steps in reverse order.
func (o *Orchestrator) Run(ctx context.Context) error {
completed := make([]int, 0, len(o.steps))
for i, step := range o.steps {
if err := step.Execute(ctx); err != nil {
// Compensate completed steps in reverse order.
for j := len(completed) - 1; j >= 0; j-- {
idx := completed[j]
if cerr := o.steps[idx].Compensate(ctx); cerr != nil {
return fmt.Errorf(
"step %q failed (%v), compensation of %q also failed: %w",
step.Name, err, o.steps[idx].Name, cerr,
)
}
}
return fmt.Errorf("step %q failed: %w", step.Name, err)
}
completed = append(completed, i)
}
return nil
}The orchestrator is generic and testable in isolation. Now let’s wire it to real services:
type OrderRequest struct {
OrderID string
CustomerID string
Items []string
TotalCents int
}
func ProcessOrder(
ctx context.Context,
inventory InventoryService,
payment PaymentService,
shipping ShippingService,
req OrderRequest,
) error {
saga := saga.New(
saga.Step{
Name: "reserve-inventory",
Execute: func(ctx context.Context) error {
return inventory.Reserve(ctx, req.OrderID, req.Items)
},
Compensate: func(ctx context.Context) error {
return inventory.Release(ctx, req.OrderID, req.Items)
},
},
saga.Step{
Name: "charge-payment",
Execute: func(ctx context.Context) error {
return payment.Charge(ctx, req.OrderID, req.CustomerID, req.TotalCents)
},
Compensate: func(ctx context.Context) error {
return payment.Refund(ctx, req.OrderID, req.TotalCents)
},
},
saga.Step{
Name: "create-shipment",
Execute: func(ctx context.Context) error {
return shipping.Create(ctx, req.OrderID, req.CustomerID)
},
Compensate: func(ctx context.Context) error {
return shipping.Cancel(ctx, req.OrderID)
},
},
)
return saga.Run(ctx)
}Each service interface is a simple contract — the implementations can call gRPC, HTTP, or a message queue behind the scenes. If charge-payment fails, the orchestrator automatically releases the inventory reservation. If create-shipment fails, it refunds the payment and releases inventory.
The Hard Part: Compensations Aren’t Undo
A common misconception is that compensation reverses a step perfectly. It doesn’t. If you charged a customer’s credit card, the compensation is a refund — but the customer still sees the charge and refund on their statement. If you sent an email, you can’t unsend it. Compensations are semantic rollbacks, not transactional ones.
This means every compensation needs to be idempotent. Network retries can deliver the same request twice, and compensation may be called even if the original step only partially completed. Design every compensating action to handle “already done” gracefully:
// Release is idempotent: releasing already-released items is a no-op.
func (s *InventoryService) Release(ctx context.Context, orderID string, items []string) error {
reservation, err := s.store.Get(ctx, orderID)
if err != nil {
if errors.Is(err, ErrNotFound) {
return nil // nothing to release
}
return err
}
if reservation.Status == "released" {
return nil
}
reservation.Status = "released"
return s.store.Save(ctx, reservation)
}Durability: Surviving Crashes
The in-memory orchestrator above works for simple cases, but it has a critical gap: if the process crashes mid-saga, you lose all state. On restart, there’s no way to know which steps completed or which compensations are pending.
Production sagas need persistent state. Before executing each step, write the saga’s current position to a durable store. After each step completes, update the status. This is sometimes called a saga log or execution journal. When the process restarts, it reads the journal and resumes from the last known position — or triggers compensation if the saga was in a failing state.
This is where durable execution platforms become valuable. Tools like Temporal and Cadence handle saga state persistence, retries, and compensation orchestration as platform features. You write the saga as ordinary sequential code, and the platform ensures it survives crashes and resumes correctly.
Isolation: Dealing with Intermediate State
Because sagas don’t hold locks, other requests can observe partial state. A customer might see their payment charged before the shipment exists. This is the lack of isolation problem, and it’s the most common source of bugs in saga-based systems.
Three techniques help manage this:
- Semantic locks: Mark in-progress records with a status like
"pending"and reject reads or modifications of pending records from other operations. - Commutative operations: Design steps so that order doesn’t matter. Reserving inventory and charging payment can happen in either order if neither depends on the other’s result.
- Reordering: Put the step most likely to fail first. If payment authorization is the riskiest part, do it before reserving inventory to minimize compensation churn.
When to Use Sagas (and When Not To)
Sagas add complexity — compensations, eventual consistency, and a new failure mode to reason about. Use them when you genuinely need transactions across service boundaries and can tolerate eventual consistency. If your services share a database, a database-per-service boundary isn’t required, and a local transaction is simpler and safer.
For read-heavy flows that don’t need atomic writes, consider whether a transaction is needed at all. An event-driven approach with CQRS can sometimes sidestep the distributed transaction problem entirely by accepting that projections catch up asynchronously.
Key Takeaways
- Sagas replace distributed locking with a sequence of local transactions plus compensations.
- Orchestration centralizes logic and is easier to reason about for complex flows; choreography works for simple event chains.
- Every compensation must be idempotent and must handle partial completion.
- Persist saga state to survive crashes — or use a durable execution platform like Temporal.
- Address isolation explicitly through semantic locks, commutative steps, or strategic ordering.
The saga pattern isn’t a silver bullet, but it’s the most reliable way to coordinate work across autonomous services. Start with the orchestration model, make your compensations bulletproof, and invest in observability early — knowing exactly where a saga failed is half the battle in keeping distributed systems reliable.