Cron at Scale in Fintech: Architecture Decisions, Failure Modes, and the Engineering Discipline Behind 1M+ Daily Financial Transactions

Author: Kuldeep Singh — Senior Engineer & Solutions Architect · AWS SAA-C03 · 7.2 YoE
System: Django 4.1 · Python 3.10 · PostgreSQL 14 · Redis · AWS (EC2, RDS, S3, SSM, Lambda)
Scale: 1M+ transactions/day · 30+ Django apps · 40+ scheduled jobs · $2M+ daily GMV · RBI PA Guidelines compliance

Most scaling discussions start with the wrong question: how do we make this faster?

In payment infrastructure, the right question is: how do we guarantee correctness under adversarial conditions — partial failures, concurrent execution, resource exhaustion, and regulatory deadlines — without sacrificing the operational simplicity that lets a small team own a large system?

This post documents the engineering discipline behind that answer. It is written for engineers who have already scaled systems and want the architectural reasoning, not the syntax.

The Problem Space: Why Financial Crons Are a Different Class of Problem

The system architecture is a deliberate monolith: 30 Django apps in a single deployable unit, shared ORM, shared connection pool, shared base models. This is not a failure of architecture — it is the architecture.

The choice optimizes for transactional consistency across integration boundaries. In a payment system that bridges multiple banking partners, payment gateways, and regulatory reporting surfaces, the cost of distributed transaction coordination (two-phase commit, saga orchestration, compensating transactions) exceeds the cost of vertical scaling and careful cron design. This is a CAP theorem trade-off made explicitly: we choose CP over AP at the data layer.

The cron jobs are not background work. They are the payment lifecycle:

Payment initiation
    → PG callback processing
    → Status reconciliation     ← cron
    → Settlement                ← cron
    → Scroll / report generation ← cron
    → Partner webhook delivery  ← cron
    → Regulatory reporting      ← cron

A failed status-check cron means transactions stay PENDING indefinitely. A failed settlement cron is an RBI PA Guidelines violation (T+1 mandate). A double-execution of the settlement cron is real money moved twice, requiring manual bank reconciliation and potentially a regulatory incident report.

This changes the engineering contract entirely. The optimization target is not throughput. It is exactly-once, bounded-time, auditable execution with zero tolerance for financial state corruption.

System Invariants That Drive Every Decision

Before solutions, enumerate the constraints:

1. Exactly-once financial mutations. Settlement, refund, and remittance operations must execute against the payment gateway exactly once per transaction, regardless of retry behavior, concurrent execution, or partial failure.
2. Bounded processing windows. Settlement has a regulatory deadline (T+1). Every cron operates within a defined temporal contract.
3. Partial failure must not cascade. If 2,000 of 10,000 settlement calls fail, the 8,000 successes must be committed. The 2,000 must be isolable, retryable, and reportable — without reprocessing the 8,000.
4. The shared connection pool is a shared resource. A cron that saturates max_connections degrades every API endpoint simultaneously.
5. Audit trail is a first-class requirement. Every state transition must be attributable, timestamped, and reconstructible for regulatory examination.

Every architectural decision in this post traces back to one or more of these invariants.

Failure Taxonomy: Five Categories at Production Scale

Category 1: Memory Exhaustion — The Django ORM Trap

Failure mode: OOM killer terminates cron process mid-iteration. No stack trace. No partial commit. Process vanishes.

Root cause: Django's QuerySet evaluation model is lazy — .filter() returns a query descriptor. But iteration via for txn in queryset: materializes the entire result set into Python heap in a single database round-trip. At scale:

100K rows × (800B model overhead + 2KB JSONField payload) = ~280MB baseline
+ downstream list accumulation of result objects           = ~400MB additional
+ GC pressure from circular FK references                  = unpredictable spikes

Peak measured: 2.1GB on a 2GB container. The OOM kill is not a misconfigured limit — it is the ORM's impedance mismatch between OLTP request-response assumptions and large-set batch iteration.

Solution — Server-Side Cursors:

queryset = Transaction.objects.filter(
    status__in=["PENDING", "INITIATED"],
    created_at__gte=timezone.now() - timedelta(days=7),
).order_by("id")

for txn in queryset.iterator(chunk_size=500):
    txn_id, success = self._process_single(txn)
    if not success:
        failure_ids.append(txn_id)  # IDs only — not model references

.iterator() opens a PostgreSQL server-side cursor (DECLARE ... CURSOR FOR SELECT ...) and fetches in bounded pages. Memory profile becomes O(chunk_size) rather than O(result_set_size).

chunk_size=500 is empirically derived: 500 rows × ~3KB average → 1.5MB per fetch, within L2 cache on c6i.xlarge. PostgreSQL cursor round-trip overhead is amortized over 500 rows.

.order_by("id") is required, not stylistic. Without a stable sort, PostgreSQL may reorder results across cursor fetches when concurrent INSERTs occur — causing rows to be skipped or processed twice. id is monotonically increasing with a primary key index: zero sort overhead, guaranteed stability.

Alternative rejected — offset/limit pagination: OFFSET N causes PostgreSQL to scan and discard N rows per page — O(N²) across full iteration. Server-side cursors maintain cursor position in the query plan executor.

Category 2: Query Planner Pathology — The 365M-Row Problem

Failure mode: Settlement cron escalates from 3 minutes to 45 minutes. pg_stat_activity shows Seq Scan on the primary transaction table. Shared connection pool exhausts. API latency spikes to 30+ seconds system-wide.

Root cause — three compounding factors:

1. Table bloat. 1M INSERTs + 2M UPDATEs/day on the transaction table. Default autovacuum threshold (20% dead tuples) triggered only after ~73M dead tuples accumulated. Dead tuples degrade planner row-count estimates, making sequential scans appear cheaper.

2. Stale statistics. ANALYZE runs on autovacuum's schedule. Between vacuum cycles, pg_statistic entries drift — selectivity estimates for status = 'SUCCESS' no longer reflected the current data distribution.

3. Index size vs. cache pressure. A full B-tree on (status, is_settlement_raised) over 365M rows is ~8GB. Cache pressure evicted index pages, causing the planner to estimate random I/O costs higher than sequential scan cost.

The planner was technically correct given its information. The information was wrong.

Solution — Partial Indexes as Working-Set Optimization:

CREATE INDEX CONCURRENTLY idx_settlement_pending
ON transaction_info (created_at)
WHERE status = 'SUCCESS' AND is_settlement_raised = FALSE;

A partial index stores only rows satisfying the WHERE predicate. The unsettled success working set is ~50K rows — index size: ~2MB, permanently resident in shared_buffers. Planner selectivity is exact because the predicate matches the index definition.

Application-level temporal bounding:

settlement_window_start = timezone.now() - timedelta(hours=26)  # T+1 + 2hr buffer
settlement_window_end   = timezone.now() - timedelta(minutes=5) # Grace period for in-flight writes

The 5-minute grace period is not arbitrary. Post-SUCCESS write fan-out (webhook delivery records, callback acknowledgments, partner notification state) has a measured P99 of ~4 minutes. Settling before this completes creates reconciliation mismatches — the settlement is correct, but the audit trail shows an inconsistent view.

Autovacuum tuning:

ALTER TABLE transaction_info SET (
    autovacuum_vacuum_scale_factor  = 0.01,  -- vacuum at 1% dead tuples vs 20% default
    autovacuum_analyze_scale_factor = 0.005, -- analyze at 0.5% change
    autovacuum_vacuum_cost_delay    = 2      -- run faster (vs 20ms default)
);

At 2M updates/day on 365M rows, default autovacuum triggers after 73M dead tuples (~36 days). The tuned thresholds trigger after ~3.65M dead tuples (~1.8 days). Dead tuples stay under 100K. The sequential scan pathology becomes structurally impossible.

Outcome: Settlement working set query: 365M row table → 800ms, stable regardless of table growth.

Category 3: Self-Induced Amplification — N+1 and Credential Cascades

Two amplification patterns. Same root cause: per-item remote operations in a tight loop without amortization.

Pattern A — N+1 database queries:

for txn in queryset.iterator(chunk_size=500):
    merchant_key  = txn.customer.merchant.key   # SELECT FROM merchant
    provider_name = txn.provider.name           # SELECT FROM provider

100K iterations × 2 FK traversals = 200K SELECT statements. At 2ms average RTT = 400 seconds of pure I/O wait before any application logic executes.

# Fix: eager load the full relation graph in a single JOIN query
qs = Transaction.objects.select_related(
    "customer",
    "customer__merchant",
    "customer__merchant__vendor",
    "provider",
).filter(...).order_by("id")

Explicit relation graph — not select_related() without arguments — prevents unintended join graph expansion as the data model evolves.

Pattern B — per-transaction credential fetches:

for txn in queryset.iterator(chunk_size=500):
    response = get_merchant_credentials(merchant_key)  # HTTP call per transaction
    salt = response.get("salt", "")

At 100K transactions for a single merchant key: 100K identical HTTP calls returning the same immutable value. The internal credential service, designed for ~100 calls/minute, received 100K calls in 3 minutes — tripping circuit breakers and taking down all 30 integration apps simultaneously.

Fix — process-scoped memoization:

class SettlementService:
    _credential_cache: dict = {}  # Class-level: scoped to process lifecycle

    def _get_credentials(self, merchant_key: str) -> dict:
        if merchant_key not in self.__class__._credential_cache:
            self.__class__._credential_cache[merchant_key] = (
                get_merchant_credentials(merchant_key)
            )
        return self.__class__._credential_cache[merchant_key]

Why class-level dict and not Redis?

The cache is bounded by process lifetime. Correct, simple, no external dependency in the hot path.

Outcome: Credential service calls: 1M+/day → <100/day. Cascading internal service failures: eliminated.

Category 4: Concurrency Hazards — The Exactly-Once Problem

This is where financial systems diverge from general web applications. The failure mode is not degraded UX — it is money moved incorrectly.

Incident: Double Settlement

Systemd timer drift + upstream API latency caused two settlement cron instances to run concurrently. Both queried the same is_settlement_raised=FALSE working set. Both called the gateway settlement API. Merchant received 2× the expected amount.

Defense-in-depth for exactly-once execution:

No single control is sufficient. The correct model is layered controls where each layer independently prevents the failure and collectively makes it structurally impossible.

Layer 1 — Process-level mutual exclusion:

LOCK_FILE = f"/tmp/{self.app_name}_settlement.lock"

def _acquire_lock(self) -> bool:
    if os.path.exists(LOCK_FILE):
        try:
            with open(LOCK_FILE, "r") as f:
                pid = int(f.read().strip())
            os.kill(pid, 0)  # signal(0) — liveness probe, no signal sent
            return False     # Live process: lock valid
        except (ValueError, OSError):
            os.remove(LOCK_FILE)  # Dead process: stale lock
    with open(LOCK_FILE, "w") as f:
        f.write(str(os.getpid()))
    return True

PID-based over flock(2): fcntl.flock() with LOCK_EX | LOCK_NB fails silently on NFS-mounted /tmp (common on certain EC2 AMIs). PID-based is filesystem-agnostic. os.kill(pid, 0) distinguishes live process (lock valid) from crashed process (stale lock) without sending any signal.

Layer 2 — Database-level compare-and-swap:

# The WHERE predicate makes this a CAS at the database level
updated = Transaction.objects.filter(
    pk=txn.pk,
    is_settlement_raised=False,
).update(
    is_settlement_raised=True,
    settlement_raised_at=timezone.now(),
)

if updated == 0:
    logger.info("Concurrent settlement detected, skipping: %s", txn.internal_transaction_id)
    continue

PostgreSQL guarantees that across concurrent transactions, exactly one UPDATE will match is_settlement_raised=False and return updated=1. All others return updated=0. This invariant holds at all standard isolation levels for single-row UPDATE operations.

Layer 3 — Row-level pessimistic locking with skip:

with transaction.atomic():
    locked_txn = (
        Transaction.objects
        .select_for_update(skip_locked=True)
        .get(pk=txn.pk)
    )
    if locked_txn.status in TERMINAL_STATUSES:
        return

    locked_txn.status = computed_status
    locked_txn.save(update_fields=["status", "updated_at"])

skip_locked=True is the critical flag. Without it: concurrent workers serialize behind the lock, creating a queue that takes hours to drain at 100K transactions. With skip_locked: locked rows are skipped, picked up on the next cycle. This is the correct behavior — the row is in-progress, not failed.

Incident: Double Refund via Thread Pool

ThreadPoolExecutor(max_workers=10) distributed work from a pre-evaluated list. A partitioning bug caused two threads to receive the same transaction ID. Both called the refund API. Customer received 2× refund.

Fix — sequential processing for financial mutations:

REFUND_WORKERS     = 1   # Financial mutations: always sequential
STATUS_CHECK_WORKERS = 10  # Idempotent reads: safely parallelizable

At WORKERS=1 and 500ms average API latency: ~120 refunds/minute. Daily volume: ~500. Completion time: ~4 minutes. The 10× parallelization speedup is not a valid trade when the failure cost is manual bank reconciliation, customer escalation, and potential regulatory disclosure.

Engineering principle: throughput is not the optimization target for financial mutation operations.

Category 5: Operational Resilience — Graceful Degradation

SIGTERM mid-execution:

Kubernetes sends SIGTERM → waits terminationGracePeriodSeconds → sends SIGKILL. A cron mid-save() at SIGKILL produces partial state: committed to the gateway, not reflected in the database.

class Command(BaseCommand):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._shutdown = False

    def handle(self, *args, **options):
        signal.signal(signal.SIGTERM, self._request_shutdown)
        signal.signal(signal.SIGINT, self._request_shutdown)

        with ThreadPoolExecutor(max_workers=WORKERS) as executor:
            futures = {}
            for txn in queryset.iterator(chunk_size=BATCH_SIZE):
                if self._shutdown:
                    logger.warning("Draining %d in-flight tasks before exit", len(futures))
                    break  # Stop accepting new work — do NOT cancel in-flight
                future = executor.submit(self._process_single, txn)
                futures[future] = txn.internal_transaction_id

            for future in as_completed(futures):
                txn_id, result = future.result()
                self._record_result(txn_id, result)

    def _request_shutdown(self, signum, frame):
        self._shutdown = True

Contract: SIGTERM stops new work submission, allows in-flight work to complete, exits cleanly. terminationGracePeriodSeconds must exceed P99 of single-transaction processing time. Ours: 90 seconds (30s gateway timeout + overhead).

Connection staleness in long-running processes:

Django's connection management assumes short-lived request-response cycles. Crons running 30+ minutes with thread pools encounter server-side connection timeouts on idle connections.

from django.db import close_old_connections

def _process_single(self, txn: Transaction) -> tuple[str, bool]:
    try:
        close_old_connections()      # Prune stale before acquiring
        service = TransactionService()
        result = service.process(txn)
        return txn.internal_transaction_id, True
    except Exception:
        logger.error("%s\n%s", txn.internal_transaction_id, traceback.format_exc())
        return txn.internal_transaction_id, False
    finally:
        close_old_connections()      # Return to pool even on exception

Both try and finally serve distinct purposes: try prunes connections that timed out while the thread was idle in the pool queue; finally prevents connection leaks on exception paths that would exhaust max_connections over the cron's lifetime.

File Generation at Scale: The Stream-or-Die Principle

The anti-pattern that works in development and fails in production:

rows = [format_row(txn) for txn in queryset]   # Materializes entire queryset
content = "\n".join(rows)                        # O(n²) string allocation
f.write(content)

At 50K rows × 500 chars: join() allocates a 25MB contiguous string. 8 integrations generating files simultaneously at 2 AM settlement windows: 200MB+ of uncontrolled allocation.

Principle: write to the sink as you read from the source. Never buffer.

CSV — direct file descriptor write:

def generate_scroll(self, queryset: QuerySet) -> str:
    with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f:
        writer = csv.writer(f)
        writer.writerow(HEADER)
        running_total = Decimal("0")

        for txn in queryset.only(*REQUIRED_FIELDS).iterator(chunk_size=500):
            writer.writerow(self._format_row(txn))
            running_total += Decimal(str(txn.net_amount_debit or 0))

        writer.writerow(FOOTER + [str(running_total)])
    return f.name

.only(*REQUIRED_FIELDS) reduces per-row memory from ~3KB to ~200 bytes. Decimal for running totals is non-negotiable — floating-point summation over 50K financial values produces paise-level drift that fails reconciliation checks.

XLSX — openpyxl write-only mode:

wb = Workbook(write_only=True)  # O(1) memory regardless of row count
ws = wb.create_sheet("Settlement Report")
ws.append(HEADER_ROW)

for txn in queryset.iterator(chunk_size=500):
    ws.append([txn.internal_transaction_id, str(txn.net_amount_debit or 0), ...])

wb.save(output_path)
wb.close()

Normal openpyxl mode persists every cell as a Python object in memory (required for random access). Write-only mode serializes each row to the underlying ZIP stream immediately and discards it. This is not a micro-optimization — it is the difference between the file completing and the process being OOM-killed mid-generation.

XLSX generation: 180s · 450MB → 12s · 25MB (15× / 18× improvement)

Settlement Consistency: A Formal Model

Invariants under all conditions:

• Exactly-once: Each transaction is settled against the payment gateway exactly once.
• Bounded scope: Every settlement run has an explicit temporal window. No unbounded queries.
• Partial failure isolation: Gateway rejection of N transactions does not prevent settlement of M−N successes.
• Idempotency: Running settlement twice for the same window produces identical state.
• Audit completeness: Every settlement is linked to a gateway batch_id.

def settle_batch(self, mode_config: dict) -> None:
    window_start = self._compute_window_start()           # T+1 lower bound
    window_end   = timezone.now() - timedelta(minutes=5)  # Grace period

    qs = Transaction.objects.filter(
        status="SUCCESS",
        is_settlement_raised=False,
        created_at__gte=window_start,
        created_at__lte=window_end,
        **mode_config["additional_filters"],
    )

    if not qs.exists():
        return

    txn_ids          = list(qs.values_list("internal_transaction_id", flat=True))
    gateway_response = self._call_settlement_api(txn_ids, mode_config)

    if not gateway_response.get("success"):
        logger.error("Settlement API failure: %s", gateway_response)
        self._page_oncall(f"Settlement failed: {len(txn_ids)} transactions")
        return

    settled_ids = gateway_response.get("settled_transaction_ids", txn_ids)
    batch_id    = gateway_response["batch_id"]

    # Atomic bulk update with CAS guard — single query, not N queries
    committed = Transaction.objects.filter(
        internal_transaction_id__in=settled_ids,
        is_settlement_raised=False,   # CAS: second run commits 0 rows
    ).update(
        is_settlement_raised=True,
        settlement_raised_at=timezone.now(),
        settlement_batch_id=batch_id,
    )

    logger.info("batch=%s committed=%d/%d window=[%s, %s]",
                batch_id, committed, len(settled_ids), window_start, window_end)

    if committed != len(settled_ids):
        self._page_oncall(f"Settlement discrepancy: {len(settled_ids) - committed} rows")

Multi-mode settlement (UPI / Cards / Net Banking) runs each mode in isolated try/except blocks — UPI failure does not block Cards settlement:

for mode in SETTLEMENT_MODES:
    try:
        self.settle_batch(mode)
    except Exception:
        logger.error("Mode %s failed:\n%s", mode["name"], traceback.format_exc())
        continue

Observability: Structured Failure Recovery

Dead-letter file pattern:

def handle(self, *args, **options) -> None:
    success_ids, failure_ids = [], []

    for txn in queryset.iterator(chunk_size=BATCH_SIZE):
        txn_id, ok = self._process_single(txn)
        (success_ids if ok else failure_ids).append(txn_id)

    if failure_ids:
        path = Path(f"/tmp/{self.command_name}_failures_{now():%Y%m%d_%H%M%S}.txt")
        path.write_text("\n".join(failure_ids))
        logger.warning("Retry: python manage.py %s --retry-file %s", self.command_name, path)

    self._emit_summary_metrics(len(success_ids), len(failure_ids))

Targeted retry of failed IDs — not a full re-run — mirrors Kafka's dead-letter topic pattern at the application layer. No infrastructure dependency, fully debuggable, replayable with a single command.

Three-state result semantics for alert fidelity:

Binary True/False conflates "newly processed" with "already in correct state" — both emit the same notification, training engineers to treat alerts as noise. Ternary semantics solve this:

def _process_single(self, txn: Transaction) -> tuple[str, bool | None]:
    status = self._poll_gateway(txn)

    if status == "dropped":
        txn.refresh_from_db(fields=["status"])
        if txn.status == "FAILURE":
            return txn.id, None   # Already terminal — suppress
        txn.status = "FAILURE"
        txn.save(update_fields=["status", "transaction_complete_date"])
        return txn.id, True       # New transition — notify

    return txn.id, False          # No action taken

for future in as_completed(futures):
    txn_id, result = future.result()
    if result is True:    notify_list.append(txn_id)   # Newly transitioned
    elif result is False: failure_list.append(txn_id)  # Failed to process
    # None = idempotent re-discovery, silently skip

On-call engineers see only new state transitions. Alert precision is an operational excellence metric.

AWS Infrastructure: Isolation as Architecture

Compute separation:

API Tier (c6i.xlarge × 4, ALB)      Cron Tier (c6i.xlarge × 2, no LB)
         │                                       │
         └────────────────┬──────────────────────┘
                          │
                  PgBouncer (connection pooler)
                  API pool:   200 connections, transaction mode
                  Cron pool:   50 connections, session mode
                          │
                  RDS PostgreSQL 14
                  db.r6g.2xlarge · io2 Block Express · 10K IOPS

PgBouncer pool separation enforces resource isolation at the connection layer. Without it, 50 cron connections directly compete with 200 API connections against max_connections=500. Session mode for crons is required — select_for_update() and advisory locks require connection persistence across multiple statements within a transaction, which is incompatible with PgBouncer's transaction pooling mode.

Key PostgreSQL parameters:

shared_buffers           = 16GB   (25% of 64GB RAM)
work_mem                 = 256MB  (large sorts in cron queries; monitor per-process ceiling)
maintenance_work_mem     = 2GB    (VACUUM on 365M-row tables)
checkpoint_completion_target = 0.9   (spread checkpoint I/O)
wal_buffers              = 64MB   (reduce WAL write latency for bulk updates)

work_mem = 256MB deserves explicit justification. Cron queries frequently sort large working sets before cursor iteration. Each sort can use up to work_mem before spilling to disk. With 50 cron connections, theoretical max sort memory = 12.8GB — within the 64GB allocation accounting for shared_buffers and OS overhead. Insufficient work_mem produces sort spills visible as I/O spikes in pg_stat_activity.

S3 lifecycle for regulatory compliance:

Standard        0–30d     (active dispute window)
Standard-IA     30–90d    (audit requests)
Glacier Instant 90d–1yr   (compliance access)
Glacier Deep    1yr–7yr   (RBI 7-year retention mandate)
Expire          7yr+1d

RBI PA Guidelines mandate 7-year retention for payment records. The S3 lifecycle policy encodes this as infrastructure rather than operational procedure — no manual intervention required, no risk of accidental deletion within the retention window.

Business Impact

The double-settlement elimination is the highest-value outcome. Each incident: gateway reversal (2–3 day SLA), bank-level reconciliation, merchant communication, internal postmortem. At $2M+ daily GMV, a 0.01% settlement error on a single day is a $200 direct liability plus unbounded operational overhead. The architectural controls in Category 4 make this structurally impossible.

Design Principles for Financial Batch Systems

1. Exactly-once requires layered controls, not a single mechanism. Process mutex prevents concurrent execution. Database CAS prevents races that slip past the mutex. Row-level SELECT FOR UPDATE SKIP LOCKED prevents interleaving in multi-statement transactions. Each layer catches failures the previous layer misses.

2. Throughput is not the optimization target for financial mutation operations. Sequential processing of settlements and refunds is a feature. Size sequential throughput against daily volume, not theoretical maximum. The cost of a double-execution vastly exceeds the cost of lower throughput.

3. Bound every operation spatially and temporally. Unbounded queries are operational liabilities. A WHERE status = 'PENDING' without a time bound processes 3-year-old exception cases alongside today's transactions. Temporal bounds are correctness requirements, not optimizations.

4. Stream from source to sink. Never buffer. Any code path that accumulates results before writing is a memory time bomb at scale. Django's .iterator(), openpyxl's write_only mode, and csv's direct fd writes are not micro-optimizations — they are the difference between the system functioning and failing under load.

5. Amortize expensive operations over the batch, not per item. N+1 queries, per-transaction credential fetches, per-row remote API calls — all are instances of the same failure: expensive operations in tight loops without amortization. Fetch what you need for the batch before the loop, not inside it.

6. The audit trail is a first-class system requirement. Every state transition must be attributable, timestamped, and linked to the external event that caused it (gateway batch ID, reconciliation run ID). This serves regulators, auditors, and the on-call engineer reconstructing an incident at midnight.

7. Alert on state transitions, not state observations. Ternary result semantics (new / failed / already-correct) prevent alert fatigue. Binary semantics conflate newly processed with re-discovered, training engineers to ignore alerts. Alert precision is an operational excellence metric.

8. Isolate batch and serving at every resource layer. Compute, connection pooling, and I/O must be isolated. Batch workloads saturate resources for sustained periods; serving workloads spike and release. Mixing them creates correlated failure modes where a cron's resource consumption directly degrades user-facing latency.

Scale Roadmap: What Changes at 10M/Day

The architecture described here is right-sized for 1M/day. At 10× scale, four components require rethinking:

File locks → distributed locks (Redis Redlock): File-based mutex works on a single host. Horizontal scaling of cron workers across EC2 instances requires network-coordinated mutual exclusion. Trade-off: Redis becomes a dependency in the critical settlement path.

Polling → event-driven settlement: At 10M/day, polling for unsettled transactions produces significant read load even with partial indexes. Publishing TransactionSettlementEligible events at terminal status + grace period converts settlement query complexity from O(working set) to O(event throughput). Natural backpressure via consumer lag metrics.

Primary reads → read replica offload: Status reconciliation and report generation are reads against historical data — primary consistency not required. Routing these to a read replica reduces primary I/O by ~60%, preserving write throughput for settlement commits.

CSV/XLSX → Arrow/Parquet: At 500K+ rows, columnar formats with Apache Arrow for in-memory processing and Parquet for storage outperform row-oriented formats on generation time and storage cost. Arrow's vectorized aggregations (running totals, payment-mode breakdowns) are materially faster than Python-level Decimal loops. Not needed at 1M/day — but plan for it.

Closing

The patterns in this post are not novel. .iterator() is in Django's documentation. Partial indexes are in PostgreSQL's documentation. Defense-in-depth for exactly-once execution is in Kleppmann's book. What is underrepresented in engineering writing is the discipline of applying these patterns systematically in financial infrastructure — where failure modes are not degraded UX but incorrect money movement and regulatory non-compliance.

The engineering investment to get this right is measured in weeks. The cost of getting it wrong is measured in trust — with merchants, with regulators, and with the engineers who have to reconstruct a double-settlement at midnight.

Build the controls. Verify the invariants. Make the failures structurally impossible before they happen.

The most reliable financial system is not the one with the most sophisticated infrastructure — it is the one where the failure modes are fully enumerated, the controls are layered, and any engineer can reconstruct exactly what happened for any transaction at any time.

References

• Martin Kleppmann — Designing Data-Intensive Applications, Ch. 7 (Transactions) and Ch. 11 (Stream Processing)
• Pat Helland — Life Beyond Distributed Transactions: An Apostate's Opinion (2007)
• PostgreSQL Documentation — Server-Side Cursors · Partial Indexes
• Django Documentation — QuerySet.iterator() · select_for_update
• AWS — RDS PostgreSQL Best Practices
• Leslie Lamport — Time, Clocks, and the Ordering of Events in a Distributed System (1978)