Scale the work. Preserve the truth.

A spreadsheet upload can be accepted while its rows are still being validated. A document binary can be stored while extraction is pending. A payment event can be recorded while downstream inventory work remains unresolved. These are legitimate intermediate states.

Falcon's case study describes imports moving from a long browser request to durable asynchronous jobs. Progress polling and clearer failure details followed, including handling for empty spreadsheets, wholly failed batches and incomplete coordinates. The result is a job that can explain accepted and failed records instead of treating file upload as proof of finished import.

DocuBex separates original storage from extraction and previews. Its documented thumbnail work removed unsuitable in-request processing from a serverless path. That makes runtime selection a responsibility decision: the API accepts work, while a suitable worker performs operations needing different time, memory or binaries.

The first scale boundary is therefore semantic. Return an accepted job identity only after it is durable. Give the UI a status endpoint or event stream that reports the actual lifecycle. Do not rename “queued” to “complete” merely to make the interaction look fast.

A useful job record includes its target identity and version, current state, attempt history, progress meaning and final outcome. If the worker process disappears, those facts should remain inspectable.

Consider a booking transaction followed by a separate queue publish. If the booking commits and publication fails, the business change exists without its follow-up. Reversing the order creates the opposite risk: a worker may act on an event whose booking never commits.

Bookalto records the domain change and its internal event intent in the same database transaction. A worker later processes pending events into durable inbox items, with replay protection against duplicate visible records. Its current worker does not deliver customer appointment messages. An internal event is a foundation for delivery, not a receipt from a messaging provider.

This is the transactional outbox pattern. AWS's pattern guidance explains the dual-write problem and the need for idempotent consumers when delivery repeats. The atomic unit is business state plus the intent to do later work.

The worker still needs its own failure model. If it performs an effect and crashes before acknowledging the event, the event can return. Use a stable effect identity and an appropriate reconciliation path. For an internal inbox row, a unique event/recipient key can enforce one record. For an external provider, use supported idempotency or inspect the provider's recorded result.

Do not call the whole system “exactly once” because one table has a unique key. Different effects have different guarantees. A printed receipt, an email submission and a database insert do not expose the same acknowledgement semantics.

Suppose worker A starts extracting document version three. Its lease expires during a long pause. Worker B claims the work. A eventually resumes and finishes. Without a guarded publication step, the old worker can overwrite a newer result.

DocuBex's implementation checks job state before publishing derived data and distinguishes superseded work. The broader pattern is to make ownership visible in durable state. A renewable lease supports recovery after a crash; a changing ownership token lets the storage boundary reject an outdated owner.

The following SQL transaction fragment is illustrative, not a DocuBex migration. Assume each job targets an immutable document version, every new claim generates a unique lease token, and worker output is uploaded under a unique attempt key before this statement.

UPDATE processing_job
SET state = 'completed',
    result_object_key = $4,
    completed_at = clock_timestamp()
WHERE id = $1
  AND state = 'running'
  AND lease_token = $2
  AND document_version = $3
  AND lease_until > clock_timestamp()
RETURNING id, document_version, result_object_key;

Zero returned rows means the worker no longer has permission to publish this job result. It must not force a successful state. Its uniquely named output can be reconciled or cleaned up under an explicit retention policy. Do not let competing attempts write the same object key before the guard.

This statement does not by itself update a document's “current result” pointer safely. That requires a separate, transactionally guarded check of the document's current version, with the document and job locked in a consistent order. Alternatively, retain results by immutable version and resolve the correct version when reading.

The important boundary is the final write, not the worker's last local boolean check. Ownership can change between checking a lease and attempting to publish.

A queue can absorb a burst, but it cannot create CPU, database connections or provider quota. Increasing worker count without considering those limits can turn a manageable backlog into connection starvation and repeated timeouts.

Separate work by resource profile. Thumbnail rendering may be CPU or native-binary intensive. Provider extraction may spend most of its time waiting on a network. Spreadsheet imports may contend on database writes. Give each class bounded concurrency, payload limits and timeouts suited to its dependency.

BullMQ's stalled-job documentation explains that jobs can lose their lock when a worker fails to renew it, including when CPU-heavy work blocks the Node.js event loop. Separating intensive processing and monitoring stalled work matters even when the queue transport itself is healthy.

Apply backpressure before accepting unlimited work. An API can reject an oversized import, limit concurrent processing per tenant or postpone enrichment while retaining the original file. Make that status visible. A queue length rising forever behind a reassuring spinner is not graceful degradation.

Fairness also matters. One organization submitting a huge batch should not monopolize every worker required for small interactive jobs. Consider per-tenant concurrency, separate queues or weighted scheduling when workload evidence justifies them. Preserve tenant scope in job records and revalidate it when resolving resources.

Retries must be bounded and classified. Rate limits can use delayed retry. Malformed documents, unsupported formats and denied authorization need explicit failure or review. Automatic retry of a permanent error consumes capacity without increasing completion.

Queue depth alone is ambiguous. A hundred tiny jobs may clear faster than one large document. Track oldest runnable job age, time from acceptance to useful result, successful completions, retries, stale-owner rejections and provider latency by work type.

A simple planning model helps expose assumptions. If two workers each complete an illustrative six jobs per minute, nominal service capacity is twelve jobs per minute. A sustained arrival rate of fifteen grows the backlog by roughly three per minute. Real jobs vary, so this arithmetic is a teaching example, not a benchmark or sizing promise.

Break end-to-end time into queue wait, execution, provider wait and publication/reconciliation. A slow dashboard may be waiting for an expensive query, a worker, a saturated provider or stale read-model refresh. Adding web replicas helps only some of those causes.

Observe the database too: connection pool saturation, lock wait, transaction duration and query plans. Bookalto's conservative branch lock is an intentional correctness tradeoff. Optimizing it requires a measured workload and preserved invariants, not a blanket decision to “make everything asynchronous.”

Trace with operation ID, job ID, target version and attempt ID. That lets a support engineer follow one user action through delayed work without logging sensitive document contents or payment payloads. Measurements should explain a failure, not create a second data-exposure problem.

The most useful tests interrupt a workflow after its effects become inconvenient to undo. Commit an outbox event and stop the dispatcher. Perform a worker effect and lose the acknowledgement. Expire a lease while the original process is paused. Publish a new document version while an old worker is running.

Then inspect the business state. Is there one internal inbox item? Does the current document still point to the correct version? Can a failed import explain which rows need correction? Does a repeated provider event converge on one commerce outcome?

Add bounded load tests only after those recovery paths are correct. Use representative job sizes, provider behavior and database contention. Record the environment, workload, duration and failure rate with the result. A historical local suite or route count cannot stand in for production throughput evidence.

Our case studies show architecture that makes asynchronous work explicit and recoverable: DocuBex's job lifecycles, Bookalto's transactional internal events, Falcon's durable imports and Varqio's provider-event recovery. They do not establish a single cross-product scale benchmark.

The scaling principle is to increase independent progress while preserving ownership at the points where truth changes. Keep durable intent, isolate expensive work, reject stale publishers and measure the path from acceptance to a useful result. That is a stronger foundation for growth than simply adding more consumers to a queue.