Skip to content
Softworks
Softworks
hi@softworks.studio
Instagram LinkedIn
← Back to the Journal

Engineering field notes 02

Offline & data integrity

Offline is a reconciliation problem.

A durable queue is only the beginning. How we preserve business intent, replay dependent work and stop stale snapshots from making committed stock reappear.

Two disconnected ledger stacks with a metal bridge, command tiles and a glass overlay.
Read the engineering story

The common problem

Local durability preserves the work. The authoritative system decides whether that work can commit.

Two disconnected terminals can both show the same machine as available. Each can capture a sale carefully and persist it successfully. They still cannot both promise that physical unit to different customers.

Hilton Kitchen's Windows workflow made this boundary explicit. Offline commands are durable and provisional; the server revalidates stock and serial availability during replay. DocuBex reveals a related problem from the other side: a remote operation can succeed while its acknowledgement disappears. In both cases, recovery starts by identifying the original intent and reconciling it with authoritative state.

Grounded in the work

  • Hilton Kitchen

    Dependency-aware Windows replay, server stock validation and revision-based POS reconciliation.

  • DocuBex

    Persisted multipart-upload manifests and recovery when storage completion is uncertain.

01 / 06

Queue a business intention, not an HTTP request.

A request is a transport event. An intention says what the employee meant to do: create this customer, sell these items, record this installment. Capturing the intention lets the application preserve meaning when the API evolves or a prerequisite has not yet synchronized.

Hilton Kitchen stores commands and their dependencies in its Windows application. The main-process coordinator handles replay independently of the visible React screen. A customer command can precede an order; an order can precede a payment. Work for an unrelated entity can continue while one chain needs attention.

The important ordering is causal. Sorting by a laptop's wall clock does not establish that the customer exists before an order references it. Store explicit dependency IDs and server identity mappings. A dependent command becomes eligible after its prerequisites are confirmed, not merely after their requests were attempted.

A useful command envelope also retains an operation ID, schema version, originating device and a stable representation of the business payload. Attempt count and the next retry time belong to delivery bookkeeping. They must not change the identity of the underlying intention.

Do not persist an expired bearer token inside every queued request. Authenticate replay using the current allowed session and recheck authorization on the server. A command captured yesterday does not carry permanent authority to mutate data today.

02 / 06

A timeout creates uncertainty, not proof of failure.

A sale can commit immediately before the connection disappears. If the client creates a new command on retry, it can accidentally describe a second sale. Reusing the same identity lets the server resolve the first intention.

That identity must be scoped. An operation key alone should not allow another tenant or user to recover somebody else's result. Bind it to an authorized operation scope and a canonical payload representation. A same-key, different-payload request is a conflict, not a convenient edit. An uncertain original intent should be resolved before the operator creates a replacement.

DocuBex exposes the equivalent problem across database and object storage. Its persisted upload manifest records expected size, parts and completion state. Recovery compares the manifest with provider state. If completion succeeded remotely but the local acknowledgement is missing, the application must recover the existing object instead of blindly uploading another document.

The Amazon S3 completion API also documents a subtle transport detail: an initial HTTP 200 can contain an embedded completion error. Use an SDK or parse the final result correctly. A status line alone is insufficient evidence of successful completion.

This is why “try again” needs a precise meaning. Retry the same intent, reconcile the known provider object, or ask the operator to resolve a genuine conflict. Those are three different actions.

03 / 06

An acknowledgement is not a new inventory snapshot.

A transparent plate with two sage command pieces lifted above a settled ledger stack.

Keep the overlay until the view catches up.

The glass layer represents provisional local intent above the server snapshot. An accepted response alone does not retire that layer; the snapshot must reach the revision that includes the command.

Ledger stack
The last authoritative snapshot the client has received.
Glass overlay
Pending effects remain visible above that snapshot.
Lifted layer
Retire an accepted effect only after its revision is included.

One of Hilton's most instructive corrections concerns the moment after success. A terminal has an inventory snapshot showing ten units. It captures a sale of two and projects eight locally. The server accepts the sale, but the next snapshot has not arrived.

If the terminal removes the pending subtraction immediately, its old snapshot makes the display jump back to ten. The sale was saved correctly; the local read model is now wrong. That can invite another sale against stock that has already left.

Hilton's POS projection keeps the accepted operation in the local overlay until an inventory revision proves that the refreshed snapshot contains it. It does not use device-clock comparisons. Physical machine units follow their own cached-unit and server-validation path; this quantity projection should not be generalized to serial inventory.

The following executable teaching example expresses that revision barrier. It assumes one ordered revision scope and signed deltas for a single stock balance. The real product's persistence and protocol are outside this sample.

typescript
type Snapshot = { quantity: number; revision: number };
type LocalChange = {
  delta: number;
  state: "pending" | "accepted" | "rejected";
  committedRevision?: number;
};

export function projectedQuantity(
  snapshot: Snapshot,
  changes: LocalChange[],
) {
  return changes.reduce((quantity, change) => {
    if (change.state === "rejected") return quantity;
    const represented =
      change.state === "accepted" &&
      change.committedRevision !== undefined &&
      change.committedRevision <= snapshot.revision;
    return represented ? quantity : quantity + change.delta;
  }, snapshot.quantity);
}

A revision comparison is meaningful only within the same authoritative stream. Do not compare an order revision with a warehouse revision or a different tenant's counter. Fetch a coherent quantity/revision pair; assembling each field from a different read would destroy the proof. Negative projections should trigger an explicit operational state, not be hidden by silently clamping to zero.

Explore the failure boundary

The stock that must not reappear.

Illustrative quantities: one snapshot, one sale of two units, and a server revision boundary.

  1. Snapshot

    10 units · r40

    The last confirmed balance.

  2. Local overlay

    −2 pending

    The command is durable but not yet accepted.

  3. Visible quantity

    8 units

    A provisional projection, not a global reservation.

The terminal preserves its own intention while other terminals remain independent.

04 / 06

Resolution is a product workflow.

A disconnected terminal cannot know that another terminal has sold the same serialized unit. When the server rejects one capture, the useful response is not an endless retry loop. Staff need to understand which item conflicted, which parts of the intention remain unresolved and what authorized alternatives exist.

Preserve the captured record and its explanation. A replacement unit may require an explicit new decision. A payment captured against the original intention may need its own review. Never silently reinterpret the customer's agreement to make a synchronization error disappear.

Separate retryable infrastructure failures from domain rejection. A temporary connection failure can use bounded exponential backoff with jitter. A stock conflict, revoked permission or unsupported envelope version requires a different path. Record next action and dependency consequences so a blocked order does not repeatedly send its dependent payment.

The UI should distinguish captured, sending, unconfirmed, accepted and needs-resolution states. Provisional documents must remain visibly provisional until server confirmation. This is especially important when paper leaves the terminal before the network returns: a printout should not promise a final transaction that the system has not accepted.

Hilton's ordinary browser sessions cache the application shell and assets; they do not originate the same offline business writes as its Windows application. Product copy must name the supported surface precisely.

05 / 06

Your oldest queued command is still an API client.

An application update does not erase work already captured on installed terminals. Hilton's documented Windows compatibility work includes an older warehouse envelope using direction while the current executor expects movementType, plus installment replay corrections.

A versioned envelope makes that difference inspectable. Migrate known versions deliberately and retain the original identity. An unknown version should remain recoverable with a useful error; guessing field semantics risks changing the business intent.

The same discipline applies to local schema upgrades. Test with an actual persisted old queue, not only a freshly created current command. Check interrupted migrations and restart behavior. Keep backups or an appropriate recovery path before rewriting durable pending work.

Database recovery has its own operational dimension. PostgreSQL's backup documentation distinguishes logical dumps, filesystem-level backups and continuous archiving. Choose and rehearse a recovery method that matches the deployment. A valid database backup does not, on its own, include commands that only exist on disconnected terminals.

That last point changes incident planning. Central recovery and device recovery are separate inventories. A team needs to know which terminals hold pending work and how that work will be reconciled after the authoritative service returns.

06 / 06

Test convergence after interruption.

A meaningful offline suite stops the system at inconvenient boundaries. Capture dependent commands, close the window, restart the coordinator and verify replay order. Commit on the server, drop the response and verify that retry resolves one result. Accept a POS sale while returning an old snapshot, then confirm that the projected balance stays stable.

Also test two devices against the same physical unit. One may be accepted; the other must retain an explicit resolution state. Then test a permission change before replay and an unsupported command version after upgrade. These cases distinguish local durability from authoritative permission and business validity.

For upload recovery, complete the object at the provider and fail the subsequent database update. Recovery should inspect the existing object and preserve a single document intention. Cleanup needs the same discipline: inability to confirm deletion is not successful deletion.

The outcome we can claim from the case studies is specific: durable provisional work, explicit dependencies, server revalidation and recovery grounded in recorded state. We are not claiming global offline reservations or replay guarantees for every deployment. A dependable offline product makes progress without hiding the point at which it needs the rest of the system.

Carry this into your next system

Three decisions to keep.

  1. Persist intention, dependency and schema identity before delivery.
  2. Retain accepted overlays until a coherent snapshot proves inclusion.
  3. Treat conflict resolution and old-envelope compatibility as first-class journeys.

Trace the reasoning

References & further reading.

Project claims link to the published case studies. External references are primary technical documentation, reviewed on 18 October 2026. Code and interactive scenarios are teaching examples with the limits described alongside them.

  1. Hilton Kitchen — offline workflows

    Dependency-aware commands, revision reconciliation and historical Windows compatibility evidence.

  2. DocuBex — recoverable intake

    Persisted upload manifests, uncertain completion and browser ownership.

  3. Amazon S3 — CompleteMultipartUpload

    Provider completion semantics, including errors embedded in a 200 response.

  4. PostgreSQL — Backup and Restore

    Official overview of database backup approaches; device-only queues require separate recovery.

Case-study evidence retains its original scope and date. This article does not represent a new test run against the underlying products, a hardware certification or a production performance benchmark.

Softworks

Softworks Studio