ADR-53: JetStream Read-after-Write

[MauriceVanVeen]

Up to this point JetStream has never been able to guarantee read-after-write or monotonic reads, unless the underlying stream was not replicated.

This ADR proposes a solution to both read-after-write and monotonic reads, for both Direct Get requests and consumers, that does not require all reads to go through replication.

Also clarified other ADRs that reference read-after-write.

Server PR: nats-io/nats-server#6970
Draft Go client implementation for read requests, consumers, and KV for both the old and new APIs (only object store is missing).

Resolves nats-io/nats-server#6557

[derekcollison]

In general really like this approach. I did have a question on deletes.

[MauriceVanVeen]

The server's read-after-write PR (nats-io/nats-server#6970) has been merged. Have updated this ADR's status to "Implemented".

Have also included a "Client Implementation" section that outlines the support required in the clients.

For reference, have created a draft Go client implementation. Please note it's incomplete, but it can be used as a reference. Have modeled that to what could be idiomatic in the Go client, other clients could improve to what's idiomatic there (the design is not strict on that).

[ripienaar]

Maybe say before 2.12?

[MauriceVanVeen]

Done, dcb9074:

JetStream does NOT support read-after-write or monotonic reads (prior to server version 2.12).

[ripienaar]

LGTM

[jnmoyne]

LGTM

[MauriceVanVeen]

Parking this for a little bit, this feature will not be included in the upcoming 2.12 release.
It's something we can revisit early on in the 2.14 cycle.

One idea we've discussed a little already is instead of everything being opt-in and on a per-request basis, instead make it a stream-setting. Something like StreamConfig.ReadConsistency: Serializable / Linearizable. That would mean there doesn't need to be client or app-level changes (except for supporting the new stream field). When enabled, that could mean certain restrictions are placed such that the guarantees can be made. We could enable this by default on object store, but have it be an opt-in on KV and streams, etc.

Lots to figure out still, but need some more options so we can discuss which would be preferable.

[jpsugar-flow]

We're very interested in this feature. Right now we're minimizing the window by leaving direct get disabled, but this would let us turn everything on.

I'm curious if there's appetite for Spanner-like client-specified consistency requirements?

[MauriceVanVeen]

We're very interested in this feature. Right now we're minimizing the window by leaving direct get disabled, but this would let us turn everything on.

I'm curious if there's appetite for Spanner-like client-specified consistency requirements?

Thanks for sharing! Am not too familiar with Spanner myself, could you perhaps share a resource for the client-specified consistency requirements?

The primary options for stronger read consistency with JetStream would be that it's either specified client-side on a per-read-request basis ("for this specific request I need reads to be linearizable, versus by default they are eventually consistent with direct get"). Or specified top-level in the stream, which would mean all reads would be linearizable if specified as such.

Is this similar to what you're suggesting/expecting?

[jpsugar-flow]

Specifically Spanner has a "max staleness" bound: https://cloud.google.com/spanner/docs/reads#perform-stale-read

Was just an idle thought, not a feature request here. :)

[jpsugar-flow]

The primary options for stronger read consistency with JetStream would be that it's either specified client-side on a per-read-request basis ("for this specific request I need reads to be linearizable, versus by default they are eventually consistent with direct get"). Or specified top-level in the stream, which would mean all reads would be linearizable if specified as such.

That's what we're looking for, definitely. We don't necessarily need to force a leader round-trip; we often have a "last seen sequence" that we need the responder to be up-to-date to.