Thinking in Hydro
Here are some tips for getting started working with Hydro. At the very highest level, Hydro's APIs steer you to write programs that behave the same regardless of network re-orderings and duplication. It does this by enforcing commutativity and idempotence when necessary. That mental model will help you understand Hydro's design.
-
Understand which parts of your distributed system are deliberately non-deterministic. Take 5-10 minutes and identify properties like the following:
- A client will choose a server depending on which one responds fastest
- The KVS will let arrival order dictate the value that wins the entry
- A client may time out (slow network, etc.) and re-subscribe to a live stream
-
Identify invariants. Here are examples that correspond to the cases above:
- The client will only connect to one of the servers
- All replicas of the KVS will hold the same eventual value
- The client must not have any gaps in its log if catchup has concluded
-
Combine the non-determinism and invariants to create simulation tests for your system.
- These might be as follows:
- Send the client a response from server 1 and from server 2. Quiesce, and assert that the client has sent a close to one server and an open to the other.
- Start up the KVS and allow it to elect a leader. Send the leader a series of PUT requests, quiesce, and then assert that every replica holds an identical value.
- Send messages for a few subscriptions (open, stream, and close messages), several clock ticks (some within, and some out of) the timeout interval(s), then quiesce and validate that the log stream that the client outputs has no gaps. See below about how to model clocks and timeouts.
- Simulation tests are an excellent way to build the skeleton for your program. You can think of them as a bounded model checker for your program.
- In a simulation test, when you send multiple messages (including across different
Streams), the simulator may re-order them. But when is that ordering chosen? That is what the points you.await(such asquiesce()and its family) mark; see Receiving Outputs and Quiescence. - Be very thoughtful about where you
.await. Avoid performing.awaitin helper functions, to give the simulator lots of room for interleaving messages. Notice how in the examples, quiescence is used to introduce and set up the search space, but there is no quiescence through the points at which the invariants must hold. - Use a happy-path test to naturally explore many orderings, instead of placing lots of
.awaitpoints that constrain the orderings that may be explored.
- These might be as follows:
-
In many programming models, servers are written as callbacks (e.g.
on_fetch_request). Hydro requires you to acknowledge that these requests may have been re-ordered and/or duplicated on the network by modeling them as aStream<FetchRequest, _, _, NoOrder, AtLeastOnce>(see Streams). Create your function signatures by creating a parameter for each of these streams. Most likely, that will beStream<GetRequest>andStream<PutRequest>instead ofStream<Request>. -
When using a clock in Hydro, prefer to model it as a
Stream<Timestamp, _>(useNoOrderif a monotonic clock is not available, so that downstream logic cannot assume a later reading is a later time). It should be delivered as an argument from the embedding code outside your dataflow.Timers should be derived from this single source of time, as it allows for more effective simulation testing. Avoid the use of
source_intervalwhen you have multiple colocated timers because the simulator will not know that they are correlated. The timestamp at which the timer last fired would be stored assliced!state.