Skip to main content

TODO

Concepts

  • p1 (Mingwei) Hydroflow and Rust: how do they go together?
    • State, control, scoping
  • p1 State over time
    • lifetimes
    • explicit deletion
  • p3 Coordination tricks?
    • End-of-stream to Distributed EOS?

Docs

  • p1 hydroflow struct and its methods
  • p2 Review the ops docs

Operators not discussed

- dest_sink
- identity
- unzip
- p1 *fold* -- add to chapter on [state](/docs/hydroflow/syntax/state)
- p1 *reduce* -- add to chapter on [state](/docs/hydroflow/syntax/state)
- p1 *fold_keyed* -- add to chapter on [state](/docs/hydroflow/syntax/state)
- p3 *sort_by_key* -- add to chapter on [state](/docs/hydroflow/syntax/state)
- p2 *next_stratum* -- add to chapter on [time](/docs/hydroflow/concepts/life_and_times)
- p2 *next_tick* -- add to chapter on [time](/docs/hydroflow/concepts/life_and_times)
- p2 *inspect* -- add to chapter on [debugging](/docs/hydroflow/concepts/debugging)
- p2 *null* -- add to chapter on [debugging](/docs/hydroflow/concepts/debugging)

How-Tos and Examples

  • p1 Lamport clocks
  • p2 Vector clocks
  • p2 A partitioned Service
  • p2 A replicated Service
  • p2 Interfacing with external data
  • p2 Interfacing with external services
  • p1 Illustrate 'static and 'tick lifetimes (KVS)
  • p3 Illustrate the next_stratum operator for atomicity (eg Bloom's upsert <+- operator)
  • p3 Illustrate ordered streams (need zip operator ... what's the example?)
  • p3 Actor model implementation (Borrow an Akka or Ray Actors example?)
  • p3 Futures emulation? (Borrow a Ray example)
  • p2 Illustrate external storage source and sink (e.g. for WAL of KVS)

Odds and ends taken out of other chapters

  • Document the methods on the hydroflow struct -- especially the run methods.

  • Make sure src/examples/echoserver is the same as the template project -- or better, find a way to do that via github actions or a github submodule

What's covered in examples

  • Concepts covered

    • cargo generate for templating
    • Hydroflow program specs embedded in Rust
    • Tokio Channels and how to use them in Hydroflow
      • Network sources and sinks (source_stream)
      • Built-in serde (source_stream_serde, dest_sink_serde)
    • Hydroflow syntax: operators, ->, variables, indexing multi-input/output operators
    • running Hydroflow via run_available and run_async
    • Recursion via cyclic dataflow
    • Fixpoints and Strata
    • Template structure: clap, message types
    • source_stdin
    • Messages and demux
    • broadcast pattern
    • the persist operator to store and replay dataflow
    • the defer_signal operator to gate a dataflow
    • bootstrapping pipelines: initialize
  • Operators covered

    • cross_join
    • defer_signal
    • demux
    • dest_sink_serde
    • difference
    • filter
    • filter_map
    • flatten
    • flat_map
    • for_each
    • initialize
    • join
    • map
    • persist
    • union
    • source_iter
    • source_stdin
    • source_stream
    • source_stream_serde
    • tee
    • union
    • unique