Introduction
Hydroflow+ layers a high-level Rust API over the Hydroflow IR, making it possible to write dataflow programs that span multiple processes with straightline, functional Rust code. Hydroflow+ is built on top of Stageleft, which allows Hydroflow+ to emit regular Hydroflow programs that are compiled into efficient Rust binaries. It also integrates with Hydro Deploy to make it easy to deploy and run Hydroflow+ programs on a cluster.
The main logic of Hydroflow+ programs manipulates streams, which capture infinite ordered sequences of elements. Streams are transformed using classic functional operators such as map
, filter
, and fold
, as well as relational operators such as join
. To build distributed dataflow programs, Hydroflow+ also introduces the concept of processes, which capture where a stream is being processed.
Setup
Hydroflow+ requires a particular workspace setup, as any crate that uses Hydroflow+ must have an supporting macro crate to drive the code generation. To get started, we recommend using the Hydroflow+ template.
cargo install cargo-generate
cargo generate hydro-project/hydroflow template/hydroflow_plus
cd
into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:
cd <my-project>
rustup update
cargo test