Introduction

Hydroflow+ layers a high-level Rust API over the Hydroflow IR, making it possible to write dataflow programs that span multiple processes with straightline, functional Rust code. Hydroflow+ is built on top of Stageleft, which allows Hydroflow+ to emit regular Hydroflow programs that are compiled into efficient Rust binaries. It also integrates with Hydro Deploy to make it easy to deploy and run Hydroflow+ programs on a cluster.

The main logic of Hydroflow+ programs manipulates streams, which capture infinite ordered sequences of elements. Streams are transformed using classic functional operators such as map, filter, and fold, as well as relational operators such as join. To build distributed dataflow programs, Hydroflow+ also introduces the concept of processes, which capture where a stream is being processed.

Setup

Hydroflow+ requires a particular workspace setup, as any crate that uses Hydroflow+ must have an supporting macro crate to drive the code generation. To get started, we recommend using the Hydroflow+ template.

cargo install cargo-generate
cargo generate hydro-project/hydroflow template/hydroflow_plus

cd into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:

cd <my-project>
rustup update
cargo test