Skip to main content

Introduction

Hydroflow

Hydroflow is a small language and compiler for low-latency dataflow programs, written in Rust. Hydroflow serves as the runtime library for the Hydro language stack, which is under development as a complete compiler stack for distributed programming languages.

Hydroflow is designed with two use cases in mind:

  • Expert developers can program directly in the Hydroflow language to build individual components that can interoperate in a distributed program or service.
  • Higher levels of the Hydro stack will offer higher-level abstractions and DSLs, and treat Hydroflow as a compiler target.

Hydroflow is targeted at supporting the following unique features:

  1. A type system that helps developers reason about progress and consistency guarantees in a distributed program. This includes an emphasis on lattice types that can allow for consistent outcomes in the face of network messages that may be interleaved, reordered, batched, and resent.
  2. A dataflow programming model, capturing the message- and data-driven nature of many distributed services.
  3. Extremely low-latency handling of asynchronously-arriving data/messages, via aggressive exploitation of Rust's monomorphization techniques.
  4. Dataflow optimizations, both to optimize single-node Hydroflow flows, and to enable distributed optimizations across multiple flows.

Hydroflow's language—the Hydroflow surface syntax—is embedded in Rust, which compiles Hydroflow code to high-efficiency machine code. As the lowest level of the Hydro stack, Hydroflow requires some knowledge of Rust to use.

The most recent release of the Hydroflow Book docs are online, providing documentation and numerous annotated examples.

You can also check out the Hydroflow Playground to see Hydroflow's surface syntax in action!

This Book​

This book will teach you how to set up your environment to get started with DFIR, and how to program in the DFIR surface syntax.

Keep in mind that DFIR is under active development. However the code in this book is tested with the DFIR library so should always be up-to-date.

If you have any questions, feel free to create an issue on Github.