Simulation Testing
Hydro comes with a built-in deterministic simulation environment that lets you test your application in various distributed scenarios and guard against concurrency bugs and race conditions. Tests written with the simulator are run against various distributed schedules to test possible concurrent executions.
In many cases, the Hydro simulator can perform exhaustive checks, which ensure that your application will behave correctly in all possible distributed executions. For particularly complex tests where there are too many scenarios to check, the Hydro simulator can use coverage-guided fuzzing to intelligently explore the space of executions and find bugs.
Beyond automatic exploration, tests can take manual control of individual decision points with simulator hooks, pinning a specific scenario while everything else continues to be explored. A test that scripts every decision can run in deterministic mode, which executes exactly one interleaving with no exploration at all.
Writing a simulation test involves three steps:
- Defining your Hydro application, with regular Hydro APIs
- Creating external inputs / outputs that you will use in the test
- Writing a test that sends inputs and makes assertions about the output
Hydro Feature Support
The simulator uses the exact same Hydro code you will run in production, and requires no changes. The simulator supports the vast majority of Hydro APIs, with some limitations around non-deterministic observations and timing. The following table highlights areas of support:
| Feature | Support Level | Notes |
|---|---|---|
| Ordered Transformations | Full | map, filter, fold, scan, etc. across all live collections |
| Networking | Full | Simulated networking is supported for both Process and Cluster locations |
| Batching / Snapshotting | Partial | Supported on all live collections except Optional |
| Unordered Reductions | Full | Built-in operators like count are accelerated, but not arbitrary {fold/reduce} |
| Non-Deterministic Observations | Limited | assume_ordering::<TotalOrder> is supported, assume_retries::<ExactlyOnce> is not supported |
Code Coverage
Simulation tests execute your Hydro code inside a dynamic library that is compiled on the fly when the test runs, rather than inside the test binary itself. The simulator makes sure code-coverage instrumentation carries over to that library: if it detects a coverage run (the LLVM_PROFILE_FILE environment variable, which coverage harnesses set for the test process, or an explicit instrument-coverage flag in RUSTFLAGS), it compiles the library with -C instrument-coverage as well, so lines exercised only through the simulator are attributed to your source files like any other test coverage.
One note for custom coverage pipelines: the coverage mapping for simulator-executed code lives in the compiled library itself, not in the test binary. During a coverage run the simulator keeps a copy of each compiled library under target/debug/deps/hydro-coverage/, so tools like grcov or llvm-cov that take explicit binary paths will find these mappings whether they are pointed at target/debug or target/debug/deps alongside the test binaries when generating reports.