This repository implements a collection of secure distributed computing protocols that serve as building blocks for larger distributed systems. The protocols are designed to provide security guarantees in adversarial environments. However, this code has been written as a research prototype and has not been vetted for security. Therefore, this repository can contain serious security vulnerabilities. Use at your own risk.
Broadcast Protocols (broadcast/)
-
CTRBC (Cachin-Tessaro's Reliable Broadcast Protocol) (
broadcast/ctrbc/) - Cachin-Tessaro's Reliable broadcast protocol based on the protocol inCT05. -
ECC-RBC (Error-Correcting Code Reliable Broadcast) (
broadcast/ecc_rbc/) - Reliable broadcast using Reed-Solomon error-correcting codes inNDD+22.
Dissemination Protocols (dissemination/)
-
ASKS (Asynchronous Secret Key Sharing)/ AwVSS (Asynchronous weak Verifiable Secret Sharing) (
dissemination/asks/) - ASKS/AwVSS protocol in theDDL+24,BBB+24. -
AVID (Asynchronous Verifiable Information Dispersal) (
dissemination/avid/) - Verifiable information dispersal protocols based on DispersedLedgerSPA+22.
Consensus Protocols (consensus/)
- ACS (Asynchronous Common Subset) (
consensus/acs/) - Implements asynchronous common subset consensus protocol in theDDL+24 - Binary Byzantine Agreement (
consensus/binary_ba/) - Asynchronous Binary BA inIBY22. - FIN-MVBA (Finite Multi-Valued Byzantine Agreement) (
consensus/fin_mvba/) - Asynchronous Multi-valued Byzantine agreement protocol in FIN (SWZ23). - IBFT (Istanbul Byzantine Fault Tolerance) (
consensus/ibft/) - PBFT-style Leader-based consensus protocol only using Message Authentication Codes inHen20. - RA (Reliable Agreement) (
consensus/ra/) - Reliable agreement protocol inDDL+24
This is a Rust project using Cargo. The compatibility between dependencies has been tested for Rust version 1.83.0. To build all components:
cargo build --releaseRun the following sequence of steps to start a protocol.
-
Generate Configuration Files: This step generates the necessary configuration files for an
$n$ party distributed system.
mkdir testdata/
./target/release/genconfig --base_port 15000 --client_base_port 19000 --client_run_port 19500 --NumNodes 4 --blocksize 100 --delay 100 --target testdata/ --local true
These instructions generate configuration files for 15000+i, listens to requests on port 19000+i, and syncs with a global synchronizer (this part is optional) on port 19500. Please ensure the directory has been created to run this command.
- Create channels and invoke protocol: The following snippet of code illustrates a basic composition of distributed protocols.
pub async fn spawn(config: Node)-> (anyhow::Result<oneshot::Sender<()>>, Vec<Result<oneshot::Sender<()>>>){
// ctrbc_req_send_channel: Request sending channel, request receiving channel. The sending channel can be used to issue message requests to the RBC module.
// ctrbc_req_recv_channel: Request receiving channel - passed as an argument. The RBC module listens to this channel.
let (ctrbc_req_send_channel, ctrbc_req_recv_channel) = channel(10000);
// ctrbc_out_send_channel: Output sending channel - passed as an argument. The RBC module sends outputs on this channel.
// ctrbc_out_recv_channel: Output receiving channel. We poll this channel to get outputs from RBC module.
let (ctrbc_out_send_channel, mut ctrbc_out_recv_channel) = channel(10000);
let mut statuses = Vec::new();
// Start Cachin-Tessaro RBC protocol
let _rbc_serv_status = ctrbc::Context::spawn(
config,
ctrbc_req_recv_channel,
ctrbc_out_send_channel,
false
);
statuses.push(_rbc_serv_status);
let _resp = ctrbc_req_send_channel.send(Vec::new()).await.unwrap();
tokio::spawn(async move {
loop {
tokio::select! {
msg = ctrbc_out_recv_channel.recv() => {
// Execute handling logic for the received message from the channel
log::debug!("Received message from CTRBC channel {:?}", msg);
// self.process_ctrbc_event(ctrbc_msg.1, ctrbc_msg.0, ctrbc_msg.2).await;
}
}
}
});
let (exit_tx, _exit_rx) = oneshot::channel();
(Ok(exit_tx), vec![])
}Protocols utilize tokio asynchronous channels or queues to receive requests and send outputs.
Each protocol takes two tokio channels as input: a receiver channel from which it receives requests (req_recv channel), and a sender channel to which it can send outputs (out_send channel).
Each protocol's invocation takes these channels as arguments.
A prominent example of protocol composition is in consensus/acs.
This folder implements an Asynchronous Common Subset (ACS) protocol from Reliable Broadcast (CTRBC), Secret Key Sharing (ASKS), and Reliable Agreement (RA).
-
Build code and run parties: After compiling the code, run
$n=4$ parties to start the protocol. Each party waits until it establishes a tcp channel with all parties. Thescripts/test.shscript can also be used to start all four parties locally.
All protocols are designed to handle Byzantine faults, where up to t out of n nodes can behave arbitrarily (where typically n ≥ 3t + 1).
Most protocols operate in asynchronous network models, making no assumptions about message delivery times or clock synchronization.
Each protocol is implemented as a separate module with well-defined interfaces, allowing them to be composed into larger systems.
The implementation includes a robust networking layer with:
- TCP-based reliable communication
- Message acknowledgments
- Automatic connection management
These protocols serve as building blocks for:
- Distributed ledgers and blockchains
- Secure asynchronous multi-party computation protocols
- Byzantine fault-tolerant state machine replication
This implementation is part of ongoing research in secure distributed computing, focusing on practical implementations of theoretically sound protocols that can handle adversarial conditions in distributed systems.
Cryptographic Primitives (crypto/)
- SHA256 Hash function, and Merkle trees based on Hardware-accelerated Hash based on AES
- Symmetric encryption (AES-based)
- Cryptographic utilities and random number generation
Configuration Management (config/)
- Network configuration and node setup
- Protocol parameter management
Type Definitions (types/)
- Common data structures and type definitions
- Replica identifiers and protocol messages
Utilities (util/)
- Helper functions and common utilities
- Networking abstractions
Tools (tools/)
- genconfig - Configuration generation utility
[1] Cachin, Christian, and Stefano Tessaro. "Asynchronous verifiable information dispersal." 24th IEEE Symposium on Reliable Distributed Systems (SRDS'05). IEEE, 2005.
[2] Alhaddad, Nicolas, Sourav Das, Sisi Duan, Ling Ren, Mayank Varia, Zhuolun Xiang, and Haibin Zhang. "Balanced byzantine reliable broadcast with near-optimal communication and improved computation." In Proceedings of the 2022 ACM Symposium on Principles of Distributed Computing, pp. 399-417. 2022.
[3] Das, Sourav, Sisi Duan, Shengqi Liu, Atsuki Momose, Ling Ren, and Victor Shoup. "Asynchronous consensus without trusted setup or public-key cryptography." In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 3242-3256. 2024.
[4] Bandarupalli, Akhil, Adithya Bhat, Saurabh Bagchi, Aniket Kate, and Michael K. Reiter. "Random beacons in monte carlo: Efficient asynchronous random beacon without threshold cryptography." In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 2621-2635. 2024.
[5] Yang, Lei, Seo Jin Park, Mohammad Alizadeh, Sreeram Kannan, and David Tse. "{DispersedLedger}:{High-Throughput} byzantine consensus on variable bandwidth networks." In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pp. 493-512. 2022.
[6] Abraham, Ittai, Naama Ben-David, and Sravya Yandamuri. "Efficient and adaptively secure asynchronous binary agreement via binding crusader agreement." In Proceedings of the 2022 ACM Symposium on Principles of Distributed Computing, pp. 381-391. 2022.
[7] Duan, Sisi, Xin Wang, and Haibin Zhang. "Fin: Practical signature-free asynchronous common subset in constant time." In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 815-829. 2023.
[8] Moniz, Henrique. "The Istanbul BFT consensus algorithm." arXiv preprint arXiv:2002.03613 (2020).