Consensus Algorithms: Distributed Agreement Under Failure
Purpose: Algorithms and theoretical frameworks for achieving agreement among distributed processes despite failures, enabling fault-tolerant distributed systems.
Core Insight: Consensus is impossible in asynchronous systems with even one crash failure (FLP), but practical with timeouts, randomization, or synchrony assumptions.
Table of Contents
- Foundational Concepts
- FLP Impossibility
- Paxos
- Raft
- Byzantine Consensus
- Practical Systems
- Stack-Specific Implementations
- Integration Points
Foundational Concepts
The Consensus Problem
Goal: N processes must agree on a single value, despite failures.
Requirements (Traditional):
- Termination: Every correct process eventually decides on a value
- Agreement: All correct processes decide the same value
- Validity: The decided value was proposed by some process
Failure Models
Crash Failures (Benign):
- Process stops executing permanently
- No incorrect behavior before crash
- Examples: power failure, OOM kill
Byzantine Failures (Malicious):
- Arbitrary behavior: send wrong messages, lie, collude
- Includes both malicious actors and software bugs
- Requires stronger protocols (BFT)
Omission Failures:
- Process fails to send or receive messages
- Includes network partitions
Timing Failures:
- Process responds too late
- Relevant in synchronous systems
System Models
Asynchronous:
- No bounds on message delay or processing speed
- No reliable failure detection
- Most realistic but hardest (FLP impossible)
Synchronous:
- Known bounds on delays and processing
- Reliable failure detection via timeouts
- Easiest but least realistic
Partially Synchronous (Practical):
- Eventually synchronous: bounds hold after some unknown time
- Models real systems (networks eventually recover)
- Algorithms like Paxos, Raft work here
FLP Impossibility
Theorem (Fischer, Lynch, Paterson, 1985):
“No deterministic consensus algorithm can guarantee termination in an asynchronous system with even one crash failure.”
FLP Proof Sketch
Setup:
- Asynchronous system, N processes
- At most 1 crash failure
- Binary consensus (decide 0 or 1)
Key Concept: Bivalent Configuration
- Configuration where both 0 and 1 are possible outcomes
- Univalent = only one outcome possible (0-valent or 1-valent)
Proof Steps:
- Initial bivalent configuration exists: Some initial state allows either outcome
- Bivalent configurations reachable forever: From any bivalent config, can always reach another bivalent config by delaying one message
- No termination: If always bivalent configs reachable, never reach univalent (decided) state
Implication: In pure asynchronous model, can’t distinguish crashed process from slow process → can’t safely terminate.
Circumventing FLP
Real systems work because they relax assumptions:
Randomization:
- Use randomness to break symmetry
- Terminate with probability 1 (not deterministic guarantee)
- Example: Bitcoin’s proof-of-work
Timeouts (Partial Synchrony):
- Assume eventual synchrony: bounds hold eventually
- Timeout to detect failures (may be wrong temporarily)
- Examples: Paxos, Raft, PBFT
Failure Detectors:
- Abstract failure detection service
- Perfect FD: never wrong (requires synchrony)
- Eventually Perfect FD: eventually accurate (achievable)
- Examples: Chandra-Toueg consensus
Weakened Termination:
- Require only “live” processes terminate
- Non-faulty but slow processes may not decide
- Example: Paxos doesn’t guarantee termination for all
Paxos
Developed by: Leslie Lamport (1989, published 1998)
Key Idea: Use quorum-based voting with proposal numbering to ensure agreement even when processes crash or messages are delayed.
Single-Decree Paxos (Basic Paxos)
Goal: Agree on a single value.
Roles:
- Proposer: Proposes values
- Acceptor: Votes on proposals
- Learner: Learns the chosen value
Phases:
Phase 1: Prepare
- Proposer selects unique proposal number N
- Sends
Prepare(N)to quorum of acceptors - Acceptor responds:
Promise(N)if N > any previous promise- Include highest-numbered accepted proposal (if any)
Phase 2: Accept
- If proposer receives promises from quorum:
- If any acceptor returned accepted value, use highest-numbered one
- Otherwise, use proposer’s own value
- Send
Accept(N, value)to quorum - Acceptor accepts if N ≥ promised number
- Respond
Accepted(N, value)
Learning:
- Learner receives
Acceptedfrom quorum → value is chosen
Invariant: If quorums overlap, at most one value can be chosen.
Multi-Paxos
Problem: Basic Paxos requires 2 round trips per decision.
Solution: Elect a leader (stable proposer) who skips Phase 1 for subsequent proposals.
Leader Election:
- Use Phase 1 to establish leadership
- Leader has exclusive proposal rights while stable
- On leader failure, run Phase 1 again
Log Replication:
- Apply Paxos to each log slot independently
- Leader coordinates appending entries across replicas
- Achieves replicated state machine
Paxos Properties
Safety (Always):
- Agreement: Only one value chosen per instance
- Validity: Chosen value was proposed
Liveness (Eventually, with stable leader):
- Progress if majority available and leader stable
Quorum Requirement:
- Need majority: ⌈(N+1)/2⌉
- Tolerates ⌊N/2⌋ crash failures
Paxos Variants
Fast Paxos:
- Clients send directly to acceptors (skip proposer)
- Requires 3f+1 acceptors for f failures
- Faster when no conflicts
Cheap Paxos:
- Subset of acceptors (main quorum) + auxiliary acceptors
- Auxiliary only activated on main failure
- Reduces number of always-active acceptors
Flexible Paxos:
- Different quorum sizes for Phase 1 and Phase 2
- Requirement: Q1 ∩ Q2 ≠ ∅ (quorums must overlap)
Raft
Developed by: Diego Ongaro and John Ousterhout (2014)
Goal: Understandable alternative to Paxos for replicated state machines.
Raft Overview
Key Innovation: Decompose consensus into:
- Leader Election
- Log Replication
- Safety
Roles:
- Leader: Handles all client requests, replicates log
- Follower: Passive, replicate leader’s log
- Candidate: Participates in leader election
Leader Election
Terms:
- Logical clock, monotonically increasing
- Each term has at most one leader
Election Process:
- Follower timeout → becomes Candidate
- Candidate increments term, votes for self
- Requests votes from all peers
- Peer grants vote if:
- Candidate’s term ≥ peer’s term
- Peer hasn’t voted this term
- Candidate’s log at least as up-to-date
- Candidate wins if receives majority votes
- Becomes Leader, sends heartbeats
Split Vote:
- No candidate gets majority → timeout, retry with new term
- Randomized timeouts reduce split votes
Log Replication
Normal Operation:
- Client sends command to Leader
- Leader appends to local log
- Leader sends
AppendEntriesRPCs to Followers - Followers append entry, respond success
- Leader commits when majority replicated
- Leader applies to state machine, replies to client
- Leader piggybacks commitIndex in heartbeats
Log Matching Property:
- If two logs have entry with same index and term → all preceding entries identical
- Maintained by Leader’s consistency check
Handling Failures:
- Follower crash: Leader retries
AppendEntriesindefinitely - Leader crash: New leader elected, log repaired
- Log inconsistencies: Leader forces followers to match its log
Safety
Election Safety:
- At most one leader per term
Leader Completeness:
- If log entry committed in term T, present in logs of leaders for all terms > T
Log Matching:
- If two logs contain entry with same index and term, they match up to that point
State Machine Safety:
- If server applies log entry at index, no other server applies different entry at same index
Commitment Rule:
- Leader can only commit entries from current term
- Once current term entry committed, all prior entries implicitly committed
Raft vs. Paxos
| Aspect | Paxos | Raft |
|---|---|---|
| Structure | Single-decree + Multi-Paxos | Leader-based from start |
| Leader | Optional, emergent | Explicit, necessary |
| Log Gaps | Allowed | Not allowed |
| Commit Rule | Value-based | Log-index-based |
| Understandability | Complex | Simpler |
Byzantine Consensus
Challenge: Achieve consensus when some nodes are Byzantine (malicious or faulty).
Requirement: ≥ 3f+1 nodes to tolerate f Byzantine failures (quorum = 2f+1).
PBFT (Practical Byzantine Fault Tolerance)
Developed by: Castro & Liskov (1999)
Key Idea: State machine replication with Byzantine fault tolerance.
Phases:
Pre-Prepare:
- Client sends request to Primary
- Primary assigns sequence number, broadcasts
PrePrepare(v, n, m)where:- v = view number
- n = sequence number
- m = request message
Prepare:
- Replica receives PrePrepare, validates
- Broadcasts
Prepare(v, n, d)where d = digest of m - Replica waits for 2f matching Prepare messages (quorum)
- “Prepared” certificate established
Commit:
- Replica broadcasts
Commit(v, n, d) - Waits for 2f+1 Commit messages (including self)
- “Committed” certificate established
- Execute request, reply to client
Client:
- Waits for f+1 matching replies (one from correct replica guaranteed)
View Change (Leader Replacement):
- Replica timeout → suspect Primary faulty
- Initiate view change protocol
- New primary elected, state transferred
PBFT Properties:
- Safety: All correct replicas agree on order
- Liveness: Clients eventually receive replies
- Performance: ~3 round trips, similar to Paxos
Tendermint
Developed by: Jae Kwon (2014)
Key Idea: BFT consensus for blockchain with immediate finality.
Mechanism:
- Rounds with designated proposer (rotating)
- Three phases: Propose, Prevote, Precommit
- Two rounds of voting for each decision
- Block finalized when 2f+1 Precommits collected
Features:
- Immediate Finality: No forks, block final when committed
- Application Agnostic: ABCI interface for any application
- Light Clients: Efficient verification via Merkle proofs
Used In: Cosmos blockchain, many app-chains.
HotStuff
Developed by: Yin, Malkhi, et al. (2018)
Key Innovation: Linear communication complexity (O(n) per decision, not O(n²) like PBFT).
Mechanism:
- Threshold Signatures: Aggregate 2f+1 signatures into single signature
- Three-Phase Voting: Prepare, Pre-Commit, Commit
- Chained Structure: Each phase’s QC becomes next phase’s justify
Phases:
- Prepare: Leader proposes, replicas vote
- Pre-Commit: Leader aggregates votes into QC, replicas vote again
- Commit: Leader aggregates, replicas commit
- Decide: Block finalized
Advantages:
- O(n) communication (vs. O(n²) in PBFT)
- Responsive: Progresses at network speed (not fixed timeout)
- Used in LibraBFT (Diem blockchain)
Practical Systems
Chubby (Google)
Purpose: Distributed lock service
Algorithm: Paxos (Multi-Paxos)
Features:
- Coarse-grained locks (lease-based)
- Small file storage (config, leader election results)
- 5-replica typical deployment
Use Cases: GFS, Bigtable (master election, metadata storage)
ZooKeeper (Apache)
Purpose: Coordination service for distributed applications
Algorithm: Zab (ZooKeeper Atomic Broadcast) - similar to Paxos/Raft
Features:
- Hierarchical namespace (like filesystem)
- Watches for change notifications
- Ephemeral nodes (disappear when client session ends)
Use Cases: Kafka (broker coordination), HBase (master election)
etcd
Purpose: Distributed key-value store
Algorithm: Raft
Features:
- Strong consistency (linearizable reads/writes)
- Watch API for change notifications
- TTL for keys (lease-based)
Use Cases: Kubernetes (cluster state), service discovery
Consul (HashiCorp)
Purpose: Service mesh, service discovery, configuration
Algorithm: Raft
Features:
- Multi-datacenter support (federated Raft clusters)
- Health checking integrated
- DNS and HTTP interfaces
Use Cases: Microservice coordination, config management
Stack-Specific Implementations
Rust: Raft Implementation (Simplified)
use serde::{Deserialize, Serialize};use std::collections::HashMap;use std::sync::{Arc, Mutex};use std::time::{Duration, Instant};
/// Raft node state#[derive(Debug, Clone, PartialEq)]enum Role { Follower, Candidate, Leader,}
/// Log entry#[derive(Debug, Clone, Serialize, Deserialize)]struct LogEntry { term: u64, index: u64, command: String,}
/// RequestVote RPC#[derive(Debug, Serialize, Deserialize)]struct RequestVoteArgs { term: u64, candidate_id: u64, last_log_index: u64, last_log_term: u64,}
#[derive(Debug, Serialize, Deserialize)]struct RequestVoteReply { term: u64, vote_granted: bool,}
/// AppendEntries RPC#[derive(Debug, Serialize, Deserialize)]struct AppendEntriesArgs { term: u64, leader_id: u64, prev_log_index: u64, prev_log_term: u64, entries: Vec<LogEntry>, leader_commit: u64,}
#[derive(Debug, Serialize, Deserialize)]struct AppendEntriesReply { term: u64, success: bool, conflict_index: Option<u64>,}
/// Raft nodestruct RaftNode { id: u64, peers: Vec<u64>,
// Persistent state current_term: u64, voted_for: Option<u64>, log: Vec<LogEntry>,
// Volatile state commit_index: u64, last_applied: u64, role: Role,
// Leader-specific next_index: HashMap<u64, u64>, match_index: HashMap<u64, u64>,
// Election timing last_heartbeat: Instant, election_timeout: Duration,}
impl RaftNode { fn new(id: u64, peers: Vec<u64>) -> Self { RaftNode { id, peers, current_term: 0, voted_for: None, log: vec![], commit_index: 0, last_applied: 0, role: Role::Follower, next_index: HashMap::new(), match_index: HashMap::new(), last_heartbeat: Instant::now(), election_timeout: Duration::from_millis(150), } }
/// Handle RequestVote RPC fn handle_request_vote( &mut self, args: RequestVoteArgs, ) -> RequestVoteReply { let mut vote_granted = false;
if args.term > self.current_term { self.current_term = args.term; self.role = Role::Follower; self.voted_for = None; }
if args.term == self.current_term { let can_vote = self.voted_for.is_none() || self.voted_for == Some(args.candidate_id);
let log_ok = self.is_log_up_to_date( args.last_log_term, args.last_log_index, );
if can_vote && log_ok { self.voted_for = Some(args.candidate_id); vote_granted = true; self.last_heartbeat = Instant::now(); } }
RequestVoteReply { term: self.current_term, vote_granted, } }
/// Check if candidate's log is at least as up-to-date fn is_log_up_to_date( &self, last_term: u64, last_index: u64, ) -> bool { let (my_last_term, my_last_index) = self.last_log_info();
if last_term != my_last_term { last_term > my_last_term } else { last_index >= my_last_index } }
fn last_log_info(&self) -> (u64, u64) { if let Some(last) = self.log.last() { (last.term, last.index) } else { (0, 0) } }
/// Handle AppendEntries RPC fn handle_append_entries( &mut self, args: AppendEntriesArgs, ) -> AppendEntriesReply { if args.term > self.current_term { self.current_term = args.term; self.role = Role::Follower; self.voted_for = None; }
let mut success = false; let mut conflict_index = None;
if args.term == self.current_term { self.role = Role::Follower; self.last_heartbeat = Instant::now();
// Check log consistency if self.log_matches(args.prev_log_index, args.prev_log_term) { success = true;
// Append new entries let mut index = args.prev_log_index; for entry in args.entries { index += 1; if let Some(existing) = self.log.get_mut(index as usize) { if existing.term != entry.term { // Conflict: delete from here onward self.log.truncate(index as usize); self.log.push(entry.clone()); } } else { self.log.push(entry.clone()); } }
// Update commit index if args.leader_commit > self.commit_index { self.commit_index = std::cmp::min( args.leader_commit, self.log.len() as u64, ); } } else { // Find conflict index conflict_index = Some(args.prev_log_index); } }
AppendEntriesReply { term: self.current_term, success, conflict_index, } }
fn log_matches(&self, index: u64, term: u64) -> bool { if index == 0 { return true; }
if let Some(entry) = self.log.get(index as usize - 1) { entry.term == term } else { false } }
/// Start leader election fn start_election(&mut self) { self.current_term += 1; self.role = Role::Candidate; self.voted_for = Some(self.id); self.last_heartbeat = Instant::now();
let (last_log_term, last_log_index) = self.last_log_info();
let args = RequestVoteArgs { term: self.current_term, candidate_id: self.id, last_log_index, last_log_term, };
// Send RequestVote RPCs to all peers // (In real implementation, send via network) println!( "Node {} starting election for term {}", self.id, self.current_term ); }
/// Become leader fn become_leader(&mut self) { self.role = Role::Leader;
// Initialize leader state let next_index = self.log.len() as u64 + 1; for peer in &self.peers { self.next_index.insert(*peer, next_index); self.match_index.insert(*peer, 0); }
println!("Node {} became leader for term {}", self.id, self.current_term); }
/// Check if election timeout elapsed fn election_timeout_elapsed(&self) -> bool { self.last_heartbeat.elapsed() > self.election_timeout }}
/// Simplified Raft example usageasync fn raft_example() { let mut node = RaftNode::new(1, vec![2, 3, 4, 5]);
// Simulate election timeout node.start_election();
// Simulate receiving votes (in real system, via network) let mut votes = 1; // Self-vote
if votes > node.peers.len() / 2 { node.become_leader(); }}TypeScript: Basic Paxos (Conceptual)
/** * Paxos proposer */class Proposer { private proposalNumber = 0; private highestAccepted: { number: number; value: any; } | null = null;
constructor( private id: number, private acceptors: Acceptor[] ) {}
async propose(value: any): Promise<boolean> { // Phase 1: Prepare this.proposalNumber++; const promisesreceived = await this.sendPrepare( this.proposalNumber );
const promises = promisesreceived.filter(p => p !== null);
// Need majority if (promises.length < Math.floor(this.acceptors.length / 2) + 1) { return false; // Failed to get quorum }
// Find highest accepted value let valueToPropose = value; for (const promise of promises) { if ( promise.acceptedNumber !== null && (this.highestAccepted === null || promise.acceptedNumber > this.highestAccepted.number) ) { this.highestAccepted = { number: promise.acceptedNumber, value: promise.acceptedValue, }; valueToPropose = promise.acceptedValue; } }
// Phase 2: Accept const accepted = await this.sendAccept( this.proposalNumber, valueToPropose );
const acceptCount = accepted.filter(a => a).length;
// Need majority return acceptCount >= Math.floor(this.acceptors.length / 2) + 1; }
private async sendPrepare( n: number ): Promise<PromiseResponse[]> { return Promise.all( this.acceptors.map(acceptor => acceptor.receivePrepare(n).catch(() => null) ) ); }
private async sendAccept( n: number, value: any ): Promise<boolean[]> { return Promise.all( this.acceptors.map(acceptor => acceptor.receiveAccept(n, value).catch(() => false) ) ); }}
/** * Paxos acceptor */class Acceptor { private promisedNumber: number | null = null; private acceptedNumber: number | null = null; private acceptedValue: any = null;
constructor(private id: number) {}
async receivePrepare(n: number): Promise<PromiseResponse> { if (this.promisedNumber === null || n > this.promisedNumber) { this.promisedNumber = n; return { promised: true, acceptedNumber: this.acceptedNumber, acceptedValue: this.acceptedValue, }; }
return { promised: false, acceptedNumber: null, acceptedValue: null }; }
async receiveAccept(n: number, value: any): Promise<boolean> { if (this.promisedNumber === null || n >= this.promisedNumber) { this.promisedNumber = n; this.acceptedNumber = n; this.acceptedValue = value; return true; // Accepted }
return false; // Rejected }
getAcceptedValue(): any { return this.acceptedValue; }}
interface PromiseResponse { promised: boolean; acceptedNumber: number | null; acceptedValue: any;}
/** * Example: Run Paxos */async function runPaxos() { // Create 5 acceptors const acceptors = [ new Acceptor(1), new Acceptor(2), new Acceptor(3), new Acceptor(4), new Acceptor(5), ];
// Create proposer const proposer = new Proposer(1, acceptors);
// Propose value const success = await proposer.propose('my-value');
if (success) { console.log('Consensus reached!'); } else { console.log('Failed to reach consensus'); }}PHP: Simple Consensus Coordinator
<?php
namespace Consensus;
/** * Simple leader election using database (similar to ZooKeeper) */class LeaderElection{ private string $lockKey = 'leader_lock'; private int $leaseSeconds = 10;
public function tryBecomeLeader(string $nodeId): bool { // Try to acquire lock using Redis $acquired = Redis::set( $this->lockKey, $nodeId, 'NX', // Only set if not exists 'EX', // Expiration $this->leaseSeconds );
return (bool) $acquired; }
public function renewLease(string $nodeId): bool { $currentLeader = Redis::get($this->lockKey);
if ($currentLeader === $nodeId) { Redis::expire($this->lockKey, $this->leaseSeconds); return true; }
return false; }
public function isLeader(string $nodeId): bool { return Redis::get($this->lockKey) === $nodeId; }
public function getLeader(): ?string { return Redis::get($this->lockKey) ?: null; }}
/** * Quorum-based configuration store */class QuorumStore{ private array $nodes; private int $quorumSize;
public function __construct(array $nodes) { $this->nodes = $nodes; $this->quorumSize = (int) floor(count($nodes) / 2) + 1; }
public function write(string $key, mixed $value): bool { $successes = 0; $version = time(); // Simple versioning
foreach ($this->nodes as $node) { if ($this->writeToNode($node, $key, $value, $version)) { $successes++; }
if ($successes >= $this->quorumSize) { return true; // Quorum reached } }
return false; }
public function read(string $key): mixed { $responses = [];
foreach ($this->nodes as $node) { $response = $this->readFromNode($node, $key); if ($response !== null) { $responses[] = $response; }
if (count($responses) >= $this->quorumSize) { break; } }
if (count($responses) < $this->quorumSize) { throw new \Exception('Failed to reach quorum for read'); }
// Return value with highest version usort($responses, fn($a, $b) => $b['version'] <=> $a['version']);
return $responses[0]['value']; }
private function writeToNode( string $node, string $key, mixed $value, int $version ): bool { // Simulate network write to node try { $response = Http::post("http://$node/write", [ 'key' => $key, 'value' => $value, 'version' => $version, ]);
return $response->successful(); } catch (\Exception $e) { return false; } }
private function readFromNode(string $node, string $key): ?array { try { $response = Http::get("http://$node/read", [ 'key' => $key, ]);
if ($response->successful()) { return $response->json(); } } catch (\Exception $e) { // Node unavailable }
return null; }}Integration Points
With Consistency Models
- Consensus provides linearizability: Agreed value visible to all
- Log replication: Consensus on operation order → consistent state
- Quorum intersection: Foundation for both consensus and consistency
With Actor Model
- Leader election: Select coordinator actor in actor system
- Distributed actors: Consensus for cross-node actor coordination
- Saga coordination: Consensus on distributed transaction outcome
With Streams
- Kafka: Log replication via quorum-based replication (Raft-like)
- Stream partitioning: Consensus on partition assignment
- Watermarks: Consensus on progress in distributed stream processing
Further Reading
Papers
- Fischer, Lynch, Paterson (1985) - “Impossibility of Distributed Consensus with One Faulty Process”
- Lamport (1998) - “The Part-Time Parliament” (Paxos)
- Ongaro & Ousterhout (2014) - “In Search of an Understandable Consensus Algorithm” (Raft)
- Castro & Liskov (1999) - “Practical Byzantine Fault Tolerance”
Books
- Van Steen & Tanenbaum - “Distributed Systems” (3rd ed.)
- Kleppmann - “Designing Data-Intensive Applications”
End of Consensus Appendix