mod chunker;mod parallel_hash;
use parallel_hash::{hash_buffer, root_hash};
const CHUNK_SIZE: usize = 64 * 1024;
fn main() {
let buffer: Vec<u8> = (0..(5 * 1024 * 1024))
.map(|i| (i % 251) as u8)
.collect();
let chunks = hash_buffer(&buffer, CHUNK_SIZE);
println!("buffer = {} bytes in {} chunks", buffer.len(), chunks.len());
for chunk in chunks.iter().take(3) {
println!(
" offset={:>8} len={:>6} {}",
chunk.offset,
chunk.len,
chunk.hash.to_hex()
);
}
let root = root_hash(&chunks);
println!("root = {}", root.to_hex());
}
use blake3::Hash;
#[derive(Debug, Clone)]
pub struct Chunk {
pub offset: usize,
pub len: usize,
pub hash: Hash,
}
pub fn chunk_ranges(total_len: usize, chunk_size: usize) -> Vec<(usize, usize)> {
assert!(chunk_size > 0, "chunk_size must be non-zero");
let mut ranges = Vec::with_capacity(total_len / chunk_size + 1);
let mut offset = 0;
while offset < total_len {
let end = (offset + chunk_size).min(total_len);
ranges.push((offset, end - offset));
offset = end;
}
ranges
}
use blake3::{Hash, Hasher};
use rayon::prelude::*;
use crate::chunker::{chunk_ranges, Chunk};
pub fn hash_buffer(buf: &[u8], chunk_size: usize) -> Vec<Chunk> {
let ranges = chunk_ranges(buf.len(), chunk_size);
ranges
.into_par_iter()
.map(|(offset, len)| {
let slice = &buf[offset..offset + len];
Chunk {
offset,
len,
hash: blake3::hash(slice),
}
})
.collect()
}
pub fn root_hash(chunks: &[Chunk]) -> Hash {
let mut hasher = Hasher::new();
for chunk in chunks {
hasher.update(chunk.hash.as_bytes());
}
hasher.finalize()
}
This snippet shows how a large in-memory byte buffer is split into fixed-size chunks and hashed independently, then combined into a stable content manifest suitable for deduplication or integrity checks. The design separates how the buffer is partitioned from how each partition is hashed, which is what makes the parallelism safe: each chunk is a disjoint, read-only slice of the original buffer, so there is no shared mutable state and Rayon can fan the work across a thread pool without locks.
In chunker.rs, the Chunk struct records an offset, the chunk len, and its hash, so the position of every chunk is preserved even though hashing happens out of order. The chunk_ranges function is a plain iterator over (offset, len) pairs computed from the buffer length and a target chunk_size; keeping range computation pure and cheap means the expensive hashing step can be expressed cleanly on top of it. The final chunk is handled by min, which clamps the last range to the true buffer end so trailing bytes are never dropped or over-read.
In parallel_hash.rs, the core work happens in hash_buffer. It borrows the buffer as &[u8], materializes the ranges, and switches into Rayon with into_par_iter. Each parallel task slices buf[offset..offset + len] and feeds it to blake3::hash, producing an independent Chunk. Because slices borrow immutably, Rust's borrow checker statically guarantees the absence of data races — the parallel closure never needs a Mutex. The results are collected with collect, and Rayon preserves input order, so the returned Vec<Chunk> is already sorted by offset without an explicit sort.
The root_hash helper folds each chunk hash into a second BLAKE3 hasher to derive a single manifest digest, giving a Merkle-style summary of the whole buffer. A key trade-off is chunk granularity: small chunks improve deduplication and parallel balance but add per-chunk hashing overhead, while large chunks reduce overhead but hurt load balancing. main.rs wires it together, printing per-chunk digests and the root, and demonstrates that identical content yields identical root_hash regardless of thread scheduling. This pattern is a good fit for backup systems, content-addressed storage, and any workload where a big buffer must be verified or deduplicated quickly.
Related snips
struct Config<'a> {
name: &'a str,
value: &'a str,
}
fn parse_config(line: &str) -> Config {Lifetime annotations for flexible borrowing in structs
require "csv"
class PeopleCsvStream
include Enumerable
HEADERS = %w[id full_name email signed_up_at plan].freezeResilient CSV Export as a Streamed Response
use crossbeam::channel::unbounded;
use std::thread;
fn main() {
let (tx, rx) = unbounded();Crossbeam for advanced concurrent data structures
use tracing::{info, instrument};
#[instrument]
fn process_request(user_id: u64) {
info!(user_id, "Processing request");
// Work happens heretracing for structured logging and distributed tracing
Rails.application.configure do
config.after_initialize do
Bullet.enable = true
Bullet.alert = false
Bullet.bullet_logger = true
Bullet.console = trueN+1 query detection with Bullet gem
use std::sync::mpsc;
use std::thread;
fn main() {
let (tx, rx) = mpsc::channel();Channels (mpsc) for message passing between threads
Share this code
Here's the card — post it anywhere.
