TorrentShare
Peer-assisted artifact distribution for GPU fleets and MLOps pipelines
When 50 nodes pull the same 70 GB model at once, TorrentShare fetches the bytes once, verifies them chunk by chunk, and spreads them peer to peer across the VPC. It runs on its own BitTorrent engine: bencode, trackers, a Kademlia DHT and the peer wire protocol, written from scratch. Formerly TorrentEdge.
- Native BitTorrent engine
- Redis fencing tokens
- SHA-256 CAS dedup
- 462 Jest tests
§1
The problem
A GPU fleet pulling the same large artifact at the same time saturates NAT gateways and VPC endpoints, multiplies egress cost, and leaves every job waiting on the slowest download.
§2
My role
Design and implementation of the control plane, the worker data plane, the lease protocol and the observability stack.
§3
Architecture
§4
Key decisions
D/01
Own the protocol
Instead of wrapping a torrent library, the engine implements bencode, .torrent parsing and creation, magnet links, tracker announces, a Kademlia DHT routing table, the peer wire and extension protocols, and piece selection, so every byte path can be measured and controlled.
D/02
Separate control and data planes
A stateless REST control plane owns orchestration with PostgreSQL as the ACID source of truth; horizontally scalable workers do the heavy lifting.
D/03
Crash-only workers with leases
Redis SETNX leases with fencing tokens let a replacement worker resume an abandoned transfer without split-brain writes.
D/04
Content-addressable storage
Verified chunks are stored by SHA-256 digest and materialised with symlinks, so identical data is deduplicated across transfers.
D/05
Kafka as the directive bus
The API publishes JOB_ASSIGNED directives to partitioned consumers, so API latency is independent of transfer work.
D/06
Retry-safe by design
X-Request-ID flows from REST ingress through Kafka metadata, so a CI/CD retry is recognised as a retry, not a new deployment.
§5
Challenges
- Stopping two workers writing the same chunk after a partition: fencing tokens let storage reject a stale lease holder.
- Tracing across an async boundary: W3C trace context is carried through Kafka headers into workers and CAS writes.
- Prometheus metrics for queue depth, lease activity, CAS hit rate and deduplicated bytes, with Grafana dashboards.
- Downloads survive restarts: piece state is checkpointed and resumed, and upload and download are throttled per transfer.
- Moved from MongoDB to PostgreSQL (Sequelize) for transfers, chunks and nodes, because chunk ownership and lease state need transactions.
§6
In action
§7
Stack
- Node.js
- Kafka
- PostgreSQL
- Redis
- Docker
- Prometheus
- OpenTelemetry
- Nginx
Next case study
JobApp
A semi-automated job-application pipeline with an AI résumé tailor