ADR 1: Peer-to-peer protocol
Date: 2024-06-25
Last updated: 2025-05-19
Status: Implementing
2024-07 decision to use Willow protocol with Earthstar. My implementation attempt revealed that Earthstar lacked peer-to-peer discovery and connectivity.
2025-02 decision to use NextGraph after alpha version addressed the functionality gaps originally documented.
If you are evaluating peer-to-peer protocols for your project, I recommend using https://www.localfirst.fm/landscape to compare your product requirements to the self-reported capability claims of various P2P options.
Context: The method for encrypted data distribution needs to be decided.
Decision: NextGraph
Consequences: The backend architecture influences the user experience for onboarding and connecting with friends, the timeliness of updates, and overall trustworthiness from non-functional requirements.
Impressions
NextGraph and Willow with Earthstar most closely match the requirements with actively maintained specifications and libraries. Only NextGraph provides a P2P solution for intermittently online peers (source). The Veilid, Freenet, and Cable protocols are the next most interesting options for Bana’s use case.
See prior post for more information on the use case.
Capabilities
| P2P option | In browser | Sync | Deletes | Access control | Score |
|---|---|---|---|---|---|
| Cable protocol | 🟢 | ✅ | ✅ | 🟡 | 3 |
| Freenet protocol | 🟢 | 🟡 | 🟡 | 🟡 | 2.5 |
| Interpeer | ❌ | – | – | – | 0 |
| NextGraph | ✅ | ✅ | ✅ | 🟢 | 3.5 |
| ODD.dev by Fission | ✅ | ✅ | ✅ | 🟢 | 0* |
| p2panda | 🟡 | 🟢 | 🟢 | 🟢 | 2 |
| Pear with Hyperdrive by Holepunch | ❌ | 🟡 | 🟡 | – | 0 |
| PZP (PPPPP by Manyverse successor) | ❌ | – | 🟡 | – | 0.5 |
| Secure Scuttlebutt | ✅ | ❌ | ❌ | – | 1 |
| Spritely | 🤷 | 🤷 | 🤷 | 🤷 | ? |
| Veilid | 🟢 | 🟡 | 🟡 | 🟡 | 2.5 |
| Willow with Earthstar | ✅ | 🟡 | ✅ | ✅ | 3 |
Key
- ✅ acceptable
- 🟡 partially meets requirements
- 🟢 will meet requirements in future
- ❌ does not meet requirements
- ? uncertain if meet requirements
- – not evaluated
- 🤷 so confusing I had no idea what I was reading
In browser requirements
- Web native: runs in an unmodified web browser
Notes
- Cable protocol: “cabal-web coming soon” (source)
- Freenet: yes, but requires 5 MB bootstrap
- Interpeer: not enough code to fully evaluate, but documentation does not suggest use within browser
- NextGraph: Web JS SDK for Svelte planned (source)
- p2panda: client applications require an aquadoggo server that can run on device or as community infrastructure (source)
- PZP: does not seem to run in-browser
- Pear: requires Bare, a Node.js-like runtime
- Veilid: WebRTC is on the roadmap, but not yet implemented (source)
- Willow: “Works in the browser.” (source)
Sync requirements
- Selective: not all the data must be synced
- Arbitrary: data can be requested as needed
- Relevant: device only hosts data and connectivity relevant to the user’s use of the app
- Resilient: data owner can backup data on multiple devices
- Relayed: peers can connect when intermittently online
Notes
- Freenet: data syncing is an app-level concern, but technically possible to implement. No way to restrict connectivity participation to only fellow users of the app.
- NextGraph: supports data syncing requirements (source) with brokers for intermittently online peers (source)
- p2panda: part of namakemono in development
- Pear: Hypercore supports a sparse mode (source 3:31) for accessing the append-only log, but requires creating own negotiation protocol.
- PZP: PPPPP planned support for requirements, but not yet released
- Veilid: Likely possible to do selective and arbitrary access, but would be app-level concern to implement. Apps can use the public Veilid network or build their own to only host data and connectivity relevant to the use case (source slide 13)
- Willow with Earthstar: all supported by specs (source, source)
Delete requirements
- Pruned: the content deleted cannot be retrieved by examining the edit history. Data should eventually leave no trace when deleted.
Notes
- Freenet: app-level concern, but technically possible to implement.
- NextGraph: based on CRDTs. Tombstones leave lingering reference, but conflict resolution can be customized to treat as deletion (source, source)
- p2panda: part of namakemono in development (source)
- Pear: Hyperdrive dependency on Hyperbee dependency on Hypercore is immutable, append-only log. Data can be removed, but its presence without metadata is kept forever. (source)
- PZP: Data is deleted, but metadata and hash remain forever. (source) PPPPP planned support for requirements, but never released (source)
- Secure Scuttlebutt: immutable, “cannot change anything that has been published” (source)
- Veilid: app-level concern, but technically possible to implement.
- Willow: supported by spec (source)
Access control requirements
- Discretionary: data owner controls who can access, create, update, delete
- (Optional) Assistive: group members can distribute data to each other when owner offline
Notes
- Cable protocol: access control enforced by creating dedicated chat channels and data owner using moderation controls. Supports assistive requirement.
- Freenet: app-level concern, but technically possible to implement.
- NextGraph: source
- ODD.dev: WNFS allows private sharing by creating a symlink to private data for each user. Private data shares are read-only. (source) Data is stored on IPFS, which allows for peer sharing. Unsure if or how private sharing affects offline peer access.
- p2panda: support for requirements planned (source)
- Veilid: app-level concern, but technically possible to implement.
- Willow: all supported by spec (source)
Additional considerations
Connectivity
- Veilid’s improved Tor-like hopping is very attractive.
- Needs more review for Willow.
- 2024-06-26 update: Willow has intentionally punted on this and said ‘however you can make a connection, go for it!’. Earthstar intends to have a few methods for establishing connections, but only server connections are currently implemented. It is possible to do WebRTC signalling to enable direct peer connections, but is not yet implemented. (source)
Threat models
- Needs more review for all of them
Momentum
- NextGraph: alpha release, continued funding from NLNet
- ODD.dev: One of the strongest potential options, but the company responsible for its primary development ceased operation in May 2024 (source)
- Secure Scuttlebutt and PPPPP: The lead developer of the successor to Secure Scuttlebutt protocol, that would allow for selective sync and deletes (source) moved on from the Manyverse project in April 2024 (source)
- PZP is carrying on PPPPP’s work (source)
- Veilid: alpha release
- Willow: Rust implementation in production use by Iroh. Earthstar implementations (Willow JS, Meadowcap JS) have active development.
Stack biases
Preferences: TypeScript, Rust, JSON, RESTful API
Disinclination: GraphQL, Ruby, Java, C#
- Freenet: Rust & WASM
- NextGraph: Rust & WASM, SPARQL, RDF
- p2panda: GraphQL
- Willow: TypeScript, Rust
Notes
- Protocol comparisons, document I wish every one of these protocols wrote for themselves:
- Cable protocol introduction 2023 video
- Freenet application architecture overview video
- More peer-to-peer technologies I was aware of, but did not fully evaluate:
- Braid with Antimatter history pruning CRDT/OT algorithm: interesting for state synchronization, but does not solve peer-to-peer connectivity
- LoFiRe: seems dormant (2 years of no activity)
- Yggdrasil: does not work offline
Post preview image features the a photo by Conny Schneider on Unsplash.
Comments?
I likely misunderstood something. Please send me your questions, clarifications, corrections, or anything else you think might be helpful.
You can respond to this post on the social web or email me.
