Biblion is a platform for creating self-defining peer-to-peer data communities. Peers within Biblion participate in a global content-addressable file sharing network. Users can share files and create communities called "libraries". Libraries are defined by a definition file that describes and configures the services provided by that community. Services can be those included with the Biblion distribution, or be defined by completely custom code. The protocols provided by a library can include incentivization systems that encourage user participation and help keep files alive. Unlike many recent distributed information systems, Biblion intentionally supports the notion of "trusting" particular peers to provide a protected service, within a library. At the low-level of Biblion is a modern networking stack that provides peer authentication and stream multiplexing.
Biblion is a platform for peer-to-peer networks built from the ground up. Taking inspiration from BitTorrent, IPFS, and various HTTP-based web applications, Biblion attempts to be a holistic solution to decentralized information hosting. Specifically, Biblion is designed for storing static immutable content. The content can be in any form, including HTML documents, PDFs, plaintext, video files, and anything else that can stored in a static file.
A peer-to-peer network is a network of computers in which participants communicate directly with each other. Unlike client-server applications, nodes in these networks are able to both service and make requests. Peer-to-peer networks can depend on the existence of centralized services that run on the side, such as "trackers" in many BitTorrent deployments, which are used to help peers find each other. File sharing is one of the most popular uses for peer-to-peer networks, since every peer can contribute pieces of their storage to participate. This has been the basis of many systems, including BitTorrent, Gnutella, IPFS, and many more. Biblion is a peer-to-peer file sharing network that attempts to provide the convenience and effectiveness of centralized services with the resilience and flexibility of decentralized systems.
At the heart of a Biblion is a content-addressable file store. Files are identified by their SHA256 hash. Given a hash, the Biblion software can find the file and download it locally. Each Biblion node maintains an offline store of file objects that caches Biblion objects. Because files are identified by a strong hash function, they are immutable. Every file in a file store has its integrity verified by validating the hash.
To support effective downloads, peers communicate using a BitTorrent-like protocol. A download occurs across multiple peers, and can work around slow peers over time.
Peers find each other by paricipating in global Kademlia-like distributed hash table (DHT). Nodes on this DHT are identified by their public key, which is in turn used to verify the node's identity during a connection.
As described, this system already provides a scalable global solution for distributing files across the internet, providing integrity protection for multiple files. The system described thusfar is nearly equivalent to the Mainline DHT used in BitTorrent, which allows clients to find peers for a download without the use of a centralized tracker. It is also extremely similar to IPFS, but is missing a piece. What we need is a solution to the "free-rider problem".
In peer-to-peer networking, the free-rider problem refers to the issue of preventing malicious nodes from taking advantage of good natured nodes in order to use the system unfairly. In a file sharing community, this generally means that "free rider" is somehow who tries to download data without offering data to upload in return. Preventing this activity on a massive scale is difficult to solve effectively, because reputation stores either require trusted participation or slow consensus mechanisms.
Biblion solves the free-rider problem by splitting the network into a potentially infinite number of heterogenous small networks that define their own solutions. While a global trusted reputation system is undesirable for the global network, such a solution can work well within the context of a smaller network. These small networks are created and defined by users themselves, and can take any form. These smaller networks are called "libraries".
A library is a peer-to-peer community defined within Biblion. Like Biblion itself, a library can be built using DHT based peer discovery, and BitTorrent style downloads. Because these smaller networks are using the same technology, the same programs are repurposed for use in both the Biblion network and the libraries. Unlike the global Biblion network, libraries can define authorization strategies for limiting activity within the network. For example, a library may maintain a record of the upload and download history of a peer, and the decision to allow a certain download would be based off of that history.
To provide these advanced services, libraries have centralized services and trusted peers. The trust and power of these peers is limited strictly within the context of a library. When a peer joins a library, they are aware of what the trust model is, and where the power within the library lies.
A library is a set of services and trust relationships to those services. In a library's internal DHT, membership can be limited by a peer whitelist. For peers participating in the DHT, they will only honor requests or add nodes that are in the whitelist. Because each node participating in this library agrees to enforce the same whitelist, security is roughly preserved. The DHT in this example is immune to corruption as long as the nodes in the whitelist are secure. Meanwhile, data leakage is possible if any singular peer has been compromised.
Libraries can also define how files and data are located within a network. A DHT is effective and robust, but can have poor performance characteristics. Instead, a library can opt to host a set of mirrors that index *all* data within the library. This puts more stress on the indexes, but is much easier to manage. We imagine that multiple clever routing strategies inspired by distributed database design will be usable.
The definitions of these files are defined in JSON documents, that specify the nature of the service, its context, and what authorization is necessary. Because the specification is itself a file, library definitions are objects within the Biblion system just like content itself.
Authorization is an important concept within Biblion. The simplest form of authorization is a user whitelist. All identities in Biblion are verified, meaning we can be confident that a user is authorized to perform a certain action or access a certain piece of information. More advanced forms of authorization are accomplished by using a payments metaphor. Users are only able to perform a certain action if they have the required balance of a virtual currency, which is spent as the download occurs. This mirrors real world economies, and especially cryptocurrencies. Unlike most popular cryptocurrencies, balances within Biblion can explicitly be centrally managed. That is, a trusted party will maintain the balance sheets and verify that payments are settled correctly.
We are finally ready to offer a real solution to the free-rider problem. Within a library, trusted nodes can run a bank that manages balances of a local currency for use within the library. A user "spends" her tokens while downloading data from another peer within the library. This ensures that a user can only download as much as she uploads. Banks can allocate initial funds to users when they join a library, or generate new money in "freeleech" events: periods of time where downloads are free but uploaders are paid regardless. There are many strategies banks can use to increase the money supply while incentivizing good work.
Because the libraries are specified customly using defintions, libraries can come up with various competing strategies for encouraging community support. Because of the community driven nature of Biblion, we intend for the network to store content metadata as well as the data itself. BitTorrent and Usenet indexes provide huge value for users looking for particular files, but are prone to attack and censorship. By making the metadata indexes files in the system as well, the sites and communities are much more robust.
They are not invincible, however. Censorship takes the form of eliminating or corrupting the leadership who have special permissions. In these events, users can take the data associated with the community - which they have been hosting cooperatively - and rally around new leadership. In the event of corruption of a bank, this will mean that the bank and its token will have to be abandonned, as transactions will no longer be processed. However, if the bank has a transparent balance sheet, the balances *could* be moved to new leadership.
Such events would surely have disastrous effects on the local currency. Even if the bank can recover the funds and adopt new leadership, there will be a confidence crisis, and the currency's value with respect to others will drop. These banks are also extremely vulnerable to temporary corruption. That is, library administrators could temporarily distribute funds illicitly to each other in an effort to exploit some market. We can avoid these situations by creating a bank with no leader: a proof-of-work blockchain. Blockchains allow us to update the funds as long as over half of the hash power committed to the chain is running correctly. This has proved extremely effective in practice, at the price of huge natural resources and performance. We implement a blockchain as a sort of central bank for Biblion. This central blockchain can be used to settle transactions across different local banks in a reliable way. We opt for proof-of-work to start, but hopefully we can switch to a useful proof-of-storage at some point in the future.
Libraries can implement their own blockchains as well. They can also define the criteria by which new blocks are accepted. This means a blockchain can be defined that uses a small set of hashers, or a small set of signatories who are the only ones allowed to update the banking records. This can then be used with blockchain consensus to maintain the state of user balances. There are several weaknesses in systems like this (bootstrap integrity being the most pressing), but they offer effective solutions to transparently managing payments in a performant way.
Libraries can define their own currencies. They can define how those currencies are spent. They can define what services are active within the library. The combination of these things means that libraries can collaboratively provide services of value to others. That is, a library defines a service which others can pay to use. Peers from outside the library could buy some of the currency local to the library and spend it on services within the library, and then sell the remainder of their currency for other things. Peers can invest in each other's libraries by buying the currency in anticipation of the value going up. Libraries can also define alternative services for payments. For example, a library could provide a VPN service that has exit points at various places worldwide, for navigating around firewalls. Other applications include video transcoding, proxying, anonymization, TURN relays, and much more. Because libraries can define arbitrary services, Biblion creates a market for collaborative services that reward communities that can provide the strongest service for the lowest price. These concepts are heavily inspired by the ideas of "agoric computing", but do not require rigid smart contracts. Rather, much like reality, the cost and rewards for services will be determined by the market, encouraging constant competition and growth.
Peers within Biblion may be members of dozens or hundreds of libraries. To have this system work effectively, peers reuse connections for all services. Services are multiplexed over the whole connection using a SPDY-like protocol. Every peer connection is authenticated when it's established. This ensures that each peer knows exactly who it is speaking with. Because services share code and connections, they can be implemented very effectively. Each service on a connection is simply an extra piece of state, which is only activated if a stream arrives that requires that particular service.
Taken together, we create a robust yet flexible file sharing network. Files can be transferred quickly and with integrity. Libraries can define incentivization structures that benefit all members. Members across libraries can collaborate by trading tokens within a global market. The community can update the code and services over time, ensuring that Biblion can evolve well into the future.
There has recently been a huge amount of development in decentralized systems. In the wake of the NSA revelations, people are more concerned about how to make data more resilient, more open, and more private. Biblion takes inspiration from both older systems and recent developments.
BitTorrent is a peer-to-peer file distribution protocol. Peers upload the content they have while they are in the process of downloading. By favoring rare pieces and choosing to connect with peers with strong connectivity, BitTorrent succeeds in providing extremely fast file distribution. Pieces are verified as they arrive, ensuring data integrity. Peer discovery was originally dependent on central services called trackers. With the development of Mainline DHT, peers can find each other by participating in a collaborative global routing table. The success of BitTorrent and Mainline DHT is a major influence on Biblion.
MojoNation, and its successor Mnet, were filesharing networks that drew upon the ideas of agorics. Users could "trade" resources amongst each other. There was a currency in place to allow users to set prices.
IPFS is the Interplanetary File System. Filecoin is a companion project to IPFS, and aims to incentivize storage on the system. These systems also draw heavy inspiration from BitTorrent and DHTs. Filecoin actively encourages research in useful proofs-of-storage, which could be extremely useful to Biblion in the future.
Bitcoin is the original blockchain currency. The primary innovation of Bitcoin is to distribute the "trust" of the network among all peers. There is no central authority that controls Bitcoin, as long as the majority of the network acts independently. Bitcoin mints coins autonomously.
Tahoe-LAFS - the "least authority file system". Tahoe stores data encrypted across various servers. It's data distribution model is based on the principle of "least authority". Each server is only able to access data that it absolutely requires. Decryption of user content happens on a trusted node, while untrusted nodes only ever see encrypted data. The model of least authority is important in cooperative peer-to-peer systems, since there are only weak links of trust among nodes.
Biblion services are built within a particular system model. In some sense, the Biblion platform is much like an operating system, It provides a consistent API for all services to use, and exposes the same primitives. Services can expect the APIs to work the same way in any Biblion implementation.
Identities represent a user within the network. An identity is tied to a assymetric key pair. It is of upmost importance that the private key of an identity is kept secure. If stolen, the thief will be able to impersonate that identity forever. A node can run multiple with multiple identities. However, it should be noted that this alone is not a true anonymization strategy: While you can create different identities to participate in certain services, your computer will advertise the same IP for each identity. Identities run services, and can be members of libraries. Running multiple identities in the same library will result in multiple service instances running.
A peer is connectable identity. Effectively, it is a tuple of an identity's public key and an object describing how to connect to it. An identity keeps tracks of its peers as it runs its services. Keeping records of peers is crucial for routing, as we need a record of who we know about, and how to connect to them. Each identity should keep track of its peers separately. Peers that aren't used or active can be cycled out of the peer store. Peers represent additional actors that can communicate in services. Within a service, all communication is done between peers.
A stream represents a communication between two peers. Streams Transports Services
Peers to need to be able to connect to each other directly.
- Multiplexing of requests, over multiplexed transports - Only one connection, per (peer, transport) - Reliable and unreliable streams - QUIC (planned) - Authentication, TLS - TLS 1.3 (planned) - UDP auth cookiesThe following services should be a part of any Biblion distribution.
Kademlia - A distributed hash table to be used in peer discovery. Gossip
SimpleDownload - An FTP-like download protocol. Useful for transferring small files.
BitTorrent -
Blockchain
SimpleBank - A simple implementation of a "bank": a service that manages balances for a set of users. The bank can distribute funds as it sees fit. Users can spend their funds to each other. They can also allocate funds to a user in pre-authorizations.
Routers
Downloaders
Metadata Sets
Authorization
Defining services
Ownership and authentication
Managing users
Managing metadata
Permissions
We define a global library that all peers should participate in. This library is limited in its features and power, but is extremely resilient to attack. It includes a central blockchain, and a global DHT. The global library can be used to host specifications for individual libraries.
Central blockchain
PoW-secured DHT
Blockchain temporary announcements
Network hardening (QUIC)
Additional configuration rules for libraries
Transparent banking
Biblion 2.0 (library configurations should be able to reference arbitrary code)
Community types and incentive strategies
Completely private/disconnected libraries
Proof of storage
Integration with other services
WASM-based service bundles
IPFS, libp2p
Beaker browser