(This is the intro page.  Try the TableOfContents to jump to a particular topic.)

In one sentence:

Persistnet (aka pnet) is a network, datastore, and messaging service.

 

Why?

Many internet services would benefit from being ported to pnet.  Persistnet users can store data, send and receive messages, and share files.  Users need not maintain separate usernames or passwords for various persistnet services, because a persistnet user has an identity (or a pseudonym) through which they access the pnet.

 

Persistnet services can live "in the cloud" - service providers typically need not run any server.  They can write the service, release it into the persistnet, and just start recruiting users!  The data lives on the persistnet, and basic functionality like keyed queries are supported.  The application itself can live in javascript in web pages, in a program the users download (from persistnet or the internet), or in routines that run in persistnet nodes.

 

What?

So much for pie in the sky descriptions of what it can bring you.  Just what is it, really?

 

Persistnet is a public network of computers that run persistnet nodes and provide storage and/or computing power.  If you install persistnet, your computer is part of the network.

 

Any computer can publish data on persistnet.  The data is broken apart and spread across the persistnet in such a way that:

• the loss of any reasonable number of computers from the pnet does not endanger your data
• data can be downloaded quickly, from many computers at once (think bittorrent)
• the data content can be kept secret, with access provided only to certain individuals or groups
• data chunks can be shared so that often the same data found in different places in many different files is shared between those files, saving space
• your data can never be compromised (unless you give someone access, for example by giving them your password or giving them explicit rights to change the data)
• the encryption used by persistnet is *more secure* than the encryption used to protect your bank information

 

(However, see PersistnetCaveats)

 

Pnet also enables the Semantic Web; see The distant future.

 

Where?

Ideally, most internet users and many service providers would run one or more pnet nodes.

 

When?

The first persistnet service (coinciding with the first release of persistnet) will be a free automated backup service.  Install the backup service, create your identity (username and password), and tell it what to back up.  Your data will be persisted automatically, and FREE!

 

Future services will include:

• secure, hostless email (never change your address again)
• javascript libraries for web service developers
• a filesystem you can mount from anywhere
• file sharing
• whatever the pnet community comes up with!

 

You!

• Join the persistnet
• Build a pnet web app
• Build a pnet app
• Join the pnet team

 

How?

How we/you ensure that data stays on the pnet:

Popular data, and data from identities with good reputations, have preference for living in "the cracks" of pnet - in the free storage on pnet nodes which is not storing data the owner is guaranteeing.

Your data can be ensured on pnet by running your own node with sufficient storage, by purchasing storage services as a commodity, or by friends sharing space on their nodes with you.  For backups, the best way is to get your friends on pnet and flag each other as friends, so they help back up your data and vice versa.

 

If you have friends storing your data and not enough of them have their nodes online to restore your data, you can automatically email offline friends to ask them to bring their nodes online (i.e. run pnet) and let you know.

 

If "the cracks" get full, and you don't have an agreement with any node that is online, data can get lost.  The expectation is that this will tend to impact abusers rather than honest users.

 

How pnet does what it does:

Pastry

Persistnet is built on top of Pastry, a Distributed Hash Table (DHT).  Pnet uses the Freepastry implementation of Pastry.

 

DHTs provide a way for many computers to very quickly find one another, and to assign a job across the network to many computers.  The most common job in pnet is storing and retrieving data, and building indexes to help you find the data you're looking for.

 

Freepastry has been modified to use SHA-512 instead of SHA-1 hashcodes. SHA-512 is the most secure hashcode available.  All data in pnet is identified (and located) by its hashcode.  Most data is read-only, so the hash you used to locate the data also validates the data integrity.  Mutable data must be digitally signed to validate its integrity.  On pnet, having the hash is presumed to mean that you have (or have been given) the original data.

 

NUTSS

The pnet uses NUTSS to allow computers that are behind firewalls to talk to one another directly.

 

Erasure codes

To make sure your data survives when a computer on the pnet goes down or leaves the pnet, we place the data on multiple computers.  However, it's not very efficient to just store multiple copies.  With erasure codes, we can store your file on 10 different computers, but split up so that the size of each chunk is only 1/3 the original file size.  The crazy thing is that *any* three of those ten pieces can be used to reconstitute the whole file.  So even if seven of those computers go down before we can back up any of the copies, we can still reconstitute your file.  This is the same way RAID arrays work.

 

Rabin encryption

2048 bit Rabin encryption, mathematically proven to be at least as strong as RSA encryption, is used to verify your identity, encrypt any secret information, and sign statements to prove who made them.  Every pnet identity has a Rabin public/private key.

 

Rabin fingerprints

Large datasets are chopped into pieces before being spread across the pnet.  By the magic of Rabin fingerprints, those pieces tend split so that the same content will end up in a distinct piece no matter which dataset or where in the dataset the content occurs.  This means that as files change, the parts that stay the same can be reused and don't need to be put on the pnet again.  It also means that we don't have to preserve the same chunk of data from many different files in different places.

 

Each piece of data on pnet is encrypted so nodes don't know what data they hold, yet by using convergent encryption, the same encrypted data from multiple sources (see Rabin fingerprints) is encrypted to the same value.  The only way to see the original data is to have the hash of the original data and to know that it goes with the encrypted data.

 

The distant future

We envision a distant future in which most internet traffic goes over the pnet or something like it.

 

Email (messaging), file sharing, and backups are more efficient, more secure, more accessible, and more reliable on pnet.  You aren't locked into one provider for any of these services, because your data remains yours, not the service provider's.

 

Applications built on the pnet scale automatically, because data access on the pnet scales automatically.  Pnet enables the semantic web proposed by Tim Berners-Lee: data provided on the pnet (such as forum comments, items you've posted for sale, who your friends are, etc.) are equally available to all applications.  When someone invents a better sale site, it can display your items for sale (if you give it permission), without having to scrape your data from some other website.