Nostr 2.0: Layer 2 Off-Chain Data Storage.

Syncing Data Across Nostr Relays with Merkle Trees & Light Wallets.

Abstract. Nostr is never going to be able to replace a layer 1 blockchain as a synchronized database or ledger. Although, we can use the bitcoin blockchain to keep Nostr relays in sync. Nostr 2.0 may be able to provide secure off-chain data storage as a Layer 2 atop bitcoin similar to how Lightning provides instant off-chain payments as a Layer 2 atop bitcoin. This paper will elucidate how Nostr relays can synchronize their data while maintaining the lightweight nature of Nostr that lets users optionally delete data, something a layer 1 blockchain cannot provide. It may also be cheaper for users to store tons of data with Nostr relays instead of in the bitcoin blockchain because of the limited capacity and speed of bitcoin blocks. This paper improves the distributed properties of the Nostr network under the standardized criteria known as the CAP theorem. CAP stands for Consistency, Availability, and Partition tolerance.

Consistency means the database synchronized across various computers is identical. Nostr relays cannot synchronize their data in a trust-minimized way, like a blockchain does block by block. Unlike bitcoin full-nodes, the database Nostr relays store is often incomplete. Nostr relays have no means of discovering what data is missing, besides blindly requesting all posts signed by a specific user’s signature.

Nostr’s Consistency/Synchronization Problem: if two users upload their individual posts to different Nostr relays, then the two users might not be able to see each other’s posts because Nostr isn’t like a blockchain. In a blockchain, all the full-nodes keep the blockchain in sync every time there is a new entry. All the full-nodes add that data, in the form of a block, to their blockchain in unison. Every full-node on the bitcoin blockchain has the exact same blockchain.

If we want Nostr users to always be able to see each other’s posts, then all the Nostr relays need a way to identify what data is missing from user profiles so that they can request the missing pieces from other Nostr relays or users.

1. Once a week or so, a user can arrange ALL their posts into a Merkle tree.

2. Each leaf in the Merkle tree contains a hash of a post, just like in bitcoin where each leaf contains a hash of a transaction.

3. Once a user arranges their entire profile into a Merkle tree, they will post the Merkle root on-chain in the OP_Code underneath a normal bitcoin transaction. This is why Nostr 2.0 does not need to hardfork the blockchain to work. The OP_Code is a section underneath all bitcoin transactions that allows for small notes to be attached to transactions before they’re signed by the sender.

4. Additionally, the user will take a hash of the entire tree and upload it on-chain with the Merkle root (in the OP_Code). The Merkle root is only a hash of the top branches, not the entire tree. The entire tree hash is vital to giving users and relays the ability to detect when profile data is missing.

5. To obtain the whole tree hash, simply put the Merkle root at the top of a text file. After that, put the Merkle branches on the lines underneath the root. After that, put the Merkle leaves on the lines underneath the branches. Once the tree is arranged as described, hash all of it at once. An example of what whole tree hashing looks like is seen below — it is a Whole Tree Hash of the merkle tree seen above.

The Merkle Root and Whole Tree Hash allow for 2 key functionalities:

• Merkle roots grant users and relays the ability to download one piece of a profile at a time, like being able to download a transaction without downloading the entire block.
• Whole tree hashes let users and relays know when a profile they are storing is incomplete. The whole tree hash only matches if you have every bit of data in the Merkle tree, unlike Merkle roots.

This inexpensive method can be used to update a user’s entire profile once a week, or however often they like. Nostr 2.0 still works without this, as it does now, but a user can pay a few sats infrequently to synchronize their data across Nostr relays if they want all users to see their posts.

Users and relays can download posts one branch at a time. After each branch, they hash the branch with another branch nearest the Merkle root to check if it matches the on-chain Merkle root (like SPV). If the branches hashed together matches this Merkle root, then they’ll know the branch is part of the user profile even if they don’t have the entire user profile yet.

Downloading one branch at a time prevents delay attacks that could cripple many distributed networks, which is why Merkle roots and branches are used in the bitcoin whitepaper to secure SPV lightwallets. Users can download different branches of the same profile from many different Nostr relays while still verifying that each branch is valid and that the profile they downloaded is complete.

Why Can’t Merkle Roots Do What a Whole Tree Hash Does?: If a Nostr relay only relies on Merkle roots, then they will not know when the Merkle tree is complete because every pair of branches nearest the Merkle root hashes into the same Merkle root.

To be sure the user’s profile is complete, relays or users hash their entire updated Merkle tree to verify that it matches the whole tree hash on-chain. If the whole tree hashes match, then the user data is complete. If the whole tree hash does not match, then the relay or user can tell other relays what their latest leaf number is and request the missing branches until the whole tree hash does match.

Since we can verify that certain branches belong to a specific user, Nostr Relays can be paid every time they deliver a small branch of data to users. To achieve this, users need to download the headers of the blockchain — like they do in SPV — so they can perform the typical functions of a lightwallet. Users will harness the lightwallet’s SPV to grab a specific transaction on-chain, but that transaction will have the Merkle root and whole tree hash of a user’s profile within its OP_Code. This will now let users pay the relays to download that person’s profile branch by branch, verifying each branch hashes into the on-chain Merkle root.

To sends sats to Nostr relays, in exchange for delivering data, we use an evolution of Gregory Maxwell’s (Prolific Bitcoin Core Developer) ZKCP design (Zero Knowledge Contingent Payment) [1] known as ZKCSP: Proof-of-Retrievability [2] combined with the Lightning network.

From the ZKCSP whitepaper: “…do not require a trusted third party… We also implement ZKCSP protocols for the case of Proof of Retrievability, where a client pays the server for providing a proof that the client’s data is correctly stored by the server.” [2]

1. User starts a Lightning smart contract with a few financiers.
2. User sends a request to the surrounding financiers. The financiers sign the request.
3. The user sends the request signed by the financiers directly to the Nostr relays those financiers are connected to.
4. The user now initiates the ZKCSP construction to make sure the Nostr relay is paid by the financier only once the correctly requested data has been delivered.

Once step 3 occurs, the original request signed by the financier will be written atop of by the user before the ZKCSP construction is initiated in step 4. The user will write atop the original request by specifying how much will be subtracted from the user’s balance and financier’s balance (they must be the same amount, plus the financier’s fee) then the user will sign the addition they wrote onto the original message.

If the user specifies that they want to send more sats than they own, or more sats than the financier has frozen with that Nostr relay, then the Nostr relay will deny the request because it will be impossible for the relay to be paid.

This way a user can be connected to many Nostr relays while only freezing their money up with a few financiers. Lightning could also work like this as well, where trust-minimized financiers are permissionless intermediaries between users and stores. Normal P2P Lightning hops can be used in Nostr 2.0 as well, but this new methodology may be far more efficient for users with less financial resources.

Nostr relays can whitelist certain keys if they want to store all the data those users’ view. If Nostr relays cannot whitelist the users they would like to store the data of, then they will store any data sent to them. If users can always send relays data for free, then users will never pay Nostr relays. The only way Nostr can have a paid option is if relays have the option to deny storing data they can’t afford to. Free relays will still exist, but paid relays are currently not an option — until now.

Paid Nostr relays can use whitelists to selectively store all the data their paid-users’ view every day, rather than attempting to store ALL Nostr data for free. Some Nostr relays will continue the free model, but that is not sustainable at the largest scales because most free relays are just passionate enthusiasts. Whitelisting, a.k.a. giving the Nostr relays the ability to decide which data they get paid for storing, will not be possible unless we can securely assign one public key per Nostr profile.

This system finally allows us to assign one public key per profile.

There is no benefit of users creating a new public key for every post… it’s all tied to their profile anyway! It’s not like bitcoin addresses. Unlike bitcoin, there’s no privacy improvement of letting users have many public keys in the same app.

One Public Key per Profile Nostr Consensus Rule: Public key of the Nostr profile must match the public key of the bitcoin transaction containing the weekly hashes (Merkle root and whole tree hash of all the user’s posts). Instead of Nostr users signing with their Schnorr signatures, they’ll need to sign with their bitcoin wallet (mobile/lightwallet or full-node).

The two-birds-with-one-stone beauty of this is every Nostr account will be represented by their bitcoin address, meaning users can send payments directly to accounts on Nostr without needing to ask for two different public keys. It reduces the mental cost in the system for newcomers.

Every Nostr relay must become a full-node. Nostr relays follow one simple rule for usernames: the first person to register a name on-chain owns it. To register a name, simply add a “register name” field underneath the Merkle root and whole tree hash in the OP_Code of your transaction. If two users try to choose the same name at the same time, the user who puts it on-chain last can post a “change username” request in the OP_Code of their next transaction because it’ll still only be the length of a sentence. After that, anyone trying to register that name to their bitcoin address will be deemed an inauthentic profile by relays and will need to change their name in order for honest Nostr relays to host their profile.

A relay can avoid labeling a conflicting profile as inauthentic, so this username method is not protected against phishing attacks unless users already know the public key/bitcoin address of the other users. Users will still need to message their public key to each other to find each other on Nostr, not usernames.

Nostr 2.0 relays are indirectly paid to store the bitcoin blockchain, strengthening the security of Bitcoin and Nostr simultaneously. If other Nostr apps use different public keys, they can all still be attached to the same DID — that way the means of identifying your account is still consistent across apps. Although, this Nostr consensus rule limits each profile on every Nostr app to 1 public key each.

There’s a chance Nostr relays may lose some user data since relays have the freedom to choose what they want to store, unlike bitcoin full-nodes. Therefore, users should only store data on Nostr relays if users can back it all up locally. Web5’s self-hosting service by Block will allow you to sync your backups across all your local devices, so that will reduce risks for users. Umrel is an interesting choice for simply self-hosting data, lightning, and a full-node all-in-one machine. At the end of the day, the blockchain is the only place where data is truly immutable. Although, Nostr is a Hybrid Layer for data that will still work well for many apps.