Arweave: An experiment in permanent storage

Arweave is a clever combination of several novel blockchain technologies, offering a tantalizing promise: pay once, use forever.

By Can Gurel

A few words about Arweave.
Pros.

Arweave is a protocol/network that attempts to achieve permanent data storage in a decentralized, censorship-resistant manner.

An innovative economic model – the ‘endowment pool’ – is built in to support long-term sustainability.

Arweave is more attractive to NFT and Web3 applications because of its permanent storage and data hosting features.

Storage prices are priced in AR, but Arweave has a well-designed pricing mechanism to combat price volatility without relying on a predictive machine.

Based on all of the above, Arweave has a wider range of application scenarios such as data traceability, data availability, proof of existence, etc.

Disadvantages.

“Perpetual” means a long period of time. It only makes sense if users pay a higher price today than other solutions because they are paying up front for future storage costs and that data will be available for decades to come.

As Arweave grows in popularity, the amount of data it stores will grow exponentially (it’s now up to 8.5 TB). Despite Arweave’s efforts to do as much as possible (e.g., full nodes do not need to synchronize data across the chain), centralization of miners is still an issue.

Sacrifice Mining” could become a new way to generate revenue based on exploitation. The risk here is that miners upload invalid data to Arweave to increase the size of the entire network in order to increase their chances of getting a block reward.

Despite the endowment pool mechanism, Arweave may need to rethink the sustainability of long-term incentives given its extremely long project lifecycle.

The story of perpetual storage requires a high degree of coordination of multiple, concurrently complex technologies. Since Arweave’s solution is still in its infancy, it cannot be ruled out that new technologies will continue to emerge that will threaten Arweave in the future. As a pioneer in this field, Arweave must be aware of this problem and be ready to respond.

An experiment in permanent storage
Arweave is an exciting project whose goal is to allow users to pay once and store their data on Arweave forever (at least 200 years). In today’s Internet, while data is growing exponentially, there are often incidents of old data being lost or stolen. This is because almost all data is currently stored centrally, and the motivations of those hosting the data may not align with those of users who wish to access their own data. In the digital age, perpetual storage will be a very novel value proposition in a future perspective. From a more distant perspective, data permanence will preserve historical truth in a way that is accessible to all but editable by none. The history of humanity will no longer be the history of the victors, and everyone will be able to keep their data forever.

Information on Arweave is designed with decentralized and sustainable incentives to ensure its sustainability. Leveraging the constant nature of PoW, Arweave enables scalable and permanent data storage. It is ambitiously designed to be able to carry a larger data holding capacity than today’s networks. However, the current blockchain is not suitable for storing large amounts of data. The more data there is, the higher the cost of synchronization across all nodes, which necessarily runs counter to decentralization and scalability. So how does Arweave ensure this? Let’s start with a small example.

A PoW chain requires full node synchronization of all blocks to verify the legitimacy of the entire chain. As time goes on, there will be more and more blocks on a chain, and the data size of the chain will get bigger and bigger, requiring more resources to maintain the whole chain. This is often referred to as state bloat (state inflation). As a result, this somewhat limits the number of full nodes and also reduces the degree of decentralization and access performance.

This relationship is shown in the figure below.

Arweave: An experiment in permanent storage

Many blockchain state growth is often limited to balance its maintenance costs, as on Bitcoin where block size and time are forced to grow linearly. However, in the case of Arweave, we see a very different growth trajectory. Here is a comparison of the two data structures. the state data on Arweave is already 25 times larger than the bitcoin network, but the entire chain can still be stored on a commercially available laptop.

Arweave: An experiment in permanent storage

Arweave’s scalability can be addressed by a variety of technologies such as Blockweave, Wildfire, Blockshadows and SPoRA, and these technologies can coexist. If this sounds like gibberish, don’t worry. In this article, we’ll talk about them from a top-level design perspective, rather than eating them up one by one.

Blockweave (or Weave for short) is a blockchain data structure similar to Arweave. It differs from traditional blockchains in that Blockweave does not require each node to synchronize the entire chain. Regardless of the size of Weave’s network, miners can start mining immediately by downloading only a portion of the blocks. This lowers the threshold for consensus, but the new question is how to ensure that the network ledger does not fork due to miner mischief?

Based on Weave’s technology, each block will be randomly selected by a cryptographic algorithm from a preorder block to verify the data in the block (recall chunk). This means that the randomly selected historical data will participate in the generation of new blocks. In this mechanism, in order to be rewarded with a block out, miners must not only win the PoW hash rate contest, but also ensure that historical data is readily available. The new mining game looks like this.

Probability (win reward) = Probability (has old data) x Probability (finds hash first)

Probability of winning a block reward = Probability of having old data x Probability of being the first to find hash

Since the cost of storage is lower than the cost of arithmetic, miners will naturally optimize their storage hardware before optimizing their arithmetic hardware, which indirectly leads to the entire network being biased towards storage rather than computation. In the future, if weave becomes too large (which was their original intention, after all) to be maintained by any single miner, then other miners will actively seek out “rare” blocks that have not been widely replicated, so that their arithmetic can be used efficiently. This incentive is at the heart of Arweave, which ensures that no matter how large the Weave network becomes, every block’s data will be saved somewhere by some miner. So, essentially, miners don’t need to scale their storage devices to the size of the Weave network, because they can store only a portion of the entire network.

This mechanism is called Proof of Access (“PoA”, not PoA as in Proof of Authority). PoA does ensure that miners are storing data on the network, but miners can also use remote servers to reduce costs, and the servers are centralized, which inevitably goes against the idea of decentralization. The team acknowledged this and subsequently upgraded the consensus to “SPoRA” (Succinct Proofs of Random Access). miners using local hardware to participate in the network will receive a higher incentive. In short, SPoRA was introduced not to prevent the concentration of miners, but to prevent the concentration of storage resources. The current market perception of SPoRA is positive, and network resources have been effectively changed since February 2021. That said, the security of SPoRA and the efficiency of synchronization when it comes to larger scale data remains to be proven.

SPoRA is one of the first mining experiments to encourage the decentralization of storage resources. That’s innovation compared to rivals that condone centralized storage.

Economic design experiments for permanent storage
In addition to network consensus and data storage, Arweave’s pass-through economy is critical to the ability of the entire network to (ideally) operate autonomously over time. arweave employs a new business model where users pay once to store their data on Arweave in perpetuity. the underlying design of the Arweave economy is tricky because:

Users buy network resources in fiat currency, while the value within the network is captured in AR.

Unlike other chains, the miner experience issues were never quickly resolved. In fact, they have never been solved.

For these reasons, the Arweave network must ensure that miners have sufficient profit margins. If profit margins become thin and lead to miners. For a healthy network, it is important to anticipate costs and balance miners’ earnings well.

Storage Costs
Arweave’s one-time payment model is based on the decreasing cost of storage across the network. According to available information, the cost of storing data on the global network is decreasing significantly over time. As the graph below shows, data storage costs have been decreasing at an average annual rate of 30.57% over the last 50 years.

Arweave: An experiment in permanent storage

More importantly, this year-to-year decline will continue as storage technology is still far from reaching the technical bottleneck of the theoretical upper limit. If we add up all the costs of the Arweave network at infinity, this predictable cost decline means that the cost margins for permanent storage are predictable. Below, I will explain in the simplest way possible how Arweave calculates the cost of permanent storage.

First, Arweave estimates the price of storing 1GB of data on a hard drive for 1 hour based on the prices of different capacities and sizes of hard drives on the market. Next, it conservatively estimates the future cost of storage based on this initial price, i.e., assuming an average annual reduction in storage costs of 0.5% instead of 30.57%, and adding up all future costs, Arweave arrives at a boundary for the future cost, which is the assumed cost of permanent storage for miners. This cost forms the basis for all user-side costs. The above costs will be recalculated periodically with market prices.

Miner Revenue Method
Miners have two sources of income in Arweave:

Network fees

Block Rewards

Block rewards follow a downward trend by algorithmic design and are not affected by external factors. The processing fee is mainly derived from a one-time user fee, and up to 14% of each user fee is distributed directly to miners and added to the circulation.

The other portion (~86%) goes into the endowment pool to support data operations and maintenance on Arweave. When miners are unable to cover their mining costs, the endowment pool issues subsidies to miners. Otherwise, the funds in the endowment pool will only accumulate indefinitely for emergencies. As a result, the passwords in the endowment pool are out of circulation for a long time. Currently, there are about $500,000 in ARs in the donation pool, and as the ARs in the pool accumulate, the donation pool subsidy may exceed the block reward. This pass-through economic experiment is shown in the following diagram.

Arweave: An experiment in permanent storage

The purpose of the permanent storage experiment is to ensure that there are always enough backups of data on the chain to avoid data loss. This can only be achieved if the miners have enough profit. As shown in the diagram, the miners’ profit funnel must not be empty.

It is important to note that the ratio in the chart above does not accurately reflect the share of fees. Currently, transaction fee revenue represents only a very small portion (<0.1%) of miners’ total revenue. Similarly, the fees accumulated in the donation pool are less than 0.1% of the circulating volume.

Anti-volatility mechanism
When the AR price goes up by a factor of 10, how can we avoid a factor of 10 increase in storage costs? Conversely, how does Arweave incentivize miners when the AR price drops?

To solve these problems, Arweave has designed a dynamic storage price adjustment model in AR/MB. The model uses mining difficulty instead of prophecy machine price as the basis for price adjustment. The principle is as follows.

When the AR price drops, the profit margin of miners becomes thinner and miners may leave the network. At this point, Arweave will make a certain degree of reduction in the hash rate and mining difficulty, indirectly increasing miners’ revenue.

Conversely, when the AR price rises and there are more miners, Arweave will increase the hash rate and mining difficulty, indirectly decreasing the miners’ revenue.

This model achieves stability for fiat currency by adjusting the mining difficulty and miners’ returns to offset price fluctuations, rather than relying solely on the prophecy machine. If AR falls, the model increases the one-time payout price of AR/MB; and vice versa. This mechanism does not prevent AR price fluctuations, but it is an excellent anti-volatility mechanism in the long run.

As shown in the figure below.

Arweave: An experiment in permanent storage

User guidance

Most data interactions on Arweave occur at the application layer, also known as the “permaweb” (permanent Internet). As the name implies, the permanent Internet is a permanent version of the regular Internet. Currently, there are hundreds of websites and applications already running on the permanent Internet. Users do not need a block browser to access these websites and applications, but simply use a traditional browser. However, each write of user data (uploading Dapps, files, URLs, etc.) requires a fee paid with the Arweave wallet. The good thing is that new users can retweet an Arweave tweet after creating a wallet to complete the verification and thus receive an initial AR from Arweave taps for free.

Arweave: An experiment in permanent storage

As an Arweave application platform, Permaweb has certain native advantages for developers building top-level applications, such as permanent data storage, data availability, data integrity, data legitimacy, censorship resistance, accessibility, and tamper resistance. This sets the stage for Arweave to attract a wide variety of applications and users, such as DeFi, researchers, lawyers, journalists, government agencies, etc.

Today, permaweb hosts hundreds of DApps that integrate one or more of these great features. here are the best of them:

ArDrive – decentralized Baidu cloud with permanent storage

Weave mail – permanent storage mailbox software

Verto.exchange – decentralized exchange for benefit-sharing pass-throughs

Koi.rocks – Virtual NFT gallery with rewards for viewing exhibitions

Pianity – Music NFT platform

In addition, most of the applications on Permaweb are implemented using mainstream programming languages such as HTML, JavaScript and CSS, eliminating the need for developers to relearn blockchain programming languages.

Profit Sharing Communities
Profit Sharing Communities is a new community model launched by Arweave to activate the developer community. In such communities, developers and founders can realize new ways to cash in through PST (Profit Sharing Tokens). PSTs can be bought and sold on exchanges.

The benefit sharing program has already incubated hundreds of projects on permaweb and has significantly boosted the demand for Arweave from developers worldwide, which will undoubtedly benefit the growth of the entire storage network in the long run.

Arweave: An experiment in permanent storage

Sacrifice Mining
One of the major growths of the Arweave network comes from “Sacrifice Mining”. The demand for this behavior stems not only from the data provider, but also from the data demander. Before we understand Sacrifice Mining, let’s review how miners are rewarded. As mentioned earlier, miners need to increase their storage space and computing power as much as possible to increase their chances of winning block rewards. And when a miner stores a relatively rare backup of data, he can also get weighted in this competition.

Recently, the Arweave team discovered a form of cheating among miners, where miners only upload data without sharing this data with other miners. By doing so, they can use the rarity of the data to increase their probability of getting a block reward. Admittedly, they need to “sacrifice” (spend) some of their AR to upload this data. This is where the term “sacrificial mining” comes from. However, this data is not productive, but simply scales the network. Miners’ earnings depend on many factors, but the main one is obviously the price of ARs. Therefore, it is difficult for Arweave to make a blanket statement about this behavior, but at the same time, it does not want miners to unnecessarily expand the network size with junk data just for their own benefit.

Importantly, blockweave’s mechanism does not allow any miner to hoard data over time. Each time a new block is created, miners must share the data they have hidden away. To one group of people, this means that the potential rewards of sacrificial mining are short-lived; but to another group, it also means that miners can continue to stuff the network with junk data, only to hide it for a short period of time, but still with the potential for profit. In theory, it’s likely that miners will continue to do so until the marginal return on probability-adjusted block rewards is less than the cost of uploading junk data. This could take a long time, and Arweave would have to continue accepting garbage data; and this behavior would do nothing to drive up the price of ARs, since miners could do this with the ARs they earn. In our opinion, the name “sacrificial mining” is still too indulgent for what is essentially a miner’s in-roll.

Arweave: An experiment in permanent storage

Arweave and NFT
Although NFTs exist on-chain, they often carry the value of “off-chain assets (e.g. digital artwork, audio/video files, GIFs, event tickets, etc.)” because it is very expensive to store these files in their entirety on-chain. A tweet from @RealAllenHena clearly illustrates the current mainstream NFT platform storage solutions. As you can see, the dominant solution is still to establish a link between the pass and the asset via IPFS or a graph bed URL link.

While IPFS is indeed a popular distributed storage solution and is somewhat censorship-resistant, IPFS does not guarantee the permanence of its data. The various key nodes that hold the file do not have the necessary reason to keep this file in perpetuity. Therefore, the storage time of data assets, whether on a centralized server or on IPFS, is dependent on the goodness of the custodian. On IPFS, if no node responds nearby, you may not be able to connect, or may lose the file forever; and if entrusted to a graph bed, the URL link may fail once it is unmaintained.

In addition, searching for content on IPFS can take several minutes. Mintbase was one of the first platforms to store NFT metadata on Arweave. Based on Arweave’s excellent features, Arweave is likely to attract more and more NFT platforms, artists and related developers.

Arweave: An experiment in permanent storage

The backbone of the encryption industry
A growing number of projects are using Arweave to store data related to the crypto industry, including some outstanding use cases in the DeFi space: Uniswap and Compound have already stored their entire front-end applications on Arweave.

Permaweb is also not a good choice for storing state data for standalone state early blockchains. For example, Solana, a well-known public chain, has partnered with third-party developers to store all transactions on the SOLAR bridge on Arweave for permanent, decentralized state data storage. There are many other projects and teams that have put source code or state data on Arweave like this, so I won’t go into details here.

Limitations
Admittedly, there are some limitations to Arweave and its ecosystem.

First, Arweave is not intended to compete on a price level with the head of centralized clouds (e.g., AWS, Google Cloud, Azure, etc.) or decentralized clouds (Filecoin, Sia, Storj, etc.) Arweave has a unique value proposition – a decentralized cloud based on economics, game theory, and incentives. Arweave is not shy about its trade-off between price and value. In fact, as noted earlier, Arweave’s costing model is very conservative, not using the industry’s 30.57% storage cost decay rate, but rather a 0.5% decay rate. The good news is that overall, storage services are cheap. Arweave’s data storage costs are now often below $0.004/MB.

Second, without the new upper layer protocols, permaweb is not ideal for a real-time user interaction experience. An operation can take several minutes to complete. This experience may be tolerable on a blog or website, but it does not make sense for more other applications.

Another more serious limitation is the speed of data retrieval. Typically, users are connected to the network through a gateway, rather than a direct connection to the miner node. The main role of the gateway is to retrieve the entire network and access specific data for the user. While Arweave encourages miner nodes to store and share data with each other, there is no direct incentive relationship between Arweave and the gateway in each user’s home, nor does the gateway ensure that users can quickly and efficiently access the data in the miner nodes. As a result, Arweave has recently introduced a new consensus protocol, Amplify, to bridge this gap.

Finally, Arweave’s permaweb is not private, and all data is permanently public and directly accessible, making it unsuitable for storing personal data or some sensitive data.

Conclusion
Many people may not know that the Internet Archive is currently the 209th largest website in the world, storing over 40 pd of data (>130,000 laptops) in 2018. The capabilities of permaweb go far beyond mere data archiving. Because of the application layer built on top of it, Permaweb has many more possibilities, such as NFT storage / NFT transactions, traffic cashing, interaction cashing, and many more scenarios than you can imagine. Today Arweave is still in its infancy, with only 8.5 TB of data stored on it, and a donation pool of $500,000, which is still less than 0.1% of the circulating volume.

As mentioned earlier, Arweave will not capture the market through price, as users will need to pay upfront for the future. If Arweave wants to achieve the perpetual storage it claims, the question is essentially one: “If I pay for the future, can Arweave guarantee me access to my data for decades and centuries to come?” But the answer is hard to come by right now, because there are so many unknowns. How big is too big? Will miners become progressively more centralized? Will there be a more expensive version of AWS’ IQ tax? Will the network incentives be designed to be sustainable? We’ll just have to wait and see on all of the above.

That said, Arweave is aiming to be an interesting application right now (especially for NFT), and the amount of positive market feedback is there for all to see. After all, a lot of NFT data needs to be stored somewhere for a long time, and the Arweave development team has come up with a well-thought-out and innovative mechanism like SPoRA to try to address the long-term challenge. This is a good thing, because there is a fundamental difference between “achieving permanent storage” and “committing to permanent storage”.

Posted by:CoinYuppie,Reprinted with attribution to:https://coinyuppie.com/arweave-an-experiment-in-permanent-storage/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.

Like (4)
Donate Buy me a coffee Buy me a coffee
Previous 2021-06-15 04:58
Next 2021-06-15 05:06

Related articles