The purpose of the RetrievalMarket Mini-Summit is to bring together all the builders of multiple Retrieval Market solutions, highlight the progress made to date, and identify how multiple systems can be combined to improve the efficiency, effectiveness and reliability of retrieval transactions for web users.
At the Retrieval Market Builders Mini-Summit in April, Raul Kripalani (core developer of libp2p and Star File System) delved into the architecture and planned changes to the Lotus node, specifically to the Lotus-miner node, to better meet the needs of customers and network participants.
The presentation aims to highlight the work needed to ensure highly reliable and scalable retrieval transactions for Lotus customers – helping to ensure that the Interstellar File System retrieval market is built on a robust foundation.
Lotus is an implementation of the Interstellar File System network, written in Go. As the first implementation of the Interstellar File System, Lotus is intended to provide a solid and resilient foundation for the entire network. One of the core functions of any Interstellar File System node is to enable the storage and retrieval of useful data – thus laying the foundation for a robust retrieval marketplace on the Interstellar File System network.
The Evolution of the Lotus Node
The interstellar file system ecosystem is evolving rapidly, and ecosystem tools such as Lotus are evolving to meet the needs of the network and to experiment with future opportunities. In particular, the retrieval market will introduce new features to the interstellar file system network, including indexing, CDNs, public search, and more.
The problem we have found is that these new retrieval market capabilities will rely on a strong customer base – today, Lotus customers tend to address certain features that they find unstable or unprofitable. Ultimately, this is good behavior for customers to witness; it helps Lotus achieve growth to meet their needs. When network participants start excising certain features, it indicates that the software itself should provide more flexibility and modularity for its users.
Through this observation, we realized that Lotus’ monolithic binaries were hindering flexible and robust operations for Lotus customers and identified opportunities to address the problem.
Up to now, the architecture of Lotus nodes has been largely monolithic. Lotus has two distinct entry points: 1) a full node, and 2) a user. Beyond that, however, network participants have little choice in what responsibilities they take on during the storage lifecycle. For example, customers are tasked with mining, proving storage, trading, etc. These are fundamentally different responsibilities, but they are coupled together through the current Lotus-miner architecture.
Problems with the monolithic architecture
In a broad sense, this monolithic architecture prevents customers from efficiently allocating their resources between different responsibilities. Forced consolidation of responsibilities contradicts the client’s core motivation of maximizing profits while minimizing operational risk. Specifically, there are many problems with monolithic binaries.
Vulnerability: When there is no isolation within the architecture, errors in one area can leak and affect other critical processes. This makes it difficult to segment between critical and non-critical processes and bring down the entire instance.
Increased attack surface: Some processes of Lotus customers need to expose public endpoints, which makes that part of the system vulnerable to attack. If the architecture is monolithic, a single public endpoint may compromise the entire system.
Functional bloat: In the mission of the Star File System, clients not only store and provide raw data, but also provide useful services such as indexing, discovery, etc. Overloading individual processes with new client-side functionality can lead to further concerns and operational risks.
Inability to scale: Different capabilities require different features and different upgrade processes. Trying to manage these different capabilities in a single system is difficult and can lead to slow iteration and growth.
Lack of fine-grained control: Since customers cannot actively opt in or out of features, they must address these issues through mechanisms such as rejecting transactions, setting high prices to dissuade transactions, or even forking code bases.
Iterative slowdown: Monolithic architectures mean that it is harder to test and innovate because there is no easy way to isolate new components and opt-in to experimental features without introducing them into the entire system.
Moving Toward Modular Customer Architectures
Looking at the responsibilities of Lotus customers behind the scenes, two distinct categories emerged to tap into the business and the market. These two layers share very different services and SLAs. for example, the Lotus customer’s mining operation is its most financially profitable activity, and is also inherently private. Marketplace operations, however, are fully public and operate under a different profitability model.
Using this initial classification, the first step toward a modular customer architecture that addresses the shortcomings of the monolithic architecture emerges: dividing Lotus customer processes into 1) Lotus market processes and 2) other Lotus customer processes.
By separating market and mining, the Lotus node architecture becomes more function-specific for customers, allowing them to align with the responsibilities that best fit their capabilities and network economic preferences.
While moving down this new path towards a more modular client architecture, Lotus will continue to adhere to a number of design principles.
Avoid the complexity of microservices-like architectures: explicitly avoid complex networks of interdependencies that can quickly translate into spaghetti-style architectures and require specialized tools (such as service grids) to manage and debug. This is the level of complexity that introduces errors into Lotus. Order and hierarchy still belong, as these processes are still part of a single cohesive unit
Simple hub/spoke architecture: Lotus is moving towards a hub-and-spoke architecture that is managed by supervisors. These supervisors know what functionality is provided in the deployment and which nodes provide that functionality. They also act as external API entry points for aggregation.
It is worth investing in many ways: the new Lotus architecture is worth the investment. In addition to profitability, some other ROI factors include reliability, security, developer agility, and robustness.
Strong distributed process lifecycle oversight: A modular architecture does not mean a completely siloed architecture. If a failure occurs and means that other downstream services will fail, the entire system must be able to notice and react.
To design this new runtime, we motivated ourselves with models such as Erlang/OTP and role-oriented programming (e.g. Akka in the JVM world) and borrowed selected concepts and abstractions from service grids and editors without actually relying exclusively on these frameworks.
Posted by:CoinYuppie，Reprinted with attribution to:https://coinyuppie.com/lotus-upgrade-reliability-security-agility-and-robustness-all-improved/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.