UPDATES:
1. Added more clarification in Summary
2. Added a ~$20k reimbursement for current node runners under the Rare RelayChainID Supplementation model.
3. Added a $8k administration cost for DA to implement these models
4. Added an hourly Discord report to the metrics shared section:
Attributes
-
Author(s): @shane
-
Recipient(s): Decentralized Authority and node runners (under the oversight of PNF)
-
Category: Reimbursement / Governance Upgrade
-
Implementer(s): DA with PNF oversight
-
Asking Amount:
One-Time Reimbursement: ~160k POKT + ~$28k
Monthly: 100k-150k POKT + ~$13k (please see Budget for more details)
Summary
Decentralized Authority (DA), the team behind Node Pilot, is excited to announce Community Chains (CC), an open, distributed, chain node pooling platform. CC allows POKT nodes and servicers to access a pool of chain nodes that are run by POKT providers, with all payments being settled with POKT. Our goal for CC is to:
- Allow node runners to serve as many chains as they desire, without adding to their already elevated infrastructure bills.
- Enable any node runner to deploy their POKT nodes in a geo-mesh configuration without having to spin up more chain infrastructure.
- Reduce the amount of underutilized chain nodes node runners would need to run to be profitable, which is one of the largest expenses on the POKT network.
- Distribute the altruist network to multiple providers in a neutral manner that is accountable to the DAO.
- Keep the POKT token at the center of chain pooling.
With this proposal we would like to introduce CC to the DAO and offer it as a way to delegate the traffic of the altruist network in an open, distributed manner, under the oversight of PNF on behalf of the DAO. Delegating the altruist network to CC would:
- Change the altruist network from a centralized network, owned and run by PNI, to a neutral, data driven, distributed, load-balanced network
- Increase resiliency and maximize capacity through CC exhaustive testing and provider rating capabilities.
- Provide the DAO with transparency reports on its capacity, usage, and cost.
- Keep the POKT token at the center of the altruist network.
- Support the development of this platform and the independent node running community.
UPDATED
Since the altruist traffic is technically traffic that should go through the protocol, the DAO can have a say since it effects POKT customers across the network. Before now the altruist has been a black box to the DAO, despite it’s important, and this proposal is to establish a precedence of DAO involvement.
This proposal is structure to enable CC to be “authorized default” of the altruist network, as it provides the best QoS for POKT customers and is providing transparency to the DAO. Any other solution that want so to be the default on chains covered with CC should follow a similar structure so the DAO can be involved. Please see the comments below (notably here)
Outline:
- Community Chains Introduction
- Distributing the Altruist Network
- Reimbursement For Current Altruist Runners
1. Community Chains Introduction
Motivation To Build Community Chains
Independent Node Runners
Independent node running needs to be preserved in the POKT ecosystem. Right now; the economics of POKT are such that only those running POKT nodes at scale, with millions of POKT staked, are positioned to weather POKT’s current market conditions and generate a profit. As time goes on, fewer and fewer folks are running POKT nodes on their own and either turning towards node hosting providers (further centralizing who runs POKT nodes) or exiting the ecosystem all together.
Community Chains provides a multi-provider distributed chain node pooling solution so any runner can utilize an existing pool of chain nodes across the world. Network average rewards through geo-mesh can be accessible to all, instead of just to the hosting provider.
Reduce Network Costs
Right now, POKT has substantial chain node overprovisioning. From our own independent testing, we have found that a load balanced cluster of 2 Ethereum nodes, using the Erigon client on a bare-metal server, can sustain around 200k requests a minute under heavy load. POKT currently does around 300k per minute. With the adoption of geo-meshing, it’s likely that most providers are running a minimum of 2 ethereum nodes in each major region, resulting in 6 nodes per provider. When just considering the top 5 hosting providers, who account for 66% of serviced relays, there are a minimum of 30 ETH nodes on the POKT network. When considering the entire network there are likely 60+ ETH nodes across the Pocket Network. This would mean POKT is utilizing just 5% of its chain node bandwidth. This overprovisioning is the largest infrastructure cost in the POKT ecosystem today.
Community Chains provides an open, distributed solution that can save resources from unneeded waste. Even large node runners would have an option in CC to reduce costs on chain infrastructure and instead focus on creating great experiences for their users. By using the POKT token for payment on CC, fewer node runners will need to exchange POKT for infrastructure costs, saving resources from unneeded waste. By enabling chain node pooling through CC, anyone can run and own a POKT node without being forced to add more chain nodes to the POKT network.
How Community Chains Works
Resilient
The core to CC is the CC Tower, a distributed network of gateways to access CC. Currently there are 7 CC Towers, spanning across 4 different node providers. Unlike the PNI Portal, which acts as a single gateway to the Pocket Network, CC Towers are operated by multiple node running companies, preventing any single entity from being a central point of failure. These towers essentially create a distributed network of chain node hubs which allow POKT nodes and services direct access to chain nodes in their region.
All traffic gets geo-routed via DNS to nearby CC Towers. Latency is minimized by having multiple CC Towers in each region, hosted directly in the same infrastructure as many of the chain nodes.
Capable
Not only is CC distributed, but it also validates its own throughput capacity. When a CC Tower is deployed, DA undergoes a significant stress test to verify that the provider has the capacity to handle altruist network level traffic. By doing these stress tests, we’ve helped providers 5x their throughput by identifying bottlenecks in their infrastructure. The only way to know if an infrastructure can handle altruist level traffic is to test it directly.
DA requires each CC Tower to be able to handle at least 25% of all of POKT traffic at any given time, and there are currently 7 CC Towers deployed (more on the way). This is done by testing each CC Tower with hundreds of thousands random requests a minute, across multiple chains.
Once a CC Tower is verified to be able to handle the required amount of traffic, it is brought online.
Open - CC Community Towers
Anyone can bring nodes onto CC through our CC Community Towers we are launching with @Breezytm. This CC Towers will be open managing outside endpoints from other node runners. If someone is running a chain node that most others aren’t running, they can sign up with the CC Community Tower. The idea of a CC Community Towers enables more to participate in CC while still ensuring that all the nodes on CC are high quality and meet our high standards.
CC Community Towers give providers 90% of the revenue generated from their endpoint, while maintaining 10% to continue operating a quality tower. (quality what?) This 10% is what pays the CC Community Tower operator for running the tower and ensuring their nodes are operating within the CC requirements.
2. Distributing the Altruist Network
Benefits To Delegate Altruist Chains To CC
-
Chains served by CC turns the altruist network from a centralized network to a neutral, data driven, distributed load-balanced network.
- Decentralized Authority has always been developing software and providing support for independent node runners. All of our software has been free and we do not host our own POKT nodes or chain nodes, thus we are able to distribute traffic from the altruist network in a neutral manner among providers.
-
Increase resiliency and maximize capacity through CC’s exhaustive testing and capability metrics.
-
We increase resilience by requiring node providers to run a CC Tower within their existing infrastructure. Calls to CC are then DNS routed to CC Towers in their region, reducing latency. By having providers run a CC Tower in their infra, it distributes access to CC and eliminates bottlenecks. If one provider crashes, they are taken out of rotation without affecting other providers.
-
We have also developed a robust testing system to rank the capacity of each provider we on-board. We stress test each provider, in each region for the following:
-
Per chain capacity (how many calls until the latency starts being affected or the nodes start throwing errors).
-
Load balancer capacity (how many calls until the LBs start getting behind on calls, dropping connection, or running out of ports)
-
We have found this testing particularly important, because while a provider’s chain nodes may be able to take millions of relays a minute, their load balancing infrastructure can be a major limiter. Identifying this limitation can only be found under intense stress testing.
-
Through this rigorous testing, we were able to help a provider improve their capacity by 5x. Being able to verify capacity is crucial as we don’t want these issues to suddenly surface when the altruist network is activated.
-
-
-
We will use the capacity data of each provider, in each of their regions, to distribute the calls in a balanced manner. This ensures that providers receive relays proportional to their capacity.
-
3. Provide the DAO with transparency reports on its capacity, usage, and cost. (UPDATED)
-
Currently only PNI has insight into the altruist network. With CC we would be able to provide an hourly report which can be fed to the POKT Discord (per PNF creating a dedicated channel and providing CC with a webhook), along with a more detailed monthly reports on the altruist’s performance and cost. At the end of each month CC would bill the DAO for the altruist’s cost and distribute the funds to the providers involved.
-
Metrics to be reported on:
1. Total Relays Served
2. Regional Data
a. Relays per chain
b. CC Towers Distribution
3. CC Tower Data
a. Relays per CC Tower
b. CCTower Stress Test Rating
4. Supplemented RelayChainID Data
a. Supplemented RelayChainID list
b. Cost per RealyChainID
-
Keep the POKT token at the center of the altruist network.
-
Payment for the altruist would be settled with POKT with CC. Since CC is utilizing existing node infrastructure from providers, there is less need for providers to liquidate POKT to cover costs.
-
Breakdown of cost model is below
-
-
Support the development of this platform and the independent node running community.
- The CC Tower and all the testing tools are open-source. Already though working with providers, we have been able to help them increase their capacity by 5x simply by stress testing their infrastructure. By utilizing CC for the altruist network, the DAO would get a more robust altruist network while also supporting development that is improving node runners.
-
Accountability to PNF
-
DA will work with PNF to ensure CC is meeting its requirements each month.
-
PNF would mediate the payment between the DAO and DA, thus not requiring new proposals each month, but only after verifying DA meeting its requirements.
-
Budget/Payment Structure
Community Chain’s Regular Payment Structure
Community Chains is already designed to pay chain node providers in POKT for the relays they serve. Multiple providers have already expressed interest in the payment structure we laid out for CC, which is as follows:
- Each POKT node using CC pays with 15% of the rewards they generate.
- Those rewards are then split 33% between DA (who develops, maintains, operates, tests, and administers the CC service) and 66% to the providers (who continue running reliable chain nodes).
Altruist Network Payment Structure
Per Relay Cost = ~100k to ~150k POKT Estimated Cost
Since CC is already set up for per relay payment, we suggest continuing with that same model. The DAO would pay per relay at a price that is 15% of the reward a POKT servicer would generate when staked in the 60k bucket. The calculation would be as follows:
Calculation: {Relays that week}{RelaysToTokensMultiplier}{ServicerAllocation}{60k weight multiplier}{CC Fee}
For example, if POKT maintains 1B relays a day for a week, about 3% would be routed to the altruist network, equalling 210M relays. Each week the cost to the DAO to supplement for these relays would be:
Calculator: 210,000,0000.000603.851.6.15= 25,833 POKT a week (about 103,330 POKT a month)
Base Infrastructure cost: = $3k per month (paid out in POKT for 4 months)
This helps bootstrap the cost of running the CC infrastructure and CC Towers while CC gets off the ground to generate its own revenue. Currently CC is used only for the altruist network, which comes at a cost for all involved, but we plan to open the system up to POKT nodes in the coming weeks.
We expect to have our infrastructure expenses covered through natural CC revenue in a matter of months. After that period, the infrastructure bootstrap would not be required.
Rare RelayChainID Supplementation
In the past PNI ran all RelayChainIDs, including nodes with few rewards, like archival nodes. This ended up being very expensive. Archival endpoints especially are used substantially less than non-archival endpoints, yet many projects rely on archival alongside non-archival. For POKT to be a competitive player in the RPC space, all RelaysChainIds need to be verifiably supported, especially rarer ones which do not generate much rewards. If a developer can create an endpoint for a chain in the Portal, it should have quality service (with the hopes that one of those customers makes a RelayChainID rewarding to run.
Instead of rare RelayChainIDs, like archival, being run by PNI and putting further pressure on PNI’s runway, the DAO should supplement this cost. Only a few RelaysChainIDs needs supplementation, and some archival endpoints get enough relays to cover their costs (like Ethereum).
How It Would Work
For each RelayChainID that is considered “underserved”, DA would provide PNF with an evaluation on the cost to run the node. DA does not run nodes ourselves but we can provide an estimation as a neutral party.
A RealyChainID will be considered underserved when the following is met:
-
There are not enough nodes in each region to provide sufficient throughput and redundancy.
-
There likely needs to be at least 2 nodes in each region to provide sufficient coverage for low volume chains.
-
Anyone can submit their nodes to CC Community Towers.
-
The RelayChainID generates less rewards than the cost of running the node.
-
Some chains require a lot more devops time than others. BSC Archival is a good example where the node itself takes troubleshooting multiple times a day. That time is a cost and is likely a reason why that chain is underserved in the network.
Any chain that is sufficiently underserved and has monthly rewards to justify its deployment cost will be eligible for Rare RelayChainID Supplementation. Only 2 nodes for each rare RelayChainID per region are eligible for supplementation. Slots will be given on a “first sync basis” meaning the first to prove to CC their node is synced. CC will then provide geo routed load balancing, so that if a rare RelayChainID in one region goes down, the traffic will be routed to the nearest region. If a node runner’s rare RelayChainID endpoints do not maintain consistent quality, they will be dropped in favor of another more reliable node runner.
DA, in accordance with PNF, will distribute 100% of the supplementation compensation to each participating archival node once a month after DA has received the monthly CC payment from PNF.
Initial Unofficial Estimations
The chains below are some initial estimations on chains we’ve identified as possibility underserved. More research will be conducted after DAN becomes an official part of the network but this is what these chains could potentially cost:
BSCA - $3,400
POLA - $3,400
SOL - $4,200
How Would DAN Be Implemented
The large portion of the altruist is already distributed through CC. For the chains currently not distributed through CC, PNI would transfer the endpoints from community folks to CC and they can be directly integrated into a CC Community Tower. Once they provide a POKT address they can expect monthly payouts for the RPC requests they served.
CC Community Towers would also be open to endpoints from other community members. Anyone can reach out to @Breezytm about getting their endpoints added.
POKT gateways would then use CC geo-routed endpoints for their altruist traffic. This proposal established a transparent way to delegate a core-service, like the altruist network, to CC’s management with PNF oversight.
The Role of PNF
PNF will play the following roles in supporting the CC distributed altruist network:
-
Remitting monthly payments from the DAO treasury
-
Oversight
-
Validating CC’s claimed traffic (and thus payouts) by cross-checking with POKT gateway operators
-
Keeping DA accountable to publishing monthly transparency reports and upholding all other requirements specified in this proposal
-
Resolving disputes if gateway operators exceed fair use of the CC endpoints (e.g. using for a purpose other than gateway relay backups)
3. Reimbursement For Current Altruist Runners
a. Per Relay Reimbursement
For the past several weeks, a diverse group of node runners has been serving altruist traffic with an expectation to be reimbursed. While CC provides a structured payment system for the future, there should be a reimbursement for the past few weeks.
However, CC’s payment model provides a good template on how node runners can be reimbursed. By adopting a model where node runners are reimbursed with 10% of the rewards that would have been generated from a POKT node, it provides fair compensation on a per relay basis. In the case of the chains that go through CC, the amount should be 15% because DA is contributing geo-routing, QoS, and distribution across multiple providers.
NOTE: The altruist is only 1-3% of traffic, being distributed across 3 other providers… so these payments are not going to be substantial nor should altruist traffic be considered a central source of revenue for any node provider. Reimbursement for altruist traffic should just be seen as a small bonus for the small addition of relays.
Once the altruist network has been transitioned to CC, current altruist runners can continue to be paid for relays via CC. If transition happens by the end of March, the cost will be around: ~160k POKT
b. Rare RelayChainID Supplementation (UPDATED)
In January PNI began outsourcing chains to community members so they could spin down expensive infrastructure. Some of these chains would qualify under Rare RelayChainID Supplementation.
To ensure that all node runners are made whole, we will retroactively apply this payment model to all node runners who supported rare RelayChainIDs on behalf of the altruist network, which began in January.
For 2 months of supplementation estimation: ~$20k
c. Administration Cost (UPDATED)
DA has been working with PNI, PNF, and node runners to create this structure to reimburse all altruist participants. Thus far this has been uncompensated work and will require a lot more time to finalize the all payment in a fair manner. Since this proposal was first posted more administration needs have arisen to make everyone whole, since both communication and data is very scattered.
This work should be supplemented by the DAO as many node runners are relying on DA to be reimbursed for their past work and establish a systematic future.
Administration Cost: $8k
TLDR Budget
One time Payment
Relay Reimbursement for altruist runners: ~160k POKT
Rare RelayChainID Supplementation: ~$20k (UPDATED)
Administration Cost: $8k (UPDATED)
Recurring Monthly Payment
Relay Cost: ~100k to ~150k POKT
Infrastructure Cost: $3k
Rare RelayChainID Supplementation: ~$10k
Dissenting Opinions
PNI should run the altruist network
PNI has already signaled they are open to offloading the altruist network to a solution like CC. CC has been in development since October to be a neutral chain node pooling platform.
Notable reasons to delegate to CC
-
Distributing core services to more companies reduces the amount of technical debt on PNI and brings in more contributors. DA has been a major contributor of free and open-source node software in the POKT’s node ecosystem since 2020 and is already running about half of the altruist network. POKT needs to move toward delegating more core-services to more competent companies.
-
CC is designed to stress test our providers to ensure CC can handle the entire POKT network. To our knowledge, the altruist network has never actually been stress tested to ensure its capacity. With the popularity of providers using their own closed-source clients and switching away from PNI’s internal nodes, it’s important that the altruist network be verifiably capable of handling POKT’s growing traffic.
-
The Altruist Network should be a natural structure that can be used once we have more POKT gateways. Currently PNI runs the only gateway on the POKT network, the POKT Portal, but the POKT ecosystem is moving towards an open gateway structure where others will be able to operate their own gateways. A 3rd party altruist network would be a benefit as it can serve the coming gateway ecosystem neutrally.
-
CC geo-routes altruist traffic which has dramatically improved altruist performance. Prior to CC all altruist traffic went to PNI infrastructure in the US. This means that applications outside of the US would see a latency spike every hour for about 30s to 60s. Since implementing CC as the primary provider of the altruist network, session role over traffic is treated with the same latency as the rest of the session since CC routes each Portal to chains nodes nearby. This also means that in the event of a chain halt, applications will not see any kind of latency changes until the network is moving forward.
DAN has been a collaboration between DA, PNI, PNF, POKTScan, Node River, Breezy, qspider, and other ecosystem contributors.
CC Is centralized
I’m putting this here just in-case. The altruist network has always been centralized and will always have a centralized component until v1 or v2. POKT is the decentralized solution to RPC but v0 has current limitations and sometimes the blockchain may have an occasional chain halt.
Until v1 is released and we know for a fact we are resilient to chain halts, we need to be prudent and provide an emergency backup. CC is about as distributed as one can get before going fully “decentralized”. To go fully decentralized, we would have to become POKT itself, which is not our goal.
While CC is a centralized service, it is more distributed, transparent, and resilient than any other solution.
Session rollover may be able to be addressed with a network upgrade
Yes, that is a possibility. You can see the recent activity here. While core contributors believe it can be addressed, it has not been coded or tested. Since right now session rollover traffic exists we need a plan until a solution is production ready.
The DAO should not pay for the altruist network
The DAO should take ownership of the altruist network and require transparency. The last chain halt like event was in November, and entire network was relying on the altruist network, which was routed to PNI’s USE infrastructure.
PNI was spending upwards of $250k a month on infrastructure up until they began outsource their infrastructure to the community at starting this year. DAN is another way to outsource from PNI at fraction of the cost and provide clear transparency.
Resources
These are the open-source tools we have built to create CC and make it altruist traffic ready:
CC-Tower
The Community Chains tower application that resides in the datacenters of CC providers and provides direct access for users to RPC endpoints within the provider’s network.
CC-API
The API that drives Community Chains on the user side as well as the provider side.
CC-Health-Check-Relay
An application which relays health check requests in and out of a closed Docker network.
Pocket-Relay-Stress-Test
A tool for load testing Pocket nodes with Ethereum-based relay chains and for testing Ethereum-based RPC endpoints directly.
Eth-Node-Simulator
A script which responds to block number and get transaction requests with static data. Useful for providing predictable back end performance for stress testing load balancers.
.
