PIP-28: DAN - Distributing the Altruist Network (Community Chains Introduction)

UPDATES:
1. Added more clarification in Summary
2. Added a ~$20k reimbursement for current node runners under the Rare RelayChainID Supplementation model.
3. Added a $8k administration cost for DA to implement these models
4. Added an hourly Discord report to the metrics shared section:

Attributes

  • Author(s): @shane

  • Recipient(s): Decentralized Authority and node runners (under the oversight of PNF)

  • Category: Reimbursement / Governance Upgrade

  • Implementer(s): DA with PNF oversight

  • Asking Amount:
    One-Time Reimbursement: ~160k POKT + ~$28k
    Monthly: 100k-150k POKT + ~$13k (please see Budget for more details)

Summary

Decentralized Authority (DA), the team behind Node Pilot, is excited to announce Community Chains (CC), an open, distributed, chain node pooling platform. CC allows POKT nodes and servicers to access a pool of chain nodes that are run by POKT providers, with all payments being settled with POKT. Our goal for CC is to:

  1. Allow node runners to serve as many chains as they desire, without adding to their already elevated infrastructure bills.
  2. Enable any node runner to deploy their POKT nodes in a geo-mesh configuration without having to spin up more chain infrastructure.
  3. Reduce the amount of underutilized chain nodes node runners would need to run to be profitable, which is one of the largest expenses on the POKT network.
  4. Distribute the altruist network to multiple providers in a neutral manner that is accountable to the DAO.
  5. Keep the POKT token at the center of chain pooling.

With this proposal we would like to introduce CC to the DAO and offer it as a way to delegate the traffic of the altruist network in an open, distributed manner, under the oversight of PNF on behalf of the DAO. Delegating the altruist network to CC would:

  1. Change the altruist network from a centralized network, owned and run by PNI, to a neutral, data driven, distributed, load-balanced network
  2. Increase resiliency and maximize capacity through CC exhaustive testing and provider rating capabilities.
  3. Provide the DAO with transparency reports on its capacity, usage, and cost.
  4. Keep the POKT token at the center of the altruist network.
  5. Support the development of this platform and the independent node running community.

UPDATED

Since the altruist traffic is technically traffic that should go through the protocol, the DAO can have a say since it effects POKT customers across the network. Before now the altruist has been a black box to the DAO, despite it’s important, and this proposal is to establish a precedence of DAO involvement.

This proposal is structure to enable CC to be “authorized default” of the altruist network, as it provides the best QoS for POKT customers and is providing transparency to the DAO. Any other solution that want so to be the default on chains covered with CC should follow a similar structure so the DAO can be involved. Please see the comments below (notably here)

Outline:

  1. Community Chains Introduction
  2. Distributing the Altruist Network
  3. Reimbursement For Current Altruist Runners

1. Community Chains Introduction

Motivation To Build Community Chains

Independent Node Runners

Independent node running needs to be preserved in the POKT ecosystem. Right now; the economics of POKT are such that only those running POKT nodes at scale, with millions of POKT staked, are positioned to weather POKT’s current market conditions and generate a profit. As time goes on, fewer and fewer folks are running POKT nodes on their own and either turning towards node hosting providers (further centralizing who runs POKT nodes) or exiting the ecosystem all together.

Community Chains provides a multi-provider distributed chain node pooling solution so any runner can utilize an existing pool of chain nodes across the world. Network average rewards through geo-mesh can be accessible to all, instead of just to the hosting provider.

Reduce Network Costs

Right now, POKT has substantial chain node overprovisioning. From our own independent testing, we have found that a load balanced cluster of 2 Ethereum nodes, using the Erigon client on a bare-metal server, can sustain around 200k requests a minute under heavy load. POKT currently does around 300k per minute. With the adoption of geo-meshing, it’s likely that most providers are running a minimum of 2 ethereum nodes in each major region, resulting in 6 nodes per provider. When just considering the top 5 hosting providers, who account for 66% of serviced relays, there are a minimum of 30 ETH nodes on the POKT network. When considering the entire network there are likely 60+ ETH nodes across the Pocket Network. This would mean POKT is utilizing just 5% of its chain node bandwidth. This overprovisioning is the largest infrastructure cost in the POKT ecosystem today.

Community Chains provides an open, distributed solution that can save resources from unneeded waste. Even large node runners would have an option in CC to reduce costs on chain infrastructure and instead focus on creating great experiences for their users. By using the POKT token for payment on CC, fewer node runners will need to exchange POKT for infrastructure costs, saving resources from unneeded waste. By enabling chain node pooling through CC, anyone can run and own a POKT node without being forced to add more chain nodes to the POKT network.

How Community Chains Works

Resilient

The core to CC is the CC Tower, a distributed network of gateways to access CC. Currently there are 7 CC Towers, spanning across 4 different node providers. Unlike the PNI Portal, which acts as a single gateway to the Pocket Network, CC Towers are operated by multiple node running companies, preventing any single entity from being a central point of failure. These towers essentially create a distributed network of chain node hubs which allow POKT nodes and services direct access to chain nodes in their region.

All traffic gets geo-routed via DNS to nearby CC Towers. Latency is minimized by having multiple CC Towers in each region, hosted directly in the same infrastructure as many of the chain nodes.

Capable

Not only is CC distributed, but it also validates its own throughput capacity. When a CC Tower is deployed, DA undergoes a significant stress test to verify that the provider has the capacity to handle altruist network level traffic. By doing these stress tests, we’ve helped providers 5x their throughput by identifying bottlenecks in their infrastructure. The only way to know if an infrastructure can handle altruist level traffic is to test it directly.

DA requires each CC Tower to be able to handle at least 25% of all of POKT traffic at any given time, and there are currently 7 CC Towers deployed (more on the way). This is done by testing each CC Tower with hundreds of thousands random requests a minute, across multiple chains.

Once a CC Tower is verified to be able to handle the required amount of traffic, it is brought online.

Open - CC Community Towers

Anyone can bring nodes onto CC through our CC Community Towers we are launching with @Breezytm. This CC Towers will be open managing outside endpoints from other node runners. If someone is running a chain node that most others aren’t running, they can sign up with the CC Community Tower. The idea of a CC Community Towers enables more to participate in CC while still ensuring that all the nodes on CC are high quality and meet our high standards.

CC Community Towers give providers 90% of the revenue generated from their endpoint, while maintaining 10% to continue operating a quality tower. (quality what?) This 10% is what pays the CC Community Tower operator for running the tower and ensuring their nodes are operating within the CC requirements.

2. Distributing the Altruist Network

Benefits To Delegate Altruist Chains To CC

  1. Chains served by CC turns the altruist network from a centralized network to a neutral, data driven, distributed load-balanced network.

    • Decentralized Authority has always been developing software and providing support for independent node runners. All of our software has been free and we do not host our own POKT nodes or chain nodes, thus we are able to distribute traffic from the altruist network in a neutral manner among providers.
  2. Increase resiliency and maximize capacity through CC’s exhaustive testing and capability metrics.

    • We increase resilience by requiring node providers to run a CC Tower within their existing infrastructure. Calls to CC are then DNS routed to CC Towers in their region, reducing latency. By having providers run a CC Tower in their infra, it distributes access to CC and eliminates bottlenecks. If one provider crashes, they are taken out of rotation without affecting other providers.

    • We have also developed a robust testing system to rank the capacity of each provider we on-board. We stress test each provider, in each region for the following:

      • Per chain capacity (how many calls until the latency starts being affected or the nodes start throwing errors).

      • Load balancer capacity (how many calls until the LBs start getting behind on calls, dropping connection, or running out of ports)

        • We have found this testing particularly important, because while a provider’s chain nodes may be able to take millions of relays a minute, their load balancing infrastructure can be a major limiter. Identifying this limitation can only be found under intense stress testing.

        • Through this rigorous testing, we were able to help a provider improve their capacity by 5x. Being able to verify capacity is crucial as we don’t want these issues to suddenly surface when the altruist network is activated.

    • We will use the capacity data of each provider, in each of their regions, to distribute the calls in a balanced manner. This ensures that providers receive relays proportional to their capacity.

3. Provide the DAO with transparency reports on its capacity, usage, and cost. (UPDATED)

  • Currently only PNI has insight into the altruist network. With CC we would be able to provide an hourly report which can be fed to the POKT Discord (per PNF creating a dedicated channel and providing CC with a webhook), along with a more detailed monthly reports on the altruist’s performance and cost. At the end of each month CC would bill the DAO for the altruist’s cost and distribute the funds to the providers involved.

  • Metrics to be reported on:

    1. Total Relays Served

    2. Regional Data
      a. Relays per chain
      b. CC Towers Distribution

    3. CC Tower Data
      a. Relays per CC Tower
      b. CCTower Stress Test Rating

    4. Supplemented RelayChainID Data
      a. Supplemented RelayChainID list
      b. Cost per RealyChainID
  1. Keep the POKT token at the center of the altruist network.

    • Payment for the altruist would be settled with POKT with CC. Since CC is utilizing existing node infrastructure from providers, there is less need for providers to liquidate POKT to cover costs.

    • Breakdown of cost model is below

  2. Support the development of this platform and the independent node running community.

    • The CC Tower and all the testing tools are open-source. Already though working with providers, we have been able to help them increase their capacity by 5x simply by stress testing their infrastructure. By utilizing CC for the altruist network, the DAO would get a more robust altruist network while also supporting development that is improving node runners.
  3. Accountability to PNF

    • DA will work with PNF to ensure CC is meeting its requirements each month.

    • PNF would mediate the payment between the DAO and DA, thus not requiring new proposals each month, but only after verifying DA meeting its requirements.

Budget/Payment Structure

Community Chain’s Regular Payment Structure

Community Chains is already designed to pay chain node providers in POKT for the relays they serve. Multiple providers have already expressed interest in the payment structure we laid out for CC, which is as follows:

  • Each POKT node using CC pays with 15% of the rewards they generate.
  • Those rewards are then split 33% between DA (who develops, maintains, operates, tests, and administers the CC service) and 66% to the providers (who continue running reliable chain nodes).

Altruist Network Payment Structure


Per Relay Cost = ~100k to ~150k POKT Estimated Cost

Since CC is already set up for per relay payment, we suggest continuing with that same model. The DAO would pay per relay at a price that is 15% of the reward a POKT servicer would generate when staked in the 60k bucket. The calculation would be as follows:

Calculation: {Relays that week}{RelaysToTokensMultiplier}{ServicerAllocation}{60k weight multiplier}{CC Fee}

For example, if POKT maintains 1B relays a day for a week, about 3% would be routed to the altruist network, equalling 210M relays. Each week the cost to the DAO to supplement for these relays would be:

Calculator: 210,000,0000.000603.851.6.15= 25,833 POKT a week (about 103,330 POKT a month)


Base Infrastructure cost: = $3k per month (paid out in POKT for 4 months)

This helps bootstrap the cost of running the CC infrastructure and CC Towers while CC gets off the ground to generate its own revenue. Currently CC is used only for the altruist network, which comes at a cost for all involved, but we plan to open the system up to POKT nodes in the coming weeks.

We expect to have our infrastructure expenses covered through natural CC revenue in a matter of months. After that period, the infrastructure bootstrap would not be required.


Rare RelayChainID Supplementation

In the past PNI ran all RelayChainIDs, including nodes with few rewards, like archival nodes. This ended up being very expensive. Archival endpoints especially are used substantially less than non-archival endpoints, yet many projects rely on archival alongside non-archival. For POKT to be a competitive player in the RPC space, all RelaysChainIds need to be verifiably supported, especially rarer ones which do not generate much rewards. If a developer can create an endpoint for a chain in the Portal, it should have quality service (with the hopes that one of those customers makes a RelayChainID rewarding to run.

Instead of rare RelayChainIDs, like archival, being run by PNI and putting further pressure on PNI’s runway, the DAO should supplement this cost. Only a few RelaysChainIDs needs supplementation, and some archival endpoints get enough relays to cover their costs (like Ethereum).

How It Would Work

For each RelayChainID that is considered “underserved”, DA would provide PNF with an evaluation on the cost to run the node. DA does not run nodes ourselves but we can provide an estimation as a neutral party.

A RealyChainID will be considered underserved when the following is met:

  1. There are not enough nodes in each region to provide sufficient throughput and redundancy.

  2. There likely needs to be at least 2 nodes in each region to provide sufficient coverage for low volume chains.

  3. Anyone can submit their nodes to CC Community Towers.

  4. The RelayChainID generates less rewards than the cost of running the node.

  5. Some chains require a lot more devops time than others. BSC Archival is a good example where the node itself takes troubleshooting multiple times a day. That time is a cost and is likely a reason why that chain is underserved in the network.

Any chain that is sufficiently underserved and has monthly rewards to justify its deployment cost will be eligible for Rare RelayChainID Supplementation. Only 2 nodes for each rare RelayChainID per region are eligible for supplementation. Slots will be given on a “first sync basis” meaning the first to prove to CC their node is synced. CC will then provide geo routed load balancing, so that if a rare RelayChainID in one region goes down, the traffic will be routed to the nearest region. If a node runner’s rare RelayChainID endpoints do not maintain consistent quality, they will be dropped in favor of another more reliable node runner.

DA, in accordance with PNF, will distribute 100% of the supplementation compensation to each participating archival node once a month after DA has received the monthly CC payment from PNF.

Initial Unofficial Estimations

The chains below are some initial estimations on chains we’ve identified as possibility underserved. More research will be conducted after DAN becomes an official part of the network but this is what these chains could potentially cost:

BSCA - $3,400

POLA - $3,400

SOL - $4,200


How Would DAN Be Implemented

The large portion of the altruist is already distributed through CC. For the chains currently not distributed through CC, PNI would transfer the endpoints from community folks to CC and they can be directly integrated into a CC Community Tower. Once they provide a POKT address they can expect monthly payouts for the RPC requests they served.

CC Community Towers would also be open to endpoints from other community members. Anyone can reach out to @Breezytm about getting their endpoints added.

POKT gateways would then use CC geo-routed endpoints for their altruist traffic. This proposal established a transparent way to delegate a core-service, like the altruist network, to CC’s management with PNF oversight.

The Role of PNF

PNF will play the following roles in supporting the CC distributed altruist network:

  1. Remitting monthly payments from the DAO treasury

  2. Oversight

  3. Validating CC’s claimed traffic (and thus payouts) by cross-checking with POKT gateway operators

  4. Keeping DA accountable to publishing monthly transparency reports and upholding all other requirements specified in this proposal

  5. Resolving disputes if gateway operators exceed fair use of the CC endpoints (e.g. using for a purpose other than gateway relay backups)

3. Reimbursement For Current Altruist Runners

a. Per Relay Reimbursement

For the past several weeks, a diverse group of node runners has been serving altruist traffic with an expectation to be reimbursed. While CC provides a structured payment system for the future, there should be a reimbursement for the past few weeks.

However, CC’s payment model provides a good template on how node runners can be reimbursed. By adopting a model where node runners are reimbursed with 10% of the rewards that would have been generated from a POKT node, it provides fair compensation on a per relay basis. In the case of the chains that go through CC, the amount should be 15% because DA is contributing geo-routing, QoS, and distribution across multiple providers.

NOTE: The altruist is only 1-3% of traffic, being distributed across 3 other providers… so these payments are not going to be substantial nor should altruist traffic be considered a central source of revenue for any node provider. Reimbursement for altruist traffic should just be seen as a small bonus for the small addition of relays.

Once the altruist network has been transitioned to CC, current altruist runners can continue to be paid for relays via CC. If transition happens by the end of March, the cost will be around: ~160k POKT

b. Rare RelayChainID Supplementation (UPDATED)

In January PNI began outsourcing chains to community members so they could spin down expensive infrastructure. Some of these chains would qualify under Rare RelayChainID Supplementation.

To ensure that all node runners are made whole, we will retroactively apply this payment model to all node runners who supported rare RelayChainIDs on behalf of the altruist network, which began in January.

For 2 months of supplementation estimation: ~$20k

c. Administration Cost (UPDATED)

DA has been working with PNI, PNF, and node runners to create this structure to reimburse all altruist participants. Thus far this has been uncompensated work and will require a lot more time to finalize the all payment in a fair manner. Since this proposal was first posted more administration needs have arisen to make everyone whole, since both communication and data is very scattered.

This work should be supplemented by the DAO as many node runners are relying on DA to be reimbursed for their past work and establish a systematic future.

Administration Cost: $8k

TLDR Budget

One time Payment

Relay Reimbursement for altruist runners: ~160k POKT
Rare RelayChainID Supplementation: ~$20k (UPDATED)
Administration Cost: $8k (UPDATED)

Recurring Monthly Payment

Relay Cost: ~100k to ~150k POKT
Infrastructure Cost: $3k
Rare RelayChainID Supplementation: ~$10k

Dissenting Opinions

PNI should run the altruist network

PNI has already signaled they are open to offloading the altruist network to a solution like CC. CC has been in development since October to be a neutral chain node pooling platform.

Notable reasons to delegate to CC

  • Distributing core services to more companies reduces the amount of technical debt on PNI and brings in more contributors. DA has been a major contributor of free and open-source node software in the POKT’s node ecosystem since 2020 and is already running about half of the altruist network. POKT needs to move toward delegating more core-services to more competent companies.

  • CC is designed to stress test our providers to ensure CC can handle the entire POKT network. To our knowledge, the altruist network has never actually been stress tested to ensure its capacity. With the popularity of providers using their own closed-source clients and switching away from PNI’s internal nodes, it’s important that the altruist network be verifiably capable of handling POKT’s growing traffic.

  • The Altruist Network should be a natural structure that can be used once we have more POKT gateways. Currently PNI runs the only gateway on the POKT network, the POKT Portal, but the POKT ecosystem is moving towards an open gateway structure where others will be able to operate their own gateways. A 3rd party altruist network would be a benefit as it can serve the coming gateway ecosystem neutrally.

  • CC geo-routes altruist traffic which has dramatically improved altruist performance. Prior to CC all altruist traffic went to PNI infrastructure in the US. This means that applications outside of the US would see a latency spike every hour for about 30s to 60s. Since implementing CC as the primary provider of the altruist network, session role over traffic is treated with the same latency as the rest of the session since CC routes each Portal to chains nodes nearby. This also means that in the event of a chain halt, applications will not see any kind of latency changes until the network is moving forward.

DAN has been a collaboration between DA, PNI, PNF, POKTScan, Node River, Breezy, qspider, and other ecosystem contributors.

CC Is centralized

I’m putting this here just in-case. The altruist network has always been centralized and will always have a centralized component until v1 or v2. POKT is the decentralized solution to RPC but v0 has current limitations and sometimes the blockchain may have an occasional chain halt.

Until v1 is released and we know for a fact we are resilient to chain halts, we need to be prudent and provide an emergency backup. CC is about as distributed as one can get before going fully “decentralized”. To go fully decentralized, we would have to become POKT itself, which is not our goal.

While CC is a centralized service, it is more distributed, transparent, and resilient than any other solution.

Session rollover may be able to be addressed with a network upgrade

Yes, that is a possibility. You can see the recent activity here. While core contributors believe it can be addressed, it has not been coded or tested. Since right now session rollover traffic exists we need a plan until a solution is production ready.

The DAO should not pay for the altruist network

The DAO should take ownership of the altruist network and require transparency. The last chain halt like event was in November, and entire network was relying on the altruist network, which was routed to PNI’s USE infrastructure.

PNI was spending upwards of $250k a month on infrastructure up until they began outsource their infrastructure to the community at starting this year. DAN is another way to outsource from PNI at fraction of the cost and provide clear transparency.

Resources

These are the open-source tools we have built to create CC and make it altruist traffic ready:

CC-Tower
The Community Chains tower application that resides in the datacenters of CC providers and provides direct access for users to RPC endpoints within the provider’s network.

CC-API
The API that drives Community Chains on the user side as well as the provider side.

CC-Health-Check-Relay
An application which relays health check requests in and out of a closed Docker network.

Pocket-Relay-Stress-Test
A tool for load testing Pocket nodes with Ethereum-based relay chains and for testing Ethereum-based RPC endpoints directly.

Eth-Node-Simulator
A script which responds to block number and get transaction requests with static data. Useful for providing predictable back end performance for stress testing load balancers.

8 Likes

Great job!
This is a very needed feature for the current Pocket Network. This is an improvement over the old PNI altruist network and enables the DAO to take control of it.

I was hoping to read exactly this lol.
It clarifies that this is not a Pocket Network within a Pocket Network, it just a safety net. Nevertheless this safety net is not centralized and low-performant as before, it’s a decentralized and fast one.

Pocket Network is now decentralized (up to a certain level) even when it is failing :exploding_head: .

This is just great…

6 Likes

The word I like to use is “distributed.” CC is a centralized network in that we hold the keys and providers have to prove themselves in order to be let in, but it is a distributed network because all RPC requests go directly to our providers. There is no central CC hub and we do not run any RPC nodes ourselves. Our whole business is providing a network where quality RPC providers can further utilize their existing infrastructure in ways that benefit the whole Pocket community.

6 Likes

Thanks for the feedback everyone. We will likely transition this proposal from Pre-Proposal to a PEP tomorrow.

Everyone is still welcome to comment!

5 Likes

Looking forward to the full proposal. I think this is an important service.

3 Likes

Shane has my full support for this proposal. He, his team, the operators in CC, and a few operators outside of CC are the reason PNI has helped extend PNI’s runway.

At present, I am confirming we are using CC for most chains. One of our engineers, Vlad, also built a load-balancer on our end to load-balance requests between CC and other providers. Shane’s current concerns is that the other endpoints may not be load-balanced across regions, like his, so we are exploring a solution here as well, since some node operators would like to remain independent, but also provide altruist support.

At the end of the day, we hope to use the results of this proposal to have the DAO compensate all operators who support the altruist system.

3 Likes

While I’m open to PNI presenting their own solution to the DAO when it is production ready, I want to make clear that this proposal is specifically designed to put the altruist traffic under the authority of the DAO (since it is protocol traffic) and then delegate to CC with PNF’s oversight.

So far with PNI’s current load-balancing between providers, customers are getting more errors and latency than if PNI were to just use CC with all the providers we are already working with. POKT customers should get the best experience possible, and already CC has been able to identify anomalies in the traffic which has been used to find Portal issues. This kind of delegating of core-services, shared responsibility, and transparency is best for the network and its customers IMO.

From what we’ve learned from the altruist network already, I’d suggest that if PNI wants to invest into another alternative to bring altruist network in-house again, there should be:

  1. Full insight into its analytics
  2. DAO transparency on that traffic
  3. Real time QoS (where if a providers goes down or is behind, minimal relays are dropped with proper rerouting)
  4. Proper LBing (where multiple providers are being used simultaneous vs switching providers ever 5 min, like it is currently)
  5. Geo-routing so each Portal is connected to local nodes (which currently only CC offers)
  6. A holistic payment system for providers that involves the DAO and PNF (as we have extensively done)
  7. Data driven stress testing (as we have done)
  8. A proposal from the DAO that can be voted on (like we are doing now)

Personally, since CC already is providing a solution to the DAO for the altruist network, I’d prefer to see resources put into other areas of the network. CC’s providers already have this covered and have gone though extensive testing to verify that it can handle multiples of POKT’s current traffic. I don’t think CC needs to be reinvented and I don’t see the network value that will be gained from investing resources into an alternative, when it accounts for such little traffic. CC is already the alternative to the original altruist system and anyone can participate via the CC Community Towers.

5 Likes

I’ve moved this proposal from a Pre-Proposal to a PIP, since there hasn’t been any suggested changes to the proposal itself. All feedback thus far has been positive.

Feedback and suggestions are still welcomed, and edits may still be made, but it’s core does not seem to require changes, thus the transition to a PIP.

6 Likes

Reiterating as PIP:

I strongly support this proposal

5 Likes

Following up that I made a mistake and how i phrased it. Our Altruist Manager is not a load-balancer at all. Originally, that’s what I thought Vlad was building so I ran with it. Vlad and Fred mentioned they will follow up in the PIP in the near future.

cc @vlad @fredt

2 Likes

Thank you @shane et al. Absolutely outstanding work!

There are a few things I would like to see cleaned up and clarified in this proposal.

  1. There is nothing in the proposal that makes clear that this proposal is the official conduit for handover of Altruist responsibility from PNI to PNF. I only found out that this was the intention in the comments section exchange with Art. It is somewhat implied by using te PIP heading instead of PEP, and the tags in the “Category” and “Implementer” attributes, but it should be made more explicit. I recommend to call this out front and center in the , summary and body of the proposal.

  2. It is not clear that this proposal in the conduit by which DA becomes the official desginee for managing Altruist. "offer it as a way to delegate is not quite the level of it being the official designee. Again, it was not until the comments section exchange with Art that this intention was clarified. I recommend clarifying this in the summary and body of the proposal.

Combining 1&2, the last paragraph of the summary may be something like the following:. "PNI currently manages the altruist network. Upon approval of this PIP, all control and management responsibility for the altruist network transfers from PNI to PNF. Currently PNI already offloads a portion of altruist network servicing to CC. Upon approval of this PIP, PNF will delegate most, if not all, of altruist network servicing responsibility to DA while retaining oversight responsibility. PNF retains the right to utilize other vendors in addition to DA if it so chooses. "

As an aside, I support both these governance improvements; I just think it best to clarify so nobody is taken by surprise as to what they are voting yes or no to.

3)The ask and the budget should be clarified (particularly the budget since that is what is normative).

(a)under budget section, do the tldr first followed by details … that way ppl don’t have to scan through an extremely long budget section just to know the tldr.

(b) consider moving all of the text in the budget section that explains or justifies budget levels to a new “Rationale” section.

(c ) under tldr, move the infrastructure cost line item from recurring to one-time in keeping with other text that says its only for a 4-month bootstrap (e.g., Infrastructure cost: $12k (to be paid in $POKT all at once or in four equal monthly instalments ")

(d) I think I understand the intention re the small chains is that this be an optional supplement. If so, that should be made explicit rather than include this into the main budget. Perhaps it should be considered and voted on separately. Or perhaps we could use the multl-option voting feature that Snapshot makes available, e.g., “Approve - baseline only; Approve - with $10k.mo supplement; Reject”

4 Likes

I would like to clarify and offer some transparency into how we are handling the Altruist currently and the development of internal tooling to help this process. This was done in isolation of Community Chains and done explicitly for the purposes of business continuity on the PNI Portal, and does not affect the relationship with Community Chains or Decentralized Authority.

PNI has developed the Altruist Manager. The Altruist Manager is a self-service tool that allows our staff and members of the community to set altruists per chain on the portal based and monitor available altruists.

Compared with a Load Balancer approach, The Altruist Manager doesn’t proxy traffic, so it has the following advantages:

  1. No latency overhead. Because Gateway connects to the Altruist Node directly

  2. No Single Point of Failure. If the Altruist Manager stops working, the Portal will use the last set Altruist for each chain, so service remains uninterrupted.

  3. Follows the principle of decentralization: anybody from the community can provide an Altruist Node; the Manager selects Altruists based on transparent and verifiable rules and data.

Now, how the Altruist Manager works:

  1. Checks all the Altruists’ health every 5 min(configurable):

All the metrics are stored in a db, and can be made available to Altruist owners with further development to the UI(3).

  1. Altruists Manager keeps records of when each altruist was set as a chain’s altruist on the Portal, this data is used to rotate altruists.

  2. Altruist selector rotates altruists (round-robin) to balance the number of times each healthy altruist was selected within the last hour. Every 5 min(configurable) for each chain it requests healthy altruists from the monitoring db(1), orders them by the number of sessions it was set on the Portal(2), and takes the altruists with the lowest amount of sessions. In the future, any selection logic could be implemented.

  3. The Altruist Manager provides a UI to manage the altruists. This allows for both our internal business users to collaborate with community members to add or remove Altruist Providers and for each altruist owner to login and manage their own Altruist offerings.

  4. The Altruist Manager could be made geo-aware and regional in a further update. Right now the Portal supports only one altruist per chain, so Altruists-manager is deployed only in the US region. This should be easy to implement and inexpensive to maintain.

Lastly, the PNI Altruist Manager source code is going through a review right now, and will be made Open Source on GitHub in the coming days.

I hope this seeks to clarify some of our motivation in building the Altruist Manager, irrespective of the engagement with Community Chains and the DAN proposal.

7 Likes

After review of this proposal and many conversations leading up to it, I am in support of this PIP.

While it may be somewhat centralizing in the short term, I believe that Community Chains is best equipped to democratize the provider layer and report out the utilization of the Altruist Network until the core Altruist use case gap is closed (session rollover).

Regardless of whether this PIP passes, PNI’s approach to the Altruist Network will be as follows:

  • If chain is offered by Community Chains, CC will be the sole altruist provider for PNI
  • If chain is not offered by Community Chains, PNI will utilize a multi-provider strategy for its Altruist support.

This will be managed entirely through our Altruist Manager described by @vlad in his previous post, and we will look to make this list of providers available publicly in the very near future (Q2).

As always, I am grateful to @shane , Node Pilot, and the entire community for their continued efforts to better and improve the Pocket Network. I look forward to leveraging another new way to lower costs on the network and provide unstoppable backup infrastructure.

6 Likes

Thanks for the feedback @msa6867. I appreicate the thought you put into this proposal.

For some context, I did have another version of this proposal where PNF was seen as the “manager” and owner of the altruist network. In working with them on the concept, they felt that they didn’t have the technical knowledge to truly be an effective manager.

With that feedback, I reworked the concept where PNF is not the owner of the altruist, but more of an overseer. PNF knows their strengths so this proposal was designed to compliment them.

The intention is not to put for every aspect of the altruist network on just one team, at least at this first pass. The idea is to have this proposal be an authorized “default” process for the altruist network. CC acts as the default for each chain it supports because we have built in QoS, multi-region, a payment model, and will provide transparency to the DAO. For the chains we support, we can provide the best default for the network ATM.

For some chains, there may a need to be some other solution depending on the situation. I think @fredt put it best with his comment :point_down:

This is the kind of flexibility we need. DA can’t be responsible for every chain on POKT at the moment, and we don’t want to be micro-managers over every chain and how providers get paid in other systems. I want to make sure this proposal doesn’t become to complex of a system where there are a bunch of hard lines.

My comment to Art was coming from a place of concern that a non-production ready product would be a default since PNI technically has control over the endpoints. As I mentioned in my comment, I feel there is tangible value to using a production ready system like CC and I believe the DAO has the right to establish it as what I’m defining now as the “authorized default”.

Happy to make this clearer in my proposal @msa6867. Will update in the next 24h :+1:

If PNI or someone else want to offer the DAO another “authorized default”, then I do believe it should be through a transparent proposal as we have done that accounts for all the elements in my comment to Art. I don’t think that we need to make it bureaucratic with a proposal to require future proposals, but I feel this proposal sets a precedent on how altruist traffic should be systematically handled, especially when it’s effecting customer QoS.

I did place a TLDR Budget section for that purpose :point_right: PIP-28: DAN - Distributing the Altruist Network (Community Chains Introduction)

I’ll see where I can make some aspects of the budge more clear. Thanks for the feedback :+1:

Thanks for the clarity here @vlad. For greater context, we’ve shared our designs plans with PNI since November, over many calls, and have made changes due to PNI’s feedback. I greatly appreciate PNI’s willingness provide the same transparency here.

Thank-you for the continued support and all the additional clarity!

4 Likes

Proposal Updated

Since posting this proposal, I’ve been learning more about what all needs to be done to reimburse node runners who started participating early on when PNI first started to outsource the altruist network. Many node runners started spinning up chains with the expectation of reimbursement.

To account for this and to make node runners whole we can use the Rare RelayChainID Supplementation model to retroactively reimburse node runners. See here.

I’ve also learned more about the administrative task to follow through with all the reimbursements needed and establishing the Rare RelayChainID Supplementation, so I’ve included an administration cost. While Rare RelayChainID Supplementation monthly fee will go 100% to node runners, there is going to be administration overhead that should be honored as altruist work. See here.

I’ve included a Updates section at the top of the proposal with links to edits. I’ve also added an updated section to the Summary to make this proposal more clear, per @msa6867’s feedback.

2 Likes

It sounds like what you’re saying is pretty soon, once CC is able to snag enough folks to run the rest of the chains they’re not running now, that no provider will be allowed to serve altruist traffic UNLESS they do it through CC, am I understanding you correctly?

1 Like

Connecting nodes to CC Community Towers vs connecting nodes to PNI is no material different, except with CC there is a method to compensate the node runners and it provides better QoS for customers.

The major difference is that CC Towers are run by a multitude of providers vs just being run by PNI.

3 Likes

Could not have said this better myself :slight_smile:
Driving the Altruist Network on CC is a short term move that will provide transparency in the billing of the DAO for the Altruist Support and also provides QoS guarantees that the PNI Altruist Management system in its current state cannot.

In addition, PNI will continue to launch chains to the Network, and with that we will bootstrap the Altruist Network for those chains until the Community (through Community Chains) is able to provide support.

I hope this helps to clarify the “why” behind PNI’s approach to the Altruist Network.

3 Likes

I’m starting to loose sight of the big picture. I thought the point of PNi divesting altruist to community players was to save money and extend runway. From reading through comments section, it is sounding like PNI is, and feels obligate to continue, running an in-house altruist capability on top of and independent of community involvement. I’m loosing sight of the overall cost savings being offered by community involvement. Setting rare-chain supplementation completely aside for the moment, can we get a sense of what PNI monthly altruist budget was before community involvement and what it will be going forward assuming community involvement continues and this proposal passes. If the delta is less than the monthly budget of this proposal, it would seem that DAO dollars would be better spent having PNI resume full in-house control of altruist and PNI submit a PEP for monthly reimbursement of the delta. Just thinking like a project manager. There are two competing models for altruist and I’m trying to figure out which has the lower overall cost.

Regarding rare chain supplement, I am not in favor of bundling it with this proposal. There are a few concerns.

(1) it doubles the monthly budget (at todays $POKT price point) and I am not convinced the spend is worth it.

(2) I am not convinced that the DAO should be serving as an enabler for PNI to close contracts with chains that will be a liability rather than an asset to the protocol . If PNI want’s to sell chains as a loss leader, then let them bear the cost and subsidize it themselves. It is ok to say no to a new chain if we are not ready for the chain. A way to sink a project is to say “yes” to anything and everything, even if the business case for it does not close. I’m not saying a good case can’t be made to support the rare chains listed. I’m saying that the community should have the opportunity to discuss and vote on it separately rather than bury the matter as a suplement to an altruist reimbursement proposal. Even better, it might be best to decide on a case-by-case bases. For example, the DAO may want to say yes to SOL but no to BSCA.

(3) From a protocol/architecture perspective, altruist is the wrong vehicle for handing rare chains. If the DAO decides it wants to support a rare chain and finds that protocol incentives are insufficient to induce node runners to support the chain, then the DAO should offer grants to support the chain first and foremost on the protocol, not on altruist. This is an important distinction. Altruist is there to provide a fail-safe; it was never intended to be used as the primary vehicle for servicing certain chains. A condition of the subsidy could be that the subsidized hw be enrolled in cc altruist in case of fail-over, but this should incur no extra cost.

2 Likes

When PNI was in-house this was the cost :point_down:


The cost to run the distributed infra is $3k (which is stated in the proposal) and around 100k POKT a month which goes to node runners an the QoS maintainers. A cheaper system would require node runners to get paid less (which that system would still need to be developed) and PNI invest their development resources (aka runway) into altruist work instead of Portal v2 work… all the while having worse QoS for customers.

I think just the QoS is worth the transition, but I also feel our cost model is very fair, especially considering this is a distributed system with multiple providers from the POKT ecosystem running CC Towers.


It will cost more to have good altruist infrastructure, but how else should rare chains be supported? I only see POKT’s reputation being hurt publicly if a number of supported chains have blackouts in service for 30s to 60 each hour… or at best, very poor service for 30s to 60 each hour without CC. Unless there is another way to incentives node runners to run infrastructure without getting supplemented, I don’t see a better option.

Just today it was suggested in Discord that the DAO provide grants for rare chains to improve QoS, and that’s what the rare relaychainID supplementation is. Happy to change the wording to grant though :wink:

I think altruist and rare chains is more closely related than one may think. To build CC we had to look at chain distribution, QoS, infrastructure capacity, etc. Whether it’s establishing a solid altruist network, or a solid rare chain network, it requires the same kind of research, development,coordination, and QoS accountability.

Because it’s all so closely related, it makes sense to combine work for the sake of efficiency. Any separate grant program would have the same coordination and infrastructure requirements, so we are actually tackling two needs at once with DAN.

1 Like