PIP-28: DAN - Distributing the Altruist Network (Community Chains Introduction)

Proposals Improvements
9

Reiterating as PIP:

I strongly support this proposal

5 Likes
10

Following up that I made a mistake and how i phrased it. Our Altruist Manager is not a load-balancer at all. Originally, that’s what I thought Vlad was building so I ran with it. Vlad and Fred mentioned they will follow up in the PIP in the near future.

cc @vlad @fredt

2 Likes
11

Thank you @shane et al. Absolutely outstanding work!

There are a few things I would like to see cleaned up and clarified in this proposal.

  1. There is nothing in the proposal that makes clear that this proposal is the official conduit for handover of Altruist responsibility from PNI to PNF. I only found out that this was the intention in the comments section exchange with Art. It is somewhat implied by using te PIP heading instead of PEP, and the tags in the “Category” and “Implementer” attributes, but it should be made more explicit. I recommend to call this out front and center in the , summary and body of the proposal.

  2. It is not clear that this proposal in the conduit by which DA becomes the official desginee for managing Altruist. "offer it as a way to delegate is not quite the level of it being the official designee. Again, it was not until the comments section exchange with Art that this intention was clarified. I recommend clarifying this in the summary and body of the proposal.

Combining 1&2, the last paragraph of the summary may be something like the following:. "PNI currently manages the altruist network. Upon approval of this PIP, all control and management responsibility for the altruist network transfers from PNI to PNF. Currently PNI already offloads a portion of altruist network servicing to CC. Upon approval of this PIP, PNF will delegate most, if not all, of altruist network servicing responsibility to DA while retaining oversight responsibility. PNF retains the right to utilize other vendors in addition to DA if it so chooses. "

As an aside, I support both these governance improvements; I just think it best to clarify so nobody is taken by surprise as to what they are voting yes or no to.

3)The ask and the budget should be clarified (particularly the budget since that is what is normative).

(a)under budget section, do the tldr first followed by details … that way ppl don’t have to scan through an extremely long budget section just to know the tldr.

(b) consider moving all of the text in the budget section that explains or justifies budget levels to a new “Rationale” section.

(c ) under tldr, move the infrastructure cost line item from recurring to one-time in keeping with other text that says its only for a 4-month bootstrap (e.g., Infrastructure cost: $12k (to be paid in $POKT all at once or in four equal monthly instalments ")

(d) I think I understand the intention re the small chains is that this be an optional supplement. If so, that should be made explicit rather than include this into the main budget. Perhaps it should be considered and voted on separately. Or perhaps we could use the multl-option voting feature that Snapshot makes available, e.g., “Approve - baseline only; Approve - with $10k.mo supplement; Reject”

4 Likes
12

I would like to clarify and offer some transparency into how we are handling the Altruist currently and the development of internal tooling to help this process. This was done in isolation of Community Chains and done explicitly for the purposes of business continuity on the PNI Portal, and does not affect the relationship with Community Chains or Decentralized Authority.

PNI has developed the Altruist Manager. The Altruist Manager is a self-service tool that allows our staff and members of the community to set altruists per chain on the portal based and monitor available altruists.

Compared with a Load Balancer approach, The Altruist Manager doesn’t proxy traffic, so it has the following advantages:

  1. No latency overhead. Because Gateway connects to the Altruist Node directly

  2. No Single Point of Failure. If the Altruist Manager stops working, the Portal will use the last set Altruist for each chain, so service remains uninterrupted.

  3. Follows the principle of decentralization: anybody from the community can provide an Altruist Node; the Manager selects Altruists based on transparent and verifiable rules and data.

Now, how the Altruist Manager works:

  1. Checks all the Altruists’ health every 5 min(configurable):

All the metrics are stored in a db, and can be made available to Altruist owners with further development to the UI(3).

  1. Altruists Manager keeps records of when each altruist was set as a chain’s altruist on the Portal, this data is used to rotate altruists.

  2. Altruist selector rotates altruists (round-robin) to balance the number of times each healthy altruist was selected within the last hour. Every 5 min(configurable) for each chain it requests healthy altruists from the monitoring db(1), orders them by the number of sessions it was set on the Portal(2), and takes the altruists with the lowest amount of sessions. In the future, any selection logic could be implemented.

  3. The Altruist Manager provides a UI to manage the altruists. This allows for both our internal business users to collaborate with community members to add or remove Altruist Providers and for each altruist owner to login and manage their own Altruist offerings.

  4. The Altruist Manager could be made geo-aware and regional in a further update. Right now the Portal supports only one altruist per chain, so Altruists-manager is deployed only in the US region. This should be easy to implement and inexpensive to maintain.

Lastly, the PNI Altruist Manager source code is going through a review right now, and will be made Open Source on GitHub in the coming days.

I hope this seeks to clarify some of our motivation in building the Altruist Manager, irrespective of the engagement with Community Chains and the DAN proposal.

7 Likes
13

After review of this proposal and many conversations leading up to it, I am in support of this PIP.

While it may be somewhat centralizing in the short term, I believe that Community Chains is best equipped to democratize the provider layer and report out the utilization of the Altruist Network until the core Altruist use case gap is closed (session rollover).

Regardless of whether this PIP passes, PNI’s approach to the Altruist Network will be as follows:

  • If chain is offered by Community Chains, CC will be the sole altruist provider for PNI
  • If chain is not offered by Community Chains, PNI will utilize a multi-provider strategy for its Altruist support.

This will be managed entirely through our Altruist Manager described by @vlad in his previous post, and we will look to make this list of providers available publicly in the very near future (Q2).

As always, I am grateful to @shane , Node Pilot, and the entire community for their continued efforts to better and improve the Pocket Network. I look forward to leveraging another new way to lower costs on the network and provide unstoppable backup infrastructure.

6 Likes
14

Thanks for the feedback @msa6867. I appreicate the thought you put into this proposal.

For some context, I did have another version of this proposal where PNF was seen as the “manager” and owner of the altruist network. In working with them on the concept, they felt that they didn’t have the technical knowledge to truly be an effective manager.

With that feedback, I reworked the concept where PNF is not the owner of the altruist, but more of an overseer. PNF knows their strengths so this proposal was designed to compliment them.

The intention is not to put for every aspect of the altruist network on just one team, at least at this first pass. The idea is to have this proposal be an authorized “default” process for the altruist network. CC acts as the default for each chain it supports because we have built in QoS, multi-region, a payment model, and will provide transparency to the DAO. For the chains we support, we can provide the best default for the network ATM.

For some chains, there may a need to be some other solution depending on the situation. I think @fredt put it best with his comment :point_down:

This is the kind of flexibility we need. DA can’t be responsible for every chain on POKT at the moment, and we don’t want to be micro-managers over every chain and how providers get paid in other systems. I want to make sure this proposal doesn’t become to complex of a system where there are a bunch of hard lines.

My comment to Art was coming from a place of concern that a non-production ready product would be a default since PNI technically has control over the endpoints. As I mentioned in my comment, I feel there is tangible value to using a production ready system like CC and I believe the DAO has the right to establish it as what I’m defining now as the “authorized default”.

Happy to make this clearer in my proposal @msa6867. Will update in the next 24h :+1:

If PNI or someone else want to offer the DAO another “authorized default”, then I do believe it should be through a transparent proposal as we have done that accounts for all the elements in my comment to Art. I don’t think that we need to make it bureaucratic with a proposal to require future proposals, but I feel this proposal sets a precedent on how altruist traffic should be systematically handled, especially when it’s effecting customer QoS.

I did place a TLDR Budget section for that purpose :point_right: PIP-28: DAN - Distributing the Altruist Network (Community Chains Introduction)

I’ll see where I can make some aspects of the budge more clear. Thanks for the feedback :+1:

Thanks for the clarity here @vlad. For greater context, we’ve shared our designs plans with PNI since November, over many calls, and have made changes due to PNI’s feedback. I greatly appreciate PNI’s willingness provide the same transparency here.

Thank-you for the continued support and all the additional clarity!

4 Likes
15

Proposal Updated

Since posting this proposal, I’ve been learning more about what all needs to be done to reimburse node runners who started participating early on when PNI first started to outsource the altruist network. Many node runners started spinning up chains with the expectation of reimbursement.

To account for this and to make node runners whole we can use the Rare RelayChainID Supplementation model to retroactively reimburse node runners. See here.

I’ve also learned more about the administrative task to follow through with all the reimbursements needed and establishing the Rare RelayChainID Supplementation, so I’ve included an administration cost. While Rare RelayChainID Supplementation monthly fee will go 100% to node runners, there is going to be administration overhead that should be honored as altruist work. See here.

I’ve included a Updates section at the top of the proposal with links to edits. I’ve also added an updated section to the Summary to make this proposal more clear, per @msa6867’s feedback.

2 Likes
16

It sounds like what you’re saying is pretty soon, once CC is able to snag enough folks to run the rest of the chains they’re not running now, that no provider will be allowed to serve altruist traffic UNLESS they do it through CC, am I understanding you correctly?

1 Like
17

Connecting nodes to CC Community Towers vs connecting nodes to PNI is no material different, except with CC there is a method to compensate the node runners and it provides better QoS for customers.

The major difference is that CC Towers are run by a multitude of providers vs just being run by PNI.

3 Likes
18

Could not have said this better myself :slight_smile:
Driving the Altruist Network on CC is a short term move that will provide transparency in the billing of the DAO for the Altruist Support and also provides QoS guarantees that the PNI Altruist Management system in its current state cannot.

In addition, PNI will continue to launch chains to the Network, and with that we will bootstrap the Altruist Network for those chains until the Community (through Community Chains) is able to provide support.

I hope this helps to clarify the “why” behind PNI’s approach to the Altruist Network.

3 Likes
19

I’m starting to loose sight of the big picture. I thought the point of PNi divesting altruist to community players was to save money and extend runway. From reading through comments section, it is sounding like PNI is, and feels obligate to continue, running an in-house altruist capability on top of and independent of community involvement. I’m loosing sight of the overall cost savings being offered by community involvement. Setting rare-chain supplementation completely aside for the moment, can we get a sense of what PNI monthly altruist budget was before community involvement and what it will be going forward assuming community involvement continues and this proposal passes. If the delta is less than the monthly budget of this proposal, it would seem that DAO dollars would be better spent having PNI resume full in-house control of altruist and PNI submit a PEP for monthly reimbursement of the delta. Just thinking like a project manager. There are two competing models for altruist and I’m trying to figure out which has the lower overall cost.

Regarding rare chain supplement, I am not in favor of bundling it with this proposal. There are a few concerns.

(1) it doubles the monthly budget (at todays $POKT price point) and I am not convinced the spend is worth it.

(2) I am not convinced that the DAO should be serving as an enabler for PNI to close contracts with chains that will be a liability rather than an asset to the protocol . If PNI want’s to sell chains as a loss leader, then let them bear the cost and subsidize it themselves. It is ok to say no to a new chain if we are not ready for the chain. A way to sink a project is to say “yes” to anything and everything, even if the business case for it does not close. I’m not saying a good case can’t be made to support the rare chains listed. I’m saying that the community should have the opportunity to discuss and vote on it separately rather than bury the matter as a suplement to an altruist reimbursement proposal. Even better, it might be best to decide on a case-by-case bases. For example, the DAO may want to say yes to SOL but no to BSCA.

(3) From a protocol/architecture perspective, altruist is the wrong vehicle for handing rare chains. If the DAO decides it wants to support a rare chain and finds that protocol incentives are insufficient to induce node runners to support the chain, then the DAO should offer grants to support the chain first and foremost on the protocol, not on altruist. This is an important distinction. Altruist is there to provide a fail-safe; it was never intended to be used as the primary vehicle for servicing certain chains. A condition of the subsidy could be that the subsidized hw be enrolled in cc altruist in case of fail-over, but this should incur no extra cost.

2 Likes
20

When PNI was in-house this was the cost :point_down:

The cost to run the distributed infra is $3k (which is stated in the proposal) and around 100k POKT a month which goes to node runners an the QoS maintainers. A cheaper system would require node runners to get paid less (which that system would still need to be developed) and PNI invest their development resources (aka runway) into altruist work instead of Portal v2 work… all the while having worse QoS for customers.

I think just the QoS is worth the transition, but I also feel our cost model is very fair, especially considering this is a distributed system with multiple providers from the POKT ecosystem running CC Towers.

It will cost more to have good altruist infrastructure, but how else should rare chains be supported? I only see POKT’s reputation being hurt publicly if a number of supported chains have blackouts in service for 30s to 60 each hour… or at best, very poor service for 30s to 60 each hour without CC. Unless there is another way to incentives node runners to run infrastructure without getting supplemented, I don’t see a better option.

Just today it was suggested in Discord that the DAO provide grants for rare chains to improve QoS, and that’s what the rare relaychainID supplementation is. Happy to change the wording to grant though :wink:

I think altruist and rare chains is more closely related than one may think. To build CC we had to look at chain distribution, QoS, infrastructure capacity, etc. Whether it’s establishing a solid altruist network, or a solid rare chain network, it requires the same kind of research, development,coordination, and QoS accountability.

Because it’s all so closely related, it makes sense to combine work for the sake of efficiency. Any separate grant program would have the same coordination and infrastructure requirements, so we are actually tackling two needs at once with DAN.

1 Like
21

Hence why my confusion around this whole proposal.

2 Likes
22

DAN isn’t about providing a primary vehicle for any chains. I think that may be a misunderstanding. Relays still go to the network and DAN helps with role-over session traffic and provides the required fail-safe :+1:

2 Likes
23

what’s the need for a pre-proposal when you can just edit in real time. love this UPDATES section :100:

2 Likes
24

@shane
If altruist is comprised of the same folks running the pocket nodes, then is it really different than having no altruist and relying solely on the nodes?

2 Likes
25

Thanks @shane for the quick reply. Neither of my questions/concerns were addressed by your reply, so let me try again.

Yes, this is very transparent which is much appreciated. The added transparency I’m looking for is how much cost savings does this provide to PNI so that I can calculate and compare the before and after combined DAO+PNI burn rate to run altruist

I am assuming this is inclusive of portal infra (eg AWS contracts) andindeed that the great majority of this was for portal infra. Just looking for the before/after delta for just the altruist portion

As I said in my first comment, this is phenomenal work you’ve done, and I am sure you are providing a better, higher QoS, more resilient altruist solution than PNI’s in-house solution and at a lower cost to boot.

That being said, in reading your proposal, one would be led to think that this community solution has allowed PNI to shed this portion of its infra costs, or at least the great majority of it. But in reading the comments left by PNI personnel, it seems that PNI plans to maintain substantial in-house altruist infra on top of this community solution. Which, if true, makes me wonder what the point was of offloading to the community in the first place.

So I guess I am more asking for @ArtSabintsev or someone else from PNI to step in and provide some clarification and context to the comments PNI has made in the comments section of this proposal and provide some transparency as to how much cost savings DAN provides to PNI.

For example if DAN allows PNI to drop monthly altruist spend from $25k to $5k, then we can deduce that the combined savings to DAO+PNI is ballpark $12k/mo (25k vs ~8k+5k)

But if DAN only allows PNI to drop monthly altruist spend by $3k (say because it feels obligated to keep capability in house “just in case”) then the result of DAN is to INCREASE combined DAO+PNI burn by ~$5k per month. Again, I am asking @ArtSabintsev or someone else from PNI to shed some light here.

3 Likes
26

some other concerns:

  1. Potential conflict of interest.
    You will sell this service to individual node operators and then also run the altruist. So those nodes will always agree with the altruist while others may not. Meaning nodes that use altruist chains can never be wrong.
  2. Centralization of the network
    If majority of the nodes started using CC, it can be terrible for the network.

I am more inclined to favor CC for altruist traffic only, only as a temporary solution. Ideally we should not be relying on altruist.

2 Likes
27

This is the concern I have with community chains, given it is a couple of steps away from bypassing the entire network entirely with a couple of “changes”.

Scenario 1: A large client asks to tap into CC’s backend and wants to create an enterprise deal to only pay FIAT for access to CC’s backends directly, not through POKT. Is CC going to accept this customer or not?

Scenario 2: If DRPC (a load-balancing service amongst node runners) also comes to the forums and proposes something very similar, would we hold the same sentiment?

This service has the opportunity to drive revenue in so many different directions. In fact, in a free market, there is not much we can do about it, and I have nothing against it. However, with funding from the DAO, I do believe there is a risk factor here on what the DAO is funding with the current terms. I would like to learn more about what the long-term plan is with CCs (outside of helping PNI with their altruist cost), and I think it is in the best interest of the DAO to set some form of limits of how much traffic this service can serve (i.e X percentage of relays) just in case in order to qualify for DAO funding.

The ideal scenario IMO is that this isn’t funded by the DAO given the nature of the product but rather funded by PNI. This also removes any restrictions from your service to monetize as it sees fit.

1 Like
28

Hey, crabman! Thanks for your comment. Our primary market is individual node runners. As Shane mentioned, one of the biggest problems in the Pocket ecosystem right now is wasted resources. Every node runner runs their own set of chain nodes and what we end up with is tons of money being sunk into infrastructure while each chain node is highly underutilized by anyone except large node runners. This favors big node running outfits dramatically since they run at scale and have enough pocket nodes to fully utilize their chains and justify the cost of their chain node infrastructure.

Community Chains is primarily about providing a way for existing node runners to make better use of their chain nodes by providing access to others in the community. Individual node runners are hurting extremely bad right now and are being priced out of the market. In order to remain decentralized, Pocket must retain these individual node runners. Community Chains allows them to run the chains that they can afford while also being able to serve relays via other chains that they cannot and in whatever region they need. This will allow individual node runners to not be priced out of node running. Lean Pokt went a long way to help node runners and this takes it the rest of the way in terms of reducing costs for node runners and providing a way to better utilize existing chain infrastructure in ways that benefit the whole network.

Our primary business has always been about empowering individual node runners and providing tools for them to do that. That is why we created Node Pilot. But, the rough crypto market has made individual node runners need more than just tools to run nodes, they need access to solid RPC infrastructure that they can choose by chain and by region in order to remain viable in the pocket ecosystem. We cannot let the entire node running community get swallowed by a few huge node running groups. Even now, I have heard people in the pocket community suggest that individual node runners should move their pokt to hosting services since running their own infrastructure is too expensive. But, as far as I am concerned that is the worst thing that could possible happen to Pocket.

We were already building community chains as a community service before anyone brought up the possibility of handling altruist traffic. Our primary business is helping node runners, not running altruist traffic. But, at Pocket’s recent state of the union gathering, they said that they wanted to find ways for the community to handle altruist traffic, since the community already is running the necessary infrastructure. It makes no sense that pocket run all the chain nodes necessary to handle all traffic in the case of a chain halt when the pocket network already has all that infrastructure there, distributed among node runners. Community Chains can provide that. If there is a chain halt, the existing chain infrastructure within the pocket network can continue handling those relays. Same goes for rollover traffic until Pocket releases their node-level solution for that.

As for where PNI gets their authoritative chain heights from, that is up to them and as far as I’m concerned that is a question for PNI, not us. We are a providing a network which allows existing node runners to further utilize and monetize their existing chain infrastructure in ways that benefit the community as a whole. Whether or not Pocket takes their authoritative chain heights from us is their decision to make. We just provide the pipes.

Regarding centralization, Pocket has been on a steep slope downward into centralization and the worst thing for the network in my opinion is for individual node runners to be priced out. Rewards are going down while chain node requirements and server costs keep going up.

Finally, I completely agree with you that ideally PNI should not be relying on altruist for much of anything. Its primary purpose is to handle rollover traffic (for the time being) and allow relays to continue to be served in the case of a chain halt. But, in the meantime while they work to solve those issues via pocket v1 and portal v2, we can handle that RPC traffic by utilizing RPC infrastructure that already exists within the network without pocket needing to spend time, energy, and money running their own nodes or building their own Community Chains type network.

1 Like