Great job!
This is a very needed feature for the current Pocket Network. This is an improvement over the old PNI altruist network and enables the DAO to take control of it.
I was hoping to read exactly this lol.
It clarifies that this is not a Pocket Network within a Pocket Network, it just a safety net. Nevertheless this safety net is not centralized and low-performant as before, it’s a decentralized and fast one.
Pocket Network is now decentralized (up to a certain level) even when it is failing .
The word I like to use is “distributed.” CC is a centralized network in that we hold the keys and providers have to prove themselves in order to be let in, but it is a distributed network because all RPC requests go directly to our providers. There is no central CC hub and we do not run any RPC nodes ourselves. Our whole business is providing a network where quality RPC providers can further utilize their existing infrastructure in ways that benefit the whole Pocket community.
Shane has my full support for this proposal. He, his team, the operators in CC, and a few operators outside of CC are the reason PNI has helped extend PNI’s runway.
At present, I am confirming we are using CC for most chains. One of our engineers, Vlad, also built a load-balancer on our end to load-balance requests between CC and other providers. Shane’s current concerns is that the other endpoints may not be load-balanced across regions, like his, so we are exploring a solution here as well, since some node operators would like to remain independent, but also provide altruist support.
At the end of the day, we hope to use the results of this proposal to have the DAO compensate all operators who support the altruist system.
While I’m open to PNI presenting their own solution to the DAO when it is production ready, I want to make clear that this proposal is specifically designed to put the altruist traffic under the authority of the DAO (since it is protocol traffic) and then delegate to CC with PNF’s oversight.
So far with PNI’s current load-balancing between providers, customers are getting more errors and latency than if PNI were to just use CC with all the providers we are already working with. POKT customers should get the best experience possible, and already CC has been able to identify anomalies in the traffic which has been used to find Portal issues. This kind of delegating of core-services, shared responsibility, and transparency is best for the network and its customers IMO.
From what we’ve learned from the altruist network already, I’d suggest that if PNI wants to invest into another alternative to bring altruist network in-house again, there should be:
Full insight into its analytics
DAO transparency on that traffic
Real time QoS (where if a providers goes down or is behind, minimal relays are dropped with proper rerouting)
Proper LBing (where multiple providers are being used simultaneous vs switching providers ever 5 min, like it is currently)
Geo-routing so each Portal is connected to local nodes (which currently only CC offers)
A holistic payment system for providers that involves the DAO and PNF (as we have extensively done)
Data driven stress testing (as we have done)
A proposal from the DAO that can be voted on (like we are doing now)
Personally, since CC already is providing a solution to the DAO for the altruist network, I’d prefer to see resources put into other areas of the network. CC’s providers already have this covered and have gone though extensive testing to verify that it can handle multiples of POKT’s current traffic. I don’t think CC needs to be reinvented and I don’t see the network value that will be gained from investing resources into an alternative, when it accounts for such little traffic. CC is already the alternative to the original altruist system and anyone can participate via the CC Community Towers.
I’ve moved this proposal from a Pre-Proposal to a PIP, since there hasn’t been any suggested changes to the proposal itself. All feedback thus far has been positive.
Feedback and suggestions are still welcomed, and edits may still be made, but it’s core does not seem to require changes, thus the transition to a PIP.
Following up that I made a mistake and how i phrased it. Our Altruist Manager is not a load-balancer at all. Originally, that’s what I thought Vlad was building so I ran with it. Vlad and Fred mentioned they will follow up in the PIP in the near future.
Thank you @shane et al. Absolutely outstanding work!
There are a few things I would like to see cleaned up and clarified in this proposal.
There is nothing in the proposal that makes clear that this proposal is the official conduit for handover of Altruist responsibility from PNI to PNF. I only found out that this was the intention in the comments section exchange with Art. It is somewhat implied by using te PIP heading instead of PEP, and the tags in the “Category” and “Implementer” attributes, but it should be made more explicit. I recommend to call this out front and center in the , summary and body of the proposal.
It is not clear that this proposal in the conduit by which DA becomes the official desginee for managing Altruist. "offer it as a way to delegate is not quite the level of it being the official designee. Again, it was not until the comments section exchange with Art that this intention was clarified. I recommend clarifying this in the summary and body of the proposal.
Combining 1&2, the last paragraph of the summary may be something like the following:. "PNI currently manages the altruist network. Upon approval of this PIP, all control and management responsibility for the altruist network transfers from PNI to PNF. Currently PNI already offloads a portion of altruist network servicing to CC. Upon approval of this PIP, PNF will delegate most, if not all, of altruist network servicing responsibility to DA while retaining oversight responsibility. PNF retains the right to utilize other vendors in addition to DA if it so chooses. "
As an aside, I support both these governance improvements; I just think it best to clarify so nobody is taken by surprise as to what they are voting yes or no to.
3)The ask and the budget should be clarified (particularly the budget since that is what is normative).
(a)under budget section, do the tldr first followed by details … that way ppl don’t have to scan through an extremely long budget section just to know the tldr.
(b) consider moving all of the text in the budget section that explains or justifies budget levels to a new “Rationale” section.
(c ) under tldr, move the infrastructure cost line item from recurring to one-time in keeping with other text that says its only for a 4-month bootstrap (e.g., Infrastructure cost: $12k (to be paid in $POKT all at once or in four equal monthly instalments ")
(d) I think I understand the intention re the small chains is that this be an optional supplement. If so, that should be made explicit rather than include this into the main budget. Perhaps it should be considered and voted on separately. Or perhaps we could use the multl-option voting feature that Snapshot makes available, e.g., “Approve - baseline only; Approve - with $10k.mo supplement; Reject”
I would like to clarify and offer some transparency into how we are handling the Altruist currently and the development of internal tooling to help this process. This was done in isolation of Community Chains and done explicitly for the purposes of business continuity on the PNI Portal, and does not affect the relationship with Community Chains or Decentralized Authority.
PNI has developed the Altruist Manager. The Altruist Manager is a self-service tool that allows our staff and members of the community to set altruists per chain on the portal based and monitor available altruists.
Compared with a Load Balancer approach, The Altruist Manager doesn’t proxy traffic, so it has the following advantages:
No latency overhead. Because Gateway connects to the Altruist Node directly
No Single Point of Failure. If the Altruist Manager stops working, the Portal will use the last set Altruist for each chain, so service remains uninterrupted.
Follows the principle of decentralization: anybody from the community can provide an Altruist Node; the Manager selects Altruists based on transparent and verifiable rules and data.
Now, how the Altruist Manager works:
Checks all the Altruists’ health every 5 min(configurable):
All the metrics are stored in a db, and can be made available to Altruist owners with further development to the UI(3).
Altruists Manager keeps records of when each altruist was set as a chain’s altruist on the Portal, this data is used to rotate altruists.
Altruist selector rotates altruists (round-robin) to balance the number of times each healthy altruist was selected within the last hour. Every 5 min(configurable) for each chain it requests healthy altruists from the monitoring db(1), orders them by the number of sessions it was set on the Portal(2), and takes the altruists with the lowest amount of sessions. In the future, any selection logic could be implemented.
The Altruist Manager provides a UI to manage the altruists. This allows for both our internal business users to collaborate with community members to add or remove Altruist Providers and for each altruist owner to login and manage their own Altruist offerings.
The Altruist Manager could be made geo-aware and regional in a further update. Right now the Portal supports only one altruist per chain, so Altruists-manager is deployed only in the US region. This should be easy to implement and inexpensive to maintain.
Lastly, the PNI Altruist Manager source code is going through a review right now, and will be made Open Source on GitHub in the coming days.
I hope this seeks to clarify some of our motivation in building the Altruist Manager, irrespective of the engagement with Community Chains and the DAN proposal.
After review of this proposal and many conversations leading up to it, I am in support of this PIP.
While it may be somewhat centralizing in the short term, I believe that Community Chains is best equipped to democratize the provider layer and report out the utilization of the Altruist Network until the core Altruist use case gap is closed (session rollover).
Regardless of whether this PIP passes, PNI’s approach to the Altruist Network will be as follows:
If chain is offered by Community Chains, CC will be the sole altruist provider for PNI
If chain is not offered by Community Chains, PNI will utilize a multi-provider strategy for its Altruist support.
This will be managed entirely through our Altruist Manager described by @vlad in his previous post, and we will look to make this list of providers available publicly in the very near future (Q2).
As always, I am grateful to @shane , Node Pilot, and the entire community for their continued efforts to better and improve the Pocket Network. I look forward to leveraging another new way to lower costs on the network and provide unstoppable backup infrastructure.
Thanks for the feedback @msa6867. I appreicate the thought you put into this proposal.
For some context, I did have another version of this proposal where PNF was seen as the “manager” and owner of the altruist network. In working with them on the concept, they felt that they didn’t have the technical knowledge to truly be an effective manager.
With that feedback, I reworked the concept where PNF is not the owner of the altruist, but more of an overseer. PNF knows their strengths so this proposal was designed to compliment them.
The intention is not to put for every aspect of the altruist network on just one team, at least at this first pass. The idea is to have this proposal be an authorized “default” process for the altruist network. CC acts as the default for each chain it supports because we have built in QoS, multi-region, a payment model, and will provide transparency to the DAO. For the chains we support, we can provide the best default for the network ATM.
For some chains, there may a need to be some other solution depending on the situation. I think @fredt put it best with his comment
This is the kind of flexibility we need. DA can’t be responsible for every chain on POKT at the moment, and we don’t want to be micro-managers over every chain and how providers get paid in other systems. I want to make sure this proposal doesn’t become to complex of a system where there are a bunch of hard lines.
My comment to Art was coming from a place of concern that a non-production ready product would be a default since PNI technically has control over the endpoints. As I mentioned in my comment, I feel there is tangible value to using a production ready system like CC and I believe the DAO has the right to establish it as what I’m defining now as the “authorized default”.
Happy to make this clearer in my proposal @msa6867. Will update in the next 24h
If PNI or someone else want to offer the DAO another “authorized default”, then I do believe it should be through a transparent proposal as we have done that accounts for all the elements in my comment to Art. I don’t think that we need to make it bureaucratic with a proposal to require future proposals, but I feel this proposal sets a precedent on how altruist traffic should be systematically handled, especially when it’s effecting customer QoS.
I’ll see where I can make some aspects of the budge more clear. Thanks for the feedback
Thanks for the clarity here @vlad. For greater context, we’ve shared our designs plans with PNI since November, over many calls, and have made changes due to PNI’s feedback. I greatly appreciate PNI’s willingness provide the same transparency here.
Thank-you for the continued support and all the additional clarity!
Since posting this proposal, I’ve been learning more about what all needs to be done to reimburse node runners who started participating early on when PNI first started to outsource the altruist network. Many node runners started spinning up chains with the expectation of reimbursement.
To account for this and to make node runners whole we can use the Rare RelayChainID Supplementation model to retroactively reimburse node runners. See here.
I’ve also learned more about the administrative task to follow through with all the reimbursements needed and establishing the Rare RelayChainID Supplementation, so I’ve included an administration cost. While Rare RelayChainID Supplementation monthly fee will go 100% to node runners, there is going to be administration overhead that should be honored as altruist work. See here.
I’ve included a Updates section at the top of the proposal with links to edits. I’ve also added an updated section to the Summary to make this proposal more clear, per @msa6867’s feedback.
It sounds like what you’re saying is pretty soon, once CC is able to snag enough folks to run the rest of the chains they’re not running now, that no provider will be allowed to serve altruist traffic UNLESS they do it through CC, am I understanding you correctly?
Connecting nodes to CC Community Towers vs connecting nodes to PNI is no material different, except with CC there is a method to compensate the node runners and it provides better QoS for customers.
The major difference is that CC Towers are run by a multitude of providers vs just being run by PNI.
Could not have said this better myself
Driving the Altruist Network on CC is a short term move that will provide transparency in the billing of the DAO for the Altruist Support and also provides QoS guarantees that the PNI Altruist Management system in its current state cannot.
In addition, PNI will continue to launch chains to the Network, and with that we will bootstrap the Altruist Network for those chains until the Community (through Community Chains) is able to provide support.
I hope this helps to clarify the “why” behind PNI’s approach to the Altruist Network.
I’m starting to loose sight of the big picture. I thought the point of PNi divesting altruist to community players was to save money and extend runway. From reading through comments section, it is sounding like PNI is, and feels obligate to continue, running an in-house altruist capability on top of and independent of community involvement. I’m loosing sight of the overall cost savings being offered by community involvement. Setting rare-chain supplementation completely aside for the moment, can we get a sense of what PNI monthly altruist budget was before community involvement and what it will be going forward assuming community involvement continues and this proposal passes. If the delta is less than the monthly budget of this proposal, it would seem that DAO dollars would be better spent having PNI resume full in-house control of altruist and PNI submit a PEP for monthly reimbursement of the delta. Just thinking like a project manager. There are two competing models for altruist and I’m trying to figure out which has the lower overall cost.
Regarding rare chain supplement, I am not in favor of bundling it with this proposal. There are a few concerns.
(1) it doubles the monthly budget (at todays $POKT price point) and I am not convinced the spend is worth it.
(2) I am not convinced that the DAO should be serving as an enabler for PNI to close contracts with chains that will be a liability rather than an asset to the protocol . If PNI want’s to sell chains as a loss leader, then let them bear the cost and subsidize it themselves. It is ok to say no to a new chain if we are not ready for the chain. A way to sink a project is to say “yes” to anything and everything, even if the business case for it does not close. I’m not saying a good case can’t be made to support the rare chains listed. I’m saying that the community should have the opportunity to discuss and vote on it separately rather than bury the matter as a suplement to an altruist reimbursement proposal. Even better, it might be best to decide on a case-by-case bases. For example, the DAO may want to say yes to SOL but no to BSCA.
(3) From a protocol/architecture perspective, altruist is the wrong vehicle for handing rare chains. If the DAO decides it wants to support a rare chain and finds that protocol incentives are insufficient to induce node runners to support the chain, then the DAO should offer grants to support the chain first and foremost on the protocol, not on altruist. This is an important distinction. Altruist is there to provide a fail-safe; it was never intended to be used as the primary vehicle for servicing certain chains. A condition of the subsidy could be that the subsidized hw be enrolled in cc altruist in case of fail-over, but this should incur no extra cost.
The cost to run the distributed infra is $3k (which is stated in the proposal) and around 100k POKT a month which goes to node runners an the QoS maintainers. A cheaper system would require node runners to get paid less (which that system would still need to be developed) and PNI invest their development resources (aka runway) into altruist work instead of Portal v2 work… all the while having worse QoS for customers.
I think just the QoS is worth the transition, but I also feel our cost model is very fair, especially considering this is a distributed system with multiple providers from the POKT ecosystem running CC Towers.
It will cost more to have good altruist infrastructure, but how else should rare chains be supported? I only see POKT’s reputation being hurt publicly if a number of supported chains have blackouts in service for 30s to 60 each hour… or at best, very poor service for 30s to 60 each hour without CC. Unless there is another way to incentives node runners to run infrastructure without getting supplemented, I don’t see a better option.
Just today it was suggested in Discord that the DAO provide grants for rare chains to improve QoS, and that’s what the rare relaychainID supplementation is. Happy to change the wording to grant though
I think altruist and rare chains is more closely related than one may think. To build CC we had to look at chain distribution, QoS, infrastructure capacity, etc. Whether it’s establishing a solid altruist network, or a solid rare chain network, it requires the same kind of research, development,coordination, and QoS accountability.
Because it’s all so closely related, it makes sense to combine work for the sake of efficiency. Any separate grant program would have the same coordination and infrastructure requirements, so we are actually tackling two needs at once with DAN.
.
