PUP-27: Raise PIP-22 Ceiling (RPC)

Attributes:

  • Author(s): @msa6867

  • Parameter: ServicerStakeWeightCeiling

  • Current Value: 60000000000

  • New Value: 75000000000

Summary:

PUP-21 set ServicerStakeWeightCeiling to 60k POKT, allowing up to 4x stake-weighted servicer rewards. This value was chosen over a more aggressive setting of 90k-120k (6-8x) due to the unknowns associated with turning on stake-weighted service rewards from the pre-PIP-22 baseline configuration.

The network now has 2.5 months of operation at the current configuration. During this time, PIP-22 is operating as expected and QoS has improved across the board between the approval of PIP-22 in June and today, as illustrated in the following data taken from cherry-picker data on the Polygon chain.

Latency (ms) Asia Pacific Europe N. America
End of June 579 268 397
2nd wk Sep 413 211` 295
Delta (ms) -166 -57 -102
Delta (%) -28.6% -21.1% -25.7%

It is proposed, therefore, to raise ServicerStakeWeightCeiling to 75k POKT. This is a sufficiently small change from current configuration that there is low probability it would trigger any degradation to QoS. Any consideration of changing the ceiling to a value higher than 75k would be deferred until network operation and QoS is first observed for a period of time at the new level of 75k POKT and would necessitate a new parameter update proposal.

Abstract:

Three questions of unknown Quality of Service (QoS) impact were raised during the July time period in which PUP-21 was formulated that led to the decision to set ServicerStakeWeightCeiling to a conservative value of 60k POKT rather than a more aggressive value:

  • Consolidation of nodes servicing lower-volume chains might decrease the number of nodes servicing those chains so much as to cause an inability to fill a session with 24 nodes, leading to potentially compromised QoS due to lack of competition / QoS differentiation within the cherry picker.
  • Lack of node consolidation on lower-volume chains compared to high-volume chains might lead to reduced rewards per relay for those chains followed by node runner decision to stop supporting those chains, leading to QoS degradations on those chains.
  • Consolidation of nodes with significantly less than, or greater than, network-average QoS may lead to diminished network-wide QoS and may cause enforcement of WAGMI/FREN inflation emission targets to negatively impact small node runners disproportionately.

The status of the above concerns, now that we have 2.5 months of operation completed at current parameter settings, are as follows:

  • There have been no reports that sessions are not being filled with 24 nodes on low-volume chains. Further with LeanPocket being made publicly available to node runners during the summer months, less consolidation has taken place to date than anticipated. There is no reason to suspect, at this point, that further raising the ceiling to 75k or 90k POKT would cause the onset of such issue. Note that generating sufficient supply-side support for certain high-cost/low-reward chains and regions is an ongoing issue that is unrelated to PIP-22-driven consolidation and should not hinder a modest increase to ServicerStakeWeightCeiling.
  • There is no evidence that levels of consolidation differ more than negligibly between low-volume and high-volume chains. Current data shows that chain-agnostic and chain-weighted levels of consolidation differ from each other by less than 1% (Currently, 1.90x vs 1.89x consolidation factor). Thus rewards per relay are comparable from one chain to the next and the expressed concern has not materialized.
  • One node provider brought network metrics to the attention of the community in late September that they felt demonstrated that issues related to the third concern above were being realized in the system and needed to be remediated. (See PUP-25). The author of this present proposal has spent extensive time over the last two months examining the claim and the data behind it and is satisfied that there is no QoS or fairness issue that hinders moving forward with raising ServicerStakeWeightCeiling to 75k. Posting this current proposal was deferred until a thorough review and disposition of all the PUP-25 claims had been completed. That disposition is complete and can be found attached within the comments section of PUP-25.

The absence of QoS detriments in response to current levels of consolidation suggests that a further modest increase to ServicerStakeWeightCeiling is safe for the network. It is proposed, therefore, to raise ServicerStakeWeightCeiling to 75k POKT.

Motivation:

Optimize stake-weighted rewards: Setting consolidation level of 4x in PUP-21 was not an endorsement of 4x being the optimal configuration point but was rather an act of cautiousness to allow observation of the system at that level before making any decision to ratchet higher. Now that the system has been observed, stake-weighted rewards is operating as expected, and Quality of Service is consistently higher than pre-PIP-22, higher levels of consolidation than 4x can be considered.

Reduce infrastructure costs: Medium to large stakeholders can reduce their node count by ~20%, over time, by moving stake-weighted rewards from 60k to 75k. While this may not seem like a large percentage compared to the initial savings of consolidating to 4x, every source of savings is important in the current market environment.

Increase Network Security: It is expected that as a result of raising ServicerStakeWeightCeiling to 75k, the amount of POKT staked to validators will increase by approximately 15%. While that may not seem much, any increase of validation-related network security is welcome, and it is low-hanging fruit involving no distraction to the development team. If the number of validators is increased to 2k via a separate proposal then the increased average staked POKT from this proposal will have a multiplicative effect when combined with raising number of validators.

End Worry that ServicerStakeWeightCeiling will be reduced to 30k or 45k POKT: There are several large stakeholders who have suboptimally allocated POKT to 30k and 45k nodes to hedge against ServicerStakeWeightCeiling being reduced to 30k or 45k and/or another mechanism being enacted that effectively eliminates highly-consolidated nodes from the network. Setting ServicerStakeWeightCeiling to 75k POKT helps ease concern that the Ceiling will be reduced in the present time frame in which the network is still over-provisioned from a node-count perspective.

Rationale:

Changing the ceiling to 75k rather than a larger value allows for observing system response for an adequate period at the new nigher level prior to considering any future proposal to change the ceiling even further.

Dissenting Opinions:

PUP-25 raises questions of system fairness and quality of service related to the current PIP-22 configuration; therefore, ServicerStakeWeightCeiling should not be considered while PUP-25 is outstanding.

PUP-25 has been posted on the Forum for almost two months. During that time, no other party in the ecosystem has introduced corroborative evidence for the claims being made. Further it has been almost one month since any person besides myself has interacted with or commented on that proposal. I have recently completed an extensive review of the PUP-25 claims and the data behind the claims, and have concluded, at least to my own satisfaction, that the concerns raised in PUP-25 are errors of interpretation and that there is no fairness or QoS issue that arises from current PUP-21 parameter settings that would stand in the way of raising ServicerStakeWeightCeiling modestly to 75k.

Furthermore, raising ServicerStakeWeightCeiling is orthogonal to any consideration to set the PIIP-22 weighting exponent to a value less than 1. If, in the future, bona fide reason to decrease the exponent is found, proposed and passed by the DAO, that new exponent value will operate just as will with a ceiling of 75k as it would with a ceiling of 60k.

Current setting of ServicerStakeWeightCeiling is already optimal.

I have outlined reasons why raising the ceiling to 75k provides marginally better optimization in the system than the current value.

Proposal does not go far enough; ServicerStakeWeightCeiling should be raised to a value greater than 75k

That is a question best left for the future. It is felt by the current author that the benefits of being safe in terms of QoS and network stability that come with increasing the ceiling by only one increment of 15k POKT outweigh arguments to make a large jump to some perceived “optimal” setting.

Copyright

Copyright and related rights waived via CC0. *

1 Like

First, I have posted an answer to your claims that there is no fairness issue with the linear model used for PUP-21 (and this proposal also). There I also comment on why that proposal was not moved forward.

Now, regarding this new proposal and ignoring the total absence of proof behind the linear model, I have some other comments.


TL-DR:

I see no reason for moving forward with this proposal , based on the following reasons (please read below for justification):

  • No real cost reduction for the large majority of the node runners.
  • No cost reduction for node runner clients.
  • No expected increase in the QoS (actually I expect the opposite).
  • The PIP-22 is not the right tool to increase the validators threshold. Using PIP-22 as the jack-of-all-trades is irresponsible.

Since the creation and approval of PIP-22 (June 2022) the Pocket Network changed a lot. At the time of the creation of the PIP-22 the cost of running a Pocket Node was high (too much sell pressure), the LeanPocket was not ready yet and the number of nodes (~56k) was creating too much bloat in the block size (and processing times). In this context the PIP-22 was able to rapidly reduce the number of nodes in the system and solve some of the mentioned problems.

The PIP-22 was chosen over simply rising the minimum stake since it was easier to accept, as it promised unchanged rewards for those that did not up-stake. I do not think that this was the best choice in the long term but it surely did its job quick enough.

Today we have LeanPocket and GeoMesh, these tools changed the paradigm of running nodes in the Pocket Network. With LeanPocket the cost of running one node, 50 or 100 is exactly the same in terms of hardware costs. Then, with GeoMesh, the cost of globally distributing a node running operation decreased dramatically, as no new Pocket Nodes were required, only the blockchains.


So, the first issue that I found is that this claim is incorrect:

Reduce infrastructure costs: Medium to large stakeholders can reduce their node count by ~20%, over time, by moving stake-weighted rewards from 60k to 75k. While this may not seem like a large percentage compared to the initial savings of consolidating to 4x, every source of savings is important in the current market environment.

As it is publicly known, the cost of running a Pocket Node or 100 or 400 is the same (using LeanPocket). Any large and medium node runner has LeanPocket (or some version of this) running in their system. This 20% reduction in cost exists only when the 20% of the total nodes of the node runner is exactly one server, this is only valid if you have 2000 nodes (only two node runners above this threshold as of block 76701, not sure if all their nodes are at 60K). No medium-size node runner will benefit from this. I see very little reduction in sell pressure coming from medium or large node runners operations.

Now, from the point of view of a client, who will sell tokens to pay the node runner provider, I see even less reduction of the cost. Most providers are charging by stake (even by unit of POKT staked) or have a reward sharing mechanism. As the cost of the node is no longer important. There is little to no impact on the sell pressure coming from node runner clients. Their sell pressure do not relates to the number of Pocket Nodes that the client owns.

Finally and most important, the author seems to be overlooking a very important factor of node running, the real cost comes from the associated blockchains not the node itself. Running a single Pocket Node requires to have one or more blockchains associated to it. Having one Pocket Node or hundreds of them make no difference, as the traffic that a well configured blockchain can handle is normally much higher than the expected traffic coming from a Pocket Node. While this highly depends on the blockchains that are being staked, the rule of thumb is that you can easily handle hundreds of Pocket Nodes per blockchain. Also, the reduction of the Pocket Nodes makes no difference in the expected number of blockchains that must be deployed. No cost reduction whatsoever.


Second, the author hints that there is an increase of QoS to be expected:

Optimize stake-weighted rewards: Setting consolidation level of 4x in PUP-21 was not an endorsement of 4x being the optimal configuration point but was rather an act of cautiousness to allow observation of the system at that level before making any decision to ratchet higher. Now that the system has been observed, stake-weighted rewards is operating as expected, and Quality of Service is consistently higher than pre-PIP-22, higher levels of consolidation than 4x can be considered.

While there was an increase in the QoS due to PIP-22, this was in a completely different network landscape. The increase in QoS (before GeoMesh) was achieved by a reduction of the number of nodes. This reduction lead to a change in how low-QoS and high-QoS nodes performed compounding and resulted in an increase QoS. Back then there was an economical incentive for the node runner client to un-stake and perform compounding, since the node runner charged them by node. This is no longer true in most cases. Also the cost of the Pocket Node itself was important, and small and medium node runners were also motivated to perform compounding. Today the cost of the Pocket Node is almost irrelevant.

Also, we are seeing that the network has divided in two kind of node runners, those with 15K stake and those with 60K stake. The node runners in 15K have lower QoS than those in the 60K group. This can be seen by looking at the average minted of each group. Probably the 60K group will be more inclined to perform compounding to 75K since it has already done it once.

The problem I see is:

  1. No economic incentive for compounding, I expect the 15K group to remain the same.
  2. The 60K group demonstrated that it is interested in perform compounding. I expect them to do it again.
  3. The 60 K group has higher QoS than the 15K group.
  4. This proposal will reduce the number of nodes in the staked in the 60K group (by a 20%) and keep the nodes in the 15K group constant.
  5. I expect the new network to have more low-QoS nodes than high-QoS nodes. Thus, the average network QoS will drop.

This motivation is difficult to prove:

End Worry that ServicerStakeWeightCeiling will be reduced to 30k or 45k POKT: There are several large stakeholders who have sub-optimally allocated POKT to 30k and 45k nodes to hedge against ServicerStakeWeightCeiling being reduced to 30k or 45k and/or another mechanism being enacted that effectively eliminates highly-consolidated nodes from the network. Setting ServicerStakeWeightCeiling to 75k POKT helps ease concern that the Ceiling will be reduced in the present time frame in which the network is still over-provisioned from a node-count perspective.

I think that many node runners are seeing that there is no point in staking in the higher bin (besides fairness issues):

  • Rewards are the same in any bin.
  • Pocket Node cost is negligible.
  • Having less nodes increase your Time Between Sessions.
  • Inflation control mechanisms (FREN, PUP-21 corrections) are being applied more frequently.

I don’t know if staking to the max bin and hope for being lucky and enter a big session is a better strategy than having lots of nodes and trying to get all the tokens I can before the next parameter change hits (inflation controls). Just a thought…

Also the author claims that: the network is over-provisioned from a node-count perspective. I would like to see where is the fundament of this claim. Right now (block 76704) we have 26140 nodes, blocks are being processed in time and block size seems controlled. As I said before, Pocket Node count has no large effect on the network cost anymore. Why are we still over provisioned?


Finally this is the only real effect this proposal will have:

Increase Network Security: It is expected that as a result of raising ServicerStakeWeightCeiling to 75k, the amount of POKT staked to validators will increase by approximately 15%. While that may not seem much, any increase of validation-related network security is welcome, and it is low-hanging fruit involving no distraction to the development team. If the number of validators is increased to 2k via a separate proposal then the increased average staked POKT from this proposal will have a multiplicative effect when combined with raising number of validators.

While increasing the validator threshold is always nice, since it secures the network, I think that PIP-22 is not the right tool for the job.

Using PIP-22 for this end brings many other “second order” effects that add complexity to the Pocket Network. We should discuss a cleaner way to increase the validators threshold (for example, assign an extra ticket to anyone with more than X POKT staked).

(I will later analyze the other proposal PUP-28)

Thank you for your comments and interaction with this proposal. Please see the MaxValidators proposal also; I would love to get your input on that as well. I will not have time this week to respond to comments in detail but can give a few thoughts on your tl;dr

Re “no real cost reduction”:
it is a one-for-one truism that smaller increments of change lead to smaller increments of cost reduction. But the eventual 20% savings in the incremental cost of node running - what ever that may be from provider to provider - is a real savings even if it does not have the “WOW” factor of doubling or tripling the paramter setting. I doubt the custodian who manages 60M POKT would agree that being able to reduce their node count by 200 nodes by virtue of this proposal would agree with the statement that there is no real cost reduction.

The spirit of both this and the MaxValidators proposal is to take smaller incremental steps that require less engineering work than shooting for giant leaps that get bogged down in engineering analysis.

Re “No cost reduction for node runner clients” This is a deceptive statement. Ultimately the cost to run nodes always gets passed to node runner clients. Even for clients who have pure rev-share SLAs, the revshare point (be it 60/40, 70/30 or whatever) has to reflect profitiability for the node provider. As node provider costs come down, via various mechanisms, of which this is but one), mor favorable rev share splits can be offered to clients - and will be if the provider wants to stay competitive in theis rather competitive markeplace.

Re “* No expected increase in the QoS (actually I expect the opposite)” I absolutely agree. I never claimed or implied that a byproduct of this proposal would be better QoS. But I do suggest that the QoS risk the system faces in going from 4x to 5x is greatly diminished, if not negligible, compared to the huge unknown in June/July associated with going from no consolidation to 4x. Now that we know that the June/July QoS concern did not materialize we can be pretty confident that the rather small step to 75k will not have negative impact. Back of the envelope calculations suggest that the worst case impact due to the bin-differential-QoS effect you describe is less than 5ms. And that is worst case and low proability to materlialize.

Re “The PIP-22 is not the right tool to increase the validators threshold. Using PIP-22 as the jack-of-all-trades is irresponsible.” The record will show that so far PIP-22 has been a very useful tool to increase validator security, in conjunction with PUP-19, POKT locked to validators has increased by a factor of 5x since June, and the lion’s share of that increase was due to PIP-22/PUP-21. I do not see what is irresponsible about that? There is no guarantee that PUP-28 (increase MaxValidators) will be received favorably or not run into engineering concerns. The tools in the toolbox to increase validator security is rather thin. It is responsible, not irresponsible, to look at all possible mechanisms to shroe up this weak spot in our network - especially ways that do not distract dev team from v1 tasks

2 Likes

I do not believe we should consider increasing PIP-22 bins until the validator threshhold has reached over 75K staked. If we do it before then, there exists a case where servicers are unintentionally or intentionally upstaking for a flipside of temporary validator rewards. This will cause an unstable validator set composed of validators and servicers who just wanted to uptake to 75K. That risk could result in consensus problems.

We saw this happen in an extreme case where a specific node provider at one point controlled over 50% of validators (no shame here, its what the network/DAO incentivized) when their intention was just to be servicers. It should never get to that point.

7 Likes

Creating another bucket is unnecessary.

As a builder in this ecosystem, the bucket system has made the life of builders significantly more complex since POKT relays no longer have standard value. Any tools in the POKT ecosystem already had to be rewritten to account for the changes of the first set of buckets, so I do not think it is wise to keep changing values within this buckets system. I would like to see all tweaking stopped unless absolutely necessary.

What might seem like a simple change of value requires A LOT of technical debt for all the tools in the ecosystem. Let us not keep burdening our builder community with unnecessary changes.

2 Likes

Agree with the blademan on this 100%.

3 Likes

I understand the concern, but this is very easily mitigated by defining a milestone gate to be met prior to changing the param value.

E.g., " The new ceiling of 75k would not take effect until validator threshold reaches (for example) 76k."

This would cause a natural bootstrapping to the new state in the desired order - those not desiring to be validators would not be incentivized to stake to 75k prior to being certain the validator gate would be met, while those desiring to validate would use passage of this proposal to rework their risk-reward balance and quickly raise the threshold to the 76k to 80k+ range.

Prior to PUP-19, validator rewards was something of an afterthought. Therefore in the wake of PUP-19 and PIP-22/PUP-21, those wishing to validate were fairly slow to react as there was a learning curve to overcome. Since then, node runners who wish to validate have become much more savvy in understanding the risk-reward balance of chasing a validator slot, and therefore can be expected very quickly to rebaseline to a higher threshold if/when this proposal (with appropriate validator gate added) is passed.

Last, a question: is it threshold>75k that you desire to see prior to considering raising ceiling or is it that 667+ validators are >75k? Note that the network is already at 575 validators > 75K, so we are very close even now.

1 Like

Thanks @shane . I understand the technical debt of PIP-22 and all the retooling in the ecosystem that resulted.

In comparison, any technical debt associated with raising the ceiling is fairly small, as this is a DAO parameter and likely not something hardcoded in any tools. Updating graphical displays is probably the only retooling needed. Further, I do not think raising the ceiling to 75k causes any strategy shift or new learning curve among node runners the way that changing the bin-size or exponent value would.

The value it brings is an eventual reduction of node count by 20%. While this may seem like a small number in an era with LeanPocket, etc, the savings add up. Using a rough-order estimate of $15/mo spend per month to maintain an extra node, allowing the network to shed ~5k nodes could reduce system-wide infra-spend (and resulting pressure on token price by ~$1M/year.