Appendix Responses
Appendix 1
This is just a recap of Mark’s presentation. Not pertinent to the document. Moreover, he said that this presentation had nothing to do with PUP-25, why is it now added to the review?
Appendix 2
The section starts with a false statement:
PoktScan postulated that, “This modification is required to …enable a stake compounding strategy that is beneficial for the quality of service of the network.” The implication is that the current PUP-21 compounding strategy is detrimental to the QoS of the network.”
This was not what we said. We know that the QoS improved and we said that in public forums and chats. This false statement seems to be a justification to add an analysis that is correct but irrelevant to the thesis of the presented document.
We only refer to the networks QoS as a whole in section 5.3. In this section we show that a non-linear stake weight can potentially improve the network QoS over the PUP-21 expected QoS. Surprisingly this section was not analyzed due to:
This violates the spirit and intention of PIP-22 and PUP-21 which is that in the current season, goal is to incentivize consolidation among as broad a swath of the community as possible, not to disincentivize consolidation.
The very same can be said from the whole document, why choose not to analyze this part?
The benefit in terms of QoS of reducing the number of nodes in the 60k bin are evident. Currently the average of the 60K bin is higher than the 15K bin, this means better QoS. If these N nodes in the 60K bin turned into 2 x N nodes in the 30K bin then the QoS rises.
Then it is stated that the change of P4 (a specific provider in the released data) affected the behavior of the network as a whole. While P4 is a big node runner, it holds only ~15% of the nodes of the network. It is unlikely that such node runner controls the behavior of all the other runners. Always entering in sessions with nodes from this provider is not as high as it seems. Moreover, the theory that many small node runners did the same (increase their QoS) at the same in the same period is not likely and very hard to prove.
There is a problem with the QoS changes statement in this appendix (that can be extended to the whole analysis), the changes in QoS do not affect (much) the linearity of a model, especially when this changes are somewhat linear, as seen in figure 2.1 of Mark’s appendix 2. While this is hard to explain in simple words it is obvious to people versed in the subject. If there is a linear relation between two variables and there is a change (of limited impact) at any given point in time or there is a linearly distributed change over time, the linearity remains strong.
This is an other reason on why we choose to use a statistical indicator (determination coefficient) over a handcrafted feature (the only one used in Marks analysis) to prove the absence of a linear relationship between the number of sessions by node and the number of relays by node.
The only way to observe a complete destruction of the linearity indicator due to QoS changes is that the changes in QoS were severe, occurred in a random fashion and distributed along the measured period also in a random fashion. This last requirement makes the argument of a massive change in QoS less likely.
Appendix 3
The appendix starts with a new calculation, called the “weighted” average. We will cover this in appendix 4, where the value is presented.
Now we will answer the different numerated statements in this section.
SECOND: The value (K_{fair} , algorithm 3) was chosen as a conservative reduction due to the uncertainty in its calculation and the negative impact in the community. We acknowledge the problem with the metric used to compute this reduction (the change of sessions by node to the change of relays by node), and thus we propose to use a more solid one to base our analysis (the determination coefficient). On the other hand you analysis is solely based on this metric that is known to be noisy.
THIRD: The issue that we rise is not related to stake bins, as we always said. This comment seems misleading. The justification of changing the linear PIP-22 to a non-linear one is to return gains to high-QoS nodes in the lower bin. All high-QoS nodes are gaining less, but only the ones in the lowest bin are earning less than before.
FOURTH: The K_{fair} calculation is not well defined. It should be based on the QoS distribution (in the network not the bins). This should be improved. Anyway it does not invalidates the need of a non-linear stake weight.
FIFTH: The need of changing the K_{fair} is only expected, as any other parameter. Optimal strategies always change, as in any dynamic system.
SIXTH: The shown curve is the expected behavior, as stated in section 5.3. It will be beneficial for the Pocket Network to change the stake distribution. In a non-linear stake weighting scenario, increasing the stake can only increase your gains up to a point, then you are forced to increase the QoS or accept to have a lower return per POKT invested.
Also, the cost of a node that is given by Mark (u$d 15) is completely arbitrary and far from reality (before and after LeanPocket). Analyzing costs of a Pocket Node is now irrelevant, having a single node or 100 nodes costs the same in terms of hardware for the Pocket Node.
SEVENTH : Once again the subject is mixed with the averages by stake bin, not the subject of the work.
The low-QoS nodes in the base bin will earn more with non-linear stake, yes, but also the low-QoS nodes in the high bin will earn less. If the token price pressure/inflation controls renders low-QoS nodes unprofitable they will start staking POKT to go the highest bin, in this scenario the non-linear staking will then punish low-QoS nodes… Relating stake bins to QoS is a repeated subject in Mark’s analysis. This argument makes no sense as the intention in the document is not to balance the rewards by stake bin for everyone, but only for high QoS nodes.
Appendix 4
This appendix reveals a fundamental (and troubling) flaw in Mark’s analysis.
Our document analyses the relays-by-node relation to sessions-by-node and its correlation to the QoS of the node runner. This means that we compare different groups of node runners.
To this end we gather information from 13 node runners, that after filtering, resulted in 10 for the Polygon chain (all calculation code is public, fell free to audit). So, we have samples from 10 different node runners. From these 10 subjects, 7 resulted in the high-QoS category and 3 in the low-QoS category. The resulting conclusions were drawn analyzing differences between these two groups.
The analysis provided by Mark ignores the QoS differentiation of the subjects and mixes all node runners into a single entity. This is a very basic sampling mistake. We are trying to achieve any conclusion over two different groups (low-QoS and high-QoS node runners in our case).
This problem can be clearly seen using the number of nodes by provider (using Mark’s average number of nodes):
- P1 : 145
- P2 : 255
- P4 : 2842
- P5 : 7693
- P6 : 919
- P8 : 986
- P9 : 457
- P11 : 217
- P12 : 1602
- P14 : 949
If we add all the node runners, the final sample will be an over-representation of P5, P4 and P12. Thus, this metric does not analyze how the different node runners behave, in fact, we can take out the node runners labels and simply calculate it using all the nodes in the network. What can we expect to see if we take the whole network? The same result, since Pocket Network is closed system, relays need to be somewhere and their number is independent of the number of nodes. This metric only proves that the sub-sample of node providers used for the analysis is accurate (thanks Mark).
The result of this metric is nothing but expected and they don’t shown anything but a trivial conclusion.
The numerous examples that follow in this appendix, all have the same problem that we have been discussing so far. The models provided by Mark (this analysis, PIP-22 and PUP-21) lack the required complexity to analyze the changes in the QoS of the network. They are all based on a linear modeling of the Cherry Picker behavior, he refuses to acknowledge that there is no proof that this is valid. There is no information in these examples. All their analysis focus on a macro view of the system that goes against a healthy and decentralized Pocket Network.
Appendix 5
The over-minting due to differences in QoS between the staked bins is irrelevant to the problematic being discussed. The subject was mentioned in 4.4 as an additional stress to high-QoS nodes in the first bin. The resolution (or not) of this particular problem is anecdotic. We have already commented on this in this thread.
Appendix 6
Once again, a linear modeling of the Cherry Picker effects, something that is not proved to be correct (in fact we proved otherwise).
Just a comment here, our original analysis of PIP-22 was criticized for including (among other things) additional stake entering the network. Now it seems that this is not a problem…