Join us in Discord for a monthly all-hands community call where we’ll discuss a range of topics, including protocol upgrades, DAO proposals, community projects, and any topic that you propose. To propose a topic, DM @jackal.
Here are the notes from the call. Thanks to @Patrick727 and @RichCL for writing these up.
1. Chain halt post mortem
May 28th - most difficult chain halt we have experienced
Why was this specifically so difficult for Pocket to overcome?
The bug that caused it.
the coordination effort. A feature of how Tendermint handles immediate consistency for every block.
On Friday - we figured out in the first hour.
Source of issue was tx indexer. Specific combination of Txs threw off the state.
Read only nature of Txs is helpful for …
But Tx Indexer, the default was storing redundant Txs. This was in 6.1.
When we went to 6.2 → we introduced a Tx Indexer that would remove redundant Txs.
Error showed an insufficient fee Tx, when we had sufficient fees.
Very niche case.
We believe it was a random collision issue.
Next step: How do we get back to consensus?
We decided to all get to 6.3, but when we got there we were unable to produce a block. Rounds got pushed back to 71.
Downside: not enough nodes were surviving to vote on the consensus rounds to produce a block.
Attempted Resolution: Brought together biggest node runners, tried to delete the wall file, then get 67% in isolation
Eventual Resolution: Code a patch and bypass those existing rounds. Start at round 0 again.
We got past the chain halt at this height.
Aftermath:
Difficult for Noderunners to
delete wal files
Txs that were living through 4 days of turmoil in the gossip
Final resolution:
we proceed with most on 0.6.3, but some with 6.3.2
Questions:
none in Slido or #community-calls
2. DAO voting period for RC 0.6.3 upgrade: 6/23/2021 - 6/30/2021
Some of the features are currently dormant because the consensus rule change has yet to be activated by the DAO
Next Wednesday 12pm EST, we will submit pocket gov upgrade tx to activate update stake, protobuf encoding, other changes mentioned in last Community Call
great time for Settlers of New Chains proposal.
Andrew: proposed changes
Merkle sum tree patch
Proposer selection patch
don’t hash in the vote data
protects us from a Grinding attack
Protobuf as the new standard for storing your evidence
As a reminder, a consensus breaking changes.
Allows UX update include Edit Stake Update
drop old chains, add new ones easily, add POKT to your stake
The reason why we haven’t been able to freely add new chains because there has been a chicken/egg problem for new or small chains
risk of targeted servicing
We initially wanted to get to a certain scale before we start issuing rewards, but then there’s an incentive bootstrapping problem before we get to that scale
First attempt: Campaign Net - isolated incentivized testnets
Second attempt: Centralized funding based on Dashboard data
Both efforts - Not super sustainable, and expensive
Instead, how about we control demand-side through Dashboard and subsidize demand for a period of time
Combine this with functionality, and this becomes MUCH easier for the Node Runners
no need to wait for 21 days period
In order to bootstrap a chain for a period of time, PNF provide a Minimum Viable Relays
once organic demand meets MVR, then we take off subsidize demand, or 3 months come about - we take off the subsidize
This is a temporary measure till future Consensus features can ameliorate targeted servicing risk
Inflation effect is negligible
<1% on total supply
small cost to bootstrap growth
Proposal summary:
would delegate power to:
choose new stakes chains
control max application parameters
give notice on new chains
We would start with the following chains
Avalanche Mainnet (0003)
BSC (0004)
BSC Archival (0010)
Fuse (0005)
xDAI (0027)
Ethereum Archival Trace (0028)
Ethereum Ropsten (0023)
Ethereum Rinkeby (0025)
Ethereum Goerli (0025)
Ethereum Kovan (0024)
Ethereum Archival (0022)
On radar for next chains:
Solana
Polygon (Matic)
Question: “Can we have the “realistic” expected volumes on each chains, to have that information taken into account into our analysis and voting?”
Shane: Reason we are choosing BSC, Fuse, Ava - we have been talking to their foundations, and DApps who have been asking for their endpoints
Now that we have Portal, we are in a good place to serve these new chains
Jack: we would like to discourage pre-staking to a chain unless you have pre-spun that specific node
Why? negatively impacting service for the Application if portal sends you traffic and your node is down
No need to pre-stake due to Edit Stake functionality
Question: “Does any other nodes providers have experience with any of those protocols, and with running them as full nodes ?”
Alex: Yes, absolutely. PNF has spun up fallback nodes for the majority of these chains.
If you run under docker, you can use one of my boxes with all of these new chains - just see in Node Runner channel chat history
Initially difficult, but getting a lot more stable
Re: resource requirements. We are running our transition from OpenEthereum to Erigon (TurboGeth)
space requirements are significantly more forgiving
separates the RPC from the data layer
easier to scale
faster than Open ETH
Also, we will be running them bare metal outside of Docker as well
Question: “Bullish 4 SOL, but big latency constraints. Will perf be sufficient?”
Have not run it,
Of course, it is enormous
Submission to DAO in the works: adjust relays for specific chains to help compensate for these more expensive chains
Koen: Re: Erigon - I have been following them but my experience is that they are not production ready. Data results vary across different versions. A bit concerned. They are pushing out patches, but there are data differences
Alex: I have been monitoring on this. Minor data differences, I would have to do more research. If you know of any major data differences, please let me know.
Syncing an archival nodes was taking 6-8 weeks. Without a filed version, it was near impossible.
