PEP-35: The v0 Optimization, LeanPocket

[addison]

Attributes

• Author(s):

  @addison Thunderhead Founder (longtime community member, ex-OTC exchange, insitutional and liquid staking)

  @Poktblade (Erick Ho) Lead Software Engineer. Contributor to Pocket’s ecosystem through PoktFund.

  @Poktdachi (Tommy Ho) Software Engineer with experience building infrastructure across big tech. Now contributing through PoktFund. Last seen at Infracon in the pool.

  @pierre - The dev behind pokt.watch and thunderpokt.fi

• Recipient(s): The funds would be dispersed as follows upon completion of a production ready client.

  47.5% Thunderhead

  47.5% PoktFund

  5% to the protocol team as they are starting to assist with QA

• Category: Reimbursement

• Fulfills:

  Addi’s post

  Blade’s post

  Adam’s letter

  There are several posts on Discord and Telegram expressing concerns about resource utilization as well.

• Asking Amount: We are asking for $2m USD in Pocket upon completion. The funds would be staked with a 3 month cliff and 15 month linear vest as we are committed to the growth of pocket.

  We understand the community may think this is an expensive proposition; however, the break-even after adoption of this client is ~1 month.

Summary

A major topic of discussion lately has been the cost of the network, and ways to optimize it.

Our two teams, Poktfund & Thunderhead, have been working independently on client optimizations since February and have created a client that could cut network costs by an order of magnitude. (See: our original idea). Although the outlook on the optimization by the protocol team was grim, our two teams decided it was feasible and worth the research. In May, our two teams joined efforts and collaborated to deliver an optimization, now coined as LeanPocket, that will reduce infrastructure costs by tens of millions USD per annum. Just recently at Infracon, we revealed the first glimpse into our progress and now this proposal serves as a discussion ground for community members to have a look at the work and voice their opinion regarding retroactive funding of this innovation. We are continuing development and working towards a complete, production ready client.

Abstract

After the two teams identified and researched potential solutions individually in February, we decided to team up and tackle this effort in early May. The core team told us this implementation would be near-impossible, so we decided it would be best to join forces to reduce duplication and drive a community collaboration. We’ve been working hand in hand everyday since then.

How it works

Context: The current state of Pocket requires you to spin up one Pocket Core Client per servicer or validator. This client is in charge of syncing the entire blockchain (currently ~240-300GB), validating blocks and transactions, listening to validator gossip, block syncs, and much more. As the network continued to grow, more Pocket Core Clients were spun up. This dramatically increased the network’s infrastructure cost unnecessarily because servicers do not require the same amount of hardware resources as a validator.

As outlined in v0 optimization, we saw an opportunity to improve the client’s code and decrease the amount of infra cost significantly. There are many different architectural design(s) to improve the client, and we brainstormed many different ways to do so as shown here in our initial design doc. The final way we decided to approach it was with a minimally invasive change, where servicers utilize the existing Pocket Core functions to handle relays, submit claims and proofs. All users need to do to get started is include the servicers private keys into a file. LeanPocket runs in the exact same way the existing client does, just with an additional keyfile.

The light client is a large optimization to the Pocket Core’s client code by allowing multiple servicers to utilize one full node. Servicers can now leverage the same state cache, blockchain data, and no longer have to validate as many transactions in a block as the number of servicers grows. This reduces the amount of resources needed for “n” nodes to a constant number, O(N) to O(1) for memory, space, io, and networking. Essentially, the amount of resources needed for a servicer converges to nothing more than a reverse proxy server sending requests to a chain node. The light client does not require any modifications to the economics of the protocol. Immediately after adoption the network becomes orders of magnitude more efficient, optimized, and cheap.

Since multiple servicers can run on one lean pocket client, there is no need to scale horizontally. You need one copy of the blockchain for n amount of servicers.

Adding additional servicers to the leanpocket client increases ram usage marginally.

This is seen on mainnet as well. The purple line denotes the memory usage of a leanpocket client with 5 servicers running on it (red) and a full node (blue). There is no demonstrable difference in memory usage between the two.

The following chart shows the cpu usage of LeanPocket with 5 servicers (red), and a normal full node (blue). There is no demonstrable difference. We will publish more comprehensive data as we produce it.

This optimization required significant research and innovative thinking into the current architecture of the v0 client, which was not easy to do given the lack of supportive resources for external contributors.

Motivation

The network currently spends a huge amount of money to function. At the time of writing this, there are 48.5k nodes on the network, at a very conservative estimate of $80 per node per month (Blockspaces and C0d3r, 43% of the network, both charge more than $200 per node per month), the network costs $3.88m a month to function. Between now and v1, the current network will conservatively spend $46-92m to function.

According to PoktBlade’s benchmarks, not perfect since there are differences on mainnet, yet still very indicative of high performance, demonstrate the possibility of putting hundreds of servicers on a single full node. This light client significantly reduces the cost of node runners, which benefits token holders and decreases the financial risk of running nodes. This is something that benefits the entire ecosystem.

Based on an AX101 server that costs ~$100 USD and allocating 20 RPS per servicer, our localnet benchmark hints that we can run approximately 50 servicers per full node for the cost of 12-13% CPU, 300GB of SSD, and 21GB of ram. This puts us around ~200-400 servicers per server and a cost of ~$0.25 to $0.50c a node under the worst case scenario that all nodes enter a session and the server can still handle all the requests (LOCAL NET TESTING). In comparison to the normal Pocket Core client, this would’ve taken ~60TB to ~120TB of storage, multiple SSD drives (more IO), 4TB to 8TB of RAM, and plenty more CPU cores. (There are differences between a localnet and the mainnet, but this is still very good).

Here is 10 servicers on one full node handling a total of 1k RPS on localnet.

There are other variable factors such as server hosting costs, mainnet overhead, etc that could skew these numbers. We are working towards gathering more production related statistics and will post them as part of this discussion as soon as they are ready.

At our ask of $2m USD and a conservative cost reduction of 50% and estimate of network spend, the ROI of this proposal is 11.5-23x. The network would break even in a little more than a month (48.5k*$80*1.04 months*.5) = $2.02m of savings. Node runners are approaching their break-even points, and this client would ensure that people never need to unstake because they can’t pay infra costs. We are approaching a critical point, and this client prevents any unstaking spiral.

This graph shows the cumulative spend until v1 (18 months) with a current network cost of $80 per node per month and 48.5k nodes on the network, a conservative light client cost reduction of 50% ($35m), Probable cost reduction of 75% ($52m), and an aggressive cost reduction of 90% ($63m).

We will continually provide new data points as the development progresses.

Budget

DAO administrator would disburse funds to each party after a production ready light client is delievered. Said administrator will coordinate with each team lead for their portion of the reimbursement.

The teams will observe a 3 month vesting cliff where all funds ($2M) will be staked on nodes.

Following the cliff, ~6.66% ($133k) will be eligible for unstake every month for a period of 15 months. (linear vest). We are committed to the success of the network, and are here for the long haul.

A break down of this would be as follows

Funds are split and disbursed to each team upon completion of the light client.

Month 1 through 3 all funds are staked into nodes.

Month 4 6.66% become available to be unstaked (though teams may opt to keep it in).

Month 5 through 14. Same as above.

Month 15 Teams have the option to unstake all funds.

Rationale

Alongside the financial breakdown in the “Motivation” section, it’s important to highlight that the light client is a non consensus breaking solution, and a very non-intrusive way for the network to be massively optimized. Other solutions such as stake weighting or stake-minimum have significant friction associated with node consolidation since they require an unstake. Although other solutions can be applied, the light client’s effect on symptoms should be observed first before applying proposals. This client leads to drastic network cost reduction immediately after being implemented and other economic proposals are supplementary.

Without the talent brought onto this optimization (@poktblade is a beast), this idea would have never made it to light before v1. Even after being identified as a potential solution by Thunderhead/Poktfund for v0, there were many who thought it was not possible/feasible and that we were better off waiting for v1 which is over a year out. The conjoined team saw differently and fast forward to today have accomplished the inconceivable. We want this proposal to serve as inspiration to other engineers to collaborate, join minds, and create waves of innovation. Likewise we would like to set the precedent that ingenuity is valued and impact is rewarded.

We are building the impossible.

Dissenting Opinions

1. This is too much money for a few months of development time spent.

While this is in fact a large ask relative to other proposals, it’s important to keep that notion of relativity when we think about the impact it will have. The amount of savings from this optimization makes the reimbursement amount look frail in comparison. And as mentioned previously, the ROI of this endeavor is tremendous, and will have immediate impacts.

@olshansky recommended the general flow for this development

1. The DAO provides a small grant (e.g. 1%-5%) to fund the research for this proposal. The deliverable for this grant will be a design document and QA plan.
2. The community and core protocol team members review and provide feedback on the proposed changes and testing plan. This can help scope the level of changes, highlight missing gaps, and give insight into the feasibility of the proposed design.
3. Assuming (2) goes well, the DAO could provide another small grant (e.g. another 1%-5%) to fund the development of a prototype.

We have completed over 66% of these recommended steps and are progressing through the last step right now.

Furthermore, we are asking for the funding to happen retroactively. We are going to get this done, and going to save the network a great deal of money in the process. The funds will also be locked for 3 months and vested linearly for another 1.25 years.

Other blockchain as a service centralized providers (infura, quicknode, etc) are backed by VC money and bringing in primo developers, we need to be competitive and encourage and incentivize more innovators into our ecosystem.

This is what the DAO’s treasury is built for. If this software was used privately, this would cause a large imbalance to the node running ecosystem increasing centralization as some node providers would be able to offer insanely cheap prices, while others could not. This optimization could have instead been sold to a large node runner. A node provider with 10k nodes paying $80 a month would save $2m from this client in 3.3 months, making purchasing this client a very good investment for them. However, we would much rather give this to the community and allow everyone to benefit.

2. This will deplete the DAO treasury too much.

We can adjust DAO allocation if this is a concern. This would push the cost back on node runners who are now saving a significant percentage of their costs from this change. With this adjustment, the DAO treasury would have earned the cost of this proposal back before the stake cliff even ended. It is possible to increase the DaoAllocation so that node runners subsidize this rather than the DAO. Node runners will have a small decrease in rewards, but a massive decrease in infrastructure costs.

3. This can cause a threat to the network.

Our primary goal is safety. We are doing extensive QA to ensure that it goes smoothly and that there are no negative externalities of running this light client. The core team will support us in this Q/A testing. One of the ways we are doing this is working with large and small scale node runners for beta testing. We are already testing the client in a test and production environment. We are also making sure that the light client works for validation as well.

4. This light client won’t actually lead to a cost reduction since there is no guarantee that providers will pass their savings onto clients.

Providers that do not pass savings onto their customers will be undercut by providers that will lower their prices drastically in order to pick up market share.

Deliverable(s)

We are ready to open source the client; however the core team recommended that we open source it once validator functionality has been implemented. We await their approval

github.com/fiagdao/leanpocket

Stay tuned

We already have a proof of concept running successfully on mainnet and have a localnet version capable of handling hundreds of servicers. The following is what we have planned.

In Progress right now ordered by priority

• Increasing RPS on one full node (est. 1 week)
  • Currently, the RPC of pocket is rate limited on mainnet which limits the number of servicers on each node. We are fixing this so that any amount of servicers can be added to a full node.
• Validator functionality - allowing multiple validator on one client (est. 0.5 - 1 week)
  • We want to make sure the light client is safe for the network under any circumstance, so we are implementing validator functionality for the extra servicers. The core team gave us this idea and suggested we do this.
• Q/A Testing (Timeline TBD)
  • We are going to undergo extensive QA testing to make sure there are no negative externalities.

Completed

• Proof of concept
  • We’ve built a client that allow for multiple servicers under one full node and it is running successfully on mainnet. We are ready to open source it.
• Validator design doc
• Localnet testing with 30 servicers on one full node.
  • We’ve completed local net benchmarking with 30 servicers on one full node (~95% cost reduction), with indications that it’s possible to handle 10x that.

Final Deliverable: Finalized light client (The three month cliff would only start here, upon completion)

• Finalized client with multiple servicers under one full node
• Validator functionality. The additional servicers will also be able to act as validators.
• Open source with documentation
• Extensive QA and testing

Community involvement

• Commitment to support until V1
• Technical talks
• Design documents
• Inspire/encourage other collaborations

We’re going to finish and build this, and so we are asking for all funding retroactively

We love pocket, and this is just the first of many contributions coming from partnerships between PF/TH and other community members.

the v0 optimization

fiag

Copyright

Copyright and related rights waived via CC0.

[shane]

I highly support the lean client effort, but I can not support this proposal.

Firstly, I am very excited about the aspects of the lean client and absolutely support your work. From what it sounds like, it’s great work and I highly commend all that has gone into it. Out of all of the initiatives out there to improve network efficiently, I currently lean heavily towards the lean client approach. It could allow scaling without having to change the balance of the protocol.

That said, I cannot support this proposal and here are a few main reasons:

Reason 1

While an innovative client is worth rewarding, we can’t disproportionately reward client work from 3rd parties over the innovations of core-devs. The reason more the core-dev team has not been pushing for more v0 client work is because they are focused primarily on v1. If the value of a new v0 client is worth $2M, then why would core-devs be motivated to work for PNI on v1? This would signal that two months of development on v0 outside of PNF is worth a 10 year salary.

@addison

it’s important to keep that notion of relativity when we think about the impact it will have. The amount of savings from this optimization makes the reimbursement amount look frail in comparison.

Using this way of measuring value, I think with the amount of expertise, man hours, and overall network impact of v1 would peg individual core-devs with a value upwards of $50M each. Thats not a sustainable evaluation in any ecosystem.

We can’t create an alternative economy within Pocket where devs are incentivized to focus on massive DAO rewards over working at PNI. Core-devs have created are underpaid compared to what is being asked of the DAO. I do not see how that is sustainable as it create counter incentives within the same ecosystem.

I believe that the lean client work should be rewarded, and I’m totally fine adding rewarding premiums for it’s value to the network and for saving PNI resources from being taken away from v0, but we can’t use DAO grants to disproportionately reward 3rd parties disproportionately to the work happening at PNI. That logic just isn’t sustainable.

The DAO’s treasury at current market value is $12.5M, so if the DAO is willing to pay about 1/6 of it’s total funds for the lean client, then how much should v0 devs be paid? These numbers are just not sustainable.

Reason 2

We need to treat proposals equally as a community. The POKT Lint proposal was give a lot of pressure from a member of this proposal team to justify and quantify the value of that proposal. Prior to that proposal, value was more generalized for each proposal, but there was now pressure get more specific with quantifying it’s value, even though it was a reimbursement.

I receded that proposal in it’s current state to created a Proposal Value Model as to address quantifying value of proposals. It’s been reviewed by many DAO members and @noproblem has used it in his pokt.tools proposal as a trail run. Once pokt.tools is finished with the feedback phase and it’s up for vote, then I plan to re-submit POKT Lint using the Proposal Value Model.

We need to have consistency with how we expect proposal team to value their proposal. We can’t require some proposal teams to quantify value in very specific terms and include man hours, while other proposal teams can be very abstract and subjective.

Reason 3

Since the lean client is reliant on PNI’s core-devs to properly QA and verify that it will be health for the network, I’d prefer that it be approved and open-sourced before we vote on this proposal. This proposal is unique since it does require PNI to help ensure it’s safety. If the core-dev team approves it, then I am stoked and would love and reward the this team. Even if it’s not approved, then I still think that the team should be rewarded for the time and effort.

TLDR

Big fan of the work and hope it succeeds, though I feel the value of this proposal is too abstract/subjective (leading to over value) and can create incentives for core-devs to focus on grants for large rewards over working v1 with PNI. This proposal also does not quantify value to the same standard that has been put on past proposals. The timing is too early as PNI has not yet released their review regarding it’s feasibility/safety.

[iaa12]

Is the intention for this to be pulled into the official Pocket codebase and supported by the Pocket team going forward as official implementation, or will it be third party software? If it will be maintained separately, will QA coordination be required with every update?

[StephenRoss]

@shane makes some excellent points.

We are ready to open source the client; however the core team recommended that we open source it once validator functionality has been implemented. We await their approval

github.com/fiagdao/leanpocket

Stay tuned

Do you have an ETA for open sourcing the client? Staking 15,000 pokt on a closed source client is seriously problematic regardless of how efficient it is.

[Poktdachi]

This will be open source. We do not offer any source code to be built and ran by the public right now and we won’t until we implement validator functionality. We wanted to open source now but core team advised not to because it was not production ready. ETA(s) are under deliverables section. I hope that adds clarity.

[luyzdeleon]

I’m thrilled by the fact that the Pocket Network community has grown to a point where such innovative work is being carried out. While the network has grown massively in the last 12 months, we on the Core Dev Team have been doing our best prioritizing stability of the core client, with releases such as cache improvements and security patches, and UX improvements for all node runners, such as config hot-reloading and non-custodial staking, which is why pursuing ideas such as a Servicer/Full Node de-coupling didn’t get to the top of the priority list.

While the entire core team and I are elated this proposal exists and it’s coming from such capable community members, we have to disagree with this proposal in its current state for the following reasons:

1. This proposal indicates that this LeanPocket client will be a parallel client to Pocket Core.
2. This proposal is based on a prototype specification, not a production-ready one.
3. This proposal assumes the Core Dev Team will receive compensation for producing QA work, which is not how the Core Dev Team operates.
4. This proposal does not contain a comprehensive Project Plan, describing responsibilities, tasks, end-to-end QA spec, testnet rehearsal and mainnet deployment plans.

I would like to propose a plan to upgrade this proposal from its current state as a prototype parallel client release to a Release Candidate (RC) of Pocket Core.

1. We’re currently developing an External Contribution program that would allow not only the authors of this proposal, but all capable community members, to make proposals such as this and continue innovating on the V0 implementation. This program while still in development has its roots in this PR which seeks to revamp the public Contribution Guide for the Pocket Core repository: Revamped Contribution Guide by luyzdeleon · Pull Request #1434 · pokt-network/pocket-core · GitHub. On top of this, an announcement will be made early next week for our Pocket Core Contributors Office Hours, which will allow any capable contributor access to the Core Dev Team to get guidance and clarity on priorities, roadmap and other details of contributing to the network in an inclusive and open manner. We would love to work with the teams behind this proposal under this framework to standardize contributions in an open and transparent way.

2. The team should start by writing a spec in the form of an issue in the Pocket Core repository. Just like any other RC release, this issue should specify the proposal technical specification and open it for discussion so any architectural flaws can be caught by the Core Dev Team or other community members on the issue for an open and transparent discussion. After that, a fork of Pocket Core (or any of its dependencies) needs to be started where the actual work will happen. This work will be entirely owned by the team behind the proposal and can be carried out in any fashion or methodology the contributors prefer. When technically implemented in their fork, a Pull Request should be done to upstream the changes to Pocket Core (or any of its dependencies), which should include a detailed report of the Regression Test Suite, alongside all the architectural, implementation and design documentation where applicable. This would merge the contribution to Pocket Core, to be picked up for the next BETA release of the Pocket Core software.

3. A testnet rehearsal and mainnet deployment plan should be drafted up, indicating possible pitfalls, dangers and recovery plans under any applicable scenarios. These plans should be discussed in the open in the Github issue outlining the specification for the proposal for a clearer audit trail.

4. The Core Dev Team will never be responsible for executing manual QA labor for external contributors. The Core Dev Team role is to provide a QA framework that’s acceptable and clear for contributors to expand with the new testing scenarios their new contributions bring to the table. This is part of the Core Dev Team duties and not something that should be reimbursed every time an external contributor contributes to the Pocket Core client. Currently as part of the Contribution Guide PR the plan is to include our 140+ scenarios we use for every RC release to ensure the quality of each release, please stay tuned for that for further clarity.

Additional Context

The Core Dev Team was approached by the Thunderhead/Poktfund team for input and technical commentary on their proposed architecture. The Core Dev Team, as with any other approach for open collaboration, responded by identifying pitfalls such as the lack of a comprehensive test suite and the fact that the prototype didn’t contemplate enabling validator functionalities, which would cause a consensus failure in the event of the feature achieving majority adoption by nodes. In response to that we created the feather branch in both Pocket Core and our Tendermint fork repositories:

GitHub - pokt-network/pocket-core at feather

GitHub - pokt-network/tendermint at feather

Which shows our own code-complete prototype, intended to serve as a guiding blueprint for Thunderhead/Poktfund to complete their prototype and then drive it to release. Because this project is not in the Core Dev Team’s roadmap, we created this prototype to incentivize the collaboration and help the team achieve the goals of their project as quickly and safely as possible for the network.

Closing thoughts

I’m personally delighted that this project is moving forward and the implications it can have on the network, and I’m looking forward to collaborating openly and transparently through all the indicated public forums with the team to drive it to completion.

[steve]

I highly support this proposal. The potential economic benefits for node runners and the network overall make this a no-brainer. I disagree with the concern @shane has about the cost. The value of a solution should be based on the economic benefits/value it provides - not how much time did or did not go into it, or who created it. Not all time spent / invested creates the same-level of return on effort and valuing contributions based on the time spent discourages people from trying to find the most efficient path to quantifiable results. But in the case of this proposal, the value is clearly measurable so the costs are absolutely justifiable in my opinion. This said, I also agree with @luyzdeleon and think that this proposal needs to include a detailed technical implementation plan that clearly shows the effort that will be required from the core-team. Without a detailed plan, my concern is that a detail could be overlooked and become an issue that distracts the core team from their focus on v1. I would also suggest that the technical implementation plan be open for comments from qualified community members.

[shane]

@steve

The value of a solution should be based on the economic benefits/value it provides - not how much time did or did not go into it, or who created it.

I can appreciate your disagreement, but measuring value in that method is unsustainable for future contributors and core-dev members. The core-team consists of devs who have worked for over a year creating immeasurable value for the network, and their features have not been rewarded with $2M from the DAO.

Why should the DAO reward outside contributors with rewards based on the value benefits to the ecosystem, while allowing the core-devs to receive significantly less for their contributions? How is that sustainable if the feather feature is worth 1/6 of the entire DAO treasury? Why should core-devs work on the clock on a feature idea they may have, when they could privately release it for payout that is 10x what they get in their day job?

[steve]

@shane

The core-team consists of devs who have worked for over a year creating immeasurable value for the network, and their features have not been rewarded with $2M from the DAO

I don’t know how the core-devs are compensated or what motivates them individually. However, if one of them came up with a proposal that made sense , I’d support it too - even if they wanted to make a lot more then they are currently making. I also don’t know how @addison, @poktblade, @Poktdachi, or @pierre are individually compensated or what motivates them. So, I’m not saying anyone is more or less valuable than anyone else - I’m just saying that the potential returns from this proposal make the cost a no-brainer in my opinion and everyone will be better off as a result.

[oskar]

Based on the responses from the core team here, you guys are wasting your time trying to be innovative for the good of the community. Everyone is complaining that node provider costs and resulting sell pressure are causing token price to collapse - you come up with an elegant solution and ask for ridiculously low compensation and all the core team does is complain that’s unfair they aren’t going to get paid or that you didn’t follow their procedures. You know what -that core dev team could have spent nights and weekends working on something similar while still working on v1 optimization. Not saying they aren’t working hard, but a project at this stage there is always more time in the day to work harder (that’s what chemical stimulants are for). Your teams have your own businesses you are running plus everything else you have going on in your lives and you still found the time to put this together. Bravo. This thing is worth triple what you are asking for. Just sell it to a node runner or implement it yourselves and just let the register ring.

[iaa12]

I agree with this. I think the methodology for determining reimbursement value that was put together is sound, and extremely generous with bonuses. It’s no different than going out and outsourcing a feature to an outside contractor, you’re going to get a rate per hour and that’s what you pay. Considering the amount of work remaining to be done on this per @luyzdeleon 's post, I think it will still be a solid reimbursement amount when it’s all said and done.

[Sevi]

Hi Shane and Steve,

Appreciate both of your points but strongly agree with Steve. The ROI on this is significant. Not only would you save millions of dollars per month in infrastructure costs, but it would lighten up the sell side pressure on the underlying asset. With the current trajectory, you will see node runners shut down and that is the last thing that should happen. Folks will be dissuaded from participating in the ecosystem.

As far as the reward amount. What the core devs get paid, or what anyone gets paid is irrelevant in this discussion. If a core dev wants to create a utility that makes them millions, then we should support it. That is awesome. Innovation should be rewarded, and innovation should not be compared to what someone gets paid. If you look on chain, you’ll be able to see what some of these core devs have been paid in terms of token distribution, I think you underestimate their upside and frankly, if you’re concerned about the core team compensation, then you should support this client with all your might as the ROI on the entire ecosystem is in the millions. Each of these devs, the foundation itself, are all running nodes. Everyone in this entire ecosystem will benefit from this client.

Lastly, the team approached the core team about this client in Feb. There was some community chat about this client, but it was largely ignored. Why? The rewards were high, and the price had strength. This same team, started to work on this and take time away from running their respective businesses. There is opportunity cost to support the ecosystem if far greater than most can imagine.

The focus right now should be on the client. I think we all agree we need a client. I think we all agree that 35m+ can be saved if implemented correctly. I think if we all realize that a private group will be able to get this to market quicker and with less distraction to the timeline of v1. I also think if we don’t do this, there will be some node runners, that buy this privately, get a multi-month headstart on the public community, and will be able to gain market share at a degree that is limited by the imagination itself.

The focus right now should be a community getting together. Getting the best minds together and getting out a solution that is effective, efficient, safe, and blessed by the team. There will always be varying degrees of debate on what is fair economically.

[iaa12]

I’m going to say something that’s probably going to be very unpopular.

Our root problem, right now, as a network, is not infrastructure costs. Our problem is exceedingly high rewards , which enable extremely high node bloat, which greatly exacerbates the inefficiency problem this proposal seeks to resolve. Folks, we’re doing 1 billion relays per day. If we monetize half of that at $400/million relays/month, that’s $200,000/ month in revenue. We’re minting 1,8 million tokens per DAY, or $216,000 worth. We are minting 30 times more tokens that we would burn if ABR was active. We are DESTROYING OUR OWN STAKES with every token we mint.

For reference, the reimbursement amount being asked for here is larger than the entire revenue of the network would have been for the past year, had it been monetized.

So, first order of business is to fix our economics. Stop making it possible for 47,000 mostly idle nodes to exist via grossly inflated rewards. Potential revenue and node rewards MUST - and WILL - converge, if not via inflation control, then by POKT going down to a cent. Rewards must be brought in line with our earning potential, and our network must be scaled appropriately based on current traffic and projected growth.

So, I think we should tackle the economics, then we can return to this and re-evaluate its value proposition when the network is not as bloated with idle nodes as it is now.

Edit: I guess my point is, there’s other stuff in the works that will significantly reduce infrastructure costs soon, like Jinx’s weighted stake . Which, combined with WAGMI, can be used to greatly reduce infrastructure costs. Thus making the estimated savings of this proposal grossly inflated. I think it’s still valuable and it should be implemented, but the ask amount is over the top.

[JackALaing]

The procedure points that Luis is reinforcing simply seek to ensure that any significant client upgrades (which this would be, given how innovative it is) are thoroughly tested before going live in production.

Some of these procedure points are misunderstandings because this is our first time coordinating with external client contributors. These will be cleared up on a call between the devs and I expect the proposal will be tweaked accordingly.

Rest assured that we could not be happier to support this team’s effort.

[shane]

@steve

So, I’m not saying anyone is more or less valuable than anyone else - I’m just saying that the potential returns from this proposal make the cost a no-brainer in my opinion and everyone will be better off as a result.

I learned about the details of the lean client at InfraCon. I was glad to hear that a lite client could improve the network without doing weighted stake (which I was originally primarily supporting). But when it comes to how the DAO should determine value, this method is completely abstract and void of structure. On an organization level I can’t see how this is a sustainable model.

With this method, for each client release, the DAO should be rewarding 10s of millions of dollars worth of DAO treasury because of the value a new client brings to the network. DAOs have to operate with organization, and there is no organization giving this contribution special treatment over others. What is the purpose of the DAO if not to be an organized entity?

@oskar

you come up with an elegant solution and ask for ridiculously low compensation and all the core team does is complain that’s unfair they aren’t going to get paid or that you didn’t follow their procedures.

I haven’t worked for PNI since December. I work full time on Node Pilot, so I wouldn’t attribute my comments to the team complaining. The DAO has always had it’s own procedures… which is why we are discussing this. This proposal is asking to be valued in a manner that isn’t sustainable to the organization or consistent with how other client contributors are treated by the DAO.

@Sevi

What the core devs get paid, or what anyone gets paid is irrelevant in this discussion.

This notion is creating two classes of contributors. Both are working on the same code, yet the 1% contribution (that isn’t from PNI) is treated differently in the DAO than the 99% contribution. Was not 99% of the code that was forked to built this client from other contributors that got paid wages from the PNI? I don’t see how the 1% should be treated to 10x the yearly salary of a senior dev when this client was started just a few months ago.

There isn’t any organization in suddenly treating the 1% of contributions as a different class than the 99%. This 1% has awesome value, so I’m down to provide an awesome reward… but not an unrealistic reward that creates imbalance.

@Sevi

Innovation should be rewarded, and innovation should not be compared to what someone gets paid.

There is not organization that I can see to this methodology. Reward should not be subjective. It’s been objective in the past when considering rewards for proposals. All proposals for innovation were rewarded according to metics like time and resources involved. This proposal is being structure more as a bounty vs a grant.

What is being ask here is to retroactively create a bounty of $2M for a improved v0 client. This can’t possibly be a sustainable model for DAO, especially one as small as ours.

Frankly, I’m also confused how this plays out in a project that is suppose to have an open-source ethos. How can the DAO treat some newer contributors to an open-source project different from all the developers before them? Gaining new developers, from 3rd parties is awesome, but the DAO can’t provide preferable treatment to different contributors.

The DAO is suppose to compare everyone’s contributions to ensure there is equality and provide an organization that is reliable to all contributors. The ask here is to now NOT compare contributions for this proposal specifically, and that just doesn’t play out as sustainable or equal in the open-source world.

@Sevi

if you’re concerned about the core team compensation, then you should support this client with all your might as the ROI on the entire ecosystem is in the millions.

This logic also applies to TH. Doesn’t TH already have the 4th largest pool of nodes? Why should the DAO provide these contributors 1/6 of it’s treasury if this will save TH millions of $ already? With TH’s level of profits already coming from the POKT ecosystem, this improvement also has a self serving element, which in the open-source world would be justification enough for contributing to the open-source code, which the business has been leveraging for profit for years.

I’m not trying to undermine the contributions here, I’m just really trying to understand why this proposal should be differently and these contributions be treated with a substantial DAO payment. I feel like I’m a broken record here but there has to be structure, suitability, and equality with the DAO funds.

[o_rourke]

Thanks for taking the time to build this guys. The sooner we can get this out, the quicker it will add relief to current node runners. I urge you to continue working with the protocol team to get this fully specced out, tested and released.

This release should without a doubt be merged upstream with the main client, and not be a separate one. Otherwise we’d be doubling QA work for no reason.

Seeing these comments, I can’t support the current ask of $2M worth of POKT for LeanPOKT.

The purpose of the DAO treasury is reimbursement for labor and services, not for profit.

Pocket is building public good infrastructure, and anyone receiving recompense for their work should never reflect on what potential savings or profit may result from their contributions. This value-based reimbursement methodology is dangerous in a public good environment.

Here’s why:

Imagine the government builds a railroad across two cities. The commerce between the two cities begins to pick up. The government puts up an RFP for solutions that solve for more commerce between the two cities. A private company designs, specs and builds out this high speed rail with the help of the government.

This solution will massively increase the economic efficiency between the two cities. Should this company respond to the RFP and ask the government to reward them based on the economic gains? No, they will submit the RFP for reimbursement on hours and materials spent. They receive outlying economic benefit for being the company that built it out.

This is a very similar situation. You are are a business contributing to a public good. Your incentives are such that by building this improve the long term prospects of the network. The amount of success you have is compounded by a successful light client. You’ve also catalyzed the core team to focus on it’s importance in relation to other aspects being focused on.

The time spent working on the client itself should determine what the payment is. This is unique, extremely niche work and should be paid equivalently (paid well). A milestone payment based on adoption of the client itself could be interesting. I would also look for the actual completion and deployment of the client itself before asking for reimbursement.

In sum, the scope of the reimbursement should be reduced to the reasonable hours spent in building out the client itself. Shane’s proposal value model is a good framework for how to consider proposal value.

On Vesting and payment

@steve

The value of a solution should be based on the economic benefits/value it provides - not how much time did or did not go into it, or who created it. Not all time spent / invested creates the same-level of return on effort and valuing contributions based on the time spent discourages people from trying to find the most efficient path to quantifiable results.

@Sevi

As far as the reward amount. What the core devs get paid, or what anyone gets paid is irrelevant in this discussion. If a core dev wants to create a utility that makes them millions, then we should support it. That is awesome. Innovation should be rewarded, and innovation should not be compared to what someone gets paid. If you look on chain, you’ll be able to see what some of these core devs have been paid in terms of token distribution, I think you underestimate their upside and frankly, if you’re concerned about the core team compensation, then you should support this client with all your might as the ROI on the entire ecosystem is in the millions. Each of these devs, the foundation itself, are all running nodes. Everyone in this entire ecosystem will benefit from this client.

As Sevi and Steve say, developers should be compensated fairly for the work they do. I completely agree with this. That said, developers should be compensated through consistent shipping and adoption of that work, as our own organization and core contributors have done. TH’s incentives are such that this meaningfully impacts their businesses and services with the successful adoption of this client. I would be happy to see a revised proposal that highlights a well paid contribution plan on supporting a client like this.

I would also be open discussing what it would look like for the DAO to fund TH’s permanent development and maintenance of a light client. This should be limited to the hours you work, and funded on a monthly or quarterly basis, and should be a completely separate proposal.

This is similar to how we work today - we have 4 year vesting for all core hires, with monthly disbursement to allow for incentives over a long enough time horizon. Why should this be any different?

This proposal does not allow the DAO to keep you accountable for your contributions, and is something that I believe is critically important for the sustainability of the ecosystem. Setting a precedent of such large upfront reimbursements for a prototype is not the way to go. The amount of POKT (even if it’s vested, there are no requirements for what it looks like to rescind if it does not get release or adopted) being asked for up front does not help keep the team long term aligned - you should be paid based on the support you continue to do over time, not a lump sum payment vested over 15 months.

That’s why the DAO treasury exists, to avoid the tragedy of the commons.

@Sevi

Lastly, the team approached the core team about this client in Feb. There was some community chat about this client, but it was largely ignored. Why? The rewards were high, and the price had strength.

Yes, the team was approached but frankly, we were focused on growth and proving out that this network can operate at scale. Nothing else mattered. It’s what has allowed us to have the very conversation happening on this forum today. We more than tripled our scale from 300m to a billion in a few short months. Throughout this time the entire (fairly small) team has been focused on keeping things alive to enable this growth.

This is why I appreciate this proposal so much. If you saw Alex’s keynote at Infracon, one thing the public doesn’t see is the amount of work that goes just to support the protocol and the quality of service to make the current situation happen. To have others help take up the slack is a massive help. We’re able to focus on the other aspects of the network that push things forward.

@addison

Asking Amount: We are asking for $2m USD in Pocket upon completion. The funds would be staked with a 3 month cliff and 15 month linear vest as we are committed to the growth of pocket.

Everyone involved should be compensated fairly for their effort in this light client. This is hard work, and you should be paid well, but this amount is unreasonable for what is being produced.

[treefiddy]

@o_rourke

The time spent working on the client itself should determine what the payment is. This is unique, extremely niche work and should be paid equivalently (paid well). A milestone payment based on adoption of the client itself could be interesting. I would also look for the actual completion and deployment of the client itself before asking for reimbursement.

I think that the amount proposed, is fair. The issue is that unfortunately, the price of pokt right now, is really low.

I would suggest that Pocket Network find a way, if possible, to bring the team that developed the light client on, as official team members. And potentially use a combination of salaries, with vested pokt, to get as close to the $2m as possible.

I am sure that the Pocket team, does not pay it’s developers, by the hour. In defi, projects have a core team, of salaried members. Any work other than that, is incentivized work, because it is niche or because it needs to get completed quicker. And because of those reasons, it is paid for by the entire task, as a whole.

[crabman]

You need at least 6+ of testing this to make sure it works. It is for the DAO and team to decide how close will they be to V0 in the next 6 months and if this client is worth the $$ you are asking.

[adam]

I’m so happy that you all are driving forward an outside contribution to Pocket Core v0. I think more community members ought to be adding value like this group is.

I’ve got some thoughts (and charts) to dissuade us that this is the silver bullet that solves all of our issues. No matter how low-cost light clients are, nodes always have a non-zero cost. Leading me to my main point, there is a dangerous assumption in the models presented: node count remains the same with lower costs. This omission paints a prettier picture and ignores one of the key issues at hand: overprovisioning.

A stable node count seems like a reasonable assumption, but let me demonstrate why it isn’t:

@addison

The justification for the value provided to the network has to be in question. It must account for the long-term future AND adoption to be valid.

I’ve done some modeling based on the numbers above that paint a more accurate picture of a just light client future. First, some assumptions:

• Growth % = Percent of node count growth monthly (we’ve been growing at a minimum of about 7% - 13% monthly and higher than that previously)
• Cost Growth % = Percent of node running costs increasing per month (node running costs rarely decrease). 4% growth monthly over 18 months = $77.92 to give you a scale.

This chart lays out multiple scenarios versus the base case given different growth and cost growth assumption.

The base case is blue (nothing changes on Pocket Network). The rest are various scenarios based on growth and cost increases assuming a 50% drop in network costs (using @addison’s numbers). As we can tell, cost can meet or exceed the network that would be stable at 48.5K nodes.

Here is the effect of that over time to network payback:

As you can tell, not addressing the core issue of over-provisioning costs the network over time. This proposal in isolation could lead to -ev and much higher actual network costs in the medium-to-long term.

That said, it’s not all doom and gloom. GOOD VIBES has the desired effect on cost savings that @addison was describing in his post, without the risk of spiraling network costs OR betting $2M on the light client:

BUT WAIT, there’s more: What if you combine them?

The combination has the optimal effect on network costs leading to the best possible short-to-medium term answer.

Now to the question of value of this proposal: while there is certainly merit to the client as whole, its value may not be as high as initially requested given the math laid out above. It’s my personal opinion that $2M feels extreme and I’d advise the creators of the proposal to consider something more reasonable for the benefit of the DAO and the future project as a whole.

[adam]

I have been thinking a lot about the proposed methodology for valuing proposals. I firmly believe that the value-based approach can get out of hand quickly by setting a dangerous precedent for DAO compensation.

Shane sums this up well in his argument.

@shane

While an innovative client is worth rewarding, we can’t disproportionately reward client work from 3rd parties over the innovations of core-devs. The reason more the core-dev team has not been pushing for more v0 client work is because they are focused primarily on v1 . If the value of a new v0 client is worth $2M, then why would core-devs be motivated to work for PNI on v1? This would signal that two months of development on v0 outside of PNF is worth a 10 year salary.

This value-based approach would have the impact of demotivating the core team or other Foundation contracted service providers. Contributors would be incentivized to work on the sexy, public-facing deliverables, rather than long-term, game-changing solutions. As Shane said, if proposals aren’t treated equally, then most contributors would focus on low-risk, short-term fixes that maximize personal returns.

Let’s look at the proposed logic provided in this proposal in defense of the $2M figure.

@addison

While this is in fact a large ask relative to other proposals, it’s important to keep that notion of relativity when we think about the impact it will have. The amount of savings from this optimization makes the reimbursement amount look frail in comparison. And as mentioned previously, the ROI of this endeavor is tremendous, and will have immediate impacts.

The logic, in essence, is impact = value = increased compensation. While I can understand the point, this can be extractive in a system not designed around capitalistic tendencies.

@o_rourke

Your incentives are such that by building this improve the long term prospects of the network. The amount of success you have is compounded by a successful light client.

To Mike’s point, the DAO is not designed to provide profit. It is designed to provide fair compensation for work. Further, Thunderhead and PoktFund should be driven by the overall success in the community. Their success should be aligned with, and driven by, the success of the community, not a single proposal.

Whatsmore, the proposers should have a financial incentive to build things like the light client to optimize their fleet of 4,000 nodes. Part of their reward is that optimization in the form of costs savings. The other part can be obtained by open-sourcing and maintaining their solution. Encouraging value-based proposals leads mercenary developers who come for a paycheck, build something, then leave that thing unsupported.

If we believe that this proposal is worth $2M in POKT, then any other proposal that accomplishes the same cost savings would be worth similar amounts.

PUP-15: GOOD VIBES - A New Economic Policy for Pocket Network - #23 by adam accomplishes similar cost savings in a parameter-driven way while having the added benefits of less inflation (more value) and it eliminates the overprovisioning problem.

To the supporters of the value-based model: is PUP-15 also worthy of a $2M+ grant?

I do not think so. You would be hard-pressed to argue that the cost savings/value are markedly different, and according to my last comment, the outcome is actually more predictable.