PEP-29 Pocket Lint (RECINDED)

WE WILL BE PUBLISHING A NEW PROPOSAL AT A LATER DATE WITH A NEW VALUE BREAKDOWN.

Attributes

Author(s): Shane Burgett

Recipient(s): Decentralized Authority

Category: Reimbursement + Funding Runway

Fulfills: RFP-7: Node Deployment Tools & RFP-8: Node Monitoring Tools

Asking Amount: 266,000 POKT

Summary

POKT Lint is an open-source tool and service that enables node runners to test their node’s latency performance against every location the POKT Portal is deployed. Since the Portal favors nodes that are able to respond to requests within 100ms, it is increasingly important that node runners be able to gauge their latency performance. Pocket Lint is the first publicly available tool for measuring node latency.

The purpose of this proposal is to reimburse for the development of Pocket Lint and fund future development plus its deployment in all 14 Portal locations as a free API service for the POKT node community. Since the beta launch in March, POKT Lint has already been used to test/troubleshoot hundreds of POKT nodes.

To see POKT Lint in action, check out our presentation and demo from the March 25th Community Call.

Background

Decentralized Authority is the development group behind Node Pilot and Node Launcher. It was started by long time POKT contributor, Shane Burget, and long time web3 developer, Ryan Burgett. POKT Lint was a joint effort between Shane, Ryan, and POKT community member @noproblem, who is the creator of the awesome https://pokt.tools.

What’s the deal with calling it POKT Lint?

Wikipedia: “Lint, or a linter, is a static code analysis tool used to flag programming errors, bugs, stylistic errors and suspicious constructs.”

We couldn’t resist! Credit: @noproblem

Motivation/Rationale

Any POKT node runner knows that the most important factors to maximizing rewards are: 1) chain selection and 2) relay latency. Fortunately, through tools like POKTscan, node runners can research chain rewards to identify which chains they want to serve… but there are no practical ways for node runners to identify their relay latency in respect to Pocket Portal locations. This leaves most node runners in the dark regarding how latency may be affecting their rewards.

Pocket Lint was built to fill this gap in node tooling by empowering node runners with the ability to identify their relay latency per chain, per Portal location. It’s being deployed as an open API for the Pocket community to freely test their POKT nodes. We’ve also created an open-source GUI that is hosted at poktlint.com which makes it simple for users to test their nodes by simply dropping their node addresses.

See instructions in how POKT Lint works here.

POKT Lint can also be added into other tools through an API. Node Pilot and pokt.tools already plan to integrate this as a feature to their platforms. Documentation for using the API can be found on the POKT Lint Github.

POKT Lint is built as a small microservice that has been packaged as an AWS Lambda function or an Docker container, making it easy for others to deploy as well. This could be useful for commercial node operators who need more flexibility for running larger scale tests. Deployment information can be found on the POKT Lint Github.

As Pocket Network’s node ecosystem grows, its important that there be tools that help node runners, large and small test latency and chain connectivity. POKT Lint does this in a way that is accessible to all types of node runners. Improved relay latency not only can help node runners with rewards, but it also improves the latency performance for apps using Pocket.

Objective(s)

  1. Reimburse for the past 5 months of development

  2. Cover costs of running and maintaining POKT Lint and the POKT Lint API as a free community service

  3. Provide a funding runway for upgrades and new features

Objective Key Results

  • Provide POKT Lint as a free, open-source tool for the Pocket community
  • Deploy POKT Lint to every region of the Pocket Portal
  • Allow applications free access to the POKT Lint API
  • Host poktlint.com as a free GUI frontend
  • Provide documentation on APIs and deployment.

Needs Being Met

POKT Lint is a tool address both RFP-7: Node Deployment Tools and RFP-8 Node Monitoring Tools. POKT Lint helps with node deployment by enabling node runners to identify the latency of both mainnet and testnet deployments. If node runners are seeing high latency through POKT Lint, they can adjust deployment strategies if a location is hindering performance. It can also be used to troubleshooting to ensure latency is not affected by changing hardware or other infrastructure changes.

Relay latency testing is frequently brought up in the weekly Node Runners Community calls on Wednesdays at 12pm EST. The node runner community has been asking for a tool like POKT Lint.

Dissenting Opinions

Shouldn’t the the Pocket Network Foundation should be the ones to operate and maintain this kind of community tools?

While PNF does create and run a number of POKT related services (example: The POKT Portal), many community services are operated and hosted by community teams. Services like POKTscan, POKTwatch, POKT.tools are all operated by different teams, adding team distribution and creative diversity to the POKT ecosystem.

Beyond providing free software for independent node runners through Node Pilot, Decentralized Authority is excited to also provide this service to the larger node community.

Deliverable(s)

  • POKT Lint (80%):
    • Completed
      • Relay Latency tester
      • Chain selection
      • Sample rate
    • In progress
      • Ping tester
      • Testnet support
  • POKT Lint API service (70%)
    • Completed
      • Deployed in most Portal regions
    • In progress
      • Deploy to all Portal regions
      • Provide public APIs for each region
  • POKTLint.com (60%)
    • Completed:
      • MVP GUI
      • Wallet address importer
      • Multiple node support
      • Auto multi-region
      • Auto multi-chain
    • In progress
      • New Professional GUI
      • Ping latency results (the ping time to the node) and the process latency results (the amount of time it take the node to process the request)
      • Testnet support

Budget

Reimbursement

POKT Lint has been in progress for 5 months. The time invested went towards:

  • Planning
  • Research
  • Development
  • Testing
  • Documentation
  • Deployment
  • Community support

Reimbursement budget: 130,000 POKT

Runway

We plan to keep growing and improving POKT Lint. New features are outlined in the Deliverables section. We also expect to update POKT Lint as we begin receiving feedback from users. To ensure we have funds to maintain development, runway funds to add new features, add new chains, and provide community support.

Runway budget: 60,000 POKT

Administrative Costs

To cover business taxes, legal expenses, the volatile nature of being paid in a crypto asset (non-stable coin), and the lack of high liquidity options for US entities, we are including a 40% administrative safety net. This is required to adhere to business best practices.

Administrative Costs: 76,000 POKT


Total Budget: 266,000 POKT


Beyond This Proposal

Depending on the needs of the POKT Validator community, there may be features, upgrades, or maintenance costs that will require further funding to complete. This current proposal covers the stated upgrades, expected community support, and expected maintenance/hosting costs.

Contributor(s)

Shane Burgett - Long time POKT contributor with a background in business development, marketing, media production, and community growth.

Ryan Burgett - Full-stack Web3 application developer with 5+ years of experience in the areas of wallets, interoperability platforms, mining, and node deployment.

Copyright

Copyright and related rights waived via CC0.

2 Likes

This post is purely on my views as a DAO Member, a person who runs nodes, and someone who has multiple years of software engineering experience and not representative of PoktFund.

TLDR: While I appreciate the efforts your team has taken to build out a latency tester, I am not sold that this is the proper solution for node runners.

I believe the PNF is best for the job and the DAO should fund the PNF for this job if bandwidth allows, because they have all the data needed. A latency tester gives a high level idea of what the networking side looks like, but it does not show you exactly where you rank or how you’re weighted in the “cherry picker” process. They have all the data in their DD logs & Redis Cache such as the weighted nodes average latency, which rpc call, etc to give us a clear picture where nodes sit in the cherry pick process. They are best fitted to be able to expose this and give node runners a better idea.

A ping/traceroute in my terminal would’ve been sufficient, or a bash script to do so. It’s not the same as a full RPC payload, but it should be far than enough. But nor does PoktLint simulation give a good idea since it only uses lightweight relays and is not indicative of what the apps are sending.

The DAO funded the team for runaway for Node Pilot and Node Runner which was suppose to cover also headcount (@noproblem, i’d assume). A good majority of the asks have been Node Pilot runners, in which, I believe this should be covered under that budget. There is a practical solution to this rather an arbitrary latency tester, anyways and I highly suggest you incorporate it into Node Pilot.

A better solution without the PNF: Any node runner including Node Pilot should just capture the nginx access logs to see the latency in a real relay scenario rather than a simulate relay test, providing realistic numbers on the average latency end to end (upstream and downstream). You will be able to capture the amount it times it takes to get a response from the proxy and return it back to the client and all sorts of statistics. There are so much more factors that come into play when it comes to the latency testing such as the RPC call method which the access logs accounts for. From there, some basic parsing through json and aggregation, and UI to show this is sufficient. This I would approve a DAO proposal for - in which the amount would still have to be significantly lower.

I’m not saying it didn’t take 5 months of development, but can you justify this more? I looked through the commits and it only dates back to Feb 18th. Since then, it has been mostly sporadic commits with most of the work showing within a week or two.

Furthermore, I don’t feel comfortable even approving this one as we’re still waiting for the team to deliver on one of the awaited asks since the approval of the Node Launcher and what the community needs the most right now, The awaited validator API so other node runners can programmatically run their own Pocket Nodes. It was marked as 80% complete at time of proposal and still not completed months later.

I’ll just end it with that asking for a 40% premium while exchanges are down and the price we are at left a bad taste in my mouth, so if there’s a need for FIAT, then the proposal should use fiat. Bad timing.

Hey @poktblade,
I’m happy to address your concerns.

I started a conversation with PNF in December regarding this tool and then provided them with a spec of what node runners need and have been asking for. They said a product like this is not on their roadmap and they did not have any time frame on when they would be begin considering this tool.

I offered to take on this tooling need and bring it to light. It was suggested that this should become a DAO funded tool since it is providing a much asked for community service.

PNF was consulted when needed during the development process and Alex, who is the tech lead for PNF, spoke highly favorable to this product and it’s relevance to node runners. POKT Lint is a tool that PNF actively refers node runners to for checking their node against an accurately replication of the cherry picker.

If I understand your question, I think you are mistaken on how cherry picker ranks. In the POKT Lint instruction we outline the rule of thumb on how the cherry picker ranks:

This is the rule of thumb that was put together with Alex for this project to give nodes runners accurate cherry picker expectations. While each session in POKT is unique, due to the randomness of node selection, this rule of thumb from PNF is accurate.

Thus, POKT Lint is an accurate depiction on where you will rank when in sessions with different Portal locations. The cherry picker derives latency by submitting a current block-height call, which is exactly how POKT Lint works. A rundown from Alex regarding the cherry picker and it’s weighting process can be found here.

There are indeed many manual/technical ways to get the information that POKT Lint provides. The challenge though is that there are many different ways to run nodes, thus requiring different tests. Doing a manual ping tests can be useful for a simple POKT node setup where everything is hosted on single machine, but for a setup where the chains are on one server or cloud and the POKT nodes are on another server or cloud, then a ping test does not show the full latency picture. There are so many ways that node infrastructure can be setup where a ping would not be accurate.

You are correct that instead of using pocket-core’s simulate relay feature, node runners could just setup their nodes and wait for a mainnet session to find the information in their proxy or networking logs. That kind of process though was seen as a pain point as it can take a user days (due to increasing time between sessions depending on the chain) to gather latency information through raw logs. POKT Lint provides all that information in a few seconds.

While there are manual/technical ways to do every aspect of node running, RFP-7: Node Deployment Tools & RFP-8: Node Monitoring Tools are specific calls from the DAO to automate and simplify node running, which is why we decided to build POKT Lint. The goal was to make node testing available to all, regardless of setup, and create a way for advanced users to automate testing with open APIs.

Noted. Using the term “development” does leave the impression that coding has been going on for 5 months. I’ll edit this proposal to make this more clear. I was thinking more in terms of development of the project as a whole. You are correct in pointing out that coding started February, which was followed by the Github repo being created on Feb 13th and the AWS work shortly after.

Discussions with PNF regarding this tool started in December, followed by research, planning, and securing contractors, which fell through in January, leading to our connecting with @noproblem.

POKT Lint is a completely different tool than Node Pilot hence why we are putting in a separate proposal. This kind of free community tooling, that is focused on all node runners (not just Node Pilot users), was not in the scope of work in our Node Pilot/Node Launcher proposal.

Regarding the team involved, @noproblem was brought on specifically for this product, not Node Pilot. The time that myself and Ryan put into POKT Lint was time away from Node Pilot. I don’t believe that funding for one specific product means that any future open-source projects for the community, from that team, should be pro bono work. This concept would stifle participation by barring teams to only one project.

Actually, we do have the Validator API in beta for Fuse. You can see the validator additions in the Node Launcher dev branch for Fuse. Here is a testnet validator that was successfully deployed via Node Launcher the end of March :point_down:

We are in the process of adding POKT validators in the coming weeks, and we have an outside contractor working on Harmony validation.

Regarding general Node Pilot/Node Launcher updates, we gave a full community update last month. Major architectural changes are underway to improve security and make way for an enterprise API. The reception we have received from the Node Pilot community regarding our development plan and progress has been very positive.

Now that I’ve operated a company that has worked exclusively within the POKT ecosystem for 6 months now, I’m in-tune with the nuances of being paid in a low volume assets (for a US entity) and the liquidation/taxation challenges. This is necessary to hedge the risk of being paid in crypto, in a volatile market.

The DAO does not pay in FIAT. All DOA funding is paid in POKT which the recipient is responsible for the liquidation, taxes, etc… hence the required safety net.

The NodePilot team is a trusted community contributor who consistently turns out high quality ecosystem offerings. I strongly support this proposal.

I’ll defer to the PNF to elaborate on the specifics of how the cherry picker works. It’s been changed multiple since December. Everything of what I know about it is from reading the source code multiple times. The latency is now < 150 ms for the best rewards with the current code base, now. From what I understand, it derives latency by measuring real relays too from RTT portal → pocket node → relay chain not just from a pure height call. Can someone confirm?

I stand by the fact a simple lightweight RPC latency tester is nearly identical to a ping test. It is not indicative of real load. Users can simulate relays to identify whether they’re connectivity is good or not and ping test to see if they are close to the portal. Nginx logs should also help aid in the a more in depth analysis (if cherry picker also uses more than block height checks for node performance)

I was talking about Pokt validators, not Fuse.

I’m all for more tools into the ecosystem, but the DAO has already funded your team, multiple times now and with runway. It really should’ve fit in the NP’s budget since it tailors to your userbase use case a lot more than someone who has basic programmatic skills to whip up some latency testing scripts

The proposed solution is still a workaround, and I’m not a fan of it. If this is something that’s massively needed by node runners then we as a community should push for it to be on their roadmap or start to collaborate and embody more of our open source nature by contributing to the portal, not spin up a product that acts as a half fix! (Admittedly, it is a bit hard to contribute to, but that’s a topic from another time)

I’ll be totally honest - when I saw 5 months, something didn’t sit right for me. - the core to this project is something that takes 24 man hours and I’m being pessimistic about that estimate. It’s a slap in the face for the DAO projects that have to produce 5x the work for the same value. I would feel more comfortable revisiting this with a lower amount and as well with the PNF public approval that this is the right direction we should be taking. I’m not convinced this is the solution to the problem and the direction we should take. Someone from the PNF would have to weigh in to convince otherwise.

Thanks for the response @poktblade.

Yes, you are indeed correct. Latency is also constantly checked through out a session and the active calls being used by the app in that session are being used.

While latency is constantly checked through the existing traffic, the Portal is also constantly using block height on it’s own to also determine if a node should get relays or not. Block height calls are the first calls that the Portal submits to nodes, so if your node doesn’t answer block height, it won’t get any relays at all. POKT Lint is simply represents the first calls the Portal already does to check health and latency.

You are correct that POKT Lint is a lightweight RPC latency tester, and not a stress tester. I respect your opinion on the value of POKT Lint. I may disagree, but thank-you for bringing it to the conversation.

Indeed, someone with programming skills can create their own latency testing scripts and deploy them in the sames regions as the Portal, similar to POKT Lint. We thought streamlining this for all level of users would be helpful.

I don’t agree that projects should be grafted into past proposals if they may be useful to a similar audiences. Happy to agree to disagree on this one.

This would be the first time that PNF would be called to make official statements regarding a contributor proposal. This has never been the role of PNF with the DAO. It’s actually the other way around… the reason PNF regularly submits proposals is because it acts on behalf of the DAO. The POKT Constitution structures the DAO as an independent agency that provide guidence to the Foundation, not the other way around.

The DAO signals directions for the ecosystem (like with RFP-7 & RFP-8). Feel free to look back at all PEP proposals and you won’t find PNF giving any official statements. PNF members may give personal statements on the forum, or on community calls (as I linked before), but those are their individual opinions and should not be considered official PNF backing.

As a voter, regardless of this proposal, I would caution restructuring the balance of power by putting PNF in the position to officially back proposals.

Yeah I agree, caution around this. But as well to keep in mind, the DAO and the community as a whole has pretty much delegated a large portion of app access to the Portal (through the form of various peps, i.e Gigastakes) that the PNF owns, and this is a very unique problem that is tied to the Portal and its cherry-picker process. This proposal stems from a problem of visibility of how nodes perform that the PNF may be able to expose.

So if we acknowledge that this is a need by the community - then it should be appropriate to bring them into the conversation as there could be a better solution that’s supported first party. Public approval was bad wording, but we should weigh the options and alternatives - to the public for visibility sake. Getting better idea of their bandwidth and how long it would take to expose the information node runners need for example.

This is a great proposal, thank you!

@poktblade: Answering some of your questions here.

  1. While we (the company) build tools, and open source most everything we do, the DAO has made a point to empower the community to build tools exactly like this kind. So, this tool and related proposal are 100% in the spirit of what the DAO hopes to spend part of its treasury on.

  2. Sure, logs can be used, but having a nice GUI to do this on behalf of end user to test the setup is a quality-of-life improvement that benefits the entire network. The work done by these folks saves time for the rest of us, so I am fine with as a DAO voter providing funding for this tooling.

  3. The 40% safety net is similar to the 30%-40% revenue share that many centralized node infrastructure providers take when standing up nodes (part of the price is due to the volatility of the token from what I’ve gathered). I’m not saying that I agree that this same amount should be used here, but I agree with the spirit of the request.

@shane the only number here that is a bit large is the 130,000k for the 5 months of work. Could that be broken down? I don’t need an essay, but maybe a paragraph or two on how it was spent and if this is where the 40% premium is being included.

3 Likes

Thank-you @ArtSabintsev. I’m happy to shed some more light on it’s development.

About 170 hours development hours went into coding of the product and building the AWS backend service for the POKT Lint API.

Beyond what is in Github, a portion of that time was dedicated to features that ended up not making it into the final product. An original objective was to also include a general purpose ping tester within the Lambda function to measure the distance latency from all Portal locations to any IP address via ICMP ping. This could be helpful with scouting out data centers. After many roadblocks this feature was sunset as we would have to venture outside of Lambda since Amazon Linux is setup without CAP_NET_RAW and workarounds would be too involved for such a niche feature.

We also sunset a variety call, pseudo stress feature. This was sunset as the adding/updating/maintaining of many calls for each chain would be more complexity and upkeep than desired. This also seemed like a niche feature.

These other features could be part of a future proposal if there is a call for these features in the future.

Beyond development, 80 non-development hours were put into POKT Lint as well for everything else mentioned in the proposal.

The 40% safety net is applied to the whole proposal as the safety net reason apply to both the reimbursement and the runway.

All in all, if there is a perceived difference in the value of this work, I’m happy to discuss :+1:

After chit chatting with Shane and learning more about how PoktLint is priced, I fully support this proposal. (Though, would still love if the PNF exposes the cherry picker information :stuck_out_tongue: )

Shane is working on some really interesting ideas on how proposals can be priced, and I look forward to seeing the adoption of that as well.

1 Like

I appreciate all the feedback everyone. This proposal in-particular has brought up a number of great discussions on how to quantify the value of a proposal and the team behind it.

As @poktblade mentioned, I’ve been playing with a value formula which I’d like to apply to my own proposal first. I’m going to withdraw this proposal from consideration and post a revised version in the near future that includes this new value model.

I’ll be keen to get everyone’s take on this value model as I hope it could help gauge value, promote best practices, and minimize risk while blustering ecosystem growth. It will also help with public discussion as specific values can discussed and tweaked upon feedback.

Catch everyone soon new version :wink:

Confirming that I support this proposal after discussing here and on Discord.