Into the Gateway-verse: Bootstrapping a Multi-Gateway Ecosystem

Announcements Foundation
1

As PNF identified in our strategic thesis for the ecosystem, decentralizing Pocket’s demand is critical because it

In other words, becoming a gateway-verse (an ecosystem comprised of multiple gateways) will drive the same optimization of the demand-side that we’ve already seen play out on the supply-side, as coopetition drives innovation in:

  • Methods for ensuring data integrity and QoS
  • Other operational quality factors that determine customer acquisition/retention - marketing, sales, DevRel, customer support
  • Cost-savings, enabling the DAO to increase the GatewayOperatorFee and scale the protocol revenue per relay
  • Differentiated strategies to product-market fit (e.g. alternate value-add services) increase the likelihood of Pocket ecosystem gaining traction

Furthermore, gateways are here to stay; v1 introduces an on-chain gateway actor, which we should begin preparing for now by:

  • developing our collective knowledge about the technical details and practicalities of operating gateways,
  • maximizing insights for the protocol developers to refine the on-chain gateway actor’s mechanisms, and
  • bootstrapping independent gateways to ensure that when v1 launches it is hitting the ground running as a gateway-verse.

Purpose & objectives

The purpose of the “Into the Gateway-verse” project is to help the Pocket Network ecosystem make the leap from a single gateway paradigm to a multi-gateway paradigm.

Our aims with “Into the Gateway-verse” are:

  1. Remove any logistical barriers to Pocket Network becoming a multi-gateway ecosystem
  2. Support a second-mover gateway to bootstrap their gateway and develop open-source IP that will lower the barrier to entry for subsequent gateways
  3. Learn more about gateway operating to inform refinement of the on-chain gateway actor mechanisms in v1
  4. Foster innovations in gateway technology, including enhanced methods for data integrity, quality assurance, cost-savings, and value-add services
  5. Diversify Pocket Network’s sources of protocol revenue

We believe that doing all of the above is the most significant impact we can have on the future growth of Pocket Network’s protocol revenue.

Implementation plan

Success metric

The north star metric for Pocket Network is protocol revenue (measured by the fees collected from gateway operators). As of 19 June 2023, this is sitting around $40k per month (per Web3Index).

While there will be a lagging impact of the objectives we outlined above, we expect this metric to improve as a result of achieving these objectives.

Technical specifications

Supporting Nodies as the Second-mover in the Gateway-verse

We began talking to Nodies (a subsidiary of BaaS Pools, LLC.) about the vision of a gateway-verse when they approached us in February with a proposal to build their own gateway. In the time since, they have formed a team of 7 community members, comprising Poktblade as the lead developer, 2 sys-op engineers, 4 software engineers, and a sales rep.

Now that they have pushed forward in soft launching their RPC webapp and we are one month into The Road to Revenue, we feel it is time to begin supporting Nodies more directly with bootstrapping grants and hands-on support by PNF and community members alike. This would include a do-not-exceed budget of $300k in grants – $50k per month for 6 months, or sooner if Nodies reaches a sustainable threshold. We expand more on the budget below.

In return, Nodies will provide the following contributions to our ecosystem:

  • Open-source R&D of data integrity and QoS assurance methods
  • An open-source QoS-weighting module for subsequent gateways to build with
  • Development of cost-saving innovations for gateways, which will ultimately enable the DAO to increase the GatewayOperatorFee and generate more protocol revenue per relay
  • Continuous knowledge sharing of their experience as a gateway operator, through regular community calls, forum posts, and open-sourcing of tools and technologies
  • Sharing insights with the core protocol developers for the refinement of the on-chain gateway actor in v1, as well as the Watcher (fka fisherman) actor
  • More protocol revenue

Nodies have proven themselves to be the perfect second-movers in our gateway-verse through a prolific history of contributions:

  • Protocol Expertise: Nodies have demonstrated a familiarity with Pocket’s protocol codebase through significant contributions such as LeanPocket and (already contributing to QoS improvements) fixing the session rollover issues
  • Readiness to Scale: Nodies have proven their capability to operate infrastructure at scale after several months as one of the largest node operators in our ecosystem
  • Trustworthiness: Nodies contributions have demonstrated their altruism in the service of the broader ecosystem’s success – open-sourcing LeanPocket when they could have used it to gain an unfair cost advantage, fixing the ChocolateRain bug when they could have exploited it for personal gain

Updating POKTscan to Measure the Gateway-verse

With the introduction of new gateways, it becomes important for POKTscan to be able to differentiate between which gateway the network’s relays are coming from, to ensure accurate tracking of gateway fees. We will therefore work with POKTscan to update their dashboard to reflect the emerging gateway-verse.

As the gateway-verse scales, we may also look to data aggregators like POKTscan to develop monitoring systems for identifying suspicious behavior by gateways.

Budget

This is an indicative “do not exceed” budget. Final numbers and costs are subject to change.

For clarity, any excess or unutilised funds from the Era Budget will be returned to the DAO Treasury at the end of the Era or at the close of the “Into the Gateway-verse” project (whichever comes first), and any remaining funds from the Era allocation will be returned to the DAO treasury at the end of each cycle.

Deliverable Responsibility Projected cost ($)
Support Nodies as a second-mover in the Gateway-verse Nodies $300k ($50k per month for 6 months)
Misc project ops (e.g. legal contracts) PNF $30k

Why are you only budgeting for one gateway?

As explained in PNF’s Era Budget proposal, the Era Budget is only for must-have (“keystone”) projects.

We must have a pioneering second-mover gateway, i.e. a team who is equipped to overcome the barrier to entry and develop open-source knowledge that others can borrow from. It has been a difficult journey for PNI and it won’t be much easier for the second-mover, so it’s important that they receive hands-on support. Through this process, they will develop open-source knowledge that will progressively lower the barrier to entry for subsequent gateways.

A third-mover gateway is a should-have prior to v1, not a must-have. This means that it is not budgeted for up-front but can still be supported through grants from the Era Allocation (80% of future DAO block rewards).

Why aren’t you budgeting for Sales/DevRel?

Each gateway team already has an incentive to do their own Sales/DevRel, which relegates supplementary incentives (e.g. to fund DAO contributors to help gateways with DevRel) to being a should-have rather than a must-have. These initiatives can be funded by the 80% Era Allocation.

Stakeholders, roles and responsibilities

Stakeholder Role and Responsibilities
PNF Steward the project by setting the objectives, maintaining oversight over all contributors and ensuring that the project is delivered in an efficient and high-quality manner.
PNI Support PNF with understanding app stake logistics, free up app stakes for Nodies, then knowledge share with Nodies and all subsequent gateways.
Nodies Develop their gateway, share their knowledge with the ecosystem, and generate revenue.
Community Developers Help PNF to implement this project by working on related POPs.
DAO voters Approve the “Into the Gateway-verse” budget of $330k included within the new ERA proposal once it goes to a vote.
Community members Provide input and comment on this project, including supporting Nodies with feedback and testing of their gateway.
PNF’s PR firm Connect PNF to the right contacts to ensure that the multi-gateway milestone gets awareness with its target audience.
PNF’s law firm Support PNF with drafting contracts.

Deliverables, stakeholders and timeline

Deliverable Stakeholder Timeline
Activate GatewayOperatorFee DAO May - Done
Scope out app stake logistics PNF & PNI May - Done
Align on objectives and approach for the “Into the Gateway-verse” project PNF June - Done
Finalise contracts between PNF and gateway operators Fox Williams (PNF’s legal firm), PNF June - Ongoing
Update Portal backend to free up app stakes for Nodies PNI July - Ongoing
Update POKTscan to segment relay tracking by gateway POKTscan July
Grant app stakes to Nodies PNI End of July
NodiesDLB begin testing of protocol integration Nodies End of July
Begin Gateway Office Hours for knowledge sharing by Nodies Nodies August
Finalise marketing and PR plan for gateway launch PNF’s HoM and PR Firm August
Rebrand PNI to avoid confusion or advantage PNI August
NodiesDLB full launch of protocol integration Nodies August/September

Conclusion and next steps

We will use this post to make continual updates as we achieve more milestones and more information and questions come to light about the final approach to the relevant social, economic and technical components relating to the project.

We encourage all community members to join us in bringing this project to fruition, so please feel free to reach out or just comment if you feel you have a valuable contribution to make. We will also announce one or two Pocket Open Priorities (POPs) in relation to “Into the Gateway-verse” in the coming weeks, so watch this space.

Thank you.

PNF

12 Likes
2

Two questions:

  1. Is the DAO treasury funding the $300k, or is it from PNF’s treasury?

  2. Why were only select companies allowed to be involved in the planning of v0 gateways (and be the benefactors of such plans) when it’s been a hot topic in every community call since the beginning of the year? Once it was announced that PNF was looking into v0 gateways, it’s been at the forefront of most POKT involved businesses.

1 Like
3

  1. It would come from the DAO treasury per the ERA Budget.

  2. Thats not a fair characterization. Poktblade came to us initially with the idea. The extent of our planning has been scoping out the logistics of pre-v1 multi-gateways, specifically in relation to how we manage app stakes, wherein we had discussions with PNI and looped Poktblade in for a third-party opinion on the technical considerations given his experience contributing to the protocol in relevant areas such as session rollovers. Nodies have developed and soft-launched their RPC webapp independently.

I would also push back on the notion that there are sole benefactors in this. Our strategy is to carve out a must-have “keystone” budget to support one gateway and in return receive open-source gateway modules and other lessons that will lower the barrier to entry for subsequent gateways. There is nothing precluding other gateways from being granted access to app stakes and receiving DAO grants for open-source contributions to gateway technology. The ERA proposal carved out 80% of future DAO rewards to be allocated towards initiatives that align with the core priorities, which the work of other gateways would. This is all outlined in the post above.

7 Likes
4

I completely support this.

Having another Gateway is an important step forward in our ecosystem, and Nodies is well placed to deliver.

Open sourcing help with QoS should additionally help other Gateways as they exist in, or enter Pocket. This is a fair trade for the grant.

Additionally, this is a better strategy to compensate Gateways for processing relays than the artificially low relay charge - discussed in more detail and generally agreed on by participants in the MINT discussion.

5 Likes
5

I do not desire to be unfair, so my apologies. I would ask you look at how those on the outside see this kind of decision making.

It was mentioned shortly after ETHDenver that plans to open v0 gateways were underway. From what was communicated, PNF members met with @poktblade in-person (not sure if others were there) about v0 gateways, which launched the idea. It excited everyone that v0 gateways would be possible.

Since then, there has been a number of community calls where both PNI and PNF members were asked for updates regarding v0 gateway plans, and it was repeatedly told that there isn’t anything to report as PNF is still trying to figure out how to share appstakes. It was always mentioned that once there has been progress it would be shared.

What was not expected, is that the update of progress would also include nearly a quarter of the Era Budget would be going to a company where a POP was not used. Some POPs are to follow… but not for this.

The ERA Budget said the support would be “regarding legal, docs, and any other small support needs they may have to go live”. It was not mentioned that this was to pay for the development of their gateway itself with 90% of the budget. From my side, I assumed anything funds for development would be focused on PNF tooling to share appstakes and ensure gateways are not gaming the system (which they easily can do unless PNF has robust monitoring).

Providing a single entity with nearly a quarter of the DAO Era Treasury, without a POP, was never communicated, hence why I ask you see things from the outsiders perspective. The POP usage for ALLOCATED PRIORITY: Mainnet Snapshots saw a lot of unique applicants with innovative value propositions. It was clear that the POP was for development, and I’m confused why a POP couldn’t have been used here, when a substantial amount of the DAO treasury is at stake.

Double funding?

I’m also struggling with seeing how this is not double funding. @BaaSPoolLLC said in their per-proposal for Leanpokt that they already have this exact team, which you are now saying PNF will fund again with more DAO treasury?

They said they will “becomes more sustainable and lucrative if we are able to get our reimbursement.”, which the DAO gave them, and now they need another $300k from the DAO for the same team? How many times is the DAO supposed to support the same team for the same roadmap? On-top of them already having one of the most lucrative node running businesses this year?

Also in their Community Call they stated they are building multiple ways to monetize with websockets and specialty APIs that do not use the POKT Protocol at all. The specially said:

There are like some creative ways where we could start taking in traffic ourselves and then create a plan to pipe that over to pocket network, but yes one of our goals is make sure that Pocket Network receives some of this traffic on top of that we want to provide sustainable support for the app users.

Yes, Pocket Network will get some of their traffic… but they propose that their main value prop is what is NOT related to the protocol itself, but related to their own centralized features. They are welcome to build whatever feature they like, but I don’t see how a gateway focused on non-protocol features should be subsidized by the DAO.

I truly have no idea how to understand what funding proposal is funding what at this point.

I also thought I had a good understanding of the Era Budget works, but now I super confused because this is not at all what was presented.

Not trying to mischaracterize anything, so please correct me where I am wrong. I’m just being factual with my understand of the information that has been given to us to date.

4 Likes
6

@JackALaing I would like to amend my comment from yesterday with some further clarification:

I do believe that open-source gateway tooling is a benefit to the community. The goal overarching goal is worth pursuing IMO.

The issue for me is how this funding came about. It is awesome that @poktblade started the ball rolling regarding v0 gateway conversation, but it appears Nodies was the only company to know that $300k was at stake for being the first alternative gateway in the ecosystem. Yes, Nodies has developed and self-launched their RPC webapp independently, but they have been the only ones with knowledge that PNF was planning to use DAO treasury to substantially subsidize gateway development in v0.

Had the rest of the ecosystem known that substantial funding from PNF was going directly to subsidize gateway development, I know many companies would have considered a gateway roadmap, my own included. Developing client facing, open-source gateway system is a lot of work and a high risk endeavor because there is no guarantee of return. From all the companies I’ve talked to, everyone in the POKT ecosystem is tightening budgets and trying to reduce risk to account for market conditions. Nodies on the other hand, has been aware of development funding plans and has been involved in the meeting between PNI and PNF, while everyone else has been patiently asking updates. From my perspective, Nodies had a clear preferential position, while the rest were essentially told to hang tight.

I also feel the LeanPOKT proposal was partially predicated on funding their future roadmap.

I’ve been under the impression that their substantial DAO reward for LeanPOKT was to support their short term future initiatives, including their gateway, which they announced prior to the proposal passing. I personally never thought that less then 3 months of receiving $353k, another $300k of DAO funds would be directly available to them without contest, especially when we have so many builders within POKT.

Clear The Air

To be clear, I don’t believe any of this was done nefariously. I do believe that PNF has the best intentions, and is looking to empower the network with gateway technology. Because of the high risk factor of developing gateway technology, without guarantee of return, it makes sense for PNF to invest in empowering the ecosystem with scalable tech. It is a solid strategy and I do believe that every company will benefit from open-source gateway tech.

My objection is how one company was able to position itself for guaranteed substantial development funding, without anyone being aware it has happening. I believe that something this substantial should utilize a POP system.

I also believe that there should already be an open technical spec for all node runners to participate in it’s design, prior to being guaranteed funding. Would it be possible for Nodies to be commissioned to create a technical spec first? Let us see the elements required and what technologies they plan to use (languages, libraries, databases, tooling, etc). From there, perhaps folks can give feedback or present their own for consideration? With such short notice, I don’t know who would be able to come up gateway plan, but I think it’s at least worth trying.

Move Quickly

I understand that the goal is to move quickly and I do not desire my feedback on this to slow anything down. I also want to help with solutions, not hinder progress and innovation. My mind is open to other thoughts on the matter, and I am very appreciate of PNF’s initiatives. I believe there was a misstep on this matter, but I believe PNF has had excellent communication and transparency the past 6 months.

2 Likes
7

Thank you for sharing your perspective @shane and for following up with clarifications. We empathize with your motives and want to clarify the facts and our reasoning.

Clarifying the Facts on the ERA Budget

Nodies did not have secret knowledge about the possibility of receiving $300k in grant funding. PNF ourselves did not know we would want to budget for this until last week.

When Nodies first came to us about their plans to build a gateway, they originally planned to self-fund. PNF also believed that all gateways should have enough incentive to take on the risk that typically comes with an entrepreneurial venture in return for the potential rewards. This is why the ERA Budget initially did not include any line items for funding any gateways.

Since then the macro environment makes it less feasible for gateways to self-fund, which prompted Nodies to seek more support from us in return for the open-source contributions we outlined in this post. In tandem with this development, we also received feedback on the ERA Budget proposal that we were not allocating enough of the budget to funding v0 QoS R&D. We took on board both developments and decided that we should be allocating a line item to ensure at least one new gateway is financially supported - in return for open-sourcing their technology to subsequent gateways - so we amended the budget.

Aside: It should be noted that $300k is a do-not-exceed. We will have wording in the contract to end the grant if either of the following occurs: 1) the price of POKT recovers to a level that enables Nodies to sustain their gateway from their node running, 2) they bring in enough revenue from gateway customers. Any excess would be returned to the DAO.

Clarifying the Origin of the v0 Gateway Strategy

When the current PNF team got started in January we immediately opened direct channels of communication with builders and conducted discovery calls with them to understand their needs and interests. It was during the discovery call with Nodies (Feb 9th, 3 weeks before EthDenver) that they first told us about their plans to build a v0 gateway service, originally envisioned as being built on top of the Portal using PNI’s proposed Gateway-as-a-Service (GaaS) model. The only other builder to express an interest in building a gateway at this time was BargSystems.

The meeting in Denver was an exploratory workshop to evaluate the merits of replacing PNI’s GaaS model with a bolder strategy to grant app stake access directly to gateways to enable more differentiated strategies (and thus innovation) at every layer of the gateway stack. Nodies & BargSystems were both part of this workshop due to previously expressing an interest in building a gateway as well as directly questioning whether GaaS was the best approach.

One thing to note is that it was important for us to have third-party perspectives in the room, besides ourselves and PNI, because we knew that PNI would naturally be reluctant to explore a strategy that would introduce operational overhead and uncertainty regarding their GaaS strategy. Inviting Nodies & BargSystems to participate at this stage was as much about eliminating bias from the discussion as it was about them being the only builders to have approached us about the idea, it was not about granting preferential access.

All of these earlier discussions laid the foundations for our ecosystem thesis, published in April, with decentralized demand as the most important component, and The Road to Revenue.

Clearing the Air, Making Bets, & Encouraging other Gateways

We’ll take the L and admit that this isn’t how we’d like to do things in an ideal world, if we were starting now and had ample time and resources, and driving demand wasn’t such an urgent priority. It is also important to recognize that Nodies was a catalyst who played a key role in formulating this strategy; without them, we would not be just weeks away from the launch of our second gateway.

Wartime decision-making requires decisive bets with a ruthless priority on the expected value from each “bet” we make. Gateway R&D is a prime example of an area where decisive bets must be taken. We need to: lower the barrier to entry for gateways, develop insights for the V1 on-chain gateway actor, improve QoS and thus grow relays/revenue. The best bet to achieve these goals is to support one pioneering second-mover gateway in return for open-sourcing the outputs of their R&D. For a must-have contribution of this nature, the only factor that matters is capability and speed. Nodies has demonstrated their capability, as outlined in the original post, and has already built momentum on their gateway and their QoS R&D (see session rollover fix).

It should be noted, as I’ve stated before, this does not preclude other companies in the Pocket ecosystem seeking grants to work on gateway R&D. We want to see more gateways! Please talk to us if you’re interested in working on this. Any relevant POPs, sockets, and bounties will be funded by both the ERA Allocation (80% of future DAO rewards) and PNF treasury.

3 Likes
8

Thank-you very much for this response! It does provide MUCH clarity, and enable there to be a conversation based on material information.

I think the primary point of miscommunication is around PNI’s Gateway-as-a-Service (GaaS) model and appstake sharing. The GaaS model was based on using PNI’s backend infrastructure with others building custom front-ends. I know that myself and others did not find that a compelling model for many reasons. I believe that is why so few folks showed interest in gateways when that was the only available model.

However, once it was announced that the GaaS model was NOT the future, but instead appstakes would be independently operated, there has been significantly more interest across the community from what I have seen. An appstakes model is a completely different beast as there isn’t a dependency on PNI (like the GaaS model would introduce).

It seems that interest for the GaaS model (dependent gateways) and the appstakes model (independent gateways) were being unintentionally conflated by PNF as the same thing. Because Nodies and BargSystems showed interest in GaaS, they were part of the internal conversations regarding appstakes, which I feel was an oversight and did provide them with preferential access to formulate their strategy and now access to uncontested funding.

I first heard about the appstakes model after ETHDenver and, and immediately expressed interest in the community call and we were told by both PNI and PNF that more information would be coming. There were little updates occasionally, but nothing that signaled there was conversations happening where interested parties could be involved. I have witnessed others expressing interest the independent gateways as well. Unfortunately there hasn’t been actionable information since, and I wrongly assumed POP’s were going to be used for the entire project, not just a small subset.

I also believe my concern about double founding the same roadmap is valid. Nodies received 6.9M POKT which has yet to be touched, and yet they have been operating with the same number of team members since February. Again, it confusing what the LeanPOKT proposal is funding if not their roadmap as indicated in their proposal and the conversation that followed.

I would also like reassurance that everything is crystal clear what they are building with any extra funding, so there isn’t a similar situation that happened with the LeanPOKT contribution. While PNI and the community was testing and improving the LeanPOKT open-souce client for everyone’s benefit, they launched their staking business entirely on a closed source client with a version of geo-meshing, unannounced to the community. While a savvy strategy, I personally saw that as the community contributing and later very lucratively rewarding one client, while another client was developed in that exact same time frame and used by their business to garner substantial market capture over their competitors.

Therefore, I feel it is in the best interest of everyone to ensure there is not the same blind spots in further funding. Everyone (including myself) have commended them for their LeanPOKT contribution and desired to reward their efforts, but it is unclear how much of the LeanPOKT funding went towards the open-source version and what went to the closed-source version they actually used for their business, since they were developed in the same time-frame covered by the LeanPOKT proposal. It would be a major loss if that same scenario played out with gateway contributions.

If they produce quality open-source tooling that has universal utility within the POKT ecosystem, then it is worth making it happen, though I would hope there be vigilant PMing with detailed technical specs that can be shared with the community, defined deliverables and roadmap, along with a clear SLA/contract with BaaS Pool LLC. I say this not at all to micro-manage, as PNF is extremely competent, I’m just being honest about blind spots we have had in the past and my perspective.

I do also believe that $300k is too much considering their LeanPOKT proposal, who’s value was partially predicated on their roadmap (which included the gateway), so I would suggest that it be taken into account, and provide more POP opportunities for others (which Nodies could apply for as well).

I leave this feedback for PNF’s consideration :+1:

2 Likes
9

Hi @shane, I believe @JackALaing already clarified a number of your questions, and we’re glad to see you’re aligned with the value proposition of gateways, as well as what an undertaking this will be. We also wanted to share our perspectives on a few things.

First, I would like to point out that the Quote you pulled was from a Pre-proposal, the details of which were quite different from the actual proposal, which the DAO actually voted for and approved. For one thing, you will find that the quote you pulled was not included in the passed proposal.

Second, there appears to be some confusion regarding the timelines involved. The development of LeanPokt by the Poktfund team, led by @poktblade, began in February 2022 and was merged into Pocket Core in December 2022. Subsequently in February 2023, we submitted a reimbursement request for the work completed by our team during that period. Currently, we are seeking funding for the upcoming 6 months of work. Therefore, if we consider the start dates of these two asks, there is actually a gap of over a year between them. To be clear, The LP Proposal (and its subsequent passage) was not intended to fund future projects. LeanPOKT was also never intended to be an all-inclusive grant for all the work we perform at Pocket. Generally speaking, we consistently do pro-bono work for Pocket that adds value to the ecosystem, but considering the scope of work required for the gateway, it made sense for us to seek funding.

Furthermore, there have been changes in personnel over time. Some of our team members who worked on LP have transitioned to our staking operation, which is unrelated to our gateway operations. Some, like @poktblade, have remained; while others, including myself, have joined as new team members specifically for Nodies DLB. I recently began my role as a business development representative for our gateway.

Thank you for transcribing from the Community Call. I believe this was misconstrued, so allow me to clarify. @poktblade’s intention was to highlight that in the event Pocket Network is unable to handle specific requests (for instance, if a customer needs websockets, but Pocket is not yet capable of handling those requests), NodiesDLB will still handle the requests to meet the client’s needs. This will allow us to incentivize DApps to use our platform (developer productivity is a common value prop), help provide market data to the protocol team and shape the roadmap, and ultimately pipe this to Pocket once it is supported. If the focus is demand, then we need to be competitive in our product offerings by strengthening our value propositions and providing DApps with what they need instead of turning them away. This in return will indirectly provide traffic to Pocket as well, as clients hardly want to use multiple providers for blockchain development. If you take a look at many other RPC providers, this is a common strategy in order to capture demand.

This is a mischaracterization based on incorrect assumptions. First, we released LP well before Nodies began running nodes, and we never stopped working on LP even as “PNI and the community” were testing the beta. Second, it is true that Nodies’ commercial staking operation began in September 2022 (and some small nodes for ourselves, friends, and family prior), and we were using a software akin to the Geo-Mesh. There were conversations to open source this work as well, but it was in alpha testing during a time when we had a relatively small amount of nodes and limited data to test. Even then, it would suffer from outages fairly often, bringing down some of our nodes with it. It took us a while to get the software to a stable enough state. We prioritized the network’s health and security similar to how LeanPocket was originally closed source as advised by the protocol team. By that point, the PoktScan had already released their Geo-Mesh in October 2022 with a much more reliable and supported codebase, hence we didn’t see the need to release ours. The timeframe between all of this is really quite short and hardly gave us much market advantage. In fact, we already alluded to this in response to a comment you made back during LeanPOKT’s Preproposal -

Lastly, we have since switched over all of our nodes to Geo-mesh in the spirit of keeping consistent with the rest of the network and encouraging open sourceness within the ecosystem. In fact, we are the second and only other contributor to Geomesh. We’ve worked with them to identify bugs, optimizations, code reviews, and have been credited multiple times directly from the main developer @Jorge_POKTscan (PR 1, PR 2, and countless of private conversations on how to architect and code).

Screenshot 2023-06-23 at 1.16.09 AM
Screenshot 2023-06-23 at 1.16.09 AM1244×434 98.3 KB

I hope this clears some things for you. We are excited to continue contributing towards enhancing and supporting Pocket Network in various aspects!

7 Likes
10

Hey @giantfrog, I don’t believe we have met before. I appreciate the response.

Oh, I’m aligned. So much so, I wrote the original ADR that the v1 integration is based on :stuck_out_tongue_winking_eye:

Pre-proposals are where the conversations happen for proposals… that is their intended design. You will find that with all the pre-proposal/proposal pairings, the real conversation for the proposal is in the first version (pre-proposal) and not the final version. It is a new perspective to try and divorce their relevancy to each other.

The final proposal for LeanPOKT was fast tracked and put to a vote after only 5 days, instead of the standard minimum of 7 days, precisely because the pre-proposal was the context, who’s conversation expanded over many weeks. Therefore I’d prefer to use the whole community conversation for context.

Unfortunately, that was not clear at the time because the proposal’s value was not predicated on measurable metrics, but more abstract concepts of value. Concepts that were partly built on notions that this proposal is worth it’s ask because of the value it would produce in the ecosystem (which thus far it has been unused). This is why I have always advocated for measurable metrics with every proposal, including LeanPOKT, so there would be universal clarity :sweat_smile:

It may be clear to you, but for myself, as a voter, outsider, and participant in the LeanPOKT proposal conversation, it is not clear. That is my point and why I provided past context.

As much I would like to take credit for transcribing, I must give it to Youtube :joy:

Yeah, I agree that if Nodies is able to capture customers for non-protocol features, it can indirectly lead to more traffic to POKT as well. I get the strategy.

I totally get that. Nodies began running nodes in August/September with geo-meshing, which means the development of the private client began before that, and active development of LeanPOKT was still underway between PNI and @poktblade in July/August, so it all overlaps. That is my point… the timing is very messy and confusing, which I why I believe it is relevant to today’s conversation. I also believe this kind of scenario should be avoided moving forward.

I would really appreciate it if we call a spade a spade and not try to now frame Nodies market capture as being an act of benevolence for the health and security of the network. From my understanding the geo-mesh code was never given to PNI to vet (LeanPOKT on the other hand was shared), nor did PNI ask to keep the geo-mesh concept private (they always encourage public discussion regarding LeanPOKT), so these are not equivalent scenarios. It was a great business move, and it took POKTScan months to reverse engineer in an open-source fashion, so I’d prefer we leave it at that.

While I can respect that the narrative at Nodies is the private client was “hardly” an advantage, it’s not fair to those who didn’t have access to it and were putting resources into reverse engineering. It was objectively an advantage that generated 2x-4x more revenue per node, and that is my only point.

Indeed, I remember seeing that Nodies started using geo-mesh in February:

image
image1324×510 56.1 KB

It’s great that everything started being open-source. Grateful for the contributions Nodies has had since :slightly_smiling_face:

1 Like
11

Ultimately, I desire to see gateways move forward. This is under PNF’s management, and not the DAOs.

I’m very grateful for their transparency and their willingness to accept feedback. I hope my feedback has constructive merit and am excited for what is next.

Thank-you @JackALaing and @giantfrog for being open to engage :+1:

3 Likes
12

Just wanted to chime in here and say Barg Systems has never wanted to develop, manage, create or sell their own gateway nor has any plans to. Nodies asked us to be a neutral party in the initial discussions around adding additional gateways and we obliged as we were already on site in Denver and it sounded an interesting conversation to be involved in.

In the end; multiple gateways are important. This will increase competition which in turn will yield better QoS, validation, accountability and scale.

7 Likes
13

@JackALaing or @poktblade, could there be a technical spec on what these modules are that Nodies is going to be producing? I asked about this over a month ago and nothing has been released yet, so I wanted to follow-up. Basic spec questions are:

What are the modules?
What do they do?
What will their features will be?
What technologies/libraries that are expected to be used?
What tech will they interface or integrate with (HAProxy, Promethius, etc)?

I understand the Nodies won’t be doing community calls until August, but there should be initial openness on what exactly is being built, so the community knows what to plan for. So far this is the most that has been communicated:

image

I still have no idea what exactly is being built, and I think there can be much more precise communication :sweat_smile:

1 Like
14

Thanks for following up @shane . We’re still ironing out the details but here is what we’ve currently defined that Nodies will provide in return for the grant:

  1. The Gateway Operator shall:

    1.1 open source technical documentation on integrating with Pocket Network including the steps below and to allow for an agnostic language implementation:
    (i) Using an app stake to retrieve a session,
    (ii) Generate an Application Authentication Token,
    (iii) Obtain hosted dispatcher URLs or spin up own dispatcher (Full Pocket Node),
    (iv) Send a request to a dispatcher for the latest nodes in a session,
    (v) Sign a relay and submit it to one of the nodes in a session,
    (vi) Receive a response from a node, and
    (vii) Proxy it back to the user;

    1.2 open source relay client implementation of the technical documentation described in clause 1.1 above in Golang,

    1.3 Open source technical documentation on the QoS and data integrity measures intended to be implemented by the Gateway Operator, as well as the open source implementation of the QoS framework implemented by the Gateway Operator;

    1.4 provide any additional information and documents as reasonably requested by PNF to help any other gateway operator portals launch on Pocket Network;

  2. take all necessary actions to open source their Gateway Operator Portal in the manner that PNF sees fit as soon as reasonably practicable after the Effective Date;

  3. The Gateway Operator must provide details that allow POKTScan, or any other data aggregator that PNF may reasonably choose to support from time to time, to verify the number of relays that the Gateway Operator Portal is sending to the Pocket Network, including for which Chain ID, as well as all relevant details about the Gateway Operator Portal’s key operating metrics such as its reliability (uptime and correctness) and performance (latency, success rate and proportion of relays sent to Pocket Network, any altruist network, or any other destination other than the Pocket Network protocol).

To expand on the QoS and data integrity measures to be open sourced:

Node Registry - consists of the current staked nodes separated by operator identity (not address) and provides long-term historical data about an operator. This will also give us information on any potential “bad actors” in the network.

Nodies Cop - Once the servicers are registered on the Node Registry, there is no guarantee that the servicers are in healthy shape. Nodies Cop will be a construct/ideology that performs “checks” for all the nodes on the registry periodically to ensure they are healthy. In the event that one of the checks fails, Nodies Cop will pull the servicer out of the servicing rotation. Once the servicer is pulled out of the rotation, Nodies Cop will periodically test the node still, exponentially increasing the retries until the servicer is healthy again. In summary, Nodies Cop is constantly looking for infractions and pulling nodes in and out of rotation. These checks would be classified as public/secret:

  • Public - health checks that users can see in order to diagnose if their servicers are healthy.
  • Secret - health checks that only administrators of the application can see. It is only important to know that we have secret checks because we do not want malicious users to ‘spoof’ or ‘workaround’ our checks to trick us into thinking their servicer is healthy. This can be further iterated to become how the first “watcher” will be implemented within the network if successful.

Here’s a working doc detailing more: Nodies DLB <> POKT Specifications - Google Docs

Note that this is a starting point and more technical details will be shared as it progresses, through channels like Gateway Office Hours, blog posts, and forum updates. Nodies also has a point person in @giantfrog who anyone is welcome to reach out to.

4 Likes
15

Thanks for providing more details Jack :+1: From the sound of it, Nodies is being contracted to build three specific tools thus far:

  1. An SDK for using AppStakes (what is being called “Bare Bones Integration”)
  2. Node Registry (a historical record of how providers respond to RPC calls)
  3. Nodies Cop (a health/liveness checker)

Those are pretty straightforward modules. I hope that these modules are being designed in an fashion where folks can pick and choose which ones they want to use, and which ones the want to use their own modules. I know that for CC we already built our own version of #2 and #3 and could potentially make our own modules available. If Nodies’ modules are primarily designed to only work with their modules, then that would limit how others could design gateways IMO.

I do hope more POPs will be utilized as well, since these types of modules could be built others, especially ones like #2 and #3.

In Nodies presentation, they focused heavily on their gateway’s primary value proposition will be the Opinionated APIs, which indubitably requires backend indexing and catching tools. Will those tools also be open sourced as part of this initiatives? Perhaps @giantfrog may be able to provide more insight?

1 Like
16

Also, could you expound on this @giantfrog? The idea of Secret health checks doesn’t make sense since nodes need to know how to identity if they are not hitting a specific performance. Example, if a “Secret health check” is actually a getlogs call and their node is failing, the node runner needs to be able to know what call they failed with to troubleshoot. A heavy call like getlogs could be effected by a number of variables, but if it’s considered a “secret” then how can anyone troubleshoot or improve their nodes? How could that also be fair if a provider like Nodies knows what the Secret health checks are, but no one else does… then Nodies node’s would have preferential treatment within your own gateway.

It also seems to be hinted that Secret health checks could be part of Watchers. Has that concept been vetted by the protocol team? Again, I’m not entirely sure what Secret health checks really means, so any further explanation would be great :+1:

1 Like
17

This is something we will have to live with in a multi-gateway landscape. If we want gateways and apps to be permissions, we cannot control that. In the end it is the gateways reputation for fair distribution of relays what is at stake.
We plan to check how each gateway is distributing their relays and show if there is favoritism in their “secret checks” , but in the end is up to the portal user.

I would not mix watchers here, it is a completely different topic (and a large one). Checks at portal level have nothing to do with checks at protocol level (watchers). The first ones decide on the distribution of the traffic and the later ones on the amount being paid for those relays.

3 Likes
18

I get your point, but it is in the spec of what they are creating for the DAO, so I’m asking for clarification on how all will benefit from “secret” health checks, as it’s not immediately obvious what they are. They obviously have thought it through to put it in the spec, so I’m just asking for clarification.

I agree with your point. Since it was mentioned in the spec, I am wanting clarification as this would have to be a larger topic that involves the larger community.

1 Like
19

Thanks for this, I gave it a read and I have some comments.

Node Registry - consists of the current staked nodes separated by operator identity (not address) and provides long-term historical data about an operator. This will also give us information on any potential “bad actors” in the network.

Grouping nodes by provider name has some drawbacks:

  • Node runners like to change and/or have multiple domains to obfuscate tracking.
  • The QoS features distributions of nodes is not well defined, we have found multi-modal distributions under a given domain name.

We suggest to use a per-address registry, as it is more flexible and provides better data. We have done this in the past (when QoS data was available), we can help you with it, it is a rather large DB but is the correct way to do it (IMO).

Nodies Cop - …

I think that you read all the watcher-related data, but just in case I will add some comments here.
This highly overlaps with the watcher and you will face very similar chalenges. Specifically I recall one relates to:

… performs “checks” for all the nodes on the registry periodically …

Periodicity of checks is not wise and is easily detected, randomize this.

is constantly looking for infractions and pulling nodes in and out of rotation

Delay actions (removal) from checks, or hide actual check among other “bloat” requests to the node before kicking them out of rotation.

I’m looking forward to see your gateway up and burning some tokens!

2 Likes
20

Emphasis on why it’s a secret check! We will start by incorporating our secret checks based on our clientele’s requests and needs, and we expect that other gateway operators will do the same. For us this is a fine balance between gamers and ensuring that we are able to ensure to our clients a specific level of assurance quality and security. It is very possible that in a world of gateway verse, some gateway operators will be more strict, and some gateway operators (i.e. non-profit or loss-leader gateways) may be less strict. Nodies DLB sits in between offering Pocket as a decentralized offering to our client base and accelerating Pocket Network’s goal of multiple gateways. I believe the spec is sufficient enough in what we will actually deliver. The mechanism for how secret checks are shared and distributed is up to gateway operators to decide. It can be further iterated by future gateway operators as Pocket continues to evolve. We’re not opposed to releasing our mechanisms, but at the same time, we have to balance that with what our clients require for security since the network has no effective slashing mechanism at the time.

Absolutely!

I’m personally not a fan of the per-node system we have in the Pocket ecosystem right now. It is often a sign of confusion about what the number of node counts actually means when in actuality, there is only a certain amount of blockchain nodes operating behind them. Ethereum is also going through a somewhat similar issue where they’re proposing to allow for stacking vertically instead of horizontally for nodes since many nodes map back to a single identity, anyways. That being said, it’s a pretty minor detail for us right now and can always be iterated on.

There have been many ideas we’ve floated around (i.e: an open dataset from us of ranked nodes that node operators can use as a baseline then build and train their own model, but this is likely not to make it into the initial bootstrapping budget, perhaps as a continuation and evolution of the initial gateway-verse). Any collaboration we can have with the best data science team in the POKT ecosystem sounds great :sunglasses:, and just shows the potential of what the world of multiple gateway operators would look like. I’m really excited to start forth on what we can do for the ecosystem and the opportunities it opens up for others.

An update on our end:

I’m happy to say we reached terms with PNF, and the last action item is a formal execution on contractual agreements in regards to our deliverables. We will finish this today and will have news to share with the community after our roadmap planning (1-2 weeks), and our first payment received for this budget will be started on the 1st of September :partying_face:

7 Likes