Into the Gateway-verse: Bootstrapping a Multi-Gateway Ecosystem

I completely support this.

Having another Gateway is an important step forward in our ecosystem, and Nodies is well placed to deliver.

Open sourcing help with QoS should additionally help other Gateways as they exist in, or enter Pocket. This is a fair trade for the grant.

Additionally, this is a better strategy to compensate Gateways for processing relays than the artificially low relay charge - discussed in more detail and generally agreed on by participants in the MINT discussion.

5 Likes

I do not desire to be unfair, so my apologies. I would ask you look at how those on the outside see this kind of decision making.

It was mentioned shortly after ETHDenver that plans to open v0 gateways were underway. From what was communicated, PNF members met with @poktblade in-person (not sure if others were there) about v0 gateways, which launched the idea. It excited everyone that v0 gateways would be possible.

Since then, there has been a number of community calls where both PNI and PNF members were asked for updates regarding v0 gateway plans, and it was repeatedly told that there isn’t anything to report as PNF is still trying to figure out how to share appstakes. It was always mentioned that once there has been progress it would be shared.

What was not expected, is that the update of progress would also include nearly a quarter of the Era Budget would be going to a company where a POP was not used. Some POPs are to follow… but not for this.

The ERA Budget said the support would be “regarding legal, docs, and any other small support needs they may have to go live”. It was not mentioned that this was to pay for the development of their gateway itself with 90% of the budget. From my side, I assumed anything funds for development would be focused on PNF tooling to share appstakes and ensure gateways are not gaming the system (which they easily can do unless PNF has robust monitoring).

Providing a single entity with nearly a quarter of the DAO Era Treasury, without a POP, was never communicated, hence why I ask you see things from the outsiders perspective. The POP usage for ALLOCATED PRIORITY: Mainnet Snapshots saw a lot of unique applicants with innovative value propositions. It was clear that the POP was for development, and I’m confused why a POP couldn’t have been used here, when a substantial amount of the DAO treasury is at stake.

Double funding?

I’m also struggling with seeing how this is not double funding. @BaaSPoolLLC said in their per-proposal for Leanpokt that they already have this exact team, which you are now saying PNF will fund again with more DAO treasury?

They said they will “becomes more sustainable and lucrative if we are able to get our reimbursement.”, which the DAO gave them, and now they need another $300k from the DAO for the same team? How many times is the DAO supposed to support the same team for the same roadmap? On-top of them already having one of the most lucrative node running businesses this year?

Also in their Community Call they stated they are building multiple ways to monetize with websockets and specialty APIs that do not use the POKT Protocol at all. The specially said:

There are like some creative ways where we could start taking in traffic ourselves and then create a plan to pipe that over to pocket network, but yes one of our goals is make sure that Pocket Network receives some of this traffic on top of that we want to provide sustainable support for the app users.

Yes, Pocket Network will get some of their traffic… but they propose that their main value prop is what is NOT related to the protocol itself, but related to their own centralized features. They are welcome to build whatever feature they like, but I don’t see how a gateway focused on non-protocol features should be subsidized by the DAO.

I truly have no idea how to understand what funding proposal is funding what at this point.

I also thought I had a good understanding of the Era Budget works, but now I super confused because this is not at all what was presented.

Not trying to mischaracterize anything, so please correct me where I am wrong. I’m just being factual with my understand of the information that has been given to us to date.

4 Likes

@JackALaing I would like to amend my comment from yesterday with some further clarification:

I do believe that open-source gateway tooling is a benefit to the community. The goal overarching goal is worth pursuing IMO.

The issue for me is how this funding came about. It is awesome that @poktblade started the ball rolling regarding v0 gateway conversation, but it appears Nodies was the only company to know that $300k was at stake for being the first alternative gateway in the ecosystem. Yes, Nodies has developed and self-launched their RPC webapp independently, but they have been the only ones with knowledge that PNF was planning to use DAO treasury to substantially subsidize gateway development in v0.

Had the rest of the ecosystem known that substantial funding from PNF was going directly to subsidize gateway development, I know many companies would have considered a gateway roadmap, my own included. Developing client facing, open-source gateway system is a lot of work and a high risk endeavor because there is no guarantee of return. From all the companies I’ve talked to, everyone in the POKT ecosystem is tightening budgets and trying to reduce risk to account for market conditions. Nodies on the other hand, has been aware of development funding plans and has been involved in the meeting between PNI and PNF, while everyone else has been patiently asking updates. From my perspective, Nodies had a clear preferential position, while the rest were essentially told to hang tight.

I also feel the LeanPOKT proposal was partially predicated on funding their future roadmap.

I’ve been under the impression that their substantial DAO reward for LeanPOKT was to support their short term future initiatives, including their gateway, which they announced prior to the proposal passing. I personally never thought that less then 3 months of receiving $353k, another $300k of DAO funds would be directly available to them without contest, especially when we have so many builders within POKT.

Clear The Air

To be clear, I don’t believe any of this was done nefariously. I do believe that PNF has the best intentions, and is looking to empower the network with gateway technology. Because of the high risk factor of developing gateway technology, without guarantee of return, it makes sense for PNF to invest in empowering the ecosystem with scalable tech. It is a solid strategy and I do believe that every company will benefit from open-source gateway tech.

My objection is how one company was able to position itself for guaranteed substantial development funding, without anyone being aware it has happening. I believe that something this substantial should utilize a POP system.

I also believe that there should already be an open technical spec for all node runners to participate in it’s design, prior to being guaranteed funding. Would it be possible for Nodies to be commissioned to create a technical spec first? Let us see the elements required and what technologies they plan to use (languages, libraries, databases, tooling, etc). From there, perhaps folks can give feedback or present their own for consideration? With such short notice, I don’t know who would be able to come up gateway plan, but I think it’s at least worth trying.

Move Quickly

I understand that the goal is to move quickly and I do not desire my feedback on this to slow anything down. I also want to help with solutions, not hinder progress and innovation. My mind is open to other thoughts on the matter, and I am very appreciate of PNF’s initiatives. I believe there was a misstep on this matter, but I believe PNF has had excellent communication and transparency the past 6 months.

2 Likes

Thank you for sharing your perspective @shane and for following up with clarifications. We empathize with your motives and want to clarify the facts and our reasoning.


Clarifying the Facts on the ERA Budget

Nodies did not have secret knowledge about the possibility of receiving $300k in grant funding. PNF ourselves did not know we would want to budget for this until last week.

When Nodies first came to us about their plans to build a gateway, they originally planned to self-fund. PNF also believed that all gateways should have enough incentive to take on the risk that typically comes with an entrepreneurial venture in return for the potential rewards. This is why the ERA Budget initially did not include any line items for funding any gateways.

Since then the macro environment makes it less feasible for gateways to self-fund, which prompted Nodies to seek more support from us in return for the open-source contributions we outlined in this post. In tandem with this development, we also received feedback on the ERA Budget proposal that we were not allocating enough of the budget to funding v0 QoS R&D. We took on board both developments and decided that we should be allocating a line item to ensure at least one new gateway is financially supported - in return for open-sourcing their technology to subsequent gateways - so we amended the budget.

Aside: It should be noted that $300k is a do-not-exceed. We will have wording in the contract to end the grant if either of the following occurs: 1) the price of POKT recovers to a level that enables Nodies to sustain their gateway from their node running, 2) they bring in enough revenue from gateway customers. Any excess would be returned to the DAO.


Clarifying the Origin of the v0 Gateway Strategy

When the current PNF team got started in January we immediately opened direct channels of communication with builders and conducted discovery calls with them to understand their needs and interests. It was during the discovery call with Nodies (Feb 9th, 3 weeks before EthDenver) that they first told us about their plans to build a v0 gateway service, originally envisioned as being built on top of the Portal using PNI’s proposed Gateway-as-a-Service (GaaS) model. The only other builder to express an interest in building a gateway at this time was BargSystems.

The meeting in Denver was an exploratory workshop to evaluate the merits of replacing PNI’s GaaS model with a bolder strategy to grant app stake access directly to gateways to enable more differentiated strategies (and thus innovation) at every layer of the gateway stack. Nodies & BargSystems were both part of this workshop due to previously expressing an interest in building a gateway as well as directly questioning whether GaaS was the best approach.

One thing to note is that it was important for us to have third-party perspectives in the room, besides ourselves and PNI, because we knew that PNI would naturally be reluctant to explore a strategy that would introduce operational overhead and uncertainty regarding their GaaS strategy. Inviting Nodies & BargSystems to participate at this stage was as much about eliminating bias from the discussion as it was about them being the only builders to have approached us about the idea, it was not about granting preferential access.

All of these earlier discussions laid the foundations for our ecosystem thesis, published in April, with decentralized demand as the most important component, and The Road to Revenue.


Clearing the Air, Making Bets, & Encouraging other Gateways

We’ll take the L and admit that this isn’t how we’d like to do things in an ideal world, if we were starting now and had ample time and resources, and driving demand wasn’t such an urgent priority. It is also important to recognize that Nodies was a catalyst who played a key role in formulating this strategy; without them, we would not be just weeks away from the launch of our second gateway.

Wartime decision-making requires decisive bets with a ruthless priority on the expected value from each “bet” we make. Gateway R&D is a prime example of an area where decisive bets must be taken. We need to: lower the barrier to entry for gateways, develop insights for the V1 on-chain gateway actor, improve QoS and thus grow relays/revenue. The best bet to achieve these goals is to support one pioneering second-mover gateway in return for open-sourcing the outputs of their R&D. For a must-have contribution of this nature, the only factor that matters is capability and speed. Nodies has demonstrated their capability, as outlined in the original post, and has already built momentum on their gateway and their QoS R&D (see session rollover fix).

It should be noted, as I’ve stated before, this does not preclude other companies in the Pocket ecosystem seeking grants to work on gateway R&D. We want to see more gateways! Please talk to us if you’re interested in working on this. Any relevant POPs, sockets, and bounties will be funded by both the ERA Allocation (80% of future DAO rewards) and PNF treasury.

3 Likes

Thank-you very much for this response! It does provide MUCH clarity, and enable there to be a conversation based on material information.

I think the primary point of miscommunication is around PNI’s Gateway-as-a-Service (GaaS) model and appstake sharing. The GaaS model was based on using PNI’s backend infrastructure with others building custom front-ends. I know that myself and others did not find that a compelling model for many reasons. I believe that is why so few folks showed interest in gateways when that was the only available model.

However, once it was announced that the GaaS model was NOT the future, but instead appstakes would be independently operated, there has been significantly more interest across the community from what I have seen. An appstakes model is a completely different beast as there isn’t a dependency on PNI (like the GaaS model would introduce).

It seems that interest for the GaaS model (dependent gateways) and the appstakes model (independent gateways) were being unintentionally conflated by PNF as the same thing. Because Nodies and BargSystems showed interest in GaaS, they were part of the internal conversations regarding appstakes, which I feel was an oversight and did provide them with preferential access to formulate their strategy and now access to uncontested funding.

I first heard about the appstakes model after ETHDenver and, and immediately expressed interest in the community call and we were told by both PNI and PNF that more information would be coming. There were little updates occasionally, but nothing that signaled there was conversations happening where interested parties could be involved. I have witnessed others expressing interest the independent gateways as well. Unfortunately there hasn’t been actionable information since, and I wrongly assumed POP’s were going to be used for the entire project, not just a small subset.

I also believe my concern about double founding the same roadmap is valid. Nodies received 6.9M POKT which has yet to be touched, and yet they have been operating with the same number of team members since February. Again, it confusing what the LeanPOKT proposal is funding if not their roadmap as indicated in their proposal and the conversation that followed.

I would also like reassurance that everything is crystal clear what they are building with any extra funding, so there isn’t a similar situation that happened with the LeanPOKT contribution. While PNI and the community was testing and improving the LeanPOKT open-souce client for everyone’s benefit, they launched their staking business entirely on a closed source client with a version of geo-meshing, unannounced to the community. While a savvy strategy, I personally saw that as the community contributing and later very lucratively rewarding one client, while another client was developed in that exact same time frame and used by their business to garner substantial market capture over their competitors.

Therefore, I feel it is in the best interest of everyone to ensure there is not the same blind spots in further funding. Everyone (including myself) have commended them for their LeanPOKT contribution and desired to reward their efforts, but it is unclear how much of the LeanPOKT funding went towards the open-source version and what went to the closed-source version they actually used for their business, since they were developed in the same time-frame covered by the LeanPOKT proposal. It would be a major loss if that same scenario played out with gateway contributions.

If they produce quality open-source tooling that has universal utility within the POKT ecosystem, then it is worth making it happen, though I would hope there be vigilant PMing with detailed technical specs that can be shared with the community, defined deliverables and roadmap, along with a clear SLA/contract with BaaS Pool LLC. I say this not at all to micro-manage, as PNF is extremely competent, I’m just being honest about blind spots we have had in the past and my perspective.

I do also believe that $300k is too much considering their LeanPOKT proposal, who’s value was partially predicated on their roadmap (which included the gateway), so I would suggest that it be taken into account, and provide more POP opportunities for others (which Nodies could apply for as well).

I leave this feedback for PNF’s consideration :+1:

2 Likes

Hi @shane, I believe @JackALaing already clarified a number of your questions, and we’re glad to see you’re aligned with the value proposition of gateways, as well as what an undertaking this will be. We also wanted to share our perspectives on a few things.

First, I would like to point out that the Quote you pulled was from a Pre-proposal, the details of which were quite different from the actual proposal, which the DAO actually voted for and approved. For one thing, you will find that the quote you pulled was not included in the passed proposal.

Second, there appears to be some confusion regarding the timelines involved. The development of LeanPokt by the Poktfund team, led by @poktblade, began in February 2022 and was merged into Pocket Core in December 2022. Subsequently in February 2023, we submitted a reimbursement request for the work completed by our team during that period. Currently, we are seeking funding for the upcoming 6 months of work. Therefore, if we consider the start dates of these two asks, there is actually a gap of over a year between them. To be clear, The LP Proposal (and its subsequent passage) was not intended to fund future projects. LeanPOKT was also never intended to be an all-inclusive grant for all the work we perform at Pocket. Generally speaking, we consistently do pro-bono work for Pocket that adds value to the ecosystem, but considering the scope of work required for the gateway, it made sense for us to seek funding.

Furthermore, there have been changes in personnel over time. Some of our team members who worked on LP have transitioned to our staking operation, which is unrelated to our gateway operations. Some, like @poktblade, have remained; while others, including myself, have joined as new team members specifically for Nodies DLB. I recently began my role as a business development representative for our gateway.

Thank you for transcribing from the Community Call. I believe this was misconstrued, so allow me to clarify. @poktblade’s intention was to highlight that in the event Pocket Network is unable to handle specific requests (for instance, if a customer needs websockets, but Pocket is not yet capable of handling those requests), NodiesDLB will still handle the requests to meet the client’s needs. This will allow us to incentivize DApps to use our platform (developer productivity is a common value prop), help provide market data to the protocol team and shape the roadmap, and ultimately pipe this to Pocket once it is supported. If the focus is demand, then we need to be competitive in our product offerings by strengthening our value propositions and providing DApps with what they need instead of turning them away. This in return will indirectly provide traffic to Pocket as well, as clients hardly want to use multiple providers for blockchain development. If you take a look at many other RPC providers, this is a common strategy in order to capture demand.

This is a mischaracterization based on incorrect assumptions. First, we released LP well before Nodies began running nodes, and we never stopped working on LP even as “PNI and the community” were testing the beta. Second, it is true that Nodies’ commercial staking operation began in September 2022 (and some small nodes for ourselves, friends, and family prior), and we were using a software akin to the Geo-Mesh. There were conversations to open source this work as well, but it was in alpha testing during a time when we had a relatively small amount of nodes and limited data to test. Even then, it would suffer from outages fairly often, bringing down some of our nodes with it. It took us a while to get the software to a stable enough state. We prioritized the network’s health and security similar to how LeanPocket was originally closed source as advised by the protocol team. By that point, the PoktScan had already released their Geo-Mesh in October 2022 with a much more reliable and supported codebase, hence we didn’t see the need to release ours. The timeframe between all of this is really quite short and hardly gave us much market advantage. In fact, we already alluded to this in response to a comment you made back during LeanPOKT’s Preproposal -

Lastly, we have since switched over all of our nodes to Geo-mesh in the spirit of keeping consistent with the rest of the network and encouraging open sourceness within the ecosystem. In fact, we are the second and only other contributor to Geomesh. We’ve worked with them to identify bugs, optimizations, code reviews, and have been credited multiple times directly from the main developer @Jorge_POKTscan (PR 1, PR 2, and countless of private conversations on how to architect and code).

I hope this clears some things for you. We are excited to continue contributing towards enhancing and supporting Pocket Network in various aspects!

7 Likes

Hey @giantfrog, I don’t believe we have met before. I appreciate the response.

Oh, I’m aligned. So much so, I wrote the original ADR that the v1 integration is based on :stuck_out_tongue_winking_eye:

Pre-proposals are where the conversations happen for proposals… that is their intended design. You will find that with all the pre-proposal/proposal pairings, the real conversation for the proposal is in the first version (pre-proposal) and not the final version. It is a new perspective to try and divorce their relevancy to each other.

The final proposal for LeanPOKT was fast tracked and put to a vote after only 5 days, instead of the standard minimum of 7 days, precisely because the pre-proposal was the context, who’s conversation expanded over many weeks. Therefore I’d prefer to use the whole community conversation for context.

Unfortunately, that was not clear at the time because the proposal’s value was not predicated on measurable metrics, but more abstract concepts of value. Concepts that were partly built on notions that this proposal is worth it’s ask because of the value it would produce in the ecosystem (which thus far it has been unused). This is why I have always advocated for measurable metrics with every proposal, including LeanPOKT, so there would be universal clarity :sweat_smile:

It may be clear to you, but for myself, as a voter, outsider, and participant in the LeanPOKT proposal conversation, it is not clear. That is my point and why I provided past context.

As much I would like to take credit for transcribing, I must give it to Youtube :joy:

Yeah, I agree that if Nodies is able to capture customers for non-protocol features, it can indirectly lead to more traffic to POKT as well. I get the strategy.

I totally get that. Nodies began running nodes in August/September with geo-meshing, which means the development of the private client began before that, and active development of LeanPOKT was still underway between PNI and @poktblade in July/August, so it all overlaps. That is my point… the timing is very messy and confusing, which I why I believe it is relevant to today’s conversation. I also believe this kind of scenario should be avoided moving forward.

I would really appreciate it if we call a spade a spade and not try to now frame Nodies market capture as being an act of benevolence for the health and security of the network. From my understanding the geo-mesh code was never given to PNI to vet (LeanPOKT on the other hand was shared), nor did PNI ask to keep the geo-mesh concept private (they always encourage public discussion regarding LeanPOKT), so these are not equivalent scenarios. It was a great business move, and it took POKTScan months to reverse engineer in an open-source fashion, so I’d prefer we leave it at that.

While I can respect that the narrative at Nodies is the private client was “hardly” an advantage, it’s not fair to those who didn’t have access to it and were putting resources into reverse engineering. It was objectively an advantage that generated 2x-4x more revenue per node, and that is my only point.

Indeed, I remember seeing that Nodies started using geo-mesh in February:

It’s great that everything started being open-source. Grateful for the contributions Nodies has had since :slightly_smiling_face:

1 Like

Ultimately, I desire to see gateways move forward. This is under PNF’s management, and not the DAOs.

I’m very grateful for their transparency and their willingness to accept feedback. I hope my feedback has constructive merit and am excited for what is next.

Thank-you @JackALaing and @giantfrog for being open to engage :+1:

3 Likes

Just wanted to chime in here and say Barg Systems has never wanted to develop, manage, create or sell their own gateway nor has any plans to. Nodies asked us to be a neutral party in the initial discussions around adding additional gateways and we obliged as we were already on site in Denver and it sounded an interesting conversation to be involved in.

In the end; multiple gateways are important. This will increase competition which in turn will yield better QoS, validation, accountability and scale.

7 Likes

@JackALaing or @poktblade, could there be a technical spec on what these modules are that Nodies is going to be producing? I asked about this over a month ago and nothing has been released yet, so I wanted to follow-up. Basic spec questions are:

What are the modules?
What do they do?
What will their features will be?
What technologies/libraries that are expected to be used?
What tech will they interface or integrate with (HAProxy, Promethius, etc)?

I understand the Nodies won’t be doing community calls until August, but there should be initial openness on what exactly is being built, so the community knows what to plan for. So far this is the most that has been communicated:

image

I still have no idea what exactly is being built, and I think there can be much more precise communication :sweat_smile:

1 Like

Thanks for following up @shane . We’re still ironing out the details but here is what we’ve currently defined that Nodies will provide in return for the grant:

  1. The Gateway Operator shall:

    1.1 open source technical documentation on integrating with Pocket Network including the steps below and to allow for an agnostic language implementation:
    (i) Using an app stake to retrieve a session,
    (ii) Generate an Application Authentication Token,
    (iii) Obtain hosted dispatcher URLs or spin up own dispatcher (Full Pocket Node),
    (iv) Send a request to a dispatcher for the latest nodes in a session,
    (v) Sign a relay and submit it to one of the nodes in a session,
    (vi) Receive a response from a node, and
    (vii) Proxy it back to the user;

    1.2 open source relay client implementation of the technical documentation described in clause 1.1 above in Golang,

    1.3 Open source technical documentation on the QoS and data integrity measures intended to be implemented by the Gateway Operator, as well as the open source implementation of the QoS framework implemented by the Gateway Operator;

    1.4 provide any additional information and documents as reasonably requested by PNF to help any other gateway operator portals launch on Pocket Network;

  2. take all necessary actions to open source their Gateway Operator Portal in the manner that PNF sees fit as soon as reasonably practicable after the Effective Date;

  3. The Gateway Operator must provide details that allow POKTScan, or any other data aggregator that PNF may reasonably choose to support from time to time, to verify the number of relays that the Gateway Operator Portal is sending to the Pocket Network, including for which Chain ID, as well as all relevant details about the Gateway Operator Portal’s key operating metrics such as its reliability (uptime and correctness) and performance (latency, success rate and proportion of relays sent to Pocket Network, any altruist network, or any other destination other than the Pocket Network protocol).

To expand on the QoS and data integrity measures to be open sourced:

Node Registry - consists of the current staked nodes separated by operator identity (not address) and provides long-term historical data about an operator. This will also give us information on any potential “bad actors” in the network.

Nodies Cop - Once the servicers are registered on the Node Registry, there is no guarantee that the servicers are in healthy shape. Nodies Cop will be a construct/ideology that performs “checks” for all the nodes on the registry periodically to ensure they are healthy. In the event that one of the checks fails, Nodies Cop will pull the servicer out of the servicing rotation. Once the servicer is pulled out of the rotation, Nodies Cop will periodically test the node still, exponentially increasing the retries until the servicer is healthy again. In summary, Nodies Cop is constantly looking for infractions and pulling nodes in and out of rotation. These checks would be classified as public/secret:

  • Public - health checks that users can see in order to diagnose if their servicers are healthy.
  • Secret - health checks that only administrators of the application can see. It is only important to know that we have secret checks because we do not want malicious users to ‘spoof’ or ‘workaround’ our checks to trick us into thinking their servicer is healthy. This can be further iterated to become how the first “watcher” will be implemented within the network if successful.

Here’s a working doc detailing more: Nodies DLB <> POKT Specifications - Google Docs

Note that this is a starting point and more technical details will be shared as it progresses, through channels like Gateway Office Hours, blog posts, and forum updates. Nodies also has a point person in @giantfrog who anyone is welcome to reach out to.

4 Likes

Thanks for providing more details Jack :+1: From the sound of it, Nodies is being contracted to build three specific tools thus far:

  1. An SDK for using AppStakes (what is being called “Bare Bones Integration”)
  2. Node Registry (a historical record of how providers respond to RPC calls)
  3. Nodies Cop (a health/liveness checker)

Those are pretty straightforward modules. I hope that these modules are being designed in an fashion where folks can pick and choose which ones they want to use, and which ones the want to use their own modules. I know that for CC we already built our own version of #2 and #3 and could potentially make our own modules available. If Nodies’ modules are primarily designed to only work with their modules, then that would limit how others could design gateways IMO.

I do hope more POPs will be utilized as well, since these types of modules could be built others, especially ones like #2 and #3.

In Nodies presentation, they focused heavily on their gateway’s primary value proposition will be the Opinionated APIs, which indubitably requires backend indexing and catching tools. Will those tools also be open sourced as part of this initiatives? Perhaps @giantfrog may be able to provide more insight?

1 Like

Also, could you expound on this @giantfrog? The idea of Secret health checks doesn’t make sense since nodes need to know how to identity if they are not hitting a specific performance. Example, if a “Secret health check” is actually a getlogs call and their node is failing, the node runner needs to be able to know what call they failed with to troubleshoot. A heavy call like getlogs could be effected by a number of variables, but if it’s considered a “secret” then how can anyone troubleshoot or improve their nodes? How could that also be fair if a provider like Nodies knows what the Secret health checks are, but no one else does… then Nodies node’s would have preferential treatment within your own gateway.

It also seems to be hinted that Secret health checks could be part of Watchers. Has that concept been vetted by the protocol team? Again, I’m not entirely sure what Secret health checks really means, so any further explanation would be great :+1:

1 Like

This is something we will have to live with in a multi-gateway landscape. If we want gateways and apps to be permissions, we cannot control that. In the end it is the gateways reputation for fair distribution of relays what is at stake.
We plan to check how each gateway is distributing their relays and show if there is favoritism in their “secret checks” , but in the end is up to the portal user.

I would not mix watchers here, it is a completely different topic (and a large one). Checks at portal level have nothing to do with checks at protocol level (watchers). The first ones decide on the distribution of the traffic and the later ones on the amount being paid for those relays.

3 Likes

I get your point, but it is in the spec of what they are creating for the DAO, so I’m asking for clarification on how all will benefit from “secret” health checks, as it’s not immediately obvious what they are. They obviously have thought it through to put it in the spec, so I’m just asking for clarification.

I agree with your point. Since it was mentioned in the spec, I am wanting clarification as this would have to be a larger topic that involves the larger community.

1 Like

Thanks for this, I gave it a read and I have some comments.

Node Registry - consists of the current staked nodes separated by operator identity (not address) and provides long-term historical data about an operator. This will also give us information on any potential “bad actors” in the network.

Grouping nodes by provider name has some drawbacks:

  • Node runners like to change and/or have multiple domains to obfuscate tracking.
  • The QoS features distributions of nodes is not well defined, we have found multi-modal distributions under a given domain name.

We suggest to use a per-address registry, as it is more flexible and provides better data. We have done this in the past (when QoS data was available), we can help you with it, it is a rather large DB but is the correct way to do it (IMO).

Nodies Cop - …

I think that you read all the watcher-related data, but just in case I will add some comments here.
This highly overlaps with the watcher and you will face very similar chalenges. Specifically I recall one relates to:

… performs “checks” for all the nodes on the registry periodically …

Periodicity of checks is not wise and is easily detected, randomize this.

is constantly looking for infractions and pulling nodes in and out of rotation

Delay actions (removal) from checks, or hide actual check among other “bloat” requests to the node before kicking them out of rotation.


I’m looking forward to see your gateway up and burning some tokens!

2 Likes

Emphasis on why it’s a secret check! We will start by incorporating our secret checks based on our clientele’s requests and needs, and we expect that other gateway operators will do the same. For us this is a fine balance between gamers and ensuring that we are able to ensure to our clients a specific level of assurance quality and security. It is very possible that in a world of gateway verse, some gateway operators will be more strict, and some gateway operators (i.e. non-profit or loss-leader gateways) may be less strict. Nodies DLB sits in between offering Pocket as a decentralized offering to our client base and accelerating Pocket Network’s goal of multiple gateways. I believe the spec is sufficient enough in what we will actually deliver. The mechanism for how secret checks are shared and distributed is up to gateway operators to decide. It can be further iterated by future gateway operators as Pocket continues to evolve. We’re not opposed to releasing our mechanisms, but at the same time, we have to balance that with what our clients require for security since the network has no effective slashing mechanism at the time.

Absolutely!

I’m personally not a fan of the per-node system we have in the Pocket ecosystem right now. It is often a sign of confusion about what the number of node counts actually means when in actuality, there is only a certain amount of blockchain nodes operating behind them. Ethereum is also going through a somewhat similar issue where they’re proposing to allow for stacking vertically instead of horizontally for nodes since many nodes map back to a single identity, anyways. That being said, it’s a pretty minor detail for us right now and can always be iterated on.

There have been many ideas we’ve floated around (i.e: an open dataset from us of ranked nodes that node operators can use as a baseline then build and train their own model, but this is likely not to make it into the initial bootstrapping budget, perhaps as a continuation and evolution of the initial gateway-verse). Any collaboration we can have with the best data science team in the POKT ecosystem sounds great :sunglasses:, and just shows the potential of what the world of multiple gateway operators would look like. I’m really excited to start forth on what we can do for the ecosystem and the opportunities it opens up for others.

An update on our end:

I’m happy to say we reached terms with PNF, and the last action item is a formal execution on contractual agreements in regards to our deliverables. We will finish this today and will have news to share with the community after our roadmap planning (1-2 weeks), and our first payment received for this budget will be started on the 1st of September :partying_face:

7 Likes

I know how much time these forum posts take from the initial post to replying to and answering all of the questions.

Shout-out to everyone involved to get this over the finish line.

Send it!

3 Likes

A brief but crucial update in spirit of open data and transparency!

  • POKT <> Nodies Integration Public Tracker can be found here , this tracker outlines all the tickets we are currently working on. This dashboard can singlehandedly be used to track all our progress.
  • POKT MVP Integration date is preliminary slated for October 31st & we’re currently a week ahead in progress.
  • User onboarding guidelines and baseline benchmarks published on our docs. We will have material for POKT before integration is complete.
  • https://nodies.app website was revamped both from a code perspective, observability, and UI enhancements. We are working towards another addition to highlight Pocket, which is staged on https://dev.nodies.app
  • @Jorge_POKTscan is graciously working on Testnet implementation for Poktscan so we can leverage it for gateway testing. (:fire: kudos to Poktscan team for taking this on). I cannot stress the importance of testnets.
  • Marketing for multi-gateway awareness is a work in progress in collaboration with @Adrienne (PNF) and our BD team!

4 of us, including myself, will be in Permissionless II, so if you will be attending and want to connect - please on Twitter/X!

Track our progression in our discord, public project tracker, through DM’s, or our socials.

3 Likes

Thanks for the update.

Can you explain why the QoS framework module will only “partially implement the ‘Cop’ model”? Why does the QoS module not include the whole Node Registry and Nodies Cop functionality that Nodies is currently building?

I understand that the you will be providing “technical documentation on how we’re currently identifying good actors”, but since only part of the code will be open source, this means that future gateways will have to spend their time and resources reverse engineering all the parts that aren’t open source.

Thanks in advance :+1:

2 Likes