Gateway Optimal Orchestration and Deployment (GOOD)

Build Research
1

authors:

This is an analysis of current Pocket Network Portal deployment and possible optimal deployment strategies.
We are requesting Pocket Network Inc. (PNI) to reduce the number of portals to reduce the cost of the network and improve the average latency.

Rationale

We have been observing the Pocket Network traffic since the Cherry Picker DB was made public (~06/2022) since that date we observed that most of the traffic was flowing through 3-4 gateways.
Accordingly, node runner aligned their nodes to capture most of this traffic. Later, with the introduction of the GeoMesh, the traffic and the node-runners node deployment aligned even further.
Today, when traffic flows from an specific area, there is always a node-runner following this surge, and a node is promptly spin up to catch all this traffic.
This behavior is is being intensified due to the lower rewards and the tighter competition among node runners, resulting in lots of effort put into service of chasing traffic and improve spot-profit detection (single chains deployed in exotic regions).
While this is not a bad thing in itself and it could be regarded as simple node-runner competition, we think that it is not optimal for the Pocket Network interests.

Summary

We are proposing to reduce the number of gateways to minimum/optimal configuration of 3 gateways:

  1. ap-southeast-1
  2. us-east-1
  3. eu-central-1

This will provide the following benefits:

  • Less node-runner effort used to track and chase chains for small gains (both in QoS and rewards)
  • Reduced infrastructure costs for node runners (less sell pressure)
  • Reduced infrastructure costs for PNI (longer runway)
  • Reduced global latency. We expect a reduction from 155 ms to 120 ms (Mean Success Latency)
  • Network cost becomes easier to estimate (very important for economic models).
  • Metrics become easier to track and understand. Cherry Picker metrics are critical to the ecosystem and economic models.
  • With fixed number of gateways (until Gateway client) (that the community agrees on having) node runners wont fear further arbitrary retirement of gateways (such as ap-northeast-1)

Also, we hope that if node-runners are able to reduce the time their spent in chasing traffic they can focus on improving the protocol.
The Pocket Network has many actors with different skill sets, if we can manage to focus them on improving the network instead on focusing on that extra 3% relays, everybody wins.

Metodology

(we encourage the interested reader to go through the full document and code here)

In a few words, the problem is tackled as a linear optimization problem based on a graph structure of the Amazon Web Services (AWS) inner ping times and the node-runners latency to the different Pocket Network Portals (PNP) located on specific AWS Availability Zones.
We show how to model the PNP deployment problem, find an optimal solution and propose a better strategy to reduce the costs for all the players of the Pocket Network ecosystem.
As data we use the Cherry Picker informed latency and the ping data of AWS from Cloud Ping. Also, the cost of “hopping” through AWS regions was modeled, adding 50 ms each time the relays need to cross a node that is not their target node (the servicers).

The initial graph is the following:

imagen
imagen1920Ă—1281 219 KB

What we did was find which were the lines coming from the blue node (servicers) that can be removed and re-routed to obtain the optimal minimum configuration. The resulting map is the following:

imagen
imagen1283Ă—806 122 KB

Also we produced a graph showing how the latency of the network evolved with the number of active portals:

imagen
imagen2586Ă—1704 215 KB

Open questions for the community

  • Are there problems that are not modeled in this report?
  • How can we improve this model if it is not realistic enough?
8 Likes
2

Hey Guys (disclaimer I haven’t read the full report yet haven’t had time this morning will try and read later)

I’m really struggling to understand why removing regions improves QoS only thing I can see which would result in this is bugs or scalability issues with how the CP is written/ deployed (is this the case?).

Take Asia for example the distance between ap-se1 and ap-ne is some 3000miles and a ping of >90ms when factoring in all the switch jumps. How routing traffic from north asia to south improves QoS for users in this region I don’t understand this claim. Let’s say you have a client in Korea my understanding currently they would be routed to AP-North in Seoul (imposing a sub 20ms latency) now they will be routed to Singapore imposing an additional latency of >90ms.

Would love to know more about the theory behind these claims that removing regions improves QoS.

3 Likes
3

Hi Andy, the problem is in the misalignment of gateways and Pocket Nodes locations and the routing of relays through public or private networks (internet or AWS).

This has nothing to do with the Cherry Picker (CP), it will remain untouched.

Let me take your example. You have a client in Korea, very near ap-notheast-2, right now if he/she connects to Pocket Network he/she would be directed there with 20ms latency (to give a number). Once there the request would be sent to the Pocket Node in some unknown location. The selected node is based on QoS using the CP. According to our observations, this relay would take 322 ms in average. So, this call, in average will take:

user ---- (+20 ms )—> ap-notheast-2 ---- (+322 ms )—> Pocket Nodes

The total average response time would be 342 ms. Why so high? because the Pocket Nodes are not normally in ap-notheast-2, the relay needs to leave AWS and travel through public internet to reach the unknown location of the Pocket Node. Even “high-QoS” Pocket Nodes in ap-notheast-2 are slow for most chains.

If instead forcing the relay to leave AWS, we propose to use their internal network, the relay would follow a different path. The client in Korea will be routed to ap-notheast-2, but from there it will be routed to ap-southeast-1 with a latency of 75 ms plus a hop time of 50 ms. Now, the relay will leave AWS in ap-southeast-1 where the Pocket Nodes respond with an average time of 75 ms. This call, in average will take:

user ---- (+20 ms )—> ap-notheast-2 ---- (+75 ms + 50ms )—> ap-southeast-1 ---- (+75 ms )—> Pocket Nodes

The total average response time would be 220 ms. Why was the time reduced? because you used the AWS infrastructure to reach a location that is closer to the Pocket Nodes locations faster. There are more chain nodes near ap-southeast-1.

This reduction is possible for two reasons:

  • AWS internal network is faster than public internet. This holds not only for AWS, any big provider (such as GCP or Azure) has optimized internal networks.
  • The node-runners did not deploy their chain nodes in each of the Pocket Network Portals locations. This means that the portals are only the entry point of the relay, not the place where they are processed.
1 Like
4

OK! this clarifies it. So the users will still point to the nearest AWS region which is then tunneled to the nearest gateway via AWS interconnects. I was under the assumption that users interacted directly with gateways (which appears to be incorrect from what you’re saying here).

This still sounds like a very drastic cut to happen based on theoretical data!, hopefully shutting down will be done gradually and monitored and not a straight up overnight cull.

Would another option not be just to increase the number of nodes per session to allow for an increased probability that nodes are closer to the gateways?. Concentrating it to just 3 benefits cloud runners even more who have flexibility to move affected nodes easily, getting nodes off centralized providers should be encouraged.

As a bare-metal service company who have invested in infrastructure in some of the culled regions it is still a little annoying to see this.

2 Likes
5

This is super great. I still have to read the longer report and have similar question as Andy.

This is music to my ears. Glad to hear someone else say it :slight_smile:

4 Likes
6
  • It averages out to 150ms because that is the cut off for the Cherry Picker’s top bucket.
  • The cost to deploy a region to AWS is minimal – Elasticache service is the only thing that requires spinning up. It only costs money once it receives traffic.
  • Pocket is changing providers so the AWS routing will no longer apply
  • Reducing the number of Gateway endpoints will undoubtedly cause poorer latency for the end users
  • You are centralizing the network and reducing the possibility of local service (user in Paris hits Paris gateway and gets Paris nodes)
  • Your assumption that users only centralized in those 3 cities is incorrect
2 Likes
7

Hi Alex, thanks for the commentaries.

If you mean the current global latency, that is 155 ms, I think that it is just coincidence. Each gateway have their own Cherry Picker process, and as seen in table 1 of the report the averages at each location greatly differ from the CP threshold of 150 ms:

imagen
imagen1622Ă—1220 119 KB

This is because the data obtained from the CP tracking process does not inform “buckets” it informs the actual latency of the relays. Also, node-runners do not seem to be targeting 150 ms, the go as low as they can. This can be seen in POKTscan’s Geo tab for any provider.

I cannot give you precision on cost reduction as I have no visibility of the involved costs. Our claim comes from a simple concept, removing things should reduce the costs. In the end is the PNI who decides if this is worth doing.

Its is true that the report as it is right now wont translate directly to GCP (the new PNI provider). However, the methodology can be easily converted to GCP by simply replacing the ping matrix used in the calculations. This matrix can be found in data/cloudping_p50_1W_02-02-2023.csv in the repository. We believe that results will not vary much, since the selected gateways represent clear clusters, but we are open to be proved otherwise.

That highly depends on the implementation, we have shown that if internal routing is applied this is not true, in fact it will improve.
If nothing is done and only the gateways are removed, the apps might align to the nodes or implement services such as Argo to improve their latency to the remaining locations. This will also reduce their ping since the round-trip to the gateway is avoided. Anyway, it is very difficult to add this to a report without making hard assumptions, for that reason we only provided the resulting scenario under controlled conditions.

This is not correct, a user in Paris hits the Paris gateway but it gets the nodes that are the best in his session.
Maybe Paris is not the best example, since Europe is very well connected and a node in Paris is the same as one in Frankfurt for the CP behavior. Take for instance Hon Kong, right now the users of Polygon in Hon Kong hit ap-east-1 and they are paired to nodes that are not in Hon Kong because there are too few of them there. The average session wont have a Hon Kong node, since there are less than half the nodes with less than 150 ms in ap-east-1, as can be seen in POKTscan’s Chains page.
We agree that it would be better to have more gateways and node-runners deploying nodes all over the world, but the current market conditions are not ideal to put that economic stress on the ecosystem.

We make no assumptions on where the nodes are deployed. The three selected cities are the result of an optimization problem not an assumption.
We believe that around 70% of the deployed nodes are in those regions, but that played no role in the results presented here. You can easily check that out by modifying the latency and/or traffic of any not selected region and re-running the algorithms (by means of modifying the grouped_df dataframe).
We never set, hint or guide the process to select a given location or a certain number of them.

1 Like
8

The numbers are clearly clustering around 150ms except in underserved regions.

I had full visibility and the cost per region is minimal. A few hundred dollars per month when it has no traffic and is scaled down. They only cost money when they receive traffic therefore the entire premise of this post is moot. You’re not saving money by spinning down regions.

Please don’t tell me how the cherry picker and sessions work. I wrote the code. There are many nodes in Paris; I ran hundreds there and in Amsterdam. The datacenters were cheaper than Frankfurt. Given that there are now dozens of nodes in each session, the likelihood that a Paris user will get a few Paris nodes is good.

Traffic shifts over time. For the first year of the protocol there was almost NO APAC traffic at all. Historically AP-SE-1 has been the biggest but over the past few months, the north regions took over. By reducing the number of gateways, you are betting that traffic doesn’t shift.

Take US-West as an easy example. With the only gateway now in the east, every single user request coming from California will enjoy an additional 80+ ms of roundtrip for each request.

1 Like
9

Thanks for the insight, this should be discussed with the PNI then.

Great then you can do the math and calculate which are those odds. The fact that you ran hundreds of nodes in Paris does not tell us anything, there are 20K nodes in the network and the CP wont care if your node is in Paris or Frankfurt as long as they have less than 150 ms latency.

We are aware that traffic shifts.

Lots have changed in the last year.
If the token was still 3 u$d, we would not be proposing this.
If the PNI wouldn’t be trying to reduce costs, we would not be proposing this.
If the V0 had enough support so node-runners could haply run their business, we would not be proposing this.

Nobody wants to drop support, ideally we would be buying hardware everyday and expanding to new regions, but that’s not the landscape today. If it wasn’t for the community (mainly node-runners) V0 would have probably crashed. I believe that this is what we need to re-group and make it to V1. We need to focus on maintaining V0 and help on V1. The node-running race can take a brake.

That is already happening, less than 10% of the nodes are local in US-West. You can get the odds of being in a session of one of your nodes in us-west.

1 Like
10

The bottom line is that this proposal does not reduce costs in any significant way. The regions cost a few hundred dollars each per month to spin up before they receive traffic. It will significantly affect service. The fact that no one on the team has raised this to Arthur is disturbing.

1 Like
11

Hmmm… not sure how to reconciling the different perspectives here. This is what PNI is saying regarding reducing costs:

image
image741Ă—297 36 KB

I was always under the impression that gateways were lightweight, but PNI is saying otherwise.

1 Like
12

hey @AlexF!

Great to hear from you. Would you mind hopping on a call with POKTscan and some folks from PNI next week?

1 Like
13

We’ll be chatting on Tuesday. Will follow up after.

2 Likes
14

They are wrong. Spinning up a new region costs around $500 per month with the minimum level of Elasticache. Nothing else costs money in a region (aside from minor costs like VPC creation) until it receives traffic.

All this proposal does is shuffle the costs of traffic to another region.

2 Likes
15

Even when the cost is shuffled and no net reduction exists for PNI, the rest of the benefits will hold:

  • Reduction of costs for node runners, both in infra and human resources (I know that this is not 500 u$d/month).
  • Reduction of latency, just by re-routing the traffic using either AWS or GCP.
  • Easier estimation of network cost.
  • (and others mentioned in first post)
1 Like
16

I don’t believe this is true. The Gateway system already uses AWS Global Accelerator which has hundreds of PoPs and keeps the traffic on the internal AWS network as soon as it reaches one of those points.

Your example in your second post is incorrect. Because the user near AP-NE-2 has to travel with their request over the public internet to AP-SE-1, nothing is gained. You’ve changed what part of the request travels over the public internet, but there is still the same amount of public internet being used.

There is no logical way that reducing the number of Gateways will reduce end-user latency. I suspect we will see this effect in action as the other Gateways are shut down.

1 Like
18

I understand that once in AWS the traffic between regions will move through the AWS Global Backbone and it is not the same as public internet (and the reason why pings are lower in the data matrix). The amount of public internet being used is different, the example is quite clear about that.

1 Like
19

@RawthiL would you mind sharing your analysis (assuming you have done so) of how eg 20 vs 3 would play out in terms of QoS in v1, where things like setting location are enabled?

1 Like
20

I’m not sure exactly what you mean by “in terms of V1”.
If you say want to know how would QoS be in a network were the relays are only served by local node, then I cannot tell.
Today it would be really difficult to know where are nodes located and in which geozones would they be staked in V1. In a perfect world, only local nodes would be staked in each geozone, resulting in better QoS, but this cannot be enforced and non-local nodes could also be staked in any geozone. This will reduce the QoS to values that are at the limits of the MinimumTestScoreThreshold.
Also, I think that V1 wont have 20 geozones, so the routing problem presented here will be present within each geozone in V1, as more than one gateway will exist in each geozone.

1 Like
21

Hey @ArtSabintsev @AlexF

Any update regarding this discussion?

1 Like