ALLOCATED PRIORITY: Mainnet Snapshots

Build RFPs
1

Priority Outline

Currently, snapshots are taken manually every now and then. Before PNI’s AWS to GCP migration, they used a script to automate the upload of data directory snapshots: GitHub - pokt-foundation/pocket-snapshotter.

As part of our continued decentralisation, we want to support and reward a community contributor to maintain these snapshots.

This Pocket Open Priority (POP) is calling for a team or individual to take on the creation and maintenance of regular mainnet snapshots for Pocket Network protocol.

This is a truly open priority and so submissions should specify the price in POKT for which the service will be provided.

Submissions should express their relevance to the Priority details and competency profile below:

Priority Details

Purpose:

Community maintenance of regular mainnet snapshots

Responsibilities:

  • Upload snapshots of the data directory to the storage account on a weekly basis… Storj is currently used for snapshot storage. Alternative solutions might be considered at the discretion of the protocol team
  • Be available to create on-demand snapshots within a 48 hour notice, when they are vital for the Pocket Network (for example, before consensus-breaking releases). This is crucial as having additional backups close to the breaking point is of utmost importance
  • In addition to the data directory archive, providing a .torrent file is required to reduce bandwidth fees and speed up snapshot delivery when they are urgently needed. The CDN/Storj HTTP endpoint could function as a “Web Seed” or “HTTP Source” to assist with downloads.
  • Maintain ownership and responsibility that the snapshots are in good condition and uploaded regularly.
  • Take security very seriously. Managing snapshots is a significant task and presents a potential attack vector.

Competencies requested:

  • Skills: Pocket Network node running, Communication
  • Knowledge: Pocket Network protocol, Snapshots, Data directories
  • Attitude: Attention to detail, High accountability, Gets shit done

Submission details:

Open: Now

Submission deadline: Close of day 14th June 2023

Submission type: Any format that addresses the requirements and competencies, up to 300 words, and must include price to complete this work denominated in POKT

Decision provided: Close of day 16th June 2023

Price: Open pricing. Please include in submission.

Funding: This priority will be funded by PNF until such time as PEP60 is passed or an alternative arrangement is made or proposed.

Submissions should be posted as replies to this thread with Submission in the first line. Submissions can be edited at any point up until the deadline.

5 Likes
2

Submission

Nodefleet is able to support this proposal. Back in the days while I worked at POKT, we used this repo to keep the snapshots up and running every 24 hours with a dynamic generated link on the README.md for downloading mainnet and testnet snapshots in compressed/uncompressed format fashions on STORJ:

https://github.com/pokt-network/pocket-snapshots

We’ll commit ourselves as a team to provide Pocket Network infrastructure support and implementation on the snapshot systems for pocket mainnet and testnet nodes in order to help PNI offload responsibilities and move forward with decentralisation

Systems to be implemented and maintained:

  • Snapshot System for Pocket Mainnet
  • Snapshot System for Pocket Testnet

Implementation will include:

  • Public github repository with snapshot system
  • Weekly automated snapshots
  • Automatically generated STORJ URLs, torrents and HTTP links for downloading mainnet and testnet snapshots on the readme.md of the repo

The snapshots will be provided in the following format:

  • Compressed .tar.gz downloadable via:
    • STORJ cli
    • HTTP download
    • Torrent download
  • Uncompressed chain folder via:
    • STORJ CLI
    • Torrent download

Maintenance will include:

  • Security and software updates/upgrades
  • Monitoring and alarms system attached to discord with notifications when a new snapshot it’s created
  • Custom channel for any engineering support regarding snapshots and custom snapshots with the Pocket team

Funds requested:

  • $2,000 USD in a monthly basis in POKT for implementation, maintenance and server and bandwidth costs.

  • Implementation:

    • 2 weeks Max
4 Likes
3

Submission

Who we are

We have been active in the Pocket community since 2021, initially directly managing custodial nodes at scale, and then for ourselves. In addition to 1st-party nodes, we also provide managed services for other large node runners in the space, supplying high-level expertise and management for POKT’s most popular backend chains.

Implementation

We will develop a system using modern design principles that will:

  • Ensure a pocket nodes’ height is current,
  • Create a snapshot of the chain from that node,
  • Upload the artifact to Storj backend
  • Perform file sanity checks once in place
  • Create a .torrent file backed by the Storj artifact as an initial seed

In addition, we will produce a flexible harness to:

  • Schedule the job to run weekly
  • Report success/failure to communication platform (Discord to start with)
  • Expose a credentialled API to allow on-demand snapshots to be taken by authorized users in the event of emergent situations

Code will be open sourced via github with a permissive license upon completion. We estimate no more than 2 weeks from commencement of work.

Deployment

Due to the security sensitivity of snapshots, we believe a small dedicated server is required. We will procure a dedicated bare-metal instance, run the official pocket client on it, deploy our snapshot service, and integrate it into our existing monitoring platform, ensuring both system and application uptime and chain sync status 24x7.

Management

Barg Systems will take on responsibility for the ongoing management and maintenance of both hardware and software components, and ultimately correctness and recency of the actual snapshot artifacts, and will provide reasonable support around remediating that correctness on-demand. We are active members of the node running community and will always be available for emergency snapshots if required.

Price
We believe this is a well-scoped project that can be developed for a fixed fee, and then an ongoing management component that will include infrastructure costs. We believe in the future of the token and will accept payment in POKT directly. If the price drops below $0.01 we would ask for the figure to be re-calculated (based on hard costs).

Initial development - 69,250 POKT
Monthly Management (including infrastructure) - 27,700 POKT

4 Likes
4

Submission

C0D3R would like to offer its services to keep the snapshots up-to-date.

Why C0D3R:

  • Trusted: C0D3R is the largest (per node count) and oldest (since day 1 of mainnet) node providers.
  • Know-How: We already have an internal mechanism to create our own snapshots (we long ago gave up on public snapshots)
  • Reliability: Our systems are highly automated and closely monitored, increasing availability and quality of the snapshots.
  • Cutting edge tech: We can offer pruned snapshots that saves cost and time by a factor of 35x!

Our offer:

  • Using the preferred storage service of PNI/PNF
    • If STORJ, snapshots will be accessible with http, storj cli, and torrent.
  • Twice-a-week snapshots for
    • Full data folder (mainnet and testnet)
    • 700GB compressed
  • Daily snapshots for
    • Pruned data folder (mainnet)
    • Only 20GB. Great for all intents and purposes except for an archival Pocket Node (e.g. serving in chain 0001). C0D3R nodes, with official builds have been running like this for many months now.
  • On demand snapshots during upgrade times per PNI guidance.
  • Discord channel for questions/concerns/comments.

Timelines:

  • Operational within 7 days of offer acceptance.

Funds requested:

  • 100,000 POKT for initial development and the first month.
  • 50,000 POKT per month for continued operational efforts, community support and quality assurance.
  • Above prices are assuming PNI/PNF is paying for actual storage fees.

Please note:

  • What makes this offer special is the inclusion of pruned snapshots that will allow much smaller nodes to operate. It will also reduce the bandwidth costs and download times by a factor of 35x!!!
5 Likes
5

Liquify is ready to step in for the proposal. As a reliable supporter of Pocket for over 2 years and counting, one of the biggest validators on the network, having a vast experience of infrastructure management and being a fully self owned bare metal infra company allows us to be confident in meeting all of the requirements and providing an exceptional service.

Services proposal:

  • Snapshot for both testnet and mainnet on a dedicated server with options to choose a region which best suits (UK, Germany, US E, US W, Japan);
  • HTTP download, .torrent download, compressed and un-compressed;
  • Weekly snapshot updates;
  • Be available to set up on-demand snapshots within a 48 hour notice;
  • Dedicated communication channel for tech support with PNF/PNI (slack, discord, telegram).

All of the mentioned services will be included in our SLA, along with a 99.95% up-time guarantee

Time to implement: upto 2 weeks (can be prioritised to 7 working days)

Pricing is the following:
Set up: $1500 in POKT one-time fee
Maintenance: $750 monthly in POKT monthly fee

5 Likes
6

Cryptonode.tools happy to provide this service if needed, and I’m happy to let the DAO decide what price they’d like to pay if they choose me to do it. I already run many mainnet pocket nodes and maintain my own regular snapshots, as any node runner should even with a small number of nodes. I’d be happy to have a reason to run a few testnet nodes and provide snapshots for that as well.

My current snapshot process already uses a variation of the pokt snapshotter script, so it would be arbitrary for me to add the uplink to storj if the dao wants to provide me an access token to upload to their storj account, I can upload a regular snapshot with no inconvenience and no extra cost to my infra. I currently upload to my own dedicated storage box, and take regular zfs snapshots of it so I can go back if necessary.

Bandwidth is the main cost for storing a snapshot and making it available to others to download. Storj bandwidth isn’t particularly cheap, but I think it’s great to utilize blockchain enabled use cases to keep innovation moving forward in that direction, and it’s a good backup to have for replication, so I think the DAO should maintain an account with a reasonable bandwidth limit so it can’t be gamed to infinity, and if they provide me a token with write access to upload, I can do that immediately and even daily with no real extra cost to me.

To provide http or .torrent access on a limited basis to trustable dao members is also relatively arbitrary, and I’m already happy to do so for friends within the ecosystem. The concern is making it public, where it might be gamed or overutilized by noobs, or a bug in some automation tool used by lots of nodes might go haywire and inadvertently ddos or drive up a huge bill for bandwidth.

As a DAO we should wonder, who are these snapshots going to be used for? New node runners? Well, we’re about to kill most of the node runners in the space when we drop inflation from 12% to 5%, as is being discussed currently, and so I can’t imagine we need to prioritize a large ask to get snapshots into every region, to be able to download unlimited snapshots super fast, as if we’re going to be needing it for new node runners onboarding quickly… Like, no one needs that. But happy to do http and .torrent hosting on 1 or 2 dedicated servers, incurring a real cost for infra in the low 3 figures to host on bare metal in EU with bare metal providers who are gracious on bandwidth.

Happy to serve whatever the DAO decides is necessary, just cover my costs then pay me what you want for labor, in any case I can implement whatever fairly quickly. I do wonder how a bunch of other projects are able to get half a dozen plus community snapshot volunteers without requiring hefty payouts, and I think this is something more than one of us could collaborate to maintain, but I digress. That said, to survive the incoming burn and churn as a small provider, only way I’ll break even let alone eat is through grants like these.

Thanks for the consideration.

2 Likes
7

DECISION


Thanks to each of the teams submitting to this first Open Priority. It’s great to see how the community activates to support network health and we appreciate you all for creating the unenviable task of PNF having to select one team/submission, when in reality each submission is highly qualified and worthy of acceptance to this work.

Our primary consideration when selecting from submissions was to:

  • Maximise impact/value to the network (i.e. Value created - Cost)
  • Propagate network health (via reputation/performance, engagement in the network and demonstration of Pocket DNA by submitters)

The team selected when weighing these variables is @Andy-Liquify.


Some specifics behind our decision include Liquify being the most experienced validator and their submission providing regional snapshots which should result in faster downloads and being more inclusive for globalised node operations. They’ve also been quite active in Discord which indicates responsiveness to needs/emergencies and their submission (while in USD) was also the cheapest.

We’ve reached out to some of the submitters around other opportunities to contribute and will be releasing other POPs soon.

As this was the first POP some learnings we will take forward are:

  • We believe the mechanism works well and plan to scale up POPs going forward. If anyone has feedback to provide on this process please share with me directly
  • Time from submission-close to decision announcement is very tight by design, but we took extra time to make sure the protocol team were aligned with and supportive of our reasoning. Going forward we will keep this time tight but add a little more buffer to be considerate of other stakeholders timelines (i.e. protocol team)
  • We showed leniency on submissions priced in USD rather than POKT. Going forward, when asking for submissions denominated in POKT we will make decisions only on submissions denominated in POKT
5 Likes
8

Thanks @b3n and PNF for the vote of confidence, myself and the whole liquify team are really looking forward to providing the community with fast and relient snapshots!

5 Likes
10

We are moving the tactical details of implementing this discussion to Discord. If you believe you should be involved, please let me know.

2 Likes
11

The snapshot service is now live at:

https://pocket-snapshot.liquify.com/#/

Full instructions can be found here:

5 Likes