CLOSED: PoktNest - Reliable POKT Blockchain Indexer/Explorer auxillary

Build Quick Grants
1

Socket Overview

This socket is initiated by cryptonode.tools to construct and improve, deploy, and maintain a blockchain indexer and explorer for the POKT network. With the recent decline in reliable indexers, there’s an urgent need for multiple reliable explorers to be available and accurate at all times.* While the primary and only remaining active explorer is a great product, there have been instances of inconsistencies in presented data. When that happens, even if rare, it sparks fear among clients and users about the health of the ecosystem and the status of their assets. Furthermore, every service in a decentralized ecosystem needs resilience, and many users prefer to have an alternative as a sanity check even when the main option is perfect.

Key Highlights and Challenges:

  1. Complexity: Building and maintaining an explorer isn’t just about coding and open sourcing a repo for others to download or deploy, which takes it’s own time and effort. It’s also about ensuring uptime for multiple services, handling many concurrent requests during indexing (unless you want it to take weeks or months), responsive for frontend users (frontend to backend to db latency and concurrency), as well as verifying data accuracy. The infrastructure cost alone might not be insignificant, and multiple components to the stack to consider (a responsive website, backend api, multiple replicated database measured in terabytes, likely a caching layer, healthy archival nodes to sync from, monitoring and logging, etc). I’m still an indie node runner and don’t outsource anything, and each component requires attention, energy, time to do well.
  2. Stack & Architecture: My solution will utilize nestjs for the backend indexer scripts and logging etc, possibly build an api as well if not postgrest on top of the postgres db, and data using pokt rpc api from my own pokt nodes.** My plan is to initially replicate pokt.watch svelte frontend using nextjs and mui as part of my cryptonode.tools website, with the backend built in nestjs/typescript with improvements for logging, error handling, and data verification using the simple poktwatch python indexer as a reference.***
  3. Self-hosted Backend API: Hosting full archival chains and multiple large databases for indexing is expensive and infeasible with cloud hosting if I hope to maintain it long term cost effectively, and so multi region logical replication on bare metal is going to be fun
  4. Selective Proprietariness: While I believe in community and collaboration, I am still just an individual; certain components re deployment and the frontend website, especially those critical for performance, security, and multi-region deployment, will remain proprietary.

Type of Socket:

Initiative: $3K, recurring/open, reviewed monthly. The costs aren’t just about development; they also cover infrastructure, data hosting, maintenance, and continuous improvements.

Commitment to Transparency:

I commit to regular updates and will self-report my progress monthly. The backend NestJS indexer service will be open source.

Link to Work:

PoktNest

Wallet:

5f2a1121cd54c25f2622b5e936e9e0159c6a48d5


* Thunderhead shut down support of pokt.watch but left the site running so some people still used it despite missing transactions and bad values, some might not know for a while. PNI/grove also scuttled the main pokt.explorer to save costs and release responsibility, leaving the ecosystem vulnerable in the meantime.
** pypokt is broken and just a dependency trap and feels like a waste of time to fix imo, is just a wrapper around the pokt rpc api anyways. Same with pocketjs, though it’s mostly usable, it still has bugs left in it forever, I know people know about it and just work around it. I started building my indexer via the pokt rpc api instead of using pocketjs or pypokt for this reason, and directly with my own nodes as using portal rpc exceeds free tier quickly, aside from other challenges using it for indexing
*** You may be familiar with public pokt watch indexer repos from th and some forks, thinking I can just copy pasta that. Neither are usable outright, like some other repos in poktland, the public stuff gets left broken or buggy and not ready for production, which slows down anyone trying to use it, and stuff like that is a HUGE time sink for someone like me, but instead for me serve as a great tool to use as a reference to build my own version, allowing me to make improvements along the way.

8 Likes
2

Thank you for a detailed socket specification and for stepping up to reinforce our ecosystem’s data. This socket is now opened.

2 Likes
3

Just a quick update to let people know the explorer for mainnet is live, and where to find it in case they see my “under construction” landing page.

Mainnet explorer can be found at POKT Mainnet Explorer

You should be able to search by address/txhash/block height in the search bar on that page, and follow links in the blocks/transactions tables to find more details about a transaction/block/address. If you have issues right away it may be a caching issue, as I’ve been making small changes frequently, and I’m working to iron out ci/cd pipeline issues in different regions. Please try refreshing or clearing your cache if you do encounter any errors. I do plan to continue adding more information to each page and improving aesthetics, but I’m open to feedback or suggestions and happy to offer support, either here or in my discord: Crypto Node Tools

1 Like
4

Thanks Ian for the update and presenting the explorer live.
However I wonder if it is OK that the explorer funded by the DAO has a direct access to stake nodes in Crypto Nodes?

imagen
imagen724×561 54.4 KB

1 Like
5

POKTScan is receiving 5x the DAO funds per month from PNF for closed-source development which also links/promotes their own paid services.

While some teams are required to compete for funds access and develop openly, others are given uncontested access to funds with no requirements to be transparent. POKTScan is in the latter, so I wouldn’t be so critical until a standard is established.

2 Likes
6

Please tell me how and where.

1 Like
7

Doesn’t POKTScan charge companies for levels of access to your API? I was considering that a paid service.

1 Like
8

@RawthiL My point is only that with so much variation between funded initiatives, we shouldn’t push to set a regulation on one type that isn’t applied across the board.

2 Likes
9

They don’t have direct access, you cannot stake from there without reaching out to me either through the contact form or via discord/telegram to request that I attach nodes to your account. Also, the explorer is usable without logging in, but to access anything under the staking portion you’d need to log in.

1 Like
10

Hey @iannn -

Reminder to get an update in by Jan 1 to keep this socket open.

1 Like
11

Hey all, sorry for the slow month around the holidays, I had hoped to accomplish more but this is where I’m at:

  • Added support for morse testnet, which can be found at cryptonode.tools/pocket_testnet/explorer
  • cryptonode.tools/pocket/mainnet_explorer and all subpaths have been moved to …/pocket/explorer; Since links have changed, redirects have been configured so the original url path should still work for old bookmarks and such
  • separated development from production deployment, to facilitate troubleshooting bugs and reducing caching issues when making frequent changes as I fiddle with things
  • Since nestjs is is a node.js app, and since node.js is single threaded, I added bull queue with redis to faciliate multithreading for the indexer. This kind of complicated things as I’ve been trying to troubleshoot a bug with some blocks not getting transactions added properly, partially why I’ve been slow to update. Trying to fix this now, and get on to making progress.

I’ll try to post another update within a week or two.

1 Like
12

This socket remains open for January.

Appreciate the update for the month. Is there anything you need support with that we or another contributor could help with?

And you’re usually in attendance but wanted to be complete with all Sockets recipients - I’m asking all open Sockets to join us next Thursday for the Builders All-hands. I’d like to encourage people getting paid via the DAO to know what’s going on with the protocol, and have a forum to ask questions and cross pollinate. Matteo will run the first half w/ developers & protocol, and then I’d take the second half to review sockets work. The ultimate goal here being more cross-pollination and collaboration, so looking forward to you showing up and sharing on Thursday!

1 Like
13

Wonder if its worth getting this explorer under its own domain, separate from the rest of the cryptonode.tools site - purchasing a domain and having this stack you’ve built run there seems like a better approach. As this is a DAO funded socket maybe the DAO can buy the domain? Just a thought

2 Likes
14

I think would make much more sense.

2 Likes
15

Hey all, I wanted to share an update on the indexer project. It’s been a journey and I’ve been tackling some interesting challenges.

I had a bit of trouble with transactions missing from the database during batch updates failing occasionally. To tackle this, I implemented a ‘execute and verify’ approach. This extra step acts as a quality check, making sure each block and transaction finds its home in the database correctly, and compares the count of entities intended for insertion with the actual saved count, raising an error if a discrepancy is found. Even then I still ran into an issue with batch inserts being too large for Postgres, the necessity to keep batch inserts within the constraints imposed by PostgreSQL led me to dynamically calculate the number of parameters in the transaction entity, to inform the chunk size during batch processing, ensuring not to exceed the 65535 parameters limit of PostgreSQL. Despite the benefits, I was still dealing with performance degradation on the primary database server due to the heavy inserts and extra reads for verification.

To alleviate the load on our master server and improve query performance of the api, I introduced read-only replicas and connection pooling into our architecture as well as built separate containers for the indexer and the api, so that api requests weren’t getting stuck behind indexer processing. With nestjs it’s easy enough to enable just the modules you need for building different service images.
For connection pooling I considered pgbouncer vs pgpool and landed on the later thanks to its promising load balancing features and watchdog capabilities, though full-fledged implementation with failover functionality is still in progress. Adding replicas though significantly improved read operations; however, I’ve still encountered challenges related to transaction log (WAL) segment retention, which caused replicas to occasionally lose sync with the master, leading to out of date data presented occasionally on the frontend.

I’ve also been exploring ways to ensure that new data to be indexed doesn’t get stuck behind older, slower-moving tasks (such as inserting or reverifying older blocks). It’s doable to set varying priorities when adding to queues with bull, but still requires extra logic to dance around for it to work well and identify what to prioritize. Alternatively, there are tools such as kafka for real time message processing and data streaming which may be better suited than trying to figure out how to jimmy the queues and still sync everything quickly. After researching and comparing kafka and alternatives, I’ve been considering Apache Pulsar, since it’s supposed to be simpler to scale horizontally and supposedly more performant than kafka.

All of this has also gotten me thinking about the possibility of direct indexing from the pokt nodes themselves, which could be a great way to offload most of the work from the external indexer/processor. Many other cosmos chains have the ability to specify a postgres db to use instead of indexing to the leveldb kv store. I’d have to learn golang and get my hands dirty at the protocl level, but it seems to make more sense than trying to sync an index externally when the node client is already doing it.

I realize also, thanks to my own critical voice, that sometimes it may seem like I’m moving at a snail’s pace, but rest assured, every step forward is through complex and dense technical terrain. On another note, early as I started trying to tackle this I encountered a peculiar case of what looked like missing data within blocks 69500 to 70000, all stake_validator transactions with null values for the msg under the stdTx field. I made many different efforts to try to resync nodes, try different snapshots, different node versions, etc. I wrote a script to identify all the transactions and their indexes and types with the missing message values. Despite my best efforts to reconcile this across every seed and peer available, the missing data and understanding it remained elusive. After discussing with several people, Toshi from C0d3r was able to shed some light on the matter: It appears this anomaly was a side-effect of the NCUST chainhalt. Specifically, it emerged as a bug associated with the rollback from that event, since originally, NCUST was enabled on 69504, but after the rollback many of these blocks became reclassified as pre-NCUST. The crux of the issue was a decoding failure for certain transactions, despite the transaction existing as encoded data, its decoded counterpart, stdTx, resulted in null upon retrieval. This deep dive into the data was not only a fascinating detective story but also an invaluable learning experience that highlights the intricacies of blockchain data management.

Here’s what I could be focusing on in the immediate future:

Fully harness pgpool’s capabilities, ensuring load balancing and standby replication are finely tuned.
Continue to monitor system performance to tailor the PostgreSQL configuration for optimal replication.
Innovate the queueing system with a more nuanced priority setup.
Add a priority queue for mempool transactions.
Add api endpoints for queries the community needs access to, open to suggestions.
Trial Apache Pulsar to evaluate its fit for our needs.
Further consider the concept of direct indexing from the core client in golang.

I have hardly been focused on the frontend work recently, but since there have been a few suggestions to isolate a separate poktnest frontned, I’m not completely opposed to eventually separating it into it’s own separate site instead of remaining at cryptonode.tools, it’s just not been my priority as I wade deeper into complexity on the backend. But it would be helpful to have some direction in what to prioritize.

I’m eager to hear feedback from the community and to collaborate more as I continue to make progress and continue to improve both my skills and the poktnest app.

3 Likes
16

Hey @iannn - we’ve DMd about moving this explorer to focus on Shannon, and I’m just going to drop a TLDR here.

This socket has been successful in creating an explorer that’s useful as a backup to POKTscan, and we’d like to keep using it until Shannon.

Some of the problems you’re tackling here seem extremely complex, and with the Cosmos ecosystem having SDKs and other tools it feels like a natural transition to focus efforts on building for Shannon instead of focusing on problems with diminishing returns.

With that, we’d like to close down this socket and understand if there’s any costs to maintaining the current version of POKTnest, and work with you to open a new socket for Shannon (if you’re interested).

We appreciate all the work you’ve done here!

1 Like