Ethereumetl tool fails when using eth-archival node endpoint

Help
1

Choose one of the following tags in the “optional tags” field to ensure that your request is shared to the appropriate Discord channel:

  • node-support
  • app-support

Description of your issue/situation

Running ethereumetl export_all ... or ethereumetl export_token_transfers ... leads to error being thrown when running against an eth-archival node. There is no issue when running the same ethereumetl commands against other providers like an Infura node for example.

Based on the error message ( ‘the method is currently not implemented: eth_newFilter’) and my cursory understanding of eth nodes, I believe the error is thrown because ethereumetl is trying to call a method that the pokt node does not expose.

Step by Step

  1. First, install the ethereumetl tool by running:
pip3 install ethereum-etl
  1. Run the following command:
ethereumetl export_token_transfers \
        --start-block 13858927 \
        --end-block 13858933 \
        --provider-uri <POKT_ETH_ARCHIVAL_NODE_ENDPOINT> \
        --output token_transfers.csv

This results in the following error:

...
ValueError: {'code': -32000, 'message': 'the method is currently not implemented: eth_newFilter'}

Setup/System Details

OS: MacOS 12.1
Memory: 32 GB
CPU: Apple M1 Pro

App version

N/A

Necessary files

N/A

Logs or Evidence

Full log output:

2022-01-13 20:44:38,188 - ProgressLogger [INFO] - Started work. Items to process: 7.
Traceback (most recent call last):
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/bin/ethereumetl", line 8, in <module>
    sys.exit(cli())
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/ethereumetl/cli/export_token_transfers.py", line 56, in export_token_transfers
    job.run()
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
    self._end()
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/ethereumetl/jobs/export_token_transfers_job.py", line 88, in _end
    self.batch_work_executor.shutdown()
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/ethereumetl/executors/batch_work_executor.py", line 97, in shutdown
    self.executor.shutdown()
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown
    self._check_completed_futures()
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/ethereumetl/executors/batch_work_executor.py", line 63, in _fail_safe_execute
    work_handler(batch)
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/ethereumetl/jobs/export_token_transfers_job.py", line 77, in _export_batch
    event_filter = self.web3.eth.filter(filter_params)
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/web3/eth.py", line 330, in filter
    _filter_id = self.web3.manager.request_blocking(
  File "/Users/xxxxx/Library/Caches/pypoetry/virtualenvs/ethereum-etl-TcpCK5yw-py3.8/lib/python3.8/site-packages/web3/manager.py", line 112, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32000, 'message': 'the method is currently not implemented: eth_newFilter'}
1 Like
2

The issue here is that the Pocket Network uses Erigon for it’s archival nodes, which doesn’t support the eth_newFilter method. The good news here is that ethereum-etl only relies on this method in 2 places, and it’s possible to accomplish this same goal without web3.eth.filter, instead using web3.eth.get_logs, and then using a handler to process those logs. The difference here is that instead of the node filtering the logs for you, you’ll be filtering the logs client side.

I’m going to take a stab at implementing that, and then sending a pull request over to ethereum-etl, that way this same behavior can be accomplished through Etheruem clients that don’t support eth_newFilter. When I do that, I’ll update you with how you can install those changes yourself, and how to know when they’re included in the main version of ethereum-etl.

Thanks again for opening the issue over here!

1 Like
3

I’ve opened a PR on etheruem-etl to address this issue here.

If you don’t want to wait for that PR to be accepted, you can clone down my fork, and then I’d recommend installing it in a virtual environment to use it.

A quick primer on virtual environments if that sounds new to you.

$ git clone https://github.com/blockjoe/ethereum-etl
$ cd etheruem-etl
$ python3 -m venv etl-env
$ source etl-env/bin/activate etl-env 
$ pip3 install .

From there, feel free to use ethereumetl as you were. And to leave the enviornment simply run:

$ deactivate

Then if you want to use this version of ethereumetl until it gets addressed in the main version, you can do the following whenever you need it:

$ cd etheruem-etl
$ source etl-env/bin/activate

and you’re ready to roll.

1 Like
4

Awesome, thank you very much @blockjoe , really appreciate it!

1 Like