Skip to content

Nightly snippet run failed #179

Description

@github-actions

The nightly snippet run failed on c1cb04dfa41e8dbca73d0069ffc5a94070936d92.

Run log: https://github.com/allora-network/docs/actions/runs/31239309367

Snippet run report

  • snippets discovered: 15
  • executed: 13
  • failed: 4

snippets/forecaster.py

exited with code 1

Traceback (most recent call last):
  File "/tmp/allora-snippets-CCJhnn/runs/forecaster.py/forecaster.py", line 38, in <module>
    asyncio.run(main())
  File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/runs/forecaster.py/forecaster.py", line 31, in main
    async for result in worker.run():
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/worker/worker.py", line 498, in run
    await self._ensure_initialized()
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/worker/worker.py", line 332, in _ensure_initialized
    self._chain_id = await self.client.raise_for_chain_id_mismatch()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/rpc_client/client.py", line 211, in raise_for_chain_id_mismatch
    chain_id = await self.chain_id()
               ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/rpc_client/client.py", line 218, in chain_id
    node_info_resp = await self.tendermint.query.get_node_info(tendermint_v1beta1.GetNodeInfoRequest())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/rpc_client/protos/cosmos/base/tendermint/v1beta1/__init__.py", line 619, in get_node_info
    return await self._unary_unary(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/betterproto2/grpc/grpclib_client.py", line 69, in _unary_unary
    response = await stream.recv_message()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/grpclib/client.py", line 426, in recv_message
    await self.recv_initial_metadata()
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/grpclib/client.py", line 394, in recv_initial_metadata
    self._raise_for_grpc_status(status, message, details)
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/grpclib/client.py", line 346, in _raise_for_grpc_status
    raise GRPCError(status, message, details)
grpclib.exceptions.GRPCError: (<Status.UNAVAILABLE: 14>, 'upstream connect error or disconnect/reset before headers. reset reason: connection termination', None)

snippets/migrate_worker.py

printed an error it swallowed (/Inference worker error:/)

_    _     _     ___  ____      _
      / \  | |   | |   / _ \|  _ \    / \
     / _ \ | |   | |  | | | | |_) |  / _ \
    / ___ \| |___| |__| |_| |  _ <  / ___ \        Chain:   allora-testnet-1
   /_/   \_\_____|_____\___/|_| \_\/_/   \_\       Topic:   PLAYGROUND: 1 day BTC/USD Price Prediction (ID: 69)
   __        _____  ____  _  _______ ____          Address: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe
   \ \      / / _ \|  _ \| |/ / ____|  _ \         Role:    INFERER
    \ \ /\ / / | | | |_) | ' /|  _| | |_) |
     \ V  V /| |_| |  _ <| . \| |___|  _ <
      \_/\_/  \___/|_| \_\_|\_\_____|_| \_\

2026-08-08 04:23:58,986 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489480469174 ALLO
2026-08-08 04:23:59,011 INF 🔄 Starting polling worker
2026-08-08 04:23:59,035 INF    Topic 69: unfulfilled nonces: {10418255}
2026-08-08 04:23:59,036 INF    Our unfulfilled nonces: {10418255}
2026-08-08 04:23:59,049 INF 👉 Found new nonce 10418255 for topic 69, submitting... account_seq=40
2026-08-08 04:23:59,073 WRN ⚠️⚠️⚠️  SANITY CHECK WARNING: Your prediction (123.450000) is 328.4 standard deviations from the network consensus (mean: 64332.131497, std: 195.490681). Please verify you're predicting the correct target variable and using the right units.
2026-08-08 04:24:09,251 INF ⚠️ Already submitted for this epoch: topic_id=69 nonce=10418255 code=1
2026-08-08 04:24:09,275 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489466969174 ALLO
Inference worker error: TxError: codespace=undefined code=1 tx_hash=A5C12FEE328A53D97D064CD5F59AF132FC6E733A4EC4648E32FE9EB2FD3DB9BB failed to execute message; message index: 0: Error planning inference admission: inference already submitted
2026-08-08 04:24:09,286 INF Received signal 15, initiating graceful shutdown

snippets/sdk_py_worker.py

printed an error it swallowed (/Inference worker error:/)

_    _     _     ___  ____      _
      / \  | |   | |   / _ \|  _ \    / \
     / _ \ | |   | |  | | | | |_) |  / _ \
    / ___ \| |___| |__| |_| |  _ <  / ___ \        Chain:   allora-testnet-1
   /_/   \_\_____|_____\___/|_| \_\/_/   \_\       Topic:   PLAYGROUND: 1 day BTC/USD Price Prediction (ID: 69)
   __        _____  ____  _  _______ ____          Address: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe
   \ \      / / _ \|  _ \| |/ / ____|  _ \         Role:    INFERER
    \ \ /\ / / | | | |_) | ' /|  _| | |_) |
     \ V  V /| |_| |  _ <| . \| |___|  _ <
      \_/\_/  \___/|_| \_\_|\_\_____|_| \_\

2026-08-08 04:27:13,473 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489453469164 ALLO
2026-08-08 04:27:13,499 INF 🔄 Starting polling worker
2026-08-08 04:27:13,524 INF    Topic 69: unfulfilled nonces: {10418309}
2026-08-08 04:27:13,524 INF    Our unfulfilled nonces: {10418309}
2026-08-08 04:27:13,549 INF 👉 Found new nonce 10418309 for topic 69, submitting... account_seq=42
2026-08-08 04:27:15,691 INF ⚠️ Already submitted for this epoch: topic_id=69 nonce=10418309 code=1
2026-08-08 04:27:15,714 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489439969164 ALLO
Inference worker error: TxError: codespace=undefined code=1 tx_hash=F75780859D3CBBCC37B364E201BA156A8ED31CDD5047298EBFF7B47E2FE67629 failed to execute message; message index: 0: Error planning inference admission: inference already submitted
2026-08-08 04:27:15,726 INF Received signal 15, initiating graceful shutdown

snippets/worker_minimal.py

printed an error it swallowed (/Inference worker error:/)

_    _     _     ___  ____      _
      / \  | |   | |   / _ \|  _ \    / \
     / _ \ | |   | |  | | | | |_) |  / _ \
    / ___ \| |___| |__| |_| |  _ <  / ___ \        Chain:   allora-testnet-1
   /_/   \_\_____|_____\___/|_| \_\/_/   \_\       Topic:   PLAYGROUND: 1 day BTC/USD Price Prediction (ID: 69)
   __        _____  ____  _  _______ ____          Address: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe
   \ \      / / _ \|  _ \| |/ / ____|  _ \         Role:    INFERER
    \ \ /\ / / | | | |_) | ' /|  _| | |_) |
     \ V  V /| |_| |  _ <| . \| |___|  _ <
      \_/\_/  \___/|_| \_\_|\_\_____|_| \_\

2026-08-08 04:27:20,913 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489439969164 ALLO
2026-08-08 04:27:20,942 INF 🔄 Starting polling worker
2026-08-08 04:27:20,968 INF    Topic 69: unfulfilled nonces: {10418309}
2026-08-08 04:27:20,968 INF    Our unfulfilled nonces: {10418309}
2026-08-08 04:27:20,994 INF 👉 Found new nonce 10418309 for topic 69, submitting... account_seq=43
2026-08-08 04:27:39,246 INF ⚠️ Already submitted for this epoch: topic_id=69 nonce=10418309 code=1
2026-08-08 04:27:39,270 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489426469164 ALLO
Inference worker error: TxError: codespace=undefined code=1 tx_hash=25701F2DCC58D983A48C97DB0A85A967445232DC2C0963AF309A81336F98AC27 failed to execute message; message index: 0: Error planning inference admission: inference already submitted
2026-08-08 04:27:39,283 INF Received signal 15, initiating graceful shutdown

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    snippets-nightlyOpened by the nightly snippet run

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions