Mainnet not syncing

[blockpi019]

Hi @ClaytonNorthey92 , I tried doing a fresh sync from scratch following the NODE_RUNNING.md tutorial. Although the tbc data synced successfully, the op-geth data hasn't been syncing at all. Here are the op-geth logs:

op-geth-l2-1  | INFO [07-08|09:47:44.890] ForkchoiceUpdated command requests we set head to 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b @ 0
op-geth-l2-1  | INFO [07-08|09:47:44.890] forkchoiceUpdated canonical block @ 'safe' height: 0: 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b
op-geth-l2-1  | INFO [07-08|09:47:44.890] forchoiceUpdated update says canonical should be: 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b @ 0
op-geth-l2-1  | INFO [07-08|09:48:05.600] Forkchoice update does not contain any transactions
op-geth-l2-1  | INFO [07-08|09:48:05.600] "Engine API forkchoice updated, head=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b, finalized=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b, safe=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b"
op-geth-l2-1  | INFO [07-08|09:48:05.600] ForkchoiceUpdated command requests we set head to 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b @ 0
op-geth-l2-1  | INFO [07-08|09:48:05.600] forkchoiceUpdated canonical block @ 'safe' height: 0: 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b
op-geth-l2-1  | INFO [07-08|09:48:05.600] forchoiceUpdated update says canonical should be: 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b @ 0
op-geth-l2-1  | INFO [07-08|09:48:27.887] Forkchoice update does not contain any transactions
op-geth-l2-1  | INFO [07-08|09:48:27.887] "Engine API forkchoice updated, head=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b, finalized=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b, safe=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b"
op-geth-l2-1  | INFO [07-08|09:48:27.887] ForkchoiceUpdated command requests we set head to 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b @ 0
op-geth-l2-1  | INFO [07-08|09:48:27.887] forkchoiceUpdated canonical block @ 'safe' height: 0: 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b
op-geth-l2-1  | INFO [07-08|09:48:27.887] forchoiceUpdated update says canonical should be: 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b @ 0
op-geth-l2-1  | INFO [07-08|09:48:48.933] Forkchoice update does not contain any transactions
op-geth-l2-1  | INFO [07-08|09:48:48.933] "Engine API forkchoice updated, head=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b, finalized=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b, safe=0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b"
op-geth-l2-1  | INFO [07-08|09:48:48.933] ForkchoiceUpdated command requests we set head to 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b @ 0
op-geth-l2-1  | INFO [07-08|09:48:48.933] forkchoiceUpdated canonical block @ 'safe' height: 0: 0x6b070d8b8f3bf81c9314ff863748e3db2662ecc0a8dbd4c394a8b646ae0c8b8b

[blockpi019]

cat docker-compose.yml

name: "hemi-stack"

networks:
  hemi:
    driver: bridge

services:

  # Bitcoin Secure Sequencer
  bssd:
    image: hemilabs/bssd:1.6.3@sha256:b6782c25b3245f1cede1d7b75bbea4d9fe36f2dc4fa708a9fae03b1f34ee7126
    profiles: ["hemi", "hemi-min", "full"]
    environment:
      BSS_BFG_URL: "ws://bfgd:8080/v1/ws/private"
      BSS_LOG_LEVEL: "INFO"
      BSS_ADDRESS: ":8081"
    ports:
      - '8081:8081'
    networks:
      - "hemi"

  # Hemi L2 node
  op-geth-l2:
    image: ghcr.io/hemilabs/op-geth:05d4d8f@sha256:92d14f0b4b89947352d57743ce9e7f2d9c9e495f0bb861a24d21c49d391c7abf
    platform: linux/amd64
    # XXX does geth need to be fully synced
    profiles: ["hemi", "hemi-min", "full"]
    depends_on:
      op-geth-l2-init:
        condition: "service_completed_successfully"
      op-geth-l2-init-tbc:
        condition: "service_completed_successfully"
    healthcheck:
      test: ["CMD", "nc", "-vz", "0.0.0.0", "18546"]
      interval: 5s
      timeout: 5s
      retries: 999999
    volumes:
      - ./genesis.json:/tmp/genesis.json
      # change this token value if you expose this service's ports for
      # connection from other machines
      - ./jwt.hex:/tmp/jwt/jwt.hex:ro
      - ./entrypoint.sh:/tmp/entrypoint.sh
      - /node/full/hemi/data/op-geth:/tmp/datadir
      - ./rollup.json:/tmp/rollup.json
      - ./l2-config.toml:/tmp/l2-config.toml
      - /node/full/hemi/data/tbc_data:/tbcdata
    environment:
      OP_GETH_L1_RPC: "${GETHL1ENDPOINT:-http://geth-l1:8545}"
      OP_GETH_L2_READINESS_RPC: "http://localhost:18546"
    ports:
      - '18546:18546'
      - '28546:28546'
      - '5555:5555'
    networks:
      - "hemi"
    entrypoint:
      - "sh"
      - "/tmp/entrypoint.sh"
    working_dir: "/tmp"
    ulimits:
      memlock: -1
    deploy:
      restart_policy:
        condition: "on-failure"

  # Init container for Hemi L2 node
  op-geth-l2-init:
    image: alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
    profiles: ["hemi", "hemi-min", "full"]
    volumes:
      - /node/full/hemi/data/op-geth:/tmp/datadir
    command:
      - "chown"
      - "-R"
      - "65532"  # geth user
      - "/tmp/datadir"

  # Init TBC container for Hemi L2 node
  op-geth-l2-init-tbc:
    image: alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
    profiles: ["hemi", "hemi-min", "full"]
    volumes:
      - /node/full/hemi/data/tbc_data:/tbcdata
    command:
      - "chown"
      - "-R"
      - "65532"  # geth user
      - "/tbcdata"

  # Hemi op-node
  op-node:
    image: ghcr.io/hemilabs/op-node:7c70d2d@sha256:7db35d5798615e572d80823a3238a8111b14a90aade876a34b6a2936b17cbd87
    platform: linux/amd64
    profiles: ["hemi", "hemi-min", "full"]
    depends_on:
      op-geth-l2:
        condition: "service_healthy"
    volumes:
      # change this token value if you expose this service's ports for
      # connection from other machines
      - ./jwt.hex:/tmp/jwt/jwt.hex
      - ./op-node-priv-key.txt:/tmp/op-node-priv-key.txt
      - ./rollup.json:/tmp/rollup.json
    environment:
      OP_NODE_BSS_WS: "http://bssd:8081/v1/ws"
    ports:
      - '8547:8547'
    networks:
      - "hemi"
    command:
      - "op-node"
      - "--l1=${GETHL1ENDPOINT:-http://geth-l1:8545}"
      - "--l1.rpckind=any"
      - "--l1.beacon=${PRYSMENDPOINT:-http://prysm:3500}"
      - "--l2=ws://op-geth-l2:8551"
      - "--l2.jwt-secret=/tmp/jwt/jwt.hex"
      - "--sequencer.enabled=false"
      - "--rollup.config=/tmp/rollup.json"
      - "--override.ecotone=1725868497"
      - "--override.canyon=1725868497"
      - "--override.delta=1725868497"
      - "--rpc.addr=0.0.0.0"
      - "--rpc.port=8547"
      - "--p2p.bootnodes=enr:-J64QACnJ0giPmPXowNCHP-FDleMMbDqYg5nuLABYfJeYbP9OA6_fZtvCsTbAwvlPD8_C6ZSXEk1-XPabDBwD8UTyNCGAZMxdy4tgmlkgnY0gmlwhCINopiHb3BzdGFja4Tn0AIAiXNlY3AyNTZrMaED22EOmUryrNnFOwq6Ve1Hpw5CMFz_TnhIkVS4Sq8JhkmDdGNwgiMrg3VkcIKjLg,enr:-J64QA20VNMfLMtbSuHYmQH2I-xaqT4-9g7lqO-VHr0fFvtSB7QybJ_7W5fuEjLAa6kh82fGLjRUdZE0hW0pfVBcxP6GAZMxdwfngmlkgnY0gmlwhCJaFfaHb3BzdGFja4Tn0AIAiXNlY3AyNTZrMaEDjt3C-gv87B5rWa5M52WUFGy16mjQvFsC7OgPkqu-rn-DdGNwgiMrg3VkcIKRXQ,enr:-J64QLQyh3lXjQLzfCbfNw0DUb4A0OEcTLmVGexMbK-2jjCtHOIlRnDqLuedQ0orNHt7zmsILELYi4ozg-0bQzc34F-GAZMxdxNogmlkgnY0gmlwhCINq4uHb3BzdGFja4Tn0AIAiXNlY3AyNTZrMaED1NV9w0EmnMXBNRnDWj6kdqzE6_4HigHopeu3ewQTwouDdGNwgiMrg3VkcIK1Iw,enr:-J64QNPfOEViWkN7U_ul5Zhw_mc5_Hta92eUufrgu6oTqSMzRsqCje-P0vPrOQ9XjVIk4VP7mmFVP6qoTrtkIwvwUV2GAZMxdv4zgmlkgnY0gmlwhCJb2HmHb3BzdGFja4Tn0AIAiXNlY3AyNTZrMaEDYAidV8rzABKrKAL9jwP1aoi3wj-GtuML4FKtUDOgzCGDdGNwgiMrg3VkcILijg,enr:-J64QFalFr9Y4r9v8Svh7XFwerJeLTRnfTixfCy_NZw3OTMVZL_dSAvcZ6JIeK0JAZf6-PU3YknV9m9Jd5V5WlnqHKWGAZMxdxT4gmlkgnY0gmlwhCKNlBOHb3BzdGFja4Tn0AIAiXNlY3AyNTZrMaECI1jqAzkQ0JRkwL-UBP2TBUfUdLyTaqAXtey82CTysdCDdGNwgiMrg3VkcILOyg"
      - "--p2p.static=/ip4/34.90.21.246/tcp/9003/p2p/16Uiu2HAmNGgNTgiFBAqH58aCT3iXWMnetYMtQgH21Ydjq2R7QRbt,/ip4/34.13.162.152/tcp/9003/p2p/16Uiu2HAm33YRmCya94zRXddxaWj25QAXW5MhuJkaEvfMLXkB4GCK,/ip4/34.13.171.139/tcp/9003/p2p/16Uiu2HAm5d71wTbQPkBA3VW9suge2afCKtrGdk7UapRHR4va8jTY,/ip4/34.91.216.121/tcp/9003/p2p/16Uiu2HAmK7s7F1ALmtXKH3LxeEENstqw8jiDzUtifasS4LkUKGVE,/ip4/34.141.148.19/tcp/9003/p2p/16Uiu2HAkwoetK83q3WNRQ4t4eV8B3DosnwcFqd9VHxz24ZZzzEgo"
      - "--p2p.ban.peers=false"
      - "--p2p.ban.duration=1s"
      - "--p2p.ban.threshold=-10000000000"
      - "--log.level=info"
      - "--syncmode=${OPSYNCMODE:-execution-layer}"
      - "--override.pectrablobschedule=1751554801"
      - "--override.isthmus=1751554801"
      - "--override.holocene=1751554801"
      - "--override.granite=1751554801"
      - "--override.fjord=1751554801"
    deploy:
      restart_policy:
        condition: "any"

[blockpi019]

our running command is:

docker compose --env-file .env -f docker-compose.yml --profile hemi-min up -d

container version is:

# grep 'ghcr.io/hemilabs' docker-compose.yml
    image: ghcr.io/hemilabs/op-geth:05d4d8f@sha256:92d14f0b4b89947352d57743ce9e7f2d9c9e495f0bb861a24d21c49d391c7abf
    image: ghcr.io/hemilabs/op-node:7c70d2d@sha256:7db35d5798615e572d80823a3238a8111b14a90aade876a34b6a2936b17cbd87

[ClaytonNorthey92]

hey @blockpi019 , thanks for opening this. taking a closer look now 👀

[ClaytonNorthey92]

hey @blockpi019 when you get a chance could you paste your last ~100 lines of op-node logs? op-node, once tbc has completed its sync, will drive the l2 sync, which looks like it hasn't started. my first guess is that something is up with op-node, possibly an invalid PRYSMENDPOINT or GETHL1ENDPOINT but I am not 100% sure at this moment. the logs would tell us.

[blockpi019]

@ClaytonNorthey92 sure, here are the 100 lines logs from my running Hemi op-node:

hemi-op-node-100-line.log

[ClaytonNorthey92]

hey @blockpi019 thanks for sending those, I should have asked this before: are you doing a snap sync or full sync?

[blockpi019]

snap sync:

cat entrypoint.sh

#!/bin/sh
# Copyright (c) 2024-2025 Hemi Labs, Inc.
# Use of this source code is governed by the MIT License,
# which can be found in the LICENSE file.

# Generated with ./scripts/gen.sh (version=no-tag commit=ad0c2cc state=clean) on Wed Jul  8 04:02:53 PM CST 2026

set -xe

if [ -d "/tmp/datadir/geth" ]; then
        echo "geth data dir exists, skipping genesis."
else
        geth init --state.scheme hash --datadir /tmp/datadir/geth /tmp/genesis.json
fi

echo "Running Hemi Node version=no-tag commit=ad0c2cc state=clean"
echo " - Network: 43111 (tbc: mainnet)"
echo " - Sync mode: snap"
geth \
        --config=/tmp/l2-config.toml \
        --http \
        --http.corsdomain=* \
        --http.vhosts=* \
        --http.addr=0.0.0.0 \
        --http.api=web3,eth,txpool,net \
        --http.port=18546 \
        --ws \
        --ws.rpcprefix=/ \
        --ws.addr=0.0.0.0 \
        --ws.port=28546 \
        --ws.origins=* \
        --ws.api=eth,txpool,net \
        --syncmode=snap \
        --gcmode=archive \
        --maxpeers=100 \
        --networkid=43111 \
        --authrpc.vhosts=* \
        --authrpc.addr=0.0.0.0 \
        --authrpc.port=8551 \
        --authrpc.jwtsecret=/tmp/jwt/jwt.hex \
        --rollup.disabletxpoolgossip=false \
        --datadir=/tmp/datadir/geth \
        --override.ecotone=1725868497 \
        --override.canyon=1725868497 \
        --override.hvm0=1739286001 \
        --override.isthmus=1751554801 \
        --override.holocene=1751554801 \
        --override.granite=1751554801 \
        --override.fjord=1751554801 \
        --tbc.network=mainnet \
        --tbc.leveldbhome=/tbcdata/data \
        --tbc.prometheusaddress='0.0.0.0:5555' \
    --hvm.headerdatadir=/tbcdata/headers \
    --hvm.genesisheight=883092 \
    --hvm.genesisheader=0000003efaaa2ba65de684c512bb67ef115298d1d16bcb49b16c02000000000000000000ed31a56788c4488afc4ee69e0791ad6aeeb9ea05f069e0fdde6159068765ad3f4128a96726770217e7f41c86 \
        --bootnodes=enode://e7970a29d89f8b158371a8d4aca909ee8c1c759e711547b797a6a6f01513c1e7c85121dd2600397ca20cebf3cea21025001be7c0f577b496caf32ea0433a1cfd@34.90.21.246:30303,enode://70877d3aa6c4ccc09d960c269846215d5dcc8bf47013ac532c1ccc3d9cfe61434c96b9d6cad88a96c3f91187fb00214d903a6be6d8e93140ac4a3c099684ce34@34.13.162.152:30303,enode://27f75e68a8c14cae2f4e12f060477c150767e98149e16a448baddc25d800c008edf8b1fefd13b206c27e5473ac9234ba1958a8267fe5272e9de3819ac080bf22@34.13.171.139:30303,enode://588ffb65f841aede8d8f69a3175f9cfed1b79d20d40a7feb8a70e574b5610fb4049bf02f3170f1ae25bff806b2c823653b28af711e1962ea3f45d99d58608191@34.91.216.121:30303,enode://ba86a76186268948bc34b7fa1c2f08c24ed60cda61346cf1a1cca278b0ef1de49e567039952e06e4887a0252974401a6d6729bfc12945c6d8c65eacbf3b11ca7@34.141.148.19:30303

[blockpi019]

generated config file by shell script

./scripts/gen.sh mainnet snap hemi

[ClaytonNorthey92]

hey @blockpi019 two things come to mind:

• is there a reason that you're using the hemi profile instead of hemi-min? I admit it's not clear, but I don't think you'll need anything from hemi unless that was intentional?
• there's a chance that you may be full syncing even though you have snap sync configured, if you restarted a sync with a (I think) finalized block in your db, op-geth will (again, I think) start up in full sync

could you try this for me:

step 1:

delete your op-geth data directory, NOT TBC. this would be the docker volume op-geth_data defined in the docker compose file

step 2:

./scripts/gen.sh mainnet snap hemi-min

then

PRYSMENDPOINT=...
GETHL1ENDPOINT=...
docker compose -f ./mainnet/docker-compose.yml --profile hemi-min up

[blockpi019]

After regenerating the hemi-min profile and starting the node, it's syncing normally now.

 # docker compose --env-file .env -f docker-compose.yml --profile hemi-min logs -f -n 99 | cat | grep 'op-geth-l2' | grep -Ev 'tbc tbc.go|"TBCBlocksAvailableToHeader'
op-geth-l2-1  | INFO [07-14|03:02:56.207] Syncing beacon headers                   downloaded=980,992 left=3,863,291 eta=3m43.102s
op-geth-l2-1  | INFO [07-14|03:02:59.552] Forkchoice update does not contain any transactions
op-geth-l2-1  | INFO [07-14|03:02:59.552] "Engine API forkchoice updated, head=0x1e84764397343ea0dcabbacdeba9446b3c4ddf05eb885eb264d18cea3b13330b, finalized=0x0000000000000000000000000000000000000000000000000000000000000000, safe=0x0000000000000000000000000000000000000000000000000000000000000000"
op-geth-l2-1  | INFO [07-14|03:02:59.552] Forkchoice requested sync to new head    number=4,844,289 hash=1e8476..13330b
op-geth-l2-1  | INFO [07-14|03:03:04.226] Syncing beacon headers                   downloaded=1,128,960 left=3,715,323 eta=3m32.828s
op-geth-l2-1  | INFO [07-14|03:03:11.555] Forkchoice update does not contain any transactions
op-geth-l2-1  | INFO [07-14|03:03:11.555] "Engine API forkchoice updated, head=0x81379ee7891c977c8892e7c90ed70a6e36f45038501aec5ddbef4aa80c3028c7, finalized=0x0000000000000000000000000000000000000000000000000000000000000000, safe=0x0000000000000000000000000000000000000000000000000000000000000000"
op-geth-l2-1  | INFO [07-14|03:03:11.555] Forkchoice requested sync to new head    number=4,844,290 hash=81379e..3028c7
op-geth-l2-1  | INFO [07-14|03:03:12.229] Syncing beacon headers                   downloaded=1,270,272 left=3,574,011 eta=3m24.474s
op-geth-l2-1  | INFO [07-14|03:03:20.313] Syncing beacon headers                   downloaded=1,415,168 left=3,429,115 eta=3m15.687s
op-geth-l2-1  | INFO [07-14|03:03:23.554] Forkchoice update does not contain any transactions
op-geth-l2-1  | INFO [07-14|03:03:23.554] "Engine API forkchoice updated, head=0x4d92469558cb151ca28c0d7a74894c0fb918338361be18b824819851c0c1bfb9, finalized=0x0000000000000000000000000000000000000000000000000000000000000000, safe=0x0000000000000000000000000000000000000000000000000000000000000000"
op-geth-l2-1  | INFO [07-14|03:03:23.554] Forkchoice requested sync to new head    number=4,844,291 hash=4d9246..c1bfb9
op-geth-l2-1  | INFO [07-14|03:03:28.346] Syncing beacon headers                   downloaded=1,560,064 left=3,284,219 eta=3m6.920s
op-geth-l2-1  | INFO [07-14|03:03:35.552] Forkchoice update does not contain any transactions
op-geth-l2-1  | INFO [07-14|03:03:35.552] "Engine API forkchoice updated, head=0x151dc2438e4295b1b149a447adbc2576ca95ffaa2cf05857ccef0881bbfdfe94, finalized=0x0000000000000000000000000000000000000000000000000000000000000000, safe=0x0000000000000000000000000000000000000000000000000000000000000000"
op-geth-l2-1  | INFO [07-14|03:03:35.552] Forkchoice requested sync to new head    number=4,844,292 hash=151dc2..fdfe94
op-geth-l2-1  | INFO [07-14|03:03:36.429] Syncing beacon headers                   downloaded=1,692,672 left=3,151,611 eta=3m0.370s
op-geth-l2-1  | INFO [07-14|03:03:44.594] Syncing beacon headers                   downloaded=1,826,816 left=3,017,467 eta=2m53.500s
op-geth-l2-1  | INFO [07-14|03:03:47.553] Forkchoice update does not contain any transactions
op-geth-l2-1  | INFO [07-14|03:03:47.553] "Engine API forkchoice updated, head=0xa2d5835db6b6ae25fbdce421db092f6001705bf0e6edfba32230f76bbfc1c860, finalized=0x0000000000000000000000000000000000000000000000000000000000000000, safe=0x0000000000000000000000000000000000000000000000000000000000000000"

[blockpi019]

Also, I've run into an issue before where, even after syncing completed, fetching the block height via the RPC interface always returned 0. I'm not sure if we'll experience that again this time. Please keep this issue open for now; I will keep you updated with any progress once the sync is finished.