eth/consensus : implement eccpow consensus engine#10
Open
mmingyeomm wants to merge 3489 commits into
Open
Conversation
Later on we can consider making these limits configurable if the use-case arose.
We can consider making this limit configurable if ever the need arose.
In this PR, we add support for protocol version eth/70, defined by EIP-7975. Overall changes: - Each response is buffered in the peer’s receipt buffer when the `lastBlockIncomplete` field is true. - Continued request uses the same request id of its original request(`RequestPartialReceipts`). - Partial responses are verified in `validateLastBlockReceipt`. - Even if all receipts for partial blocks of the request are collected, those partial results are not sinked to the downloader, to avoid complexity. This assumes that partial response and buffering occur only in exceptional cases. --------- Co-authored-by: Gary Rong <garyrong0905@gmail.com> Co-authored-by: Felix Lange <fjl@twurst.com>
Add persistent storage for Block Access Lists (BALs) in `core/rawdb/`. This provides read/write/delete accessors for BALs in the active key-value store. --------- Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This is a breaking change in the opcode (structLog) tracer. Several fields will have a slight formatting difference to conform to the newly established spec at: ethereum/execution-apis#762. The differences include: - `memory`: words will have the 0x prefix. Also last word of memory will be padded to 32-bytes. - `storage`: keys and values will have the 0x prefix. --------- Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
This PR changes the blsync checkpoint init logic so that even if the initialization fails with a certain server and an error log message is printed, the server goes back to its initial state and is allowed to retry initialization after the failure delay period. The previous logic had an `ssDone` server state that did put the server in a permanently unusable state once the checkpoint init failed for an apparently permanent reason. This was not the correct behavior because different servers behave differently in case of overload and sometimes the response to a permanently missing item is not clearly distinguishable from an overload response. A safer logic is to never assume anything to be permanent and always give a chance to retry. The failure delay formula is also fixed; now it is properly capped at `maxFailureDelay`. The previous formula did allow the delay to grow unlimited if a retry was attempted immediately after each delay period.
Block overrides were to a great extent ignored by the gasestimator. This PR fixes that.
This PR implements the missing functionality for archive nodes by pruning stale index data. The current mechanism is relatively simple but sufficient for now: it periodically iterates over index entries and deletes outdated data on a per-block basis. The pruning process is triggered every 90,000 new blocks (approximately every 12 days), and the iteration typically takes ~30 minutes on a mainnet node. This mechanism is only applied with `gcmode=archive` enabled, having no impact on normal full node.
In this PR, the Database interface in `core/state` has been extended with one more function: ```go // Iteratee returns a state iteratee associated with the specified state root, // through which the account iterator and storage iterator can be created. Iteratee(root common.Hash) (Iteratee, error) ``` With this additional abstraction layer, the implementation details can be hidden behind the interface. For example, state traversal can now operate directly on the flat state for Verkle or binary trees, which do not natively support traversal. Moreover, state dumping will now prefer using the flat state iterator as the primary option, offering better efficiency. Edit: this PR also fixes a tiny issue in the state dump, marshalling the next field in the correct way.
Implement the snap/2 wire protocol with BAL serving --------- Co-authored-by: Gary Rong <garyrong0905@gmail.com>
…34649) This PR adds Bytes field back to GetAccesListsPacket
The trienode history indexing progress is also exposed via an RPC endpoint and contributes to the eth_syncing status.
👋
This PR makes it possible to run "Amsterdam" in statetests. I'm aware
that they'll be failing and not in consensus with other clients, yet,
but it's nice to be able to run tests and see what works and what
doesn't
Before the change:
```
$ go run ./cmd/evm statetest ./amsterdam.json
[
{
"name": "00000019-mixed-1",
"pass": false,
"fork": "Amsterdam",
"error": "unexpected error: unsupported fork \"Amsterdam\""
}
]
```
After
```
$ go run ./cmd/evm statetest ./amsterdam.json
{"stateRoot": "0x25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e"}
[
{
"name": "00000019-mixed-1",
"pass": false,
"stateRoot": "0x25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e",
"fork": "Amsterdam",
"error": "post state root mismatch: got 25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e, want 0000000000000000000000000000000000000000000000000000000000000000"
}
]
```
…ared pipeline into triedb/internal (#34654) This PR adds `GenerateTrie(db, scheme, root)` to the `triedb` package, which rebuilds all tries from flat snapshot KV data. This is needed by snap/2 sync so it can rebuild the trie after downloading the flat state. The shared trie generation pipeline from `pathdb/verifier.go` was moved into `triedb/internal/conversion.go` so both `GenerateTrie` and `VerifyState` reuse the same code.
This PR refactors the encoding rules for `AccessListsPacket` in the wire protocol. Specifically: - The response is now encoded as a list of `rlp.RawValue` - `rlp.EmptyString` is used as a placeholder for unavailable BAL objects
PathDB keys diff layers by state root, not by block hash. That means a side-chain block can legitimately collide with an existing canonical diff layer when both blocks produce the same post-state (for example same parent, same coinbase, no txs). Today `layerTree.add` blindly inserts that second layer. If the root already exists, this overwrites `tree.layers[root]` and appends the same root to the mutation lookup again. Later account/storage lookups resolve that root to the wrong diff layer, which can corrupt reads for descendant canonical states. At runtime, the corruption is silent: no error is logged and no invariant check fires. State reads against affected descendants simply return stale data from the wrong diff layer (for example, an account balance that reflects one fewer block reward), which can propagate into RPC responses and block validation. This change makes duplicate-root inserts idempotent. A second layer with the same state root does not add any new retrievable state to a tree that is already keyed by root; keeping the original layer preserves the existing parent chain and avoids polluting the lookup history with duplicate roots. The regression test imports a canonical chain of two layers followed by a fork layer at height 1 with the same state root but a different block hash. Before the fix, account and storage lookups at the head resolve the fork layer instead of the canonical one. After the fix, the duplicate insert is skipped and lookups remain correct.
Co-authored-by: Felix Lange <fjl@twurst.com>
…33884) Changes JSON serialization of FilterCriteria to exclude "address" when it is empty.
ProcessBeaconBlockRoot (EIP-4788) and processRequestsSystemCall (EIP-7002/7251) do not merge the EVM access events into the state after execution. ProcessParentBlockHash (EIP-2935) already does this correctly at line 290-291. Without this merge, the Verkle witness will be missing the storage accesses from the beacon root and request system calls, leading to incomplete witnesses and potential consensus issues when Verkle activates.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR fixes #34623 by changing the `vm.StateDB` interface: Instead of `EmitLogsForBurnAccounts()` emitting burn logs, `LogsForBurnAccounts() []*types.Log` just returns these logs which are then emitted by the caller. This way when tracing is used, `hookedStateDB.AddLog` will be used automatically and there is no need to duplicate either the burn log logic or the `OnLog` tracing hook.
This is to fix a crasher in keeper.
This tool is designed for the offline translation of an MPT database to a binary trie. This is to be used for users who e.g. want to prove equivalence of a binary tree chain shadowing the MPT chain. It adds a `bintrie` command, cleanly separating the concerns.
already check on line 40 before.
`BinaryTrie.DeleteAccount` was a no-op, silently ignoring the caller's deletion request and leaving the old `BasicData` and `CodeHash` in the trie. Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Update the EraE (ere) reader and builder to the latest e2store ere spec (eth-clients/e2store-format-specs#16). The reader now derives the component layout from the on-disk e2store type tags via the dynamic block index, rather than assuming fixed slot positions. This makes the optional components (receipts, td, proof) resolvable in any supported subset. --------- Co-authored-by: lightclient <lightclient@protonmail.com>
…35100) Make the `Block` parameter optional on the six state-reading methods, defaulting to `latest` when omitted: - `eth_getBalance` - `eth_getCode` - `eth_getStorageAt` - `eth_getTransactionCount` - `eth_getProof` - `eth_getStorageValues` This implements the behavior proposed in ethereum/execution-apis#812. --------- Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
With Osaka being a while ago I believe we can drop this transition and drop the tx instead.
`clef` is a great tool, however: * It is no longer maintained * No one else in the team can pronounce it properly We are however receiving some slop PRs for it, so I think it's time - with infinite sadness - to say goodbye.
The checksum count during EraE import is off by one when `checksums.txt`
ends its last line on a newline, as the pandaops file does. The current
code would result in one empty string after the final `\n`, something
like
```
[]string{
"line1",
"line2",
"line3",
"",
}
```
Trim off the final `\n`, if it exists: `return
strings.Split(strings.TrimRight(string(b), "\n"), "\n"), nil`
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
check the len of BlobVersionedHashed in blob tx.
### Summary `TestServerWebsocketReadLimit/limit_with_large_request_-_should_fail` is flaky on `windows/amd64` (see [run 25364841576](https://github.com/ethereum/go-ethereum/actions/runs/25364841576/job/74378334589) referenced in #34877): ``` --- FAIL: TestServerWebsocketReadLimit/limit_with_large_request_-_should_fail (0.02s) server_test.go:279: unexpected error for read limit violation: read tcp 127.0.0.1:56703->127.0.0.1:56700: wsarecv: An existing connection was forcibly closed by the remote host. ``` When the server enforces the read limit and tears the connection down, the client's read can race the close frame. On Windows the OS surfaces that race as `wsarecv: An existing connection was forcibly closed by the remote host` instead of the gorilla `CloseError(1009)`, `websocket.ErrReadLimit`, or the POSIX `connection reset by peer` the test already tolerates. This change adds `"forcibly closed"` to the set of acceptable error substrings for the failure case, so the Windows reset is recognized as a valid signal that the server enforced the limit. ### Fixes #34877 ### Test plan - [x] `go test -count=5 -run TestServerWebsocketReadLimit ./rpc/` (darwin/arm64) — pass - [x] `go test ./rpc/...` — pass - [x] `go vet ./rpc/...` / `gofmt -l rpc/server_test.go` — clean - [ ] CI on `windows/amd64` confirms the flake no longer trips --------- Co-authored-by: lightclient <lightclient@protonmail.com>
Previously was iterating Blobs, but that could cause panic if the sidecar is malformed.
) ## Overview This PR fixes a race condition during blockchain shutdown where snapshot generation could continue accessing the trie database after it has been closed, leading to iterator errors. We noticed this in one of our nodes on https://github.com/ava-labs/avalanchego, which relies on an older version of geth with the same issue (so this behavior does happen!). During node shutdown, the following sequence occurs: 1. `BlockChain.Stop()` calls `snaps.Release()` to clean up snapshot resources 2. `Release()` only resets the cache but doesn't stop the generator goroutine 3. The trie database is then closed via `triedb.Close()` 4. The still-running generator attempts to iterate storage tries 5. Iterator fails because the database is closed (`"Generator failed to iterate storage trie"`) ## Problem There are three related bugs: 1. `Release()` doesn't stop generation: The `diskLayer.Release()` method only resets the cache without stopping ongoing snapshot generation, leaving the generator goroutine running after database closure. 2. `stopGeneration()` has an incorrect completion check: The `stopGeneration()` method checks `genMarker != nil` to determine if generation is running. However, `genMarker` is set to nil when generation completes successfully, even though the generator goroutine is still waiting for the abort signal at the end of `generate()`. See line 705 in `generate.go`: https://github.com/ethereum/go-ethereum/blob/eaaa5b716dcf97e94eb17a1469a7385a7101ffab/core/state/snapshot/generate.go#L699-L707 This means `stopGeneration()` returns early without sending the abort signal. 3. Node shutdown doesn't stop generation: During shutdown, no code path calls `stopGeneration()` or sends the abort signal to the generator, causing the generator to access a closed database and error. ## Fix - Modified `diskLayer.Release()` to call `stopGeneration()` before releasing resources - Added cancelation architecture, removing reliance on someone having to wait - Fixed `stopGeneration()` to properly and safely stop snapshot generation - Added `TestGenerateGoroutineLeak` to verify the fix and prevent regression. The test fails without the fix and passes with it. - The test creates a snapshot with active generation, waits for completion, then calls `Release()`, and uses `go.uber.org/goleak` to assert no generator goroutine survives. - Without the fix, the test fails: `Release()` returns without stopping the generator, which stays parked at `generate.go:705` waiting for an abort signal that never comes: ``` --- FAIL: TestGenerateGoroutineLeak (0.88s) generate_test.go: found unexpected goroutines: [Goroutine 6 in state chan receive, with core/state/snapshot.(*diskLayer).generate on top of the stack: core/state/snapshot.(*diskLayer).generate(...) core/state/snapshot/generate.go:705 created by core/state/snapshot.generateSnapshot core/state/snapshot/generate.go:79 ] ``` - With the fix, the test passes: `Release()` -> `stopGeneration()` blocks until the generator goroutine has fully exited, so nothing leaks Note that this fix follows the same pattern used in `Tree.Disable()` in #30040, which introduced `stopGeneration()` for use in `Disable()` and `Rebuild()` but didn't address the shutdown path. The test follows the same pattern used in `TestCheckSimBackendGoroutineLeak`
…rrect variable usages in error strings (#35121) This PR is trying to stem further slop PRs by going over all incorrect strings and fixing them. --------- Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
Better error messages on the engine api
Implement a similar RPC as what reth offers paradigmxyz/reth#18539, to clear the tx pool.
Fixes an issue where we would falsely return 400 when we should return 200 because the request was served successfully --------- Co-authored-by: Felix Lange <fjl@twurst.com>
This is a PR that removes all correctly flagged typos, in order to stop an onslaught of slop PRs in its tracks. It should be followed by #34994 but the latter needs more configuration work and I want to limit the stem of PRs right now.
This adds a client option to configure trace context propagation via the `traceparent` HTTP header. I'm adding this so that prysm can enable distributed tracing on their engine API client.
… crash (#35140) Found while updating to go-ethereum master in prysm.
…5074) In old code, mu is struct not pointer, it caused create new mutex event with same writer. Change to use pointer to sync.Mutex, so that the mutex is shared between handler with same writer.
Adds snap/2 (EIP-8189), a block-access-list (BAL) based state sync, and wires it to run side by side with snap/1. It's opt-in (for now) behind a new --snap.v2 flag and chosen at startup. https://eips.ethereum.org/EIPS/eip-8189 --------- Co-authored-by: Toni Wahrstätter <info@toniwahrstaetter.com> Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR introduces a cache for GetBlobs request. The main purpose of this PR is to reduce the getBlobs latency by reading and decoding blobs from the pool in advance of the actual query. This is important especially in the context of a sparse blobpool, since it may be necessary to recover blobs from cells on a getBlobs request. Previously, the Engine API read and decoded blobs from the pool on every call. Now those calls check the cache and only fall back to the pool on a miss. The cache has two modes: - In topK mode (default), it wakes up periodically, picks the most profitable pending blob transactions up to the current fork's maxBlobsPerBlock, and loads their blobs. The selection logic is shared with the miner's block-building logic. The selection size is derived from eip4844.MaxBlobsPerBlock at the current head. - When the CL calls HasBlobs, the cache switches to hasBlobs mode and tries to pin the set it just reported as available. Cache updates (read, decode, and optionally conversion in the future) run in background goroutines. --------- Co-authored-by: Felix Lange <fjl@twurst.com>
This PR adds an optimization to the `findNodeByID` function in `p2p/discover`. There is already an open PR (#33205) for similar improvements, and I have further optimized the function to get better performance. I have attached the benchmark results comparing the current `main` branch with my `optimized version`, and the results show clear improvements. --------- Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Currently geth ignores the docker `--memory` directive and doesn't adjust its cache size downward when necessary, potentially running into OOM. while gopsutil has functions like `docker/CgroupMem()` they are rather for reading cgroup memory limit of a container from the host.
Implements https://eips.ethereum.org/EIPS/eip-8037 mainly done in order to judge the complexity of the EIP and to act as a jumping off point, since the eip will likely change. --------- Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
implements eccpow consensus engine for Worldland Network