Add per-slice and per-partition breakdowns to the query profiler by prudhvigodithi · Pull Request #22613 · opensearch-project/OpenSearch

prudhvigodithi · 2026-07-30T20:33:35Z

Description

Adds a nested slices[] → partitions[] breakdown to the query profiler, showing per-slice and (under intra-segment search) per-partition timings that were previously collapsed into only the max_/min_/avg_slice_* aggregates.

This change adds per-slice/per-partition detail to the query section only (profile.shards[].searches[].query[]). The same pattern can be extended, as follow-ups, to the collector and aggregation sections so the whole profile output shows slice/partition granularity consistently.

Related Issues

Related to [Feature Request] Segment Profiling #20344
Part to [RFC] Intra-Segment Search: Query and Aggregation Performance Analysis #20202

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>

github-actions · 2026-07-30T20:35:10Z

PR Reviewer Guide 🔍

(Review updated until commit `fb113e3`)

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ Recommended focus areas for review Thread ID Reuse Collision The breakdown key `ThreadLeafKey(threadId, leaf)` uses `Thread.currentThread().threadId()`, which may be reused after a thread terminates. If a search thread that processed a leaf terminates and a new thread with the same id later processes a different partition of the same segment (e.g., across sequential queries reusing this breakdown, or a slow leaf-level operation completing after thread pool churn), the two searches will collide on the same key, defeating the fix's purpose. Confidence is limited on whether this can occur within a single query's lifetime given search thread pooling, but the potential for silent Timer corruption warrants review. final Object key = contextKey(Thread.currentThread().threadId(), context); // See please https://bugs.openjdk.java.net/browse/JDK-8161372 final AbstractProfileBreakdown profile = contexts.get(key); if (profile != null) { return profile; } return contexts.computeIfAbsent(key, ctx -> new QueryProfileBreakdown(metricSuppliers)); } /** * Builds the key under which a leaf's breakdown is stored: the searching thread's id paired with * the segment (leaf). When no thread id is known (e.g. a reduce lookup for a collector whose * thread was never recorded), the leaf is used directly, preserving the original single-key * behavior. Keying by thread id mirrors how {@link ConcurrentQueryProfiler} already separates * concurrent work. / static Object contextKey(Long threadId, Object leaf) { return (threadId == null) ? leaf : new ThreadLeafKey(threadId, leaf); } /* Composite key pairing the searching thread's id with the segment (leaf) it searched. / private record ThreadLeafKey(long threadId, Object leaf) { } Potential ClassCastException* In the `BREAKDOWN` xContent parser, values are cast to `Number` via `((Number) entry.getValue()).longValue()`. If a breakdown value is parsed as a non-Number type (e.g., a string on malformed input from a foreign source), this throws ClassCastException rather than a friendly parse error. Consider validating or using a typed parser. parser.declareObject(constructorArg(), (p, c) -> { final Map<String, Object> raw = p.map(); final Map<String, Long> breakdown = new HashMap<>(raw.size()); for (Map.Entry<String, Object> entry : raw.entrySet()) { breakdown.put(entry.getKey(), ((Number) entry.getValue()).longValue()); } return breakdown; }, BREAKDOWN); BWC Version Placeholder The stream and xContent gating uses `Version.V_3_8_0`. Confirm this constant exists and correctly represents the intended minimum version; if this PR merges into a branch where 3.8.0 is not yet defined or the actual release version differs, cross-cluster/mixed-cluster serialization will break silently (writer emits data older readers cannot parse, or vice versa). if (in.getVersion().onOrAfter(Version.V_3_8_0)) { this.sliceProfileResults = in.readList(SliceProfileResult::new); } else { this.sliceProfileResults = List.of(); } } @Override public void writeTo(StreamOutput out) throws IOException { out.writeString(type); out.writeString(description); out.writeLong(nodeTime); // not Vlong because can be negative out.writeMap(breakdown, StreamOutput::writeString, StreamOutput::writeLong); out.writeMap(debug, StreamOutput::writeString, StreamOutput::writeGenericValue); out.writeList(children); if (out.getVersion().onOrAfter(Version.V_2_10_0)) { out.writeOptionalLong(maxSliceNodeTime); out.writeOptionalLong(minSliceNodeTime); out.writeOptionalLong(avgSliceNodeTime); } if (out.getVersion().onOrAfter(Version.V_3_8_0)) { out.writeList(sliceProfileResults); }

github-actions · 2026-07-30T20:35:40Z

PR Code Suggestions ✨

Latest suggestions up to fb113e3

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix thread-safety of inner doc-range map The inner `HashMap` used for `sliceLeafDocRanges` values is not thread-safe. Under intra-segment search, multiple threads may concurrently call `associateCollectorToLeaves` for different collectors, and while the outer `ConcurrentHashMap` protects the top-level keys, `computeIfAbsent` can return the same inner `HashMap` to concurrent callers for the same collector key, causing a race. Use a `ConcurrentHashMap` for the inner map to ensure thread safety. server/src/main/java/org/opensearch/search/profile/query/ConcurrentQueryProfileBreakdown.java [606] -sliceLeafDocRanges.computeIfAbsent(collector, k -> new HashMap<>()).put(leaf, new int[] { minDocId, maxDocId }); +sliceLeafDocRanges.computeIfAbsent(collector, k -> new ConcurrentHashMap<>()).put(leaf, new int[] { minDocId, maxDocId }); Suggestion importance[1-10]: 6 __ Why: The inner `HashMap` for `sliceLeafDocRanges` values could be accessed concurrently if multiple threads call `associateCollectorToLeaves` for the same collector. Using `ConcurrentHashMap` for the inner map would be safer, though in practice each collector is searched by a single thread (one-thread-per-slice contract), making this a defensive improvement rather than a critical fix.	Low
	Verify parser constructor argument ordering matches The `SLICES` field is declared as `optionalConstructorArg()` and is the last constructor argument added to the parser, but the canonical constructor now has `sliceProfileResults` as the last parameter. The `ConstructingObjectParser` assigns constructor arguments positionally in declaration order. Verify that the declaration order of all `constructorArg`/`optionalConstructorArg` calls exactly matches the parameter order of the constructor that `PARSER` targets; a mismatch would silently pass the wrong value (e.g., `sliceProfileResults` list) to a different parameter. server/src/main/java/org/opensearch/search/profile/ProfileResult.java [339-340] +// Ensure declaration order matches constructor parameter order: +// type, description, breakdown, debug, nodeTime, children, +// maxSliceNodeTime, minSliceNodeTime, avgSliceNodeTime, sliceProfileResults parser.declareObjectArray(optionalConstructorArg(), (p, c) -> SliceProfileResult.fromXContent(p), SLICES); PARSER = parser.build(); Suggestion importance[1-10]: 5 __ Why: The suggestion raises a valid concern about positional argument ordering in `ConstructingObjectParser`, which could silently pass wrong values to constructor parameters. However, the `improved_code` only adds a comment without actually changing any code, making it a verification suggestion rather than a concrete fix.	Low
	Safely cast numeric XContent values to int The `ConstructingObjectParser` lambda casts `args[0]` with `(int)` directly from `Object`, but `declareInt` stores values as `Integer` objects. The cast `(int) args[0]` will unbox correctly, but `(List) args[1]` relies on the parser returning `Integer` elements from `declareIntArray`. If the XContent parser returns `Number` instead of `Integer` for array elements (which can happen with some parsers), `docRange.get(0)` would throw a `ClassCastException`. Use `((Number) docRange.get(i)).intValue()` for safety. server/src/main/java/org/opensearch/search/profile/SliceProfileResult.java [113-121] @SuppressWarnings("unchecked") private static final ConstructingObjectParser<PartitionInfo, Void> PARSER = new ConstructingObjectParser<>( "partition_info", true, args -> { - final List<Integer> docRange = (List<Integer>) args[1]; - return new PartitionInfo((int) args[0], docRange.get(0), docRange.get(1)); + final List<?> docRange = (List<?>) args[1]; + return new PartitionInfo( + ((Number) args[0]).intValue(), + ((Number) docRange.get(0)).intValue(), + ((Number) docRange.get(1)).intValue() + ); } ); Suggestion importance[1-10]: 5 __ Why: The concern about `ClassCastException` when XContent parsers return `Number` instead of `Integer` for array elements is valid, and using `((Number) ...).intValue()` is a safer pattern. However, `declareIntArray` in OpenSearch's `ConstructingObjectParser` typically guarantees `Integer` elements, making this a defensive improvement rather than a critical bug fix.	Low
General	Assert single-thread-per-slice invariant on recording The `sliceCollectorThreads.putIfAbsent` silently ignores subsequent calls for the same collector, even if they come from a different thread. Since the correctness of the reduce depends on the recorded thread matching the one that actually searched the slice, add an assertion to catch violations of the one-thread-per-slice contract during development. server/src/main/java/org/opensearch/search/profile/query/ConcurrentQueryProfileBreakdown.java [596] -sliceCollectorThreads.putIfAbsent(collector, Thread.currentThread().threadId()); +final Long existingThread = sliceCollectorThreads.putIfAbsent(collector, Thread.currentThread().threadId()); +assert existingThread == null \|\| existingThread == Thread.currentThread().threadId() + : "Collector " + collector + " was searched by two different threads: " + existingThread + " and " + Thread.currentThread().threadId(); Suggestion importance[1-10]: 3 __ Why: Adding an assertion to catch violations of the one-thread-per-slice contract is a defensive development aid, but it's a minor improvement since the contract is already documented and the code relies on it throughout. The `improved_code` also has a syntax issue (missing semicolon on the assert statement).	Low

Previous suggestions

Suggestions up to commit 5f31ef8

Category	Suggestion	Impact
Possible issue	Use thread-safe map for inner doc-ranges The inner `HashMap` is not thread-safe, but `sliceLeafDocRanges` (a `ConcurrentHashMap`) can be accessed concurrently by different slice threads, and under intra-segment search the same collector could theoretically be updated by multiple threads. Use a `ConcurrentHashMap` for the inner map to prevent potential data corruption or visibility issues during concurrent puts and reads (via `getSliceLeafDocRanges()` propagation). server/src/main/java/org/opensearch/search/profile/query/ConcurrentQueryProfileBreakdown.java [604] @Override public void associateCollectorToLeaves(Collector collector, LeafReaderContext leaf, int minDocId, int maxDocId) { associateCollectorToLeaves(collector, leaf); - // Additively record the doc-id range this (collector=slice, leaf) was searched with, from the - // searchLeaf seam where the bounds are in scope. Keyed by (collector, leaf) so that under - // intra-segment search — where one segment is split across multiple slices — each slice's - // partition of that leaf keeps its own range. Does not affect the existing reduce. - sliceLeafDocRanges.computeIfAbsent(collector, k -> new HashMap<>()).put(leaf, new int[] { minDocId, maxDocId }); + sliceLeafDocRanges.computeIfAbsent(collector, k -> new ConcurrentHashMap<>()).put(leaf, new int[] { minDocId, maxDocId }); } Suggestion importance[1-10]: 3 __ Why: Since each slice/collector is executed by a single thread (as documented in `associateCollectorToLeaves`), the inner map is unlikely to face concurrent writes for the same collector. The suggestion adds defensive safety but the actual risk is low.	Low
General	Validate doc_range array size during parsing The parser assumes `docRange` always has exactly 2 elements, but no validation is performed. If a malformed document contains fewer or more elements in `doc_range`, this will throw an `IndexOutOfBoundsException` at runtime instead of a clear parse error. Validate the array size and throw an informative exception. server/src/main/java/org/opensearch/search/profile/SliceProfileResult.java [114-121] @SuppressWarnings("unchecked") private static final ConstructingObjectParser<PartitionInfo, Void> PARSER = new ConstructingObjectParser<>( "partition_info", true, args -> { final List<Integer> docRange = (List<Integer>) args[1]; + if (docRange.size() != 2) { + throw new IllegalArgumentException("doc_range must contain exactly 2 elements, got: " + docRange.size()); + } return new PartitionInfo((int) args[0], docRange.get(0), docRange.get(1)); } ); Suggestion importance[1-10]: 3 __ Why: Adding validation improves error clarity for malformed input, but this is a minor robustness improvement in an internal parser typically fed by trusted output.	Low

Suggestions up to commit d663f7e

Category	Suggestion	Impact
General	Support nested searchLeaf invocations safely The `ThreadLocal` is set unconditionally when profiling, but if `searchLeaf` is invoked reentrantly (nested queries or recursion through subqueries during profiling), the outer slice collector will be lost when the inner call clears it in `finally`. Save and restore the previous value instead of clearing to null, to correctly support nested invocations. server/src/main/java/org/opensearch/search/internal/ContextIndexSearcher.java [355-358] final boolean profiling = weight instanceof ProfileWeight; + final Collector previousSliceCollector = profiling ? ConcurrentQueryProfileBreakdown.getCurrentSliceCollector() : null; if (profiling) { ConcurrentQueryProfileBreakdown.setCurrentSliceCollector(collector); } Suggestion importance[1-10]: 6 __ Why: Save/restore semantics for the ThreadLocal would be safer for potential reentrant invocations, though in practice `searchLeaf` is not typically called reentrantly. Also, the referenced getter method does not exist and would need to be added.	Low
General	Verify BWC version gate is correct The version gate references `Version.V_3_8_0`, but if this PR is merged into a different release the version constant may be wrong, causing serialization mismatches between nodes. Verify the target version constant matches the actual release version to prevent stream corruption between mixed-version nodes. server/src/main/java/org/opensearch/search/profile/ProfileResult.java [166-170] + if (in.getVersion().onOrAfter(Version.V_3_8_0)) { + this.sliceProfileResults = in.readList(SliceProfileResult::new); + } else { + this.sliceProfileResults = List.of(); + } - Suggestion importance[1-10]: 3 __ Why: The suggestion only asks to verify the version constant and provides no actual code change (`existing_code` equals `improved_code`), limiting its usefulness.	Low
Possible issue	Guard against division by zero Dividing by `sliceCollectorsToLeaves.size()` without a guard risks an `ArithmeticException` if `buildSliceLevelBreakdown` is ever invoked with no slices registered (e.g. edge cases where the map is empty). Guard against division by zero to prevent an unexpected exception from breaking profiling. server/src/main/java/org/opensearch/search/profile/query/ConcurrentQueryProfileBreakdown.java [387] - avgSliceNodeTime = totalSliceNodeTime / sliceCollectorsToLeaves.size(); + avgSliceNodeTime = sliceCollectorsToLeaves.isEmpty() ? 0L : totalSliceNodeTime / sliceCollectorsToLeaves.size(); return sliceLevelBreakdowns; } Suggestion importance[1-10]: 5 __ Why: A valid defensive fix: if `sliceCollectorsToLeaves` is empty, dividing would throw `ArithmeticException`. While likely not reachable in practice (the method is invoked in contexts where slices exist), the guard is cheap and reasonable.	Low

prudhvigodithi · 2026-07-30T20:42:53Z

Sample output

curl -s -X POST "localhost:9200/big5/_search" -H 'Content-Type: application/json' -d '{                                                         
      "profile": true,                                                                                                                                                                                    
      "size": 0,                                                                                                                                                                                          
      "aggs": {                                                                                                                                                                                           
        "c": { "cardinality": { "field": "host.name" } }                                                                                                                                                  
      }                                                                                                                                                                                                   
    }'

{
  "took": 1712,
  "timed_out": false,
  "terminated_early": true,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "c": {
      "value": 26832
    }
  },
  "profile": {
    "shards": [
      {
        "id": "[cuGqvARwSxScvm_BahkJ0Q][big5][0]",
        "inbound_network_time_in_millis": 0,
        "outbound_network_time_in_millis": 0,
        "searches": [
          {
            "query": [
              {
                "type": "ConstantScoreQuery",
                "description": "ConstantScore(*:*)",
                "time_in_nanos": 1327070424,
                "max_slice_time_in_nanos": 1325619099,
                "min_slice_time_in_nanos": 1137197544,
                "avg_slice_time_in_nanos": 1180361203,
                "breakdown": {
                  "advance": 293180566,
                  "advance_count": 4,
                  "avg_advance": 282,
                  "avg_advance_count": 0,
                  "avg_build_scorer": 364072924,
                  "avg_build_scorer_count": 4,
                  "avg_compute_max_score": 0,
                  "avg_compute_max_score_count": 0,
                  "avg_match": 0,
                  "avg_match_count": 0,
                  "avg_next_doc": 1180330032,
                  "avg_next_doc_count": 14500001,
                  "avg_score": 0,
                  "avg_score_count": 0,
                  "avg_set_min_competitive_score": 0,
                  "avg_set_min_competitive_score_count": 0,
                  "avg_shallow_advance": 0,
                  "avg_shallow_advance_count": 0,
                  "build_scorer": 533987120,
                  "build_scorer_count": 34,
                  "compute_max_score": 0,
                  "compute_max_score_count": 0,
                  "create_weight": 19670,
                  "create_weight_count": 1,
                  "match": 0,
                  "match_count": 0,
                  "max_advance": 660,
                  "max_advance_count": 1,
                  "max_build_scorer": 533584518,
                  "max_build_scorer_count": 6,
                  "max_compute_max_score": 0,
                  "max_compute_max_score_count": 0,
                  "max_match": 0,
                  "max_match_count": 0,
                  "max_next_doc": 1325593109,
                  "max_next_doc_count": 15990157,
                  "max_score": 0,
                  "max_score_count": 0,
                  "max_set_min_competitive_score": 0,
                  "max_set_min_competitive_score_count": 0,
                  "max_shallow_advance": 0,
                  "max_shallow_advance_count": 0,
                  "min_advance": 0,
                  "min_advance_count": 0,
                  "min_build_scorer": 240800974,
                  "min_build_scorer_count": 4,
                  "min_compute_max_score": 0,
                  "min_compute_max_score_count": 0,
                  "min_match": 0,
                  "min_match_count": 0,
                  "min_next_doc": 1137155974,
                  "min_next_doc_count": 13183985,
                  "min_score": 0,
                  "min_score_count": 0,
                  "min_set_min_competitive_score": 0,
                  "min_set_min_competitive_score_count": 0,
                  "min_shallow_advance": 0,
                  "min_shallow_advance_count": 0,
                  "next_doc": 1325593499,
                  "next_doc_count": 116000013,
                  "score": 0,
                  "score_count": 0,
                  "set_min_competitive_score": 0,
                  "set_min_competitive_score_count": 0,
                  "shallow_advance": 0,
                  "shallow_advance_count": 0
                },
                "slices": [
                  {
                    "slice_id": 0,
                    "slice_time_in_nanos": 1150105102,
                    "partitions": [
                      {
                        "segment_ord": 1,
                        "doc_range": [
                          0,
                          2749679
                        ]
                      },
                      {
                        "segment_ord": 7,
                        "doc_range": [
                          11685489,
                          23370979
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance": 390,
                      "advance_count": 1,
                      "build_scorer": 298871749,
                      "build_scorer_count": 4,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1150065842,
                      "next_doc_count": 14435170,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  },
                  {
                    "slice_id": 1,
                    "slice_time_in_nanos": 1156203792,
                    "partitions": [
                      {
                        "segment_ord": 3,
                        "doc_range": [
                          0,
                          2241135
                        ]
                      },
                      {
                        "segment_ord": 12,
                        "doc_range": [
                          0,
                          12330000
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance_count": 0,
                      "build_scorer": 258342305,
                      "build_scorer_count": 4,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1156164712,
                      "next_doc_count": 14571137,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  },
                  {
                    "slice_id": 2,
                    "slice_time_in_nanos": 1325619099,
                    "partitions": [
                      {
                        "segment_ord": 6,
                        "doc_range": [
                          0,
                          4304666
                        ]
                      },
                      {
                        "segment_ord": 7,
                        "doc_range": [
                          0,
                          11685489
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance_count": 0,
                      "build_scorer": 462176867,
                      "build_scorer_count": 4,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1325593109,
                      "next_doc_count": 15990157,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  },
                  {
                    "slice_id": 3,
                    "slice_time_in_nanos": 1137403221,
                    "partitions": [
                      {
                        "segment_ord": 0,
                        "doc_range": [
                          0,
                          499331
                        ]
                      },
                      {
                        "segment_ord": 10,
                        "doc_range": [
                          0,
                          4481017
                        ]
                      },
                      {
                        "segment_ord": 11,
                        "doc_range": [
                          8215000,
                          16430000
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance": 630,
                      "advance_count": 1,
                      "build_scorer": 533584518,
                      "build_scorer_count": 6,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1137383911,
                      "next_doc_count": 13195350,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  },
                  {
                    "slice_id": 4,
                    "slice_time_in_nanos": 1146163094,
                    "partitions": [
                      {
                        "segment_ord": 9,
                        "doc_range": [
                          0,
                          4968983
                        ]
                      },
                      {
                        "segment_ord": 11,
                        "doc_range": [
                          0,
                          8215000
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance_count": 0,
                      "build_scorer": 533270608,
                      "build_scorer_count": 4,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1146136984,
                      "next_doc_count": 13183985,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  },
                  {
                    "slice_id": 5,
                    "slice_time_in_nanos": 1192280596,
                    "partitions": [
                      {
                        "segment_ord": 4,
                        "doc_range": [
                          0,
                          2706749
                        ]
                      },
                      {
                        "segment_ord": 8,
                        "doc_range": [
                          12309510,
                          24619021
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance": 580,
                      "advance_count": 1,
                      "build_scorer": 290224488,
                      "build_scorer_count": 4,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1192241196,
                      "next_doc_count": 15016261,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  },
                  {
                    "slice_id": 6,
                    "slice_time_in_nanos": 1197917180,
                    "partitions": [
                      {
                        "segment_ord": 5,
                        "doc_range": [
                          0,
                          2735334
                        ]
                      },
                      {
                        "segment_ord": 8,
                        "doc_range": [
                          0,
                          12309510
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance_count": 0,
                      "build_scorer": 295311886,
                      "build_scorer_count": 4,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1197898530,
                      "next_doc_count": 15044846,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  },
                  {
                    "slice_id": 7,
                    "slice_time_in_nanos": 1137197544,
                    "partitions": [
                      {
                        "segment_ord": 2,
                        "doc_range": [
                          0,
                          2233106
                        ]
                      },
                      {
                        "segment_ord": 12,
                        "doc_range": [
                          12330000,
                          24660000
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance": 660,
                      "advance_count": 1,
                      "build_scorer": 240800974,
                      "build_scorer_count": 4,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1137155974,
                      "next_doc_count": 14563107,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  }
                ],
                "children": [
                  {
                    "type": "MatchAllDocsQuery",
                    "description": "*:*",
                    "time_in_nanos": 742342994,
                    "max_slice_time_in_nanos": 740902109,
                    "min_slice_time_in_nanos": 535214839,
                    "avg_slice_time_in_nanos": 629029081,
                    "breakdown": {
                      "advance": 293180376,
                      "advance_count": 4,
                      "avg_advance": 191,
                      "avg_advance_count": 0,
                      "avg_build_scorer": 364063864,
                      "avg_build_scorer_count": 4,
                      "avg_compute_max_score": 0,
                      "avg_compute_max_score_count": 0,
                      "avg_match": 0,
                      "avg_match_count": 0,
                      "avg_next_doc": 629004663,
                      "avg_next_doc_count": 14500001,
                      "avg_score": 0,
                      "avg_score_count": 0,
                      "avg_set_min_competitive_score": 0,
                      "avg_set_min_competitive_score_count": 0,
                      "avg_shallow_advance": 0,
                      "avg_shallow_advance_count": 0,
                      "build_scorer": 533981430,
                      "build_scorer_count": 34,
                      "compute_max_score": 0,
                      "compute_max_score_count": 0,
                      "create_weight": 3320,
                      "create_weight_count": 1,
                      "match": 0,
                      "match_count": 0,
                      "max_advance": 470,
                      "max_advance_count": 1,
                      "max_build_scorer": 533579398,
                      "max_build_scorer_count": 6,
                      "max_compute_max_score": 0,
                      "max_compute_max_score_count": 0,
                      "max_match": 0,
                      "max_match_count": 0,
                      "max_next_doc": 740879459,
                      "max_next_doc_count": 15990157,
                      "max_score": 0,
                      "max_score_count": 0,
                      "max_set_min_competitive_score": 0,
                      "max_set_min_competitive_score_count": 0,
                      "max_shallow_advance": 0,
                      "max_shallow_advance_count": 0,
                      "min_advance": 0,
                      "min_advance_count": 0,
                      "min_build_scorer": 240794604,
                      "min_build_scorer_count": 4,
                      "min_compute_max_score": 0,
                      "min_compute_max_score_count": 0,
                      "min_match": 0,
                      "min_match_count": 0,
                      "min_next_doc": 535177079,
                      "min_next_doc_count": 13183985,
                      "min_score": 0,
                      "min_score_count": 0,
                      "min_set_min_competitive_score": 0,
                      "min_set_min_competitive_score_count": 0,
                      "min_shallow_advance": 0,
                      "min_shallow_advance_count": 0,
                      "next_doc": 740879849,
                      "next_doc_count": 116000013,
                      "score": 0,
                      "score_count": 0,
                      "set_min_competitive_score": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance": 0,
                      "shallow_advance_count": 0
                    },
                    "slices": [
                      {
                        "slice_id": 0,
                        "slice_time_in_nanos": 740902109,
                        "partitions": [
                          {
                            "segment_ord": 6,
                            "doc_range": [
                              0,
                              4304666
                            ]
                          },
                          {
                            "segment_ord": 7,
                            "doc_range": [
                              0,
                              11685489
                            ]
                          }
                        ],
                        "breakdown": {
                          "advance_count": 0,
                          "build_scorer": 462170767,
                          "build_scorer_count": 4,
                          "compute_max_score_count": 0,
                          "match_count": 0,
                          "next_doc": 740879459,
                          "next_doc_count": 15990157,
                          "score_count": 0,
                          "set_min_competitive_score_count": 0,
                          "shallow_advance_count": 0
                        }
                      },
                      {
                        "slice_id": 1,
                        "slice_time_in_nanos": 728941448,
                        "partitions": [
                          {
                            "segment_ord": 0,
                            "doc_range": [
                              0,
                              499331
                            ]
                          },
                          {
                            "segment_ord": 10,
                            "doc_range": [
                              0,
                              4481017
                            ]
                          },
                          {
                            "segment_ord": 11,
                            "doc_range": [
                              8215000,
                              16430000
                            ]
                          }
                        ],
                        "breakdown": {
                          "advance": 450,
                          "advance_count": 1,
                          "build_scorer": 533579398,
                          "build_scorer_count": 6,
                          "compute_max_score_count": 0,
                          "match_count": 0,
                          "next_doc": 728925378,
                          "next_doc_count": 13195350,
                          "score_count": 0,
                          "set_min_competitive_score_count": 0,
                          "shallow_advance_count": 0
                        }
                      },
                      {
                        "slice_id": 2,
                        "slice_time_in_nanos": 583653689,
                        "partitions": [
                          {
                            "segment_ord": 4,
                            "doc_range": [
                              0,
                              2706749
                            ]
                          },
                          {
                            "segment_ord": 8,
                            "doc_range": [
                              12309510,
                              24619021
                            ]
                          }
                        ],
                        "breakdown": {
                          "advance": 410,
                          "advance_count": 1,
                          "build_scorer": 290209878,
                          "build_scorer_count": 4,
                          "compute_max_score_count": 0,
                          "match_count": 0,
                          "next_doc": 583626639,
                          "next_doc_count": 15016261,
                          "score_count": 0,
                          "set_min_competitive_score_count": 0,
                          "shallow_advance_count": 0
                        }
                      },
                      {
                        "slice_id": 3,
                        "slice_time_in_nanos": 587533712,
                        "partitions": [
                          {
                            "segment_ord": 5,
                            "doc_range": [
                              0,
                              2735334
                            ]
                          },
                          {
                            "segment_ord": 8,
                            "doc_range": [
                              0,
                              12309510
                            ]
                          }
                        ],
                        "breakdown": {
                          "advance_count": 0,
                          "build_scorer": 295305696,
                          "build_scorer_count": 4,
                          "compute_max_score_count": 0,
                          "match_count": 0,
                          "next_doc": 587519092,
                          "next_doc_count": 15044846,
                          "score_count": 0,
                          "set_min_competitive_score_count": 0,
                          "shallow_advance_count": 0
                        }
                      },
                      {
                        "slice_id": 4,
                        "slice_time_in_nanos": 535214839,
                        "partitions": [
                          {
                            "segment_ord": 2,
                            "doc_range": [
                              0,
                              2233106
                            ]
                          },
                          {
                            "segment_ord": 12,
                            "doc_range": [
                              12330000,
                              24660000
                            ]
                          }
                        ],
                        "breakdown": {
                          "advance": 470,
                          "advance_count": 1,
                          "build_scorer": 240794604,
                          "build_scorer_count": 4,
                          "compute_max_score_count": 0,
                          "match_count": 0,
                          "next_doc": 535177079,
                          "next_doc_count": 14563107,
                          "score_count": 0,
                          "set_min_competitive_score_count": 0,
                          "shallow_advance_count": 0
                        }
                      },
                      {
                        "slice_id": 5,
                        "slice_time_in_nanos": 575937738,
                        "partitions": [
                          {
                            "segment_ord": 1,
                            "doc_range": [
                              0,
                              2749679
                            ]
                          },
                          {
                            "segment_ord": 7,
                            "doc_range": [
                              11685489,
                              23370979
                            ]
                          }
                        ],
                        "breakdown": {
                          "advance": 200,
                          "advance_count": 1,
                          "build_scorer": 298857419,
                          "build_scorer_count": 4,
                          "compute_max_score_count": 0,
                          "match_count": 0,
                          "next_doc": 575910758,
                          "next_doc_count": 14435170,
                          "score_count": 0,
                          "set_min_competitive_score_count": 0,
                          "shallow_advance_count": 0
                        }
                      },
                      {
                        "slice_id": 6,
                        "slice_time_in_nanos": 551471552,
                        "partitions": [
                          {
                            "segment_ord": 3,
                            "doc_range": [
                              0,
                              2241135
                            ]
                          },
                          {
                            "segment_ord": 12,
                            "doc_range": [
                              0,
                              12330000
                            ]
                          }
                        ],
                        "breakdown": {
                          "advance_count": 0,
                          "build_scorer": 258328585,
                          "build_scorer_count": 4,
                          "compute_max_score_count": 0,
                          "match_count": 0,
                          "next_doc": 551444062,
                          "next_doc_count": 14571137,
                          "score_count": 0,
                          "set_min_competitive_score_count": 0,
                          "shallow_advance_count": 0
                        }
                      },
                      {
                        "slice_id": 7,
                        "slice_time_in_nanos": 728577561,
                        "partitions": [
                          {
                            "segment_ord": 9,
                            "doc_range": [
                              0,
                              4968983
                            ]
                          },
                          {
                            "segment_ord": 11,
                            "doc_range": [
                              0,
                              8215000
                            ]
                          }
                        ],
                        "breakdown": {
                          "advance_count": 0,
                          "build_scorer": 533264568,
                          "build_scorer_count": 4,
                          "compute_max_score_count": 0,
                          "match_count": 0,
                          "next_doc": 728554841,
                          "next_doc_count": 13183985,
                          "score_count": 0,
                          "set_min_competitive_score_count": 0,
                          "shallow_advance_count": 0
                        }
                      }
                    ]
                  }
                ]
              }
            ],
            "rewrite_time": 8260,
            "collector": [
              {
                "name": "QueryCollectorManager",
                "reason": "search_multi",
                "time_in_nanos": 1284525675,
                "reduce_time_in_nanos": 297231,
                "max_slice_time_in_nanos": 1283460121,
                "min_slice_time_in_nanos": 1100999210,
                "avg_slice_time_in_nanos": 1184261280,
                "slice_count": 8,
                "children": [
                  {
                    "name": "EarlyTerminatingCollectorManager",
                    "reason": "search_count",
                    "time_in_nanos": 1131994,
                    "reduce_time_in_nanos": 14640,
                    "max_slice_time_in_nanos": 15860,
                    "min_slice_time_in_nanos": 6400,
                    "avg_slice_time_in_nanos": 10722,
                    "slice_count": 8
                  },
                  {
                    "name": "NonGlobalAggCollectorManager: [c]",
                    "reason": "aggregation",
                    "time_in_nanos": 520425303,
                    "reduce_time_in_nanos": 253261,
                    "max_slice_time_in_nanos": 519358029,
                    "min_slice_time_in_nanos": 467946347,
                    "avg_slice_time_in_nanos": 488668110,
                    "slice_count": 8
                  }
                ]
              }
            ]
          }
        ],
        "aggregations": [
          {
            "type": "CardinalityAggregator",
            "description": "c",
            "time_in_nanos": 514102722,
            "max_slice_time_in_nanos": 512934808,
            "min_slice_time_in_nanos": 448443154,
            "avg_slice_time_in_nanos": 470547178,
            "breakdown": {
              "min_build_leaf_collector": 302941,
              "build_aggregation_count": 8,
              "post_collection": 256933740,
              "max_collect_count": 15990155,
              "initialize_count": 8,
              "reduce_count": 0,
              "avg_collect": 466983256,
              "max_build_aggregation": 94351,
              "avg_collect_count": 14500000,
              "max_build_leaf_collector": 18361754,
              "min_build_leaf_collector_count": 2,
              "build_aggregation": 255820696,
              "min_initialize": 260,
              "max_reduce": 0,
              "build_leaf_collector_count": 17,
              "avg_reduce": 0,
              "min_collect_count": 13183983,
              "avg_build_leaf_collector_count": 2,
              "avg_build_leaf_collector": 2771147,
              "max_collect": 511471553,
              "reduce": 0,
              "avg_build_aggregation": 82159,
              "min_post_collection": 423602,
              "max_initialize": 1890,
              "max_post_collection": 1770986,
              "collect_count": 116000000,
              "avg_post_collection": 710027,
              "avg_initialize": 587,
              "post_collection_count": 8,
              "build_leaf_collector": 18361754,
              "min_collect": 446719657,
              "min_build_aggregation": 66071,
              "initialize": 1216444,
              "max_build_leaf_collector_count": 3,
              "min_reduce": 0,
              "collect": 511471603
            },
            "debug": {
              "ordinals_collectors_used": 0,
              "ordinals_collectors_overhead_too_high": 0,
              "string_hashing_collectors_used": 0,
              "dynamic_pruned_segments": 0,
              "numeric_collectors_used": 0,
              "empty_collectors_used": 0,
              "hybrid_collectors_used": 3
            }
          }
        ],
        "fetch": []
      }
    ]
  }
}

github-actions · 2026-07-30T21:10:18Z

❌ Gradle check result for d663f7e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

asimmahmood1 · 2026-07-30T22:27:27Z

Very nice.

Just to confirm, this will add slice_time_in_nanos metric. This is the cumulative spent on for that slice? So the assumption is that highest value will be the slowest slice? I know you're calculator the start and end of each slice, do you do think its worth outputting that as well? I think we can start with this, even if we know the start and end time there's little we can do change it, the distribution of slice_time_in_nanos gives us an idea how well the slices have been constructed, i.e. range of docs.

I'm looking forward to see what visualization we can create out of this data.

"slices": [
                  {
                    "slice_id": 0,
                    "slice_time_in_nanos": 1150105102,
                    "partitions": [
                      {
                        "segment_ord": 1,
                        "doc_range": [
                          0,
                          2749679
                        ]
                      },
                      {
                        "segment_ord": 7,
                        "doc_range": [
                          11685489,
                          23370979
                        ]
                      }
                    ],
                    "breakdown": {
                      "advance": 390,
                      "advance_count": 1,
                      "build_scorer": 298871749,
                      "build_scorer_count": 4,
                      "compute_max_score_count": 0,
                      "match_count": 0,
                      "next_doc": 1150065842,
                      "next_doc_count": 14435170,
                      "score_count": 0,
                      "set_min_competitive_score_count": 0,
                      "shallow_advance_count": 0
                    }
                  },

Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>

github-actions · 2026-07-30T23:53:32Z

Persistent review updated to latest commit 5f31ef8

prudhvigodithi · 2026-07-31T00:48:52Z

Just to confirm, this will add slice_time_in_nanos metric. This is the cumulative spent on for that slice?

Its the slice's earliest operation started to when its latest one finished sliceMaxEndTime − sliceMinStartTime.

So the assumption is that highest value will be the slowest slice?

Yes, this is true. The idea is get/debug the slowest slice which would further help with partition data with intra segment.

github-actions · 2026-07-31T00:49:58Z

✅ Gradle check result for 5f31ef8: SUCCESS

codecov · 2026-07-31T00:50:46Z

Codecov Report

❌ Patch coverage is 79.20792% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.43%. Comparing base (d19f68e) to head (fb113e3).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...profile/query/ConcurrentQueryProfileBreakdown.java	81.53%	7 Missing and 5 partials ⚠️
...a/org/opensearch/search/profile/ProfileResult.java	61.11%	6 Missing and 1 partial ⚠️
...ensearch/search/internal/ContextIndexSearcher.java	50.00%	0 Missing and 1 partial ⚠️
...arch/profile/query/ConcurrentQueryProfileTree.java	91.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #22613      +/-   ##
============================================
- Coverage     71.43%   71.43%   -0.01%     
- Complexity    76760    76823      +63     
============================================
  Files          6142     6142              
  Lines        357766   357880     +114     
  Branches      52148    52178      +30     
============================================
+ Hits         255581   255657      +76     
- Misses        81861    81890      +29     
- Partials      20324    20333       +9

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

prudhvigodithi · 2026-07-31T00:53:21Z

Thanks @asimmahmood1 I have updated to fix the tests will do some cleanup, adding @jainankitk @sohami

Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>

github-actions · 2026-07-31T18:56:34Z

Persistent review updated to latest commit fb113e3

github-actions · 2026-07-31T19:58:54Z

✅ Gradle check result for fb113e3: SUCCESS

prudhvigodithi added 2 commits July 29, 2026 14:20

query phase per slice level profile breakdown

69630ad

Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>

query phase per slice level profile breakdown

d663f7e

Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>

fix tests

5f31ef8

Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>

prudhvigodithi self-assigned this Jul 31, 2026

prudhvigodithi added the feature New feature or request label Jul 31, 2026

use java threadId

fb113e3

Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>

prudhvigodithi changed the title ~~[Draft] Add per-slice and per-partition breakdowns to the query profiler~~ Add per-slice and per-partition breakdowns to the query profiler Jul 31, 2026

prudhvigodithi marked this pull request as ready for review July 31, 2026 18:55

prudhvigodithi requested review from Bukhtawar, alchemist51, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh and owaiskazi19 as code owners July 31, 2026 18:55

prudhvigodithi requested review from a team, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners July 31, 2026 18:55

Navigation Menu

Uh oh!

Add per-slice and per-partition breakdowns to the query profiler - #22613

Add per-slice and per-partition breakdowns to the query profiler#22613
prudhvigodithi wants to merge 4 commits into
opensearch-project:mainfrom
prudhvigodithi:profile

prudhvigodithi commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

prudhvigodithi commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

asimmahmood1 commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

prudhvigodithi commented Jul 31, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

codecov Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

prudhvigodithi commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

prudhvigodithi commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Reviewer Guide 🔍

(Review updated until commit fb113e3)

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Code Suggestions ✨

Previous suggestions

Uh oh!

prudhvigodithi commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sample output

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

asimmahmood1 commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

prudhvigodithi commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

codecov Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

prudhvigodithi commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

prudhvigodithi commented Jul 30, 2026 •

edited

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading

(Review updated until commit `fb113e3`)

github-actions Bot commented Jul 30, 2026 •

edited

Loading

prudhvigodithi commented Jul 30, 2026 •

edited

Loading

prudhvigodithi commented Jul 31, 2026 •

edited

Loading

codecov Bot commented Jul 31, 2026 •

edited

Loading