[SPARK-58552][SQL][UI] Add total task time column to the SQL / DataFrame tab

[ulysses-you]

What changes were proposed in this pull request?

Add a Total Task Time column to the SQL / DataFrame tab, shown right after the existing Duration column. The value is the SQL-level total task time aggregated across all stages of the execution: it sums executorRunTime (the "Total Time Across All Tasks" stage metric) over every attempt of every stage, since a retried/failed attempt also genuinely consumed task time.

The same column also appears in the summary table on the SQL execution detail page, because the list page and the detail page share one set of column definitions.

When an execution has no stages to aggregate (e.g. a query that never launched a job), the value is reported as -1 and rendered as N/A in the UI, so it is not confused with a genuine 0 ms.

Why are the changes needed?

Duration mixes scheduling/queueing overhead together with the actual compute time, so it cannot tell how much real task time a query consumed. The per-stage executorRunTime is already collected and shown on each stage page, but the SQL tab did not aggregate it up to the query level. A SQL-level total task time lets users compare the real compute cost of queries at a glance and identify compute-heavy plans.

Does this PR introduce any user-facing change?

Yes:

• The SQL / DataFrame tab and the SQL execution detail page gain a new "Total Task Time" column (rendered as N/A when the execution has no stages).
• The SQL v1 REST API's ExecutionData gains a totalTaskTime field (in milliseconds, -1 when unknown). The field defaults to -1, so existing API consumers are unaffected.

How was this patch tested?

• New unit test in SqlResourceSuite — totalTaskTime aggregates executorRunTime across all attempts of all stages — verifies the aggregation across multiple stages and attempts (including a retried attempt).
• Extended SqlResourceWithActualMetricsSuite to assert the totalTaskTime field of the sqlTable endpoint with real queries.
• Both suites pass: SqlResourceSuite (9 tests) and SqlResourceWithActualMetricsSuite (10 tests).
• dev/lint-js passes.

Snapshot:

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (deepseek-v4-flash)

[ulysses-you]

cc @dongjoon-hyun @LuciferYang @sarutak thank you

[uros-b]

Nice change, thank you @ulysses-you! Leaving up to others to also review ^^

[sarutak]

When sorting by totalTaskTime, the value is computed for all executions here, and then computed again for each execution on the current page in execToRow. Each computation does KVStore lookups for all stage attempts.

With the default page size of 20, the duplicated computation is limited to 20 executions × their stages, so this is unlikely to be a practical issue. sortBy's Schwartzian transform ensures the sort itself computes the key only once per element.

One approach would be to precompute a Map[Long, Long] (executionId -> totalTaskTime) and reuse it
in both sort and row construction.

[sarutak]

The Scaladoc says:

Sums `executorRunTime` (the "Total Time Across All Tasks" metric)

executorRunTime specifically measures the time executors spent running task code. It excludes deserialization time, result serialization time, and GC time. The "Total Time Across All Tasks" label used on the Stages page is the same metric, so the description isn't wrong, but a slightly more precise
phrasing might avoid confusion:

Sums `executorRunTime` (the cumulative time executors spent running tasks, which is the "Total Time Across All Tasks" stage-level metric)