[DirectTls] Add metrics, tracing, and logging on par with the Sockets transport and runtime epoll engine

[DeagleGross]

Follow-up for the experimental DirectTls transport (proposal #67909, draft implementation #67912).

DirectTls currently ships with almost no telemetry. Because it terminates TLS inside the transport (on the TlsEventPump epoll threads) rather than through SslStream in HttpsConnectionMiddleware, it both loses existing observability and adds a new, unobserved execution surface (the pump). This issue tracks bringing metrics / tracing / logging up to parity with what the Sockets transport and the runtime's epoll socket engine already provide.

Motivation / gaps

1. TLS handshake metrics are silently lost

kestrel.tls_handshake.duration (Histogram) and kestrel.active_tls_handshakes (UpDownCounter), on Meter Microsoft.AspNetCore.Server.Kestrel, are emitted only from HttpsConnectionMiddleware (_metrics.TlsHandshakeStart/Stop(...)). DirectTls makes that middleware a no-op — the handshake happens on the pump thread against TlsSocketSession. So for DirectTls endpoints these two metrics are never recorded.

That's the wrong thing to lose for a transport whose entire purpose is TLS performance: handshake latency and in-flight handshake count are exactly what operators need to see. The pump must record the same instruments at handshake begin / complete / fail (including ConnectionEndReason.TlsHandshakeFailed on failures), reusing KestrelMetrics so the data lands on the existing meter/tags and dashboards keep working.

Note: transport-agnostic connection metrics (kestrel.active_connections, kestrel.connection.duration, kestrel.rejected_connections, kestrel.queued_connections, kestrel.queued_requests) are recorded in Kestrel core's connection pipeline and continue to work for DirectTls — no action needed there. The gap is specifically the TLS-handshake instruments above.

2. Connection lifecycle tracing, including backpressure

The Sockets transport has a full LoggerMessage-based trace surface in SocketsLog.cs (category Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets) that DirectTls has no equivalent for:

Event (Sockets SocketsLog)	Meaning	DirectTls equivalent to add
ConnectionPause / ConnectionResume	backpressure — input pipe hit PauseWriterThreshold / recovered	receive loop pauses/resumes the fd (deregister/re-arm EPOLLIN) when Application.Output flush backpressures
ConnectionReadFin	peer half-close (read FIN)	SSL_read / TlsSocketSession reports EOF
ConnectionWriteFin / ConnectionWriteRst	graceful vs abortive close	send loop / shutdown path
ConnectionError / ConnectionReset	I/O error / RST	pump + ConnectionIoState error paths

Backpressure is the one we care about most. In DirectTls the receive loop decrypts on the pump thread and flushes into Application.Output; when the app/Kestrel is slow the pipe's PauseWriterThreshold trips and the flush stops completing. The pump must then stop reading that fd (drop EPOLLIN) and re-arm when the reader drains — and we need to see that happen (how often, how long connections sit paused), just like ConnectionPause/ConnectionResume do for sockets. Without it, a stalled or backpressured pump is invisible.

sequenceDiagram
    participant Pump as epoll pump thread
    participant Pipe as Application.Output pipe
    participant App as Kestrel / app

    Pump->>Pipe: FlushAsync decrypted bytes
    Note over Pipe: PauseWriterThreshold reached, flush does not complete
    Pipe-->>Pump: backpressure - flush pending
    Note over Pump: stop reading fd, drop EPOLLIN<br/>log ConnectionPause, bump paused-connections gauge
    App->>Pipe: reads and drains
    Pipe-->>Pump: ResumeWriterThreshold, flush completes
    Note over Pump: re-arm EPOLLIN<br/>log ConnectionResume, clear gauge

Loading

3. Pump / engine health telemetry (inspired by the runtime epoll engine)

The runtime's SocketAsyncEngine (Linux epoll) is the direct analogue of our TlsEventPump — a small fixed set of dedicated epoll threads pulling readiness events — and System.Net.Sockets (SocketsTelemetry EventSource) exposes engine-level counters (connections established, bytes sent/received, etc.). We have zero visibility into the pump. We should add engine-level telemetry, e.g.:

• connections currently owned per pump / total, accepts per second
• connections currently paused for backpressure (gauge, ties to §2)
• bytes read / written through the pump
• optional: epoll_wait iteration/wakeup counts, ready-event batch sizes, pending-op depth

These can start as an EventSource / EventCounters surface (like the runtime's), and graduate the operationally-important ones (paused connections, handshake duration) to System.Diagnostics.Metrics instruments.