Algorithm Analysis: How Kepler and Scaphandre Calculate VM/Container Power Attribution

[dipankardas011]

Deep Dive: Power Attribution Algorithms on Bare Metal

This issue analyzes the algorithms used by Kepler, Scaphandre, and vrapl for attributing power consumption to VMs/containers when running on bare metal with RAPL access.

---

1. Scaphandre Algorithm

Core Formula

Process_Power = (Process_CPU_Time / Total_CPU_Time) × Total_Power_from_RAPL

Detailed Algorithm

Step 1: Data Collection

- Read RAPL energy counters from /sys/class/powercap/intel-rapl
- Read global CPU time from /proc/stat (in jiffies)
- Read per-process CPU time from /proc/<PID>/stat (in jiffies)

Step 2: Calculate Ratios

CPU_Ratio = Process_Jiffies / Total_Active_Jiffies

Step 3: Attribution

Process_Energy = Total_Energy_Delta × CPU_Ratio
Process_Power = Process_Energy / Time_Interval

Implementation Details

Jiffy-based Timing:

• Uses Linux kernel time units (jiffies)
• Typical jiffy = 10ms (100 Hz) or 4ms (250 Hz)
• Reads from /proc/stat for system-wide CPU time
• Reads from /proc/<PID>/stat for process-specific time

Energy Sources:

• Primary: RAPL via powercap subsystem
• Supports: CPU package, DRAM, uncore domains
• Sampling interval: User-configurable (default ~5 seconds)

Why This Approach?

Design Rationale:

1. Low Overhead: /proc reads are cheap, no kernel modifications
2. Simplicity: Linear proportional model, easy to understand
3. Platform Independence: Works on any Linux system with /proc
4. No Special Hardware: Only needs RAPL (standard on modern x86)

Developer Quote:

"This core mechanism combines time (jiffies) and energy (joules) to estimate process-level power consumption with minimal overhead."

Assumptions

1. Linear Power-Time Relationship: Assumes power consumption scales linearly with CPU time
2. Uniform Instruction Power: All CPU instructions consume roughly the same power
3. CPU Time = Energy: Ignores instruction-level power differences
4. Idle Power: Currently doesn't explicitly handle idle power attribution
5. Single Allocation Key: Uses only CPU time (doesn't account for memory, I/O)

Constraints

Technical:

• Requires read access to /sys/class/powercap (root or proper permissions)
• RAPL must be available (Intel Sandy Bridge+, AMD Zen+)
• Linux kernel 5.4.0-53.59+: powercap only accessible by root
• Must load intel_rapl or intel_rapl_common kernel module

Accuracy:

• Jiffy granularity limits precision
• /proc/stat updates at HZ rate (100-1000 Hz)
• RAPL updates ~1000 Hz but with smoothing
• Proportional model breaks down when CPU time != energy

Known Limitations

Issue #20: CPU usage sum can exceed 100% × CPU count, leading to power sum > RAPL total

Issue #38: "The main allocation key is CPU time, and as host level power metrics include more components on the machine, this allocation key becomes more inaccurate."

From CNCF Blog (2023):

"Future versions should include keys regarding the activity of other components than CPU."

---

2. Kepler Algorithm

Core Formula (Ratio Power Model)

Container_Power = Idle_Power_Share + Dynamic_Power_Share

Where:
Dynamic_Power_Share = (Container_Resource_Utilization / Total_Resource_Utilization) × Dynamic_Power
Idle_Power_Share = (Container_Size / Total_Container_Size) × Idle_Power

Detailed Algorithm

Step 1: Collect Hardware Metrics via eBPF

Per-container metrics:
- CPU instructions (performance counter)
- CPU cycles (performance counter)  
- Cache misses (last-level cache)
- CPU time (scheduler stats)
- Memory access patterns

Step 2: Calculate Dynamic Power

Total_Power = RAPL_Energy_Delta / Time_Interval
Node_CPU_Usage = Active_CPU_Time / Total_CPU_Time
Dynamic_Power = Total_Power × Node_CPU_Usage

Step 3: Calculate Idle Power

Idle_Power = Total_Power × (1 - Node_CPU_Usage)

Step 4: Attribute Dynamic Power

Container_Dynamic = Dynamic_Power × (Container_Instructions / Total_Instructions)

Note: Can use cycles, cache misses, or CPU time - configurable

Step 5: Attribute Idle Power (GHG Protocol)

Container_Idle = Idle_Power × (Container_Memory_Size / Total_Memory_Allocated)

Step 6: Total Attribution

Container_Total_Power = Container_Dynamic + Container_Idle

Implementation Details

eBPF Collection:

// Pseudo-code for eBPF program
struct process_metrics {
    u64 cpu_cycles;
    u64 cpu_instructions;
    u64 cache_references;
    u64 cache_misses;
    u64 cpu_time_us;
};

// Hook into scheduler events
int on_sched_switch(struct task_struct *prev, struct task_struct *next) {
    // Read hardware performance counters
    metrics.cpu_cycles = read_perf_counter(PERF_COUNT_HW_CPU_CYCLES);
    metrics.cpu_instructions = read_perf_counter(PERF_COUNT_HW_INSTRUCTIONS);
    metrics.cache_misses = read_perf_counter(PERF_COUNT_HW_CACHE_MISSES);
    
    // Store per-container
    update_container_metrics(next->cgroup, &metrics);
}

Hardware Performance Counters:

• PERF_COUNT_HW_CPU_CYCLES: Total CPU cycles
• PERF_COUNT_HW_INSTRUCTIONS: Retired instructions
• PERF_COUNT_HW_CACHE_REFERENCES: Cache accesses
• PERF_COUNT_HW_CACHE_MISSES: Last-level cache misses
• PERF_COUNT_HW_BRANCH_MISSES: Branch prediction failures

Why This Approach?

Design Rationale:

1. Instruction-Level Accuracy: Different instructions consume different power
2. Memory Awareness: Cache misses indicate memory-bound workloads
3. Idle Power Handling: Explicit idle power attribution
4. Kubernetes Native: Built for container orchestration
5. Cloud-Friendly: Can fall back to ML models

Developer Discussion (#548):

"CPU time only explains part of the energy usage... the nature of the CPU instructions accounts for the delta between different types of workloads when their CPU time are the same."

Research Finding:

"There is evidence that CPU instructions can estimate power well"

Assumptions

1. Instruction-Power Correlation: CPU instructions better correlate with power than time
2. Linear Dynamic Power: Dynamic power scales with resource utilization
3. Idle Power by Size: Idle power should be split by container memory size (GHG Protocol)
4. Hardware Counters Available: Assumes access to performance counters
5. Stable Workloads: Models trained on specific workload patterns

Constraints

Technical:

• Requires eBPF support (Linux kernel 4.14+)
• Needs access to perf events subsystem
• Performance counters must be exposed (may not be in VMs)
• Kernel with BPF LSM or appropriate security settings
• cgroup v2 for container tracking

Accuracy:

• ML models require training data
• Estimator accuracy varies by workload type
• Idle power attribution is controversial
• Hardware counter overhead (though minimal with eBPF)

Known Limitations

From 2024 Research Paper:

"Kepler's metrics are not accurate where total cluster power is concerned... the estimates exhibit consistency for showing variations between runs of the same workload, but they are approximations of actual power consumption."

Idle Power Challenge (CNCF Blog 2024):

"Idle power attribution is challenging in public cloud environment where computation is shared among multiple tenants, and its contribution to total power for a node is significant and can range from 20% to 60% of the power at maximum utilization."

Discussion #1284:

"Dynamic power consumption is directly related to resource utilization, but determining CPU utilization can vary based on the type of instructions and cache operations, with CPU cycles typically exhibiting better correlation."

---

3. vrapl Algorithm

Core Formula

VM_Power = (VM_CPU_Time / Total_CPU_Time) × Host_RAPL_Power

Detailed Algorithm

Step 1: Discover VM Process

// Use libvirt to get VM UUID
uuid := libvirt.GetVMUUID(vmName)

// Find QEMU process with that UUID
for process in allProcesses {
    if process.cmdline.contains(uuid) {
        vmPid = process.pid
    }
}

Step 2: Collect Metrics

// Host CPU time (all cores)
hostCPU = readCPUTimes()  // user + system time
hostTotal = hostCPU.user + hostCPU.system

// VM process CPU time
vmCPU = readProcessTimes(vmPid)  // via gopsutil
vmTotal = vmCPU.user + vmCPU.system

// Host RAPL energy
raplEnergy = readFile("/sys/class/powercap/intel-rapl:0/energy_uj")

Step 3: Calculate Deltas

deltaHostCPU = currentHostTotal - lastHostTotal
deltaVMCPU = currentVMTotal - lastVMTotal
deltaEnergy = currentRaplEnergy - lastRaplEnergy

Step 4: Attribution

if deltaHostCPU > 0 {
    vmCPUFraction = deltaVMCPU / deltaHostCPU
    vmEnergy = deltaEnergy * vmCPUFraction
    vmPower = vmEnergy / timeInterval / 1_000_000  // Convert μJ to W
}

Step 5: Communication

// Stream to VM via serial port
serialPort.WriteString(fmt.Sprintf("vRAPL_Watts: %.2f\n", vmPower))

Why This Approach?

Design Rationale:

1. Simplicity: Easiest possible implementation
2. Direct Communication: Serial port avoids filesystem complexity
3. Real-time: Continuous streaming, no polling
4. Single Purpose: Focused on VM attribution only
5. Educational: Clear code for learning

Assumptions

1. CPU Time Proportionality: Same as Scaphandre
2. Single VM Focus: Designed for monitoring one VM
3. Serial Port Available: VM has virtio-serial configured
4. QEMU/KVM: Uses libvirt, assumes QEMU hypervisor
5. Package-Level RAPL: Uses first RAPL package found

Constraints

Technical:

• Requires host access (cannot run inside VM)
• Needs libvirt connection
• Serial port must be configured in VM
• RAPL access (same as Scaphandre)
• Linux host only

Accuracy:

• Same limitations as CPU-time-based approaches
• No idle power handling
• No memory/IO attribution
• Single RAPL zone (doesn't handle multi-socket)

---

Algorithm Comparison Matrix

Feature	vrapl	Scaphandre	Kepler
Allocation Metric	CPU time	CPU time (jiffies)	CPU instructions*
Idle Power	❌ No	❌ No (planned)	✅ Yes (by size)
Dynamic Power	✅ Proportional	✅ Proportional	✅ Ratio model
Memory Attribution	❌ No	❌ No	⚠️ Indirect (idle)
I/O Attribution	❌ No	❌ No	❌ No
Hardware Counters	❌ No	❌ No	✅ Yes (eBPF)
Instruction Awareness	❌ No	❌ No	✅ Yes
Cache Miss Tracking	❌ No	❌ No	✅ Yes
Multi-Socket	⚠️ Single zone	✅ Yes	✅ Yes
Overhead	🟢 Minimal	🟢 Low	🟡 Medium (eBPF)
Complexity	🟢 ~200 LOC	🟡 ~10K LOC	🔴 ~50K LOC
Data Source	/proc, gopsutil	/proc/stat	eBPF + perf
Sampling Rate	2s (configurable)	~5s (configurable)	~3s (configurable)

* Kepler is configurable - can use cycles, instructions, or CPU time

---

Accuracy Analysis

CPU Time vs CPU Instructions

The Debate (from Kepler Discussion #548):

Pro CPU Instructions (Kepler):

"CPU time only explains part of the energy usage... the nature of the CPU instructions accounts for the delta between different types of workloads when their CPU time are the same."

• Different instructions have different power draw
• Integer ops < Floating point < SIMD/AVX
• Cache misses cause memory accesses (higher power)
• Branch mispredictions waste energy

Pro CPU Time (Scaphandre/vrapl):

"The approach by Scaphandre using elapsed jiffies is at least off in absolute value from harder metrics like Instructions."

• Simpler to implement and understand
• Lower overhead (no performance counters)
• Works reliably across all systems
• Good enough for comparative analysis

Empirical Evidence

From 2024 Research:

• Kepler: "Estimates exhibit consistency for showing variations... but they are approximations"
• Scaphandre: "Generally accurate enough for comparative analysis"

Both approaches are approximations. Neither is perfect.

When Each Works Best

CPU Time Better For:

• Uniform workloads (web servers, databases)
• Comparative analysis (before/after)
• Relative measurements (container A vs B)
• Systems without hardware counters

CPU Instructions Better For:

• Heterogeneous workloads (mixed int/float)
• Absolute power values
• Memory-bound vs CPU-bound distinction
• Fine-grained attribution

---

Pros and Cons

vrapl

Pros:

• ✅ Simplest implementation possible
• ✅ Easy to understand and modify
• ✅ Minimal dependencies
• ✅ Real-time serial streaming
• ✅ No filesystem complexity
• ✅ Perfect for learning/teaching

Cons:

• ❌ Single VM only
• ❌ No idle power handling
• ❌ No memory/IO attribution
• ❌ CPU time limitations (same as Scaphandre)
• ❌ No Prometheus export
• ❌ No multi-socket support

Best For:

• Development/testing environments
• Single VM monitoring
• Learning VM power attribution
• Simple deployments

---

Scaphandre

Pros:

• ✅ Production-ready and mature
• ✅ Multiple exporters (Prometheus, JSON, Riemann)
• ✅ Multi-process/VM support
• ✅ Low overhead
• ✅ Active community
• ✅ Comprehensive documentation
• ✅ Multi-socket support

Cons:

• ❌ CPU time limitations (see Issue #20)
• ❌ No idle power handling yet
• ❌ No instruction-level awareness
• ❌ Filesystem sharing for VMs (more complex than serial)
• ❌ Sum of process power can exceed RAPL total
• ❌ Single allocation key (CPU time)

Best For:

• Production monitoring
• Multi-process environments
• Integration with monitoring stacks
• When simplicity matters more than instruction-level accuracy

---

Kepler

Pros:

• ✅ Most sophisticated approach
• ✅ Instruction-level power awareness
• ✅ Idle power attribution (GHG Protocol)
• ✅ Hardware performance counters
• ✅ Cache miss tracking
• ✅ Kubernetes-native
• ✅ ML fallback for cloud
• ✅ Active CNCF project

Cons:

• ❌ Most complex implementation
• ❌ Requires eBPF (kernel 4.14+)
• ❌ Requires performance counters
• ❌ ML models need training
• ❌ Higher overhead than alternatives
• ❌ Idle power attribution controversial
• ❌ Accuracy issues documented (2024 research)
• ❌ Container-focused, not VM-optimized

Best For:

• Kubernetes environments
• Heterogeneous workloads
• When instruction-level accuracy needed
• Cloud environments (with ML fallback)
• Organizations with ML/data science teams

---

Recommendations Based on Use Case

Choose vrapl when:

• 🎯 Monitoring a single VM
• 🎯 Learning VM power attribution
• 🎯 Development/testing environment
• 🎯 Need simplest possible solution
• 🎯 Want to customize the code easily

Choose Scaphandre when:

• 🎯 Production bare-metal deployment
• 🎯 Multiple VMs/processes to monitor
• 🎯 Need Prometheus integration
• 🎯 Simplicity and reliability are priorities
• 🎯 CPU time attribution is "good enough"

Choose Kepler when:

• 🎯 Kubernetes cluster
• 🎯 Container workloads
• 🎯 Need instruction-level accuracy
• 🎯 Heterogeneous workloads (ML, HPC)
• 🎯 Cloud deployment (can use ML estimator)
• 🎯 Have resources for eBPF/ML complexity

---

Improving vrapl Based on This Analysis

Quick Wins (Keep Simplicity):

1. Add multi-VM support (monitor N VMs simultaneously)
2. Add DRAM RAPL domain (memory attribution)
3. Add Prometheus exporter (compatibility)
4. Handle RAPL counter overflow

Medium Term (Add Sophistication):

1. Implement idle power attribution (Kepler approach)
2. Track VM memory size for better idle split
3. Add multi-socket RAPL support
4. Benchmark vs Scaphandre accuracy

Advanced (Optional):

1. Add eBPF for instruction counting (Kepler approach)
2. Hybrid model: CPU time + instructions
3. ML estimator for cloud deployment
4. Per-vCPU attribution

---

References

Scaphandre

• Algorithm Docs: https://hubblo-org.github.io/scaphandre-documentation/explanations/how-scaph-computes-per-process-power-consumption.html
• Issue #20: Investigate and at least document why a compilation process may create a difference between total host power and sum of processes power hubblo-org/scaphandre#20
• Issue #38: Clarification: *process* level power usage stats or *processor* level stats ? hubblo-org/scaphandre#38

Kepler

• Power Model: https://sustainable-computing.io/design/power_model/
• eBPF Details: https://sustainable-computing.io/design/ebpf_in_kepler/
• Discussion #548: Sources / Design decisions for energy splitting sustainable-computing-io/kepler#548
• Discussion #1284: Relation between CPU Instructions and Power Consumption sustainable-computing-io/kepler#1284
• 2024 Research: https://arxiv.org/html/2504.10702v1
• Idle Power Blog: https://tag-env-sustainability.cncf.io/blog/2024-06-idle-power-matters-kepler-metrics-for-public-cloud-energy-efficiency/

General

• RAPL in Action: https://dl.acm.org/doi/10.1145/3177754
• GHG Protocol: Greenhouse Gas Protocol guidelines for idle power

[dipankardas011]

🔄 IMPORTANT CONTEXT

This algorithm analysis is excellent educational content, but needs context about vrapl's actual purpose. See Issue #5.

What This Issue Gets Right:

✅ Fantastic deep-dive into power attribution algorithms
✅ All the technical details are accurate
✅ Great learning resource for understanding:

• How Scaphandre uses CPU time
• How Kepler uses CPU instructions
• Trade-offs between approaches
• Pros and cons of each

Keep all this content - it's valuable!

The Missing Context:

This makes it seem like vrapl is trying to compete on algorithm sophistication.

The Correct Understanding:

vrapl doesn't care WHICH algorithm is used - it's about getting the data TO the VM!

┌─────────────────────────────────────────────┐
│ Host Side (Cloud Provider)                  │
│                                             │
│ Use ANY attribution algorithm:              │
│   • CPU time (Scaphandre approach) ✅       │
│   • CPU instructions (Kepler approach) ✅   │
│   • Hybrid model ✅                         │
│   • Cloud provider's custom algo ✅         │
│                                             │
│ vrapl doesn't care! Pick the best one.     │
└────────────────┬────────────────────────────┘
                 │
                 │ vrapl cares about THIS part
                 │ (transport mechanism)
                 ↓
┌─────────────────────────────────────────────┐
│ Guest VM                                    │
│                                             │
│ Receives the attributed power value         │
│ Then user installs their favorite tool:    │
│   • Kepler ✅                               │
│   • Scaphandre ✅                           │
│   • Custom ✅                               │
└─────────────────────────────────────────────┘

vrapl's Current Algorithm Choice:

VM_Power = (VM_CPU_Time / Host_CPU_Time) × Host_RAPL_Power

Why this algorithm?

• ✅ Simplest possible (demonstration purposes)
• ✅ Good enough for proof-of-concept
• ✅ Easy for cloud providers to understand

Is it the best algorithm?

• Maybe not! Kepler's instruction-based might be better
• But that's NOT THE POINT of vrapl

The Real Point:

Regardless of which algorithm the host uses, once the data reaches the guest VM:

Guest can run:

Option 1: Kepler
  └─ Gets attributed VM power from host
  └─ Does its own container-level attribution with eBPF
  └─ Users get container-level power

Option 2: Scaphandre  
  └─ Gets attributed VM power from host
  └─ Does its own process-level attribution
  └─ Users get process-level power

Option 3: Custom tool
  └─ Gets attributed VM power from host
  └─ Does whatever they want with it

Two Separate Problems:

Problem 1: Host-to-VM Attribution (cloud provider's job)

• Host must attribute power to each VM
• Can use CPU time, instructions, ML, whatever
• vrapl shows one simple approach
• Cloud providers can use better algorithms

Problem 2: VM-internal Attribution (user's choice)

• Inside VM, attribute to containers/processes
• User installs Kepler, Scaphandre, or custom
• These tools do their own attribution
• vrapl doesn't interfere

Why This Analysis is Still Valuable:

For Cloud Providers reading this:

• 📖 Learn different attribution approaches
• 📖 Understand trade-offs
• 📖 See that CPU time is simple but instruction-level is more accurate
• 📖 Choose the best algorithm for your infrastructure

For Users reading this:

• 📖 Understand how tools work
• 📖 Learn why instruction-level is better than CPU time
• 📖 Make informed choices about which tool to use in the VM
• 📖 Appreciate the complexity of power attribution

Updated Recommendations:

"Improving vrapl Based on This Analysis"

Correct framing:

For vrapl (the transport):

1. Demonstrate alternative transport mechanisms
2. Show it works regardless of attribution algorithm
3. Let cloud providers choose their algorithm

For cloud providers implementing it:

1. Consider instruction-level attribution (Kepler approach)
2. Handle idle power properly
3. Add memory/IO attribution
4. Use the most accurate algorithm available

For users (inside VM):

1. Once you have the data, install your preferred tool
2. Kepler for container-level + instruction awareness
3. Scaphandre for process-level + simplicity
4. Build custom solutions

The Bottom Line:

This issue teaches: "Here's how different power attribution algorithms work"

vrapl's message: "Cloud providers can pick any of these, just give the result to VMs!"

Ecosystem benefit: Once VMs have the data, users can install tools that do even more sophisticated attribution internally

---

TL;DR: This algorithm analysis is great education, but vrapl is about transport, not algorithms. Cloud providers implement the algorithm (host-side), users choose their tools (guest-side).

See Issue #5 for vrapl's true purpose.