Research: Existing VM Power Monitoring Projects and Comparison

[dipankardas011]

Comprehensive Research: VM Power Monitoring Projects

This issue documents existing projects doing similar work to vrapl and provides a detailed comparison.

---

Open Source Tools (Actively Maintained)

1. Scaphandre ⭐ (Most Similar to vrapl)

• URL: https://github.com/hubblo-org/scaphandre
• Focus: Energy consumption metrology agent
• Architecture:
  • Uses RAPL on host
  • Qemu exporter for VM-specific metrics
  • Shares metrics via mounted filesystem to guest VMs
  • VM can access metrics with --vm flag
• Attribution Method: CPU time-based proportional allocation (similar to vrapl!)
• Status: Active, used in production
• Comparison to vrapl:
  • ✅ Similar: CPU proportional attribution
  • ✅ Similar: RAPL-based measurement
  • ➕ More features: Prometheus exporter, multiple exporters
  • ➖ More complex: Requires filesystem sharing setup
  • vrapl advantage: Simpler, direct serial communication

2. Kepler (Kubernetes Efficient Power Level Exporter) ⭐

• URL: CNCF Sandbox Project
• Focus: Kubernetes container power monitoring
• Architecture:
  • Uses eBPF to probe CPU performance counters
  • ML-based power models for instruction-level estimation
  • Works without direct hardware access (cgroups fallback)
  • Prometheus exporter
• Attribution Method: Per-instruction power estimation (more sophisticated)
• Status: Active CNCF project, growing adoption
• Comparison to vrapl:
  • ➕ More accurate: Tracks individual CPU instructions
  • ➕ Cloud-friendly: Works without RAPL access
  • ➖ More complex: ML models, eBPF requirements
  • ➖ Different scope: Container-focused, not VM-focused
  • vrapl advantage: Direct hardware measurement, simpler

Caution

Currently Version 0.10.0+ the nonRAPL based systems cannot work and it need ML based estimations and there is a lot of ERROR

3. PowerJoular

• URL: https://github.com/joular/powerjoular
• Focus: Multi-platform process power monitoring
• Architecture:
  • Two-step VM approach:
    1. Host monitors VM process power
    2. Shares data file with guest
    3. Guest reads from shared file
  • Supports RAPL, NVIDIA GPUs, Raspberry Pi
• Attribution Method: Process-level monitoring
• Status: Active (v1.0.0+ supports VMs)
• Comparison to vrapl:
  • ✅ Similar: Two-step host/guest approach
  • ➖ Different: File sharing vs serial communication
  • ➕ More platforms: Works beyond x86
  • vrapl advantage: Real-time serial stream, libvirt integration

4. vJoule

• URL: https://github.com/davidson-consulting/vjoule
• Focus: Process energy consumption estimation
• Architecture: Hardware-based measurements for processes
• Status: Available but less documentation
• Comparison to vrapl: Limited VM-specific features

---

Academic Research & Tools

5. BitWatts (EuroSys 2015)

• Paper: "Process-level Power Estimation in VM-based Systems"
• Institution: INRIA, University of Neuchâtel
• Architecture:
  • Distributed actor-based middleware
  • Automatic power model learning
  • Multi-level virtualization support
• Attribution Method: Application-agnostic power model
• Status: Research project
• Comparison to vrapl:
  • ➕ More sophisticated: Multi-level virtualization
  • ➕ Academic validation
  • ➖ Not production-ready
  • vrapl advantage: Simpler, production-ready

6. Joulemeter (Microsoft Research) [DEPRECATED]

• Paper: "Joulemeter: Virtual Machine Power Measurement and Management"
• Focus: VM power measurement through modeling
• Architecture: Software-only power models from resource usage
• Status: ❌ Deprecated (capability moved to Visual Studio)
• Historical Significance: Pioneer in VM power measurement (2009)
• Comparison to vrapl:
  • ➖ Model-based (less accurate than RAPL)
  • ➖ Windows-focused
  • vrapl advantage: Hardware-based measurement

---

Cloud Provider Solutions

7. Cloud Carbon Footprint (Multi-cloud)

• URL: https://www.cloudcarbonfootprint.org/
• Focus: Carbon emissions estimation across AWS, GCP, Azure
• Architecture: API-based, estimates from cloud metrics
• Attribution Method: Model-based estimation
• Status: ⚠️ Uncertain (last release 1+ year ago)
• Comparison to vrapl:
  • ➕ Multi-cloud: Works across providers
  • ➖ No direct measurement: Pure estimation
  • ➖ External tool: Not in-VM
  • vrapl advantage: Direct measurement, real-time

8. AWS Customer Carbon Footprint

• Native AWS tool
• ❌ No direct power metrics to VMs
• Model-based emissions estimates

9. GCP Carbon Footprint

• Hourly emissions tracking
• ❌ No direct power metrics to VMs
• Most detailed of big 3

10. Azure Sustainability Calculator

• Application carbon footprint estimation
• ❌ No direct power metrics to VMs

---

QEMU/KVM Specific

11. QEMU RAPL MSR Support (Built-in)

• Feature: qemu-vmsr-helper
• Status: Official QEMU feature
• Architecture:
  • Exposes host RAPL MSRs to guest
  • Requires privileged helper daemon
  • Guest can read RAPL directly
• Comparison to vrapl:
  • ✅ Similar goal: VM access to RAPL
  • ➕ Official: Part of QEMU
  • ➖ Read-only MSR access: Still needs calculation
  • ➖ Security concerns: Direct MSR exposure
  • vrapl advantage: Attribution done on host, safer

12. kvmtop

• URL: https://cha87de.github.io/kvmtop/
• Focus: Real-time KVM VM monitoring (top-like)
• Metrics: CPU, memory, disk, network
• Status: Active
• ❌ No power monitoring

13. vmtop (DigitalOcean)

• URL: https://github.com/digitalocean/vmtop
• Focus: Real-time KVM monitoring
• ❌ No power monitoring

---

Comparison Matrix

Tool	RAPL Direct	CPU Proportional	VM Support	Cloud Ready	Complexity	Status
vrapl	✅ Yes	✅ Yes	✅ Yes	⚠️ Needs host access	🟢 Low	✅ Working
Scaphandre	✅ Yes	✅ Yes	✅ Yes	⚠️ Needs host access	🟡 Medium	✅ Active
Kepler	⚠️ Fallback	❌ No (instruction-based)	⚠️ Containers	✅ Yes	🔴 High	✅ Active
PowerJoular	✅ Yes	✅ Yes	✅ Yes	⚠️ Needs host access	🟡 Medium	✅ Active
QEMU vRAPL	✅ Yes	❌ No	✅ Yes	⚠️ Needs helper	🟡 Medium	✅ Official
BitWatts	⚠️ Model	⚠️ Learned	✅ Yes	❌ No	🔴 High	⚠️ Research
Joulemeter	❌ Model	❌ Model	✅ Yes	❌ No	🟡 Medium	❌ Deprecated
Cloud Tools	❌ No	❌ Model	✅ Yes	✅ Yes	🟢 Low	✅ Active

---

Key Findings

1. Scaphandre is the Closest Competitor ⭐

• Same core approach: RAPL + CPU proportional attribution
• More mature: Prometheus exporters, production deployments
• More complex setup: Requires qemu exporter + filesystem sharing
• vrapl is simpler and more direct

2. Kepler is the Most Sophisticated

• Instruction-level power estimation via eBPF
• Works in cloud without RAPL access
• Kubernetes/container focused
• Different use case than vrapl

3. Nobody Does It Exactly Like vrapl

Your serial communication approach is unique:

• ✅ Real-time streaming to guest
• ✅ Minimal dependencies (no filesystem sharing)
• ✅ Simple architecture
• ✅ Direct libvirt integration

4. The Industry Needs This

All cloud providers use model-based estimation:

• No direct power metrics exposed to VMs
• AWS, GCP, Azure: All use approximations
• vrapl demonstrates hardware-based measurement is possible

5. Academic Validation Exists

Multiple papers (BitWatts/EuroSys, Joulemeter) validate:

• ✅ CPU proportional attribution works
• ✅ Process-level power estimation is feasible
• ✅ Multi-level virtualization can be monitored

---

vrapl's Unique Position

Strengths

1. Simplest implementation of RAPL VM attribution
2. Serial communication - novel, low-overhead
3. Direct libvirt integration - production-ready
4. Real-time streaming - no polling, no file I/O
5. Educational value - clear, understandable code

Gaps Compared to Competition

1. ❌ No Prometheus exporter (Scaphandre has this)
2. ❌ No multi-VM monitoring (monitors one at a time)
3. ❌ No memory/IO attribution (only CPU)
4. ❌ No web dashboard (PowerJoular has this)
5. ❌ No container support (Kepler's domain)

Opportunities

1. Reference implementation for cloud providers
2. Simpler alternative to Scaphandre for single-VM use
3. Educational tool for understanding VM power
4. Basis for virtio-power device specification

---

Recommendations

Immediate Enhancements

1. Multi-VM support - Monitor N VMs simultaneously
2. Memory attribution - Add RAPL DRAM domain
3. Prometheus exporter - Make metrics standard
4. Comparison benchmark - vrapl vs Scaphandre accuracy

Long-term Vision

1. Collaborate with Scaphandre - Your approach could complement theirs
2. Propose to QEMU - Serial virtio-power device
3. Academic paper - Validate CPU proportional attribution
4. Industry outreach - AWS, GCP, Azure for standardization

---

Conclusions

What We Learned:

1. Your approach is validated - Scaphandre/PowerJoular use similar methods
2. You're not reinventing the wheel - But you have a unique angle
3. There's no standard - Industry needs a common interface
4. Direct measurement beats models - All tools confirm this

vrapl's Role:

vrapl is the simplest, most direct implementation of CPU-proportional VM power attribution using RAPL

It demonstrates that:

• ✅ Hardware-based VM power measurement is feasible
• ✅ CPU proportional attribution is accurate enough
• ✅ Simple architectures can work
• ✅ Cloud providers could expose this natively

Next Steps:

1. Document comparison with Scaphandre
2. Add Prometheus exporter for compatibility
3. Write blog post explaining the approach
4. Propose virtio-power device to QEMU/KVM
5. Submit paper to sustainability conference (e.g., CNCF TAG Environmental Sustainability)

---

References

Tools

• Scaphandre: https://github.com/hubblo-org/scaphandre
• Kepler: https://github.com/sustainable-computing-io/kepler
• PowerJoular: https://github.com/joular/powerjoular
• vJoule: https://github.com/davidson-consulting/vjoule
• Cloud Carbon Footprint: https://www.cloudcarbonfootprint.org/

Academic Papers

• BitWatts (EuroSys 2015): https://dl.acm.org/doi/10.1145/2741948.2741971
• Joulemeter (Microsoft Research): https://www.microsoft.com/en-us/research/publication/joulemeter-virtual-machine-power-measurement-and-management/
• RAPL in Action: https://dl.acm.org/doi/10.1145/3177754

Documentation

• QEMU RAPL MSR: https://www.qemu.org/docs/master/specs/rapl-msr.html
• Linux Power Capping Framework: https://www.kernel.org/doc/html/latest/power/powercap/powercap.html
• CNCF TAG Environmental Sustainability: https://tag-env-sustainability.cncf.io/

[dipankardas011]

CORRECTION: Fair Comparison with Bare Metal Requirements

Important clarification: The original comparison was misleading about cloud provider support. Here's the corrected analysis:

---

The Truth About Bare Metal Requirements

All RAPL-based tools require bare metal or special cloud provider support:

Tool	RAPL Access Required?	Works on Standard Cloud VMs?	Fallback Mode?
vrapl	✅ Yes (bare metal)	❌ No	❌ No
Scaphandre	✅ Yes (bare metal)	❌ No	⚠️ Planned (Issue #25)
Kepler (with RAPL)	✅ Yes (bare metal)	❌ No	✅ Yes (ML estimator)

What This Means:

For Direct Hardware Measurement (Most Accurate):

• ✅ vrapl: Requires bare metal or host access
• ✅ Scaphandre: Requires bare metal or host access
• ✅ Kepler: Requires bare metal or host access

All three are on equal footing when using RAPL!

For Cloud VMs (Standard AWS EC2, GCP Compute Engine):

• ❌ vrapl: Does not work
• ❌ Scaphandre: Does not work (Issue #25 open since 2020 requesting this)
• ⚠️ Kepler: Falls back to ML estimator (less accurate, experimental)

---

Kepler 0.10+ Improvements

Version 0.10 Release (Complete Rewrite)

New Features:

• Dynamic detection of RAPL zones (no hardcoded zones)
• More accurate power attribution based on active CPU usage
• Improved VM, Container, and Pod detection
• Better label values for identification

BUT the fundamental limitation remains:

• Still requires RAPL for accurate measurement
• Without RAPL, falls back to ML estimator
• Estimator is marked as "experimental, not supported" (Red Hat docs)

Kepler Estimator Accuracy

From recent research (2024):

"Kepler's metrics are not accurate where total cluster power is concerned... the estimates exhibit consistency for showing variations between runs of the same workload, but they are approximations of actual power consumption."

Key Finding: Kepler's estimator shows trends well but is not accurate for absolute power values.

---

Scaphandre Cloud VM Support

Current Status (Issue #25)

Scaphandre has an open issue since December 2020 titled:

"Integrate power consumption estimation on public cloud virtual machines as a degraded mode"

Proposed Solutions:

1. Integrate statistical models (like Etsy's cloud-jewels for GCP)
2. Implement ratio-based approach similar to powertop
3. Mix with provider hardware information

Status: Still open, needs help, no implementation yet

Quote from issue:

"Until the cloud provider does install scaphandre on its hypervisors, we should enable cloud customers to estimate their power usage and thus their climate impact."

---

Corrected Comparison Matrix

Mode 1: Direct RAPL Measurement (Bare Metal)

Feature	vrapl	Scaphandre	Kepler
Bare Metal Required	✅ Yes	✅ Yes	✅ Yes
RAPL Access	✅ Direct	✅ Direct	✅ Direct
Attribution Method	CPU proportional	CPU proportional	eBPF + CPU counters
Accuracy	🟢 High	🟢 High	🟢 High
VM Support	✅ Yes	✅ Yes	⚠️ Containers
Complexity	🟢 Low	🟡 Medium	🔴 High
Communication	Serial	Filesystem	Prometheus
Dependencies	libvirt, gopsutil	Multiple exporters	eBPF, K8s, ML

Verdict: All three require bare metal. vrapl is simplest, Kepler is most sophisticated.

Mode 2: Cloud VM Estimation (No RAPL)

Feature	vrapl	Scaphandre	Kepler Estimator
Works on Cloud VMs?	❌ No	❌ No	✅ Yes
Measurement Type	N/A	N/A	ML estimation
Accuracy	N/A	N/A	⚠️ Low (trends only)
Status	N/A	Planned (Issue #25)	⚠️ Experimental
Production Ready?	N/A	❌ No	❌ No (per Red Hat)

Verdict: Only Kepler has a fallback, but it's not accurate or production-ready.

---

What vrapl Actually Competes With

For Bare Metal / Private Cloud (Direct RAPL):

• ✅ Scaphandre: Same approach, more features, more complex
• ✅ Kepler: Different approach (eBPF), container-focused
• ✅ PowerJoular: Similar approach, multi-platform

For Public Cloud (No RAPL):

• ❌ vrapl doesn't compete here (no estimator mode)
• ⚠️ Kepler has experimental estimator
• ❌ Scaphandre doesn't compete here yet

---

vrapl's Strengths (Fair Assessment)

1. Simplicity ⭐⭐⭐

• Simplest RAPL VM attribution tool
• ~200 lines of Go vs thousands in Scaphandre/Kepler
• Easy to understand and modify

2. Novel Approach ⭐⭐⭐

• Serial communication is unique
• No filesystem sharing needed
• Real-time streaming

3. Direct Integration ⭐⭐

• Built-in libvirt integration
• Automatic PID discovery
• Production-ready code

4. Accuracy ⭐⭐⭐

• Same accuracy as Scaphandre (both use CPU proportional)
• Better than Kepler estimator (when Kepler lacks RAPL)
• Hardware-based measurement

5. Educational Value ⭐⭐⭐

• Clear, readable code
• Demonstrates the concept perfectly
• Good reference implementation

---

Where vrapl Falls Short

1. No Cloud Fallback

• Scaphandre: Planned (Issue #25)
• Kepler: Has it (experimental)
• vrapl: No estimator mode

2. Single VM Only

• Can only monitor one VM at a time
• Scaphandre/Kepler: Monitor all containers/VMs

3. Limited Metrics

• Only CPU power
• Scaphandre/Kepler: Also track memory, GPU

4. No Standard Exports

• Serial output only
• Scaphandre/Kepler: Prometheus, JSON, etc.

---

Corrected Conclusions

What We Actually Learned:

1. vrapl, Scaphandre, and Kepler all need bare metal for RAPL ✅
2. Only Kepler has a cloud fallback (but it's not accurate) ⚠️
3. Scaphandre has been trying to add cloud support since 2020 (Issue #25)
4. Standard cloud VMs (AWS EC2, GCP Compute) don't expose RAPL ❌

vrapl's True Position:

vrapl is the simplest bare-metal RAPL VM power attribution tool

Compared to Scaphandre (closest competitor on bare metal):

• ✅ Simpler architecture
• ✅ Novel serial communication
• ✅ Easier to deploy
• ➖ Fewer features
• ➖ No Prometheus export
• ➖ Single VM only

Compared to Kepler:

• ✅ Much simpler
• ✅ Direct measurement (no ML)
• ✅ VM-focused (Kepler is container-focused)
• ➖ No cloud fallback
• ➖ No eBPF sophistication

The Real Opportunity:

None of these tools solve the cloud provider problem perfectly:

• Scaphandre: Doesn't work on cloud VMs (yet)
• Kepler: Works but with low accuracy
• vrapl: Doesn't work on cloud VMs

This validates the vision in Issue #2: Cloud providers need to expose power metrics natively. All current tools are workarounds.

---

Updated Recommendations

Immediate (Maintain Parity):

1. Add Prometheus exporter (compatibility with Scaphandre/Kepler)
2. Multi-VM support (monitor N VMs simultaneously)
3. Add DRAM RAPL domain (memory attribution)

Medium-term (Differentiation):

1. Benchmark accuracy vs Scaphandre (prove CPU proportional works)
2. Add estimator mode for cloud VMs (learn from Kepler)
3. Write comparison blog post

Long-term (Industry Impact):

1. Propose virtio-power device to QEMU
2. Collaborate with Scaphandre on serial transport
3. Submit to CNCF TAG Environmental Sustainability
4. Academic paper on CPU proportional attribution accuracy

---

References

• Scaphandre Issue #25: Integrate power consumption estimation on public cloud virtual machines as a degraded mode hubblo-org/scaphandre#25
• Kepler 0.10 Release: https://github.com/sustainable-computing-io/kepler/releases
• Container-level Energy Observability (2024): https://arxiv.org/html/2504.10702v1
• Kepler Power Model Docs: https://sustainable-computing.io/design/power_model/

[dipankardas011]

🔄 IMPORTANT REFRAMING

This research is valuable, but the positioning needs correction. See Issue #5 for vrapl's true purpose.

What This Issue Got Wrong:

This positions vrapl as a competitor to Scaphandre/Kepler, comparing features and finding gaps.

The Correct Understanding:

vrapl is NOT competing with these tools - it's ENABLING them to work in cloud environments!

❌ WRONG: "vrapl vs Scaphandre vs Kepler" (competitors)
✅ RIGHT: "vrapl enables Scaphandre + Kepler" (enabler)

Corrected Comparison:

On Bare Metal (Current):

Tool	Works?	Accuracy
Scaphandre	✅ Yes	🟢 High
Kepler	✅ Yes	🟢 High
vrapl	✅ Yes	🟢 High

All three work fine on bare metal. No competition here.

On Cloud VMs (Problem):

Tool	Works?	Accuracy
Scaphandre	❌ No	N/A
Kepler	⚠️ ML fallback	🔴 Low (trends only)
vrapl	❌ No (host-side only)	N/A

Problem: No RAPL access in guest!

With vrapl's Transport (Solution):

Tool	Works?	Accuracy
Scaphandre	✅ Yes	🟢 High (hardware data)
Kepler	✅ Yes	🟢 High (hardware data)
Any tool	✅ Yes	🟢 High (hardware data)

vrapl shows cloud providers how to expose RAPL, enabling all tools!

What's Still Valuable:

All the research on these tools is excellent! The information about:

• ✅ How Scaphandre works
• ✅ How Kepler works
• ✅ What features they have
• ✅ Their limitations

This is all useful for understanding the ecosystem.

The Real Insight:

The problem isn't the tools - it's the infrastructure!

• Scaphandre Issue #25: "Need cloud VM support" (open since 2020)
• Kepler: Works but uses inaccurate ML estimator
• PowerJoular: Doesn't work on cloud VMs

They're all trying to solve the same problem: No RAPL in cloud VMs

vrapl's Unique Position:

Scaphandre:  ┐
Kepler:      ├─ Power monitoring/attribution tools
PowerJoular: ┘  (Choose your favorite)
                       ↑
                       │ needs data from
                       ↓
vrapl:           Transport mechanism
                 (Exposes RAPL to VM)
                       ↑
                       │ reads from
                       ↓
Cloud Host:      Hardware RAPL

Updated "Competition":

vrapl doesn't compete with monitoring tools. It competes with:

• ❌ Status quo: No RAPL in cloud VMs
• ❌ ML estimators: Inaccurate approximations
• ❌ Model-based approaches: No hardware data

vrapl's competitors are actually:

• QEMU vRAPL (MSR exposure - different approach)
• Cloud provider APIs (if they existed)
• virtio-power device (doesn't exist yet - vrapl proposes it)

Corrected Recommendations:

1. Add Prometheus exporter (to compete with Scaphandre)
2. Add eBPF support (to compete with Kepler)

✅ Correct recommendations:

1. Demonstrate alternative transport mechanisms (virtio, sysfs)
2. Show guest-side integration examples (how to consume the data)
3. Propose to cloud providers
4. Write virtio-power spec
5. Let users choose Kepler/Scaphandre/etc in the VM

The Big Picture:

Once cloud providers adopt vrapl's approach (or similar):

• ✅ Scaphandre works in cloud (solves their Issue #25)
• ✅ Kepler gets accurate data (no ML needed)
• ✅ PowerJoular works in cloud
• ✅ New tools can be built
• ✅ Users choose what they want

vrapl enables the ecosystem, it doesn't replace it.

See Issue #5 for the complete corrected framing.