pci_resource_assignment: static BAR address assignment for P2P DMA#3671
Open
jstarks wants to merge 16 commits into
Open
pci_resource_assignment: static BAR address assignment for P2P DMA#3671jstarks wants to merge 16 commits into
jstarks wants to merge 16 commits into
Conversation
Allow specific PCI BARs to be pinned to their physical addresses so that VFIO passthrough devices can participate in peer-to-peer DMA without ATS. When a VFIO device is configured with `bar0=pt` (through `bar5=pt`) on the CLI, its virtual BAR is pre-programmed with the host physical BAR address read from sysfs. The `preserve_bars` flag on the root complex tells the resource allocator to treat non-zero BAR values found during probing as fixed--pinned BARs stay at their physical addresses (GPA=HPA) while dynamic BARs are bump-allocated after the pinned region. The allocation algorithm extends the existing hierarchical bottom-up sizing / top-down assignment approach. During sizing, pinned BARs establish a constrained base for their bridge window; dynamic demands are summed on top. During assignment, pinned demands are placed at their fixed addresses and dynamic demands bump-allocate forward from the highest pinned end. Bridges with pinned descendants propagate their position constraint upward so parent windows are correctly sized and placed. When no BARs are pinned, the algorithm produces identical results to the previous code. New CLI surface: - `--pcie-root-complex` gains `low_mmio_base=`, `high_mmio_base=` (producing Fixed MMIO ranges), and `preserve_bars` - `--vfio` gains `bar0=pt` through `bar5=pt` Validation rejects pinned BARs that are misaligned, overlapping, or outside the MMIO aperture. Physical BAR addresses are read from the kernel's `/sys/bus/pci/devices/<bdf>/resource` table rather than VFIO config space, which returns cleared BARs after device reset.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds support for “pinned” PCI BAR address assignment to enable VFIO passthrough devices to participate in peer-to-peer DMA (GPA=HPA) without ATS. It introduces CLI/config surface to (a) pre-program VFIO virtual BARs with host physical BAR addresses and (b) have the PCI resource allocator preserve (treat as fixed) non-zero BAR values discovered during probing, extending the allocator to size/assign bridge windows with pinned descendants.
Changes:
- Add
preserve_barsplumbing from CLI/config → topology → resource assignment to treat discovered non-zero BARs as pinned/fixed addresses. - Add VFIO
bar0=pt..bar5=ptflags to seed virtual BARs from host/sys/bus/pci/devices/<bdf>/resource. - Extend the PCI resource assignment algorithm and tests to validate and place pinned BARs, and to size/position bridge windows accordingly.
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| vmm_core/src/acpi_builder.rs | Adds TODO notes for ACPI behavior when firmware BAR assignments must be preserved. |
| vm/vmcore/vm_topology/src/pcie.rs | Extends PcieHostBridge with preserve_bars. |
| vm/devices/pci/vfio_assigned_device/src/resolver.rs | Threads bar_pt through VFIO device resolver paths. |
| vm/devices/pci/vfio_assigned_device/src/lib.rs | Implements BAR “passthrough” seeding from sysfs resource table and uses it for initial virtual BAR state. |
| vm/devices/pci/vfio_assigned_device_resources/src/lib.rs | Extends VFIO handle resources with bar_pt. |
| vm/devices/pci/pci_resource_assignment/src/tests.rs | Adds extensive pinned-BAR unit tests and updates params with preserve_bars. |
| vm/devices/pci/pci_resource_assignment/src/lib.rs | Adds preserve_bars parameter and new pinned-BAR error variants. |
| vm/devices/pci/pci_resource_assignment/src/enumerate.rs | Captures pre-probe BAR values as pinned_address when preserve_bars is enabled. |
| vm/devices/pci/pci_resource_assignment/src/assign.rs | Updates sizing/assignment to account for fixed-position (pinned) demands; adds pinned BAR validation helpers. |
| petri/src/vm/openvmm/modify.rs | Wires new preserve_bars field into Petri OpenVMM config generation. |
| openvmm/openvmm_entry/src/lib.rs | Adds low_mmio_base / high_mmio_base fixed MMIO range handling and threads preserve_bars; passes bar_pt into VFIO resources. |
| openvmm/openvmm_entry/src/cli_args.rs | Adds CLI parsing for low_mmio_base, high_mmio_base, preserve_bars, and --vfio barX=pt. |
| openvmm/openvmm_entry/Cargo.toml | Adds memory_range dependency for fixed MMIO range construction. |
| openvmm/openvmm_defs/src/config.rs | Adds preserve_bars to PcieRootComplexConfig. |
| openvmm/openvmm_core/src/worker/memory_layout.rs | Updates test configs to include preserve_bars. |
| openvmm/openvmm_core/src/worker/dispatch/ecam_config_access.rs | Plumbs preserve_bars into pci_resource_assignment::AssignmentParams. |
| openvmm/openvmm_core/src/worker/dispatch.rs | Threads preserve_bars from config into PcieHostBridge. |
| Cargo.lock | Records new workspace dependency usage (memory_range). |
jstarks
commented
Jun 9, 2026
jstarks
commented
Jun 9, 2026
jstarks
commented
Jun 9, 2026
shenw0000
reviewed
Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow specific PCI BARs to be pinned to their physical addresses so that VFIO passthrough devices can participate in peer-to-peer DMA without ATS. When a VFIO device is configured with
bar0=pt(throughbar5=pt) on the CLI, its virtual BAR is pre-programmed with the host physical BAR address read from sysfs. Thepreserve_barsflag on the root complex tells the resource allocator to treat non-zero BAR values found during probing as fixed--pinned BARs stay at their physical addresses (GPA=HPA) while dynamic BARs are bump-allocated after the pinned region.The allocation algorithm extends the existing hierarchical bottom-up sizing / top-down assignment approach. During sizing, pinned BARs establish a constrained base for their bridge window; dynamic demands are summed on top. During assignment, pinned demands are placed at their fixed addresses and dynamic demands bump-allocate forward from the highest pinned end. Bridges with pinned descendants propagate their position constraint upward so parent windows are correctly sized and placed. When no BARs are pinned, the algorithm produces identical results to the previous code.
When
preserve_barsis set, the ACPI SSDT emits a_DSMmethod telling the guest OS to preserve firmware-assigned BAR values, and the device tree path setslinux,pci-probe-onlyfor the same effect.New CLI surface:
--pcie-root-complexgainslow_mmio_base=,high_mmio_base=(producing Fixed MMIO ranges), andpreserve_bars--vfiogainsbar0=ptthroughbar5=ptValidation rejects pinned BARs that are misaligned, overlapping, or outside the MMIO aperture. Physical BAR addresses are read from the kernel's
/sys/bus/pci/devices//resourcetable rather than VFIO config space, which returns cleared BARs after device reset.