Xen: assigning PCI devices to a domain
From BeSTGRID
In Xen, it is possible to assign a PCI device to a user domain. As there some tricks were necessary on top of the instructions in the Xen User's Manual, I decided to write them down here.
Contents |
[edit] Basics & System requirements
To assign a PCI device to a domain, Xen uses a PCI frontend and backend driver. In dom0, the device must be controlled by the PCI backend driver, and when a user domain can be assigned such devices at the time it is created. For that, the user domain kernel must have the PCI frontend driver, and of course the PCI driver for the specific device. The standard Xen dom0 kernel has all the necessary drivers (backend, frontend and device drivers), and I recommend running this kernel in a user domain which needs direct access to PCI devices. The CentOS distribution xenU kernel does not have the frontend and PCI device drivers.
[edit] Exporting a device
It is generally not necessary to reboot dom0 to export a device. It is sufficient just to:
- Determine the PCI Id of the device (short form 00:1d.7 with lspci, and long form 0000:00:1d.7 by simply prepending "0000:" to the short form)
- Make sure the device is not controlled by any driver: there should be no driver symlink in /sys/bus/pci/devices/nnnn:nn:nn.n
- Load the pciback driver, and pass it the list of devices to control in the hide parameter:
modprobe pciback 'hide=(0000:00:1d.0)(0000:00:1d.1)(0000:00:1d.2)(0000:00:1d.3)(0000:00:1d.7)'
[edit] Assigning the device to a domain
The devices to be assigned are passed with the pci=nn:nn.nn parameter to xm create (and the parameter can occur multiple times). The only requirement is that the kernel has the PCI frontend (which, e.g., the dom0 kernel has).
Thus, a domain should be created with
xm create -c NGTest-dom0kernel pci=00:1d.0 pci=00:1d.1 pci=00:1d.2 pci=00:1d.3 pci=00:1d.7
After it boots, the device(s) should be visible with lspci and in /sys/bus/pci/devices/.
Alternatively, one may achieve the same effect by adding the list of PCI devices to the domain configuration file:pci = [ '00:1d.0', '00:1d.1', '00:1d.2', '00:1d.3', '00:1d.7' ]
[edit] Hints and troubleshooting
- The pciback driver will silently ignore any failure to grab the device (if already controlled by a driver). Check the output of dmesg: you should see:
pciback 0000:00:1d.0: seizing device pciback 0000:00:1d.1: seizing device pciback 0000:00:1d.2: seizing device pciback 0000:00:1d.3: seizing device pciback 0000:00:1d.7: seizing device
- PCI permissions: you may have to grant extra permissions to let the domain write to the devices PCI configuration registers - either by listing it in the quirks or permissive list. See the Xen User's Manual (page 27) for more information - I have not done this myself.
- To check if a user domain has the PCI frontend driver: check the kernel boot messages. The output of dmesg should include:
pcifront pci-0: Installing PCI frontend pcifront pci-0: Creating PCI Frontend Bus 0000:00
If it instead contains
kernel: XENBUS: Device with no driver: device/pci/0
then the user domain is getting the PCI devices as a xen virtual PCI bus, but does not have the frontend driver available.
