How to choose specific address for PCI Passthrough

Community Support
1

Hi

We are playing around with the PCI Passthrough using Mellanox network cards. In our case we have 2 NICs available in our hyps. The first one is used by the hypervisor byt the system and the second one should be attached to a VM via PCI PT. The problem that we found was that the class, device and vendor are the same for those cards:

PCI=[
CLASS=“15b3”,
DEVICE=“1004”,
VENDOR=“15b3” ]

The only difference is the address 0000:08:00.0 instead 0000:07:00.0
But the scheduler chooses the first device available, 07 and we do not want that. It is possible to force to use an specific address from the PCI section when the class,device and vendor are the same between two devices? like ADDRESS=“0000:08:00.0”?

Or should we change the filter from /var/lib/one/remotes/im/kvm-probes.d/pci.rb to just show an specific address to OpenNebula scheduler?

Thanks in advance
Álvaro

2

Hi

Is there a solution for this yet?
We also need to select between different PCI devices with the same vendor/device/class values.

Moritz

3

Hi @stratster

What we did was just modify the current pci probe from /var/lib/one/remotes/im/kvm-probes.d/pci.rb (in one 5.6.1) with these changes:

$ diff /var/lib/one/remotes/im/kvm-probes.d/pci.rb /var/lib/one/remotes/im/kvm-probes.d/pci.rb.orig

87,98d86
< # vendor / device => incr value per different bus
< rename = {
<   '15b3' => { # Mellanox
<     '1003' => '1000', # connectx3
<     '1004' => '1000', # connectx3 VF
<     '1013' => '1000', # connectx4
<     '1014' => '1000', # connectx4 VF
<   }
< }
< 
< count = {}
< 
110,133d97
<     type = dev[:type]
<     device = dev[:device]
<     vendor = dev[:vendor]
<     if rename.key?(vendor) and rename[vendor].key?(device)
<       if not count.key?(vendor)
<         count[vendor] = {}
<       end
<       if not count[vendor].key?(device)
<         count[vendor][device] = {}
<       end
< 
<       # keep track of devices per bus
<       bus = dev[:bus]
<       if count[vendor][device].key?(bus)
<         device = count[vendor][device][bus]
<       else
<         deviceint = device.hex + rename[vendor][device].hex * count[vendor][device].length
<         newdevice = "%04x" % deviceint
<         count[vendor][device][bus] = newdevice
<         device = newdevice
<       end
<       type = [dev[:vendor], device, dev[:class]].join(':')
<     end
< 
136c100
<         pval('TYPE', type),
---
>         pval('TYPE', dev[:type]),
139c103
<         pval('DEVICE', device),
---
>         pval('DEVICE', dev[:device]),

That provides a different address if you have several devices from the same vendor (in our case mellanox cards) so you can choose a specific one.

Cheers
Álvaro

4

Hi Álvaro

Thanks for your quick response :slight_smile:
But i think your approach does not work for us.

In your case “the user”, the one creating the VM template, is in charge of selecting the specific device.
We don’t want that. The user should only be able to query devices based on vendor/device/class values.

Our own scheduler then decides which host and device is chosen for the VMs and needs to set the specific device address when it deploys the VM on the host.