Monday, May 28, 2012

VGA Passthrough KVM Fedora 17 and Benchmarks

Check out Arch Forum, the method in this guide is beeing deprecated in favor of vfio, which at this point requires a bit more work but it might work better for you.


Well how do I start?
"Huston, the Eagle has landed"


I think this is the first time in the last 12 month since I "felt the urge" to do this, that I feel that I've done it, that VGA Passthrough is not only possible, but it also stable enough (99% of the time, although a reboot is sometime necessary) for daily use:
- USB passthrough is working fine
- the sound controller passthrough is also working fine
- VGA passthrough is alright and well I just played Crysis 1280x1024 all setting Ultra High without any problems.

Once you have the hardware, the knowledge is out there (and here), you just need to play with it and make it work for you.

Howto 

The latest version of the guide is at: https://docs.google.com/document/d/1ef_nfl652L0HLn_wGvnpgjsBJd9LZzaV_-rIcEEoK8Y/edit


Benchmarks

Proper benchmarking is hard, and I might have done something wrong, because almost all the benchmarks are showing >90% of native Windows performance, except Crysis (I played it in the vm with all setting on ultra high and it was fine, at least in my humble opinion, so i don’t know why I got so low result in the benchmark ) and Stawker Call of Prypiat (this is a total mistery).

You can find the results at: https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFdOLU1Yc0trT29JdUxac21QcXg3Qnc#gid=6

You can always do your own ;)

Q&A 
From the last post about VGA passthrough.

Anonim1
    Xen benchmarks, well I think I've had enough of using 'sombody's else code' for the time beeing, but I wouldn't exclude a Xen comparison sometime in the future, if I have the time.
Anonim2
    https://help.ubuntu.com/community/KVM/Installation this + intel_iommu=on to grub (grub2-update), and not installing ubuntu with uefi should be the same thing as fedora, I don't have the time to test ubuntu right now.
Evan Fraser
    Check how many cpus does the VM see? If you get the same bad performance no mather what video card you use, maybe the game is cpu bound that's where I would start.

19 comments:

  1. Hi!

    I've tried to passthrough my HD6930 inside Win7 guest on KVM, and so far it seems that its working. But, when i trying to turn off guest, the host freeze completely. Do u know how to solve this problem?

    My hardware CPU->Phenom2 1090T+MB->GA990FX-D3 (IOMMU enabld in BIOS)

    OS: Fedora 17

    ReplyDelete
  2. Hi, have you looked at the logs? Is there something strange there?

    ReplyDelete
  3. Hi!

    Ok, i tried to look at logs after freeze in recovery mode, so far it seems nothing suspicious in /var/log/messages.

    Also i cant pass to the guest my pci ati audio device (01:00.1). When i'm add device in virt-manager and launch vm, virt-manager hangs, dunno why. Only 01:00.0 HD6930 passed to the guest.

    Also i didn't blacklist my radeon driver cause i do have 2 videocards inside(X1950PRO and HD6930).

    My kernel 3.4.0-1.fc17.x86_64
    Libvirt 0.9.11.3
    qemu-kvm 1.0
    virt-manager 0.9.11

    Any guesses?

    ReplyDelete
  4. Hi, can you blacklist the radeon module, or temporarily unload it mdeprobe -r radeon before starting the VM or trying to add the audio, just for testing and see if you have the same problems. I'm afraid I haven't had this sort of problems.

    ReplyDelete
  5. i tried to passthrough HD4850 to my win7 vm on fedora 17 KVM, and got BSOD in guest win7 OS.

    ReplyDelete
  6. Well, I just installed Fedora Desktop 17 and trying to do so.
    I've just disabled Xorg by switching from runlevel 5 to 3:
    http://superuser.com/questions/364260/how-to-disable-x-window-on-fedora-15
    now I'm trying to do:
    sudo modprobe -r radeon
    FATAL: Module radeon is in use.
    however, it is already blacklisted:
    cat /etc/modprobe.d/blacklist.conf | grep radeon
    blacklist radeonfb
    blacklist radeon

    ReplyDelete
  7. I've managed to remove driver from memory with:
    /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --install $(uname -r)
    after blacklisting it, now I'm fighting with problem how to unbind card from the host:
    błąd: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
    do_spice_init: starting 0.10.1
    spice_server_add_interface: SPICE_INTERFACE_MIGRATION
    spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
    spice_server_add_interface: SPICE_INTERFACE_MOUSE
    spice_server_add_interface: SPICE_INTERFACE_QXL
    red_worker_main: begin
    display_channel_create: create display channel
    cursor_channel_create: create cursor channel
    spice_server_add_interface: SPICE_INTERFACE_PLAYBACK
    spice_server_add_interface: SPICE_INTERFACE_RECORD
    Failed to assign device "hostdev1" : Device or resource busy
    *** The driver 'pci-stub' is occupying your device 0000:01:00.0.
    ***
    *** You can try the following commands to free it:
    ***
    *** $ echo "1002 9460" > /sys/bus/pci/drivers/pci-stub/new_id
    *** $ echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/unbind
    *** $ echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind
    *** $ echo "1002 9460" > /sys/bus/pci/drivers/pci-stub/remove_id
    ***
    qemu-kvm: -device pci-assign,host=01:00.0,

    ReplyDelete
  8. Ok, I missed some things from:
    http://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#PCI_device_assignment
    Now, according to Your entry:
    If you get “Connection reset by peers” when booting up the VM whit a pci device attached IOMMU might not be enabled in the BIOS, grub, or you installed using UEFI (dmesg | grep -e DMAR -e IOMMU shows only IOMMU).

    But IOMMU is enabled in BIOS (it's AWARD), also Virtualization is Enabled.
    I'm getting:
    [ 0.000000] Please enable the IOMMU option in the BIOS setup
    [ 1.166451] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40

    ReplyDelete
  9. Well, I've tried with Xen and it also failed.
    Have no idea what next.. thx for great how-to anyway :(

    ReplyDelete
  10. Sorry for the late response, If you do dmesg | grep AMD-Vi do you get AMD-Vi: Initialized for Passthrough Mode ? AMD uses different kernel parameter than Intel, on AMD you need to pass "iommu=pt iommu=1" - from the kvm wiki, you have to put this on grub command line, if you haven't already done this.

    AMD Machine
    dmesg | grep AMD-Vi

    ...
    AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
    AMD-Vi: Lazy IO/TLB flushing enabled
    AMD-Vi: Initialized for Passthrough Mode
    ...

    ReplyDelete
    Replies
    1. I had only:
      AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
      AMD-Vi: Initialized for Passthrough Mode
      via DMESG..

      Delete
    2. That is exactly what I get and I'm having the same problem. It seems the "AMD-Vi: Lazy IO/TLB flushing enabled" line is missing. What is required to enable that line. I'm not above compiling a custom kernel build but I don't like going that route unless necessary.

      Delete
    3. I got it to work on my Fedora 16 KVM host! I needed to pass an argument to the KVM kernel module "options kvm allow_unsafe_assigned_interrupts=1" I created the file "/etc/modprobe.d/kvm.conf" and inserted the above quoted line and it made all the device redirection start working. I hope this helps someone else as I was stumped on this for days.

      Delete
    4. I've tried again, because IT WORKS on ESXi 5.0 (won't work with 5.1!! important)
      However, I don't know GPU assign is failing with Fedora 17 all-the-time..
      New posts at:
      http://toomeek.waw.pl/blog/?p=2296
      http://toomeek.waw.pl/blog/?p=2285
      and for ESXi 5.0
      http://toomeek.waw.pl/blog/?p=2260

      Delete
    5. .. but I'm also trying to assign 2 x GPU cards to 2 x VMs ..
      without luck.

      Delete
  11. Thanks for sharing your how-to!!! I wish I had seen it earlier, but now I have everything setup on Xen and it works nicely. Is there a reason you chose KVM over Xen? By the way, I agree with you on the state of KVM documentation, which was one reason I chose Xen. Still, I'd like to give KVM a try and perhaps compare the two.

    ReplyDelete
  12. I chose KVM because I couldn't figure out how to make VGA passthrough work with XEN and because KVM was quite easy to set up, and there are other applications I use that benefit from KVM and Qemu.

    ReplyDelete
  13. I managed to blacklist the radeon driver but it seems impossible to assign my graphic card.

    When I use virt-manager and try to add the radeon 5850 with its hdmi module, it ends with:
    "Error starting domain: Unable to read from monitor: Connection rest by peer." And some python errors.

    If I try this:
    qemu-kvm -m 512 -boot c -net none -hda /dev/fedora/WinVM -device pci-assign,host=01:00.0 -device pci-assign,host=01:00.1
    Failed to assign device "(null)" : Device or resource busy
    *** The driver 'pci-stub' is occupying your device 0000:01:00.0.
    ***
    *** You can try the following commands to free it:
    ***
    *** $ echo "1002 68a1" > /sys/bus/pci/drivers/pci-stub/new_id
    *** $ echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/unbind
    *** $ echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind
    *** $ echo "1002 68a1" > /sys/bus/pci/drivers/pci-stub/remove_id
    ***
    qemu-kvm: -device pci-assign,host=01:00.0: Device 'pci-assign' could not be initialized


    And if I blacklist pci-stub:
    qemu-kvm -m 512 -boot c -net none -hda /dev/fedora/WinVM -device pci-assign,host=01:00.0 -device pci-assign,host=01:00.1
    Failed to assign device "(null)" : Device or resource busy
    Couldn't find out why.
    qemu-kvm: -device pci-assign,host=01:00.0: Device 'pci-assign' could not be initialized


    Do you have any idea whats wrong?

    ReplyDelete
  14. Any plans on trying this with Fedora 19/20 and kernel 3.12+ with vfio, yet?
    Like here: https://bbs.archlinux.org/viewtopic.php?id=162768
    I'm using your howto on F19, but I can't figure out how to use vfio instead^^

    ReplyDelete

I reserve the right to delete at any time any comment I deem inappropiate.