Get those FSYNC numbers up on your ZFS pool

For the last week, I've been trying to figure out why our 10 drive ZFS zpool has been delivering such lousy NFS performance Proxmox KVM cluster.

Here's what pveperf was returning:

pveperf /mnt/pve/kvm-images/
CPU BOGOMIPS:      76608.87
REGEX/SECOND:      896132
HD SIZE:           7977.14 GB (xxx.xxx.xxx.xxx:/volumes/vol0/kvm-images)
FSYNCS/SECOND:     23.15
DNS EXT:           58.84 ms
DNS INT:           1.50 ms (my.company.com)

The zpool looked like this:

zpool status vol0
  pool: vol0
 state: ONLINE
 scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        vol0                       ONLINE       0     0     0
          raidz2-0                 ONLINE       0     0     0
            c0t5000C50010377B5Bd0  ONLINE       0     0     0
            c0t5000C5001037C317d0  ONLINE       0     0     0
            c0t5000C5001037EED7d0  ONLINE       0     0     0
            c0t5000C50010381737d0  ONLINE       0     0     0
            c0t5000C50010381BBBd0  ONLINE       0     0     0
            c0t5000C50010382777d0  ONLINE       0     0     0
            c0t5000C5001038291Fd0  ONLINE       0     0     0
            c0t5000C500103870A3d0  ONLINE       0     0     0
            c0t5000C500103871C3d0  ONLINE       0     0     0
            c0t5000C500103924E3d0  ONLINE       0     0     0
            c0t5000C500103941F7d0  ONLINE       0     0     0
        cache
          c0t50015179591D9AEFd0    ONLINE       0     0     0
          c0t50015179591DACA1d0    ONLINE       0     0     0
          c1t2d0                   ONLINE       0     0     0
        spares
          c0t5000C50010395057d0    AVAIL   

errors: No known data errors

Raw write speed wasn't a problem. Tests of copying DVD iso files were supper fast over the 10G network backbone. But the performance of creating new files and folders really hurt. This was very apparent when I started using bonnie++ on the NFS shares from the Proxmox nodes. Bonnie++ zipped along until it started its "Create files..." tests. The Linux client would practically lock up.

So a little Goolge ZFS keyword searching later, I came across Joe Little's blog post, ZFS Log Devices: A Review of the DDRdrive X1. This got me thinking about my zpool setup. Looking at the configuration again, I realized that I'd made a mistake and added the second Intel X25M SSD to the cache pool instead of the log pool. :)

Thanks to ZFS awesomeness it was real easy to pull the SSD out of the cache and designate it as part of the log pool. No down time for the production system and no wasted weird weekend hours staring at glowing terminal console.

Oh man, did that make a difference in performance.

Here's what the reconfigured vol0 zpool looks like:

zpool status vol0
  pool: vol0
 state: ONLINE
 scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        vol0                       ONLINE       0     0     0
          raidz2-0                 ONLINE       0     0     0
            c0t5000C50010377B5Bd0  ONLINE       0     0     0
            c0t5000C5001037C317d0  ONLINE       0     0     0
            c0t5000C5001037EED7d0  ONLINE       0     0     0
            c0t5000C50010381737d0  ONLINE       0     0     0
            c0t5000C50010381BBBd0  ONLINE       0     0     0
            c0t5000C50010382777d0  ONLINE       0     0     0
            c0t5000C5001038291Fd0  ONLINE       0     0     0
            c0t5000C500103870A3d0  ONLINE       0     0     0
            c0t5000C500103871C3d0  ONLINE       0     0     0
            c0t5000C500103924E3d0  ONLINE       0     0     0
            c0t5000C500103941F7d0  ONLINE       0     0     0
        logs
          c1t2d0                   ONLINE       0     0     0
        cache
          c0t50015179591D9AEFd0    ONLINE       0     0     0
          c0t50015179591DACA1d0    ONLINE       0     0     0
        spares
          c0t5000C50010395057d0    AVAIL   

errors: No known data errors

Now ZFS can properly feed all of the Linux FSYNC disk requests. Check out the Proxmox performance test improvements.

pveperf /mnt/pve/kvm-images/
CPU BOGOMIPS:      76608.87
REGEX/SECOND:      896132
HD SIZE:           7977.14 GB (xxx.xxx.xxx.xxx:/volumes/vol0/kvm-images)
FSYNCS/SECOND:     1403.21
DNS EXT:           58.84 ms
DNS INT:           1.50 ms (my.company.com)

Get your CentOS 5.5 mouse to behave as Linux KVM guest

Just spent 30 minutes trying to figure out why CentOS 5.5 wasn't playing nice with QEMU/KVM's USB tablet emulator.

All you need to do is edit the xorg.conf old school style. My thanks to dyasny for posting his xorg.conf code snipit.

Here's a copy of my working configuration.

# Xorg configuration created by pyxf86config

Section "ServerLayout"
        Identifier     "Default Layout"
        Screen      0  "Screen0" 0 0
        InputDevice    "Keyboard0" "CoreKeyboard"
        InputDevice "Tablet" "SendCoreEvents"
        InputDevice "Mouse" "CorePointer"
EndSection

Section "InputDevice"
        Identifier  "Keyboard0"
        Driver      "kbd"
        Option      "XkbModel" "pc105"
        Option      "XkbLayout" "us"
EndSection

Section "InputDevice"
        Identifier "Mouse"
        Driver "void"
        #Option "Device" "/dev/input/mice"
        #Option "Emulate3Buttons" "yes"
EndSection

Section "InputDevice"
        Identifier "Tablet"
        Driver "evdev"
        Option "Device" "/dev/input/event2"
        Option "CorePointer" "true"
        Option "Name" "Adomax QEMU USB Tablet"
EndSection

Section "Device"
        Identifier  "Videocard0"
        Driver      "cirrus"
EndSection

Section "Screen"
        Identifier "Screen0"
        Device     "Videocard0"
        DefaultDepth     24
        SubSection "Display"
                Viewport   0 0
                Depth     24
        EndSubSection
EndSection

Hope this helps.

Use Clonezilla to image KVM VirtIO disks

I'm always seeking to squeeze more speed out of common administrator tasks like disk imaging and P2V conversions. Today I tried using my favorite FOSS cloning software, Clonezilla, to restore an image to a KVM running VirtIO disks. What I found was that the current stable release (20110113-maverick) doesn't recognize VirtIO's /dev/vh[a,b,c...] disk naming syntax. You get used to this working with KVM and I'm still on the fence about VirtIO's name convention verses the more common /dev/sd[a,b,c...] method.

Luckily, another Clonezilla user already submitted a patch for VirtIO drives back in December. It should make it into a future stable release in a few months.

http://sourceforge.net/tracker/index.php?func=detail&aid=3112544&group_id=115473&atid=671650

I was in rush to get a P2V complete, so I used a quick sed onliner to modify stable Clonezilla's Perl scripts to recognize the /dev/vda disk. You'll need to drop into the shell mode to execute this.

sudo sed -i '/\[hs\]/\[vhs\]/' /opt/drbl/sbin/ocs-*

Keep in mind that these changes will be lost if your booting from a Live CDROM.

Using the VirtIO disk drivers improve the disk imaging throughput for my machine by about 15 percent. Also, don't forget to preload the VirtIO drivers on a Windows machine before imaging and restoring. Otherwise you'll get BSOD on boot.

Blurry Linux KVM screen fix

QEMU/KVM scales the guest's screen to fit a re-sized window. This is one of the small "paper cuts" that I've been living with since moving to KVM for my virtualization needs.

Having the screen get resized fine for occasions where you want to lessen the screen real-estate of a VM but still keep an eye want is going on. The problem is once you try drag the window back to it's 1:1 size. With a free hand you won't be able to get the window size exactly back to a 1:1 ratio so the everything in the VM will look slightly blurry.

Here's an example of my trying to get this 1152 x 864 Windows XP KVM back to actual screen resolution. It's close but still blurry.

I've lived with this blurriness for months now but today I came across Al Dimond's post about the problem. He took the time to investigate the problem and found a quick workaround using xdotool to resize the KVM window to a width and height one pixel less than the guest.

First get the window ID of your KVM

xdotool search --title QEMU

Then use the windowsize option to set the window to an exact size. The window ID in my example is 90177539.

xdotool windowsize 90177539 1151 863

Following the one pixel less workaround, you would use this command for a 1024x768 guest.

xdotool windowsize 90177539 1023 767

Now the guest screen is sharp and crisp again.

Run Windows virtual machines on Ubuntu/Debian desktop with KVM

Both at home and at work, I use Ubuntu as my operating system. There are times when I'm forced use Windows for some reason and there are several solutions for host Windows OS virtual machines on an Ubuntu laptop. Several years ago, I used what I most understood, VMware's workstation offering for Linux. Later when Virtualbox-ose (open source edition) caught up with VMware's features and hosted from Ubuntu's repositories, I switched to it.

These days, I'm much more technically adept with FOSS virtualization technologies and made the switch to using Linux KVM on my newer machines which support Intel's VT and AMD's AMD-V acceleration. I don't have any Phoronix style detailed comparisons but KVM feels faster and lighter than Virtualbox or VMware.

Quick Setup

Install the qemu-kvm package

sudo apt-get install qemu-kvm

Create a directory to hold your virtual machines.

mkdir -p ~/VM/WinXP

Move to that directory and create a disk image file.

cd ~/VM/WinXP
qemu-img create -f raw windows_xp.img 12G

Options:
-f raw = creates raw IO driver format image (You could also use the qcow2 mode. It has more features but doesn't perform as fast as raw)
windows_xp.img = name of the image file
12G = The virtual disk size.

Now create a bash script using your favourite text editor. I like vim but you could just as easily use gedit from GNOME.

vim Windows_XP.sh

Here's how my script looks:

#!/bin/bash
#
# Description: Launches Windows XP QEMU64
#
# Verion: 1
# Author: Clayton Kramer clayton.kramer @ gmail.com
# Modified: Fri 23 Apr 2010 11:43:35 AM EDT 
#

# Ubuntu Karmic tweek - Prepare audio to use Pulse driver instead of ALSA 
export QEMU_AUDIO_DRV=pa

# Launch Windows XP KVM
kvm  \
    -name "Windows XP Guest" \
    -m 1024 \
    -smp 1 \
    -localtime \
    -drive file=~/VM/WinXP/windows_xp.img,if=virtio,index=0,boot=on,cache=writeback \
    -drive file=~/ISO/windows_xp_sp2.iso,if=ide,media=cdrom,index=2 \
    -fda ~/ISO/viostor-31-03-2010-floppy.img \
    -net nic,model=virtio \
    -net user \
    -soundhw ac97 \
    -usb \
    -usbdevice tablet 

By default Ubuntu 9.10's qemu-kvm will use ALSA drivers which can lead to some choppy sound. You can change this behavior by setting the QEMU_AUDIO_DRV environmental variable to pa before launching the KVM.

I am using the VirtIO drivers in the script above. They improve the IO performance for Windows guests. Haydn Solomon provides some detailed instructions on setting them up in his KVM blog. I've decided to live a little dangerous and enabled the writeback option for the block driver.

http://www.linux-kvm.com/content/block-driver-updates-install-drivers-during-windows-installation

After the Windows installation is complete you can ommit the virtual floppy disk device line.

You may also want to take note that my script also configures the paravirtualized network device. You'll need to get the latest driver for that from:

http://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers

If you wanted wanted to get a Windows XP install going without using the VirtIO drivers you can use this compatibility script. It uses IDE for the IO controller bus and Intel e1000 driver for the NIC.

# Launch Windows XP KVM (compatibility)
kvm  \
    -name "Windows XP Guest" \
    -m 1024 \
    -smp 1 \
    -localtime \
    -drive file=~/VM/WinXP/windows_xp.img,if=ide,index=0,boot=on \
    -drive file=~/ISO/windows_xp_sp2.iso,if=ide,media=cdrom,index=2 \
    -net nic,model=e1000 \
    -net user \
    -soundhw ac97 \
    -usb \
    -usbdevice tablet