Wednesday, August 4, 2010

Ubuntu 10.04 MySQL Server startup bug

I encountered an issue yesterday with the mysql-server package on a server when I attempted to use the my.large.cnf settings file in place of the default.

See Launchpad:

There is a bug with Ubuntu 10.04's MySQL server. If you have it bind to a specific interface, it will fail to start on reboot because it attempts to start after any network interface (such as is initialized. If the interface that MySQL is bound to isn't reinitialized, it will hang. If you try to remove any specific interface bindings from the my.cnf settings, you'll run into another problem port assignment. You need to make sure that the upstart init script matches the bind-address value in the my.cnf file.

My edits:

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            =

start on (net-device-up IFACE=lo
          and local-filesystems
          and runlevel [2345])
stop on runlevel [016]

Note the IFACE=lo addition to the start on line.

My thanks to cdenley's post on the Ubuntu forums for shedding light on the problem.

Tuesday, July 27, 2010

Grub_puts not found

Two of our Ubuntu 10.4 Lucid workstations ran into Grub2 errors today. Something must have gone wrong with the grub2 apt scripts while they were updating to the latest kernel. Both of the machines with the problem were created from the same Clonezilla image but a few of the other cloned machines weren't affected.

After running the apt-get dist-upgrade command and rebooting, my users encountered the "fix symbol 'grub_puts' not found" error.


Burn the Ubuntu desktop ISO to CDROM or use the System > Administrator > Startup Disk Creator to create a bootable USB stick.

Boot from your live disk.

Open a terminal and get a list of the available partitions.

sudo fdisk -l

You should see results that look something like this:

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e0719

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          32      248832   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              32        9730    77899777    5  Extended
/dev/sda5              32        9730    77899776   83  Linux

Disk /dev/sdb: 8053 MB, 8053063680 bytes
255 heads, 63 sectors/track, 979 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00009233

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1         255     2048256    b  W95 FAT32
/dev/sdb2             256         979     5815530    b  W95 FAT32

In my example above, you can see the system drive is listed as /dev/sda and the bootable USB is /dev/sdb. You may, like me, have a separate /boot partition because you are running encrypted LVM volumes. In that case you need to pay attention to which is your root volume.

Mount your "root" partition or volume first. Standard Linux partitions are simple.

sudo mount /dev/sda1 /mnt

An encrypted LVM is a little more complicated. The Ubuntu Live CD doesn't have the LVM crypto packages installed so run these commands to get it working.

sudo apt-get install lvm2 cryptsetup

Load the dm-crypt module.

sudo modprobe dm-crypte

Now unlock your encrypted volume. Enter your LUKS passphrase when prompted.

sudo cryptsetup luksOpen /dev/sda2 foo

Load the LVM Kernel module.

sudo modprobe dm-mod

Scan for all of the available volume groups.

sudo vgscan

Active the volume group.

sudo vgchange -a

Now list the logical volumes along with their /dev paths. In the example below, note that my laptop is named "falcon" and yours is most likely something else.

sudo lvscan
  ACTIVE            '/dev/falcon/root' [71.22 GiB] inherit
  ACTIVE            '/dev/falcon/swap_1' [3.07 GiB] inherit

Now mount the root volume to /mnt. Replace falcon to match your own results of the previous command.

sudo mount /dev/falcon/root /mnt

Chroot Prep

Now mount the /dev, /proc, /sys folders for os-prober and grub to work properly in a chrooted jail.

sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys

If you had separate /boot partition because of LVM then mount it now.

sudo mount /dev/sda1 /mnt/boot

Now chroot yourself.

sudo chroot /mnt

Repair Grub2

Run the grub-mkconfig command to generate a new grub2 configuration file. This might be what got corrupted and left in this lurch.

grub-mkconfig -o /boot/grub/grub.cfg

Make sure no errors were generated. Then install grub2 in the hard drive MBR.

grub-install /dev/sda

Again make sure didn't get any errors. If you want a warm and fuzzy test your repair with the recheck option.

grub-install --recheck /dev/sda

Exit out of chroot with an exit or Crt+D command.

Unmount the directories.

sudo umount /mnt/dev
sudo umount /mnt

Now reboot and you should have your system back.

Wednesday, June 2, 2010

Blurry Linux KVM screen fix

QEMU/KVM scales the guest's screen to fit a re-sized window. This is one of the small "paper cuts" that I've been living with since moving to KVM for my virtualization needs.

Having the screen get resized fine for occasions where you want to lessen the screen real-estate of a VM but still keep an eye want is going on. The problem is once you try drag the window back to it's 1:1 size. With a free hand you won't be able to get the window size exactly back to a 1:1 ratio so the everything in the VM will look slightly blurry.

Here's an example of my trying to get this 1152 x 864 Windows XP KVM back to actual screen resolution. It's close but still blurry.

I've lived with this blurriness for months now but today I came across Al Dimond's post about the problem. He took the time to investigate the problem and found a quick workaround using xdotool to resize the KVM window to a width and height one pixel less than the guest.

First get the window ID of your KVM
xdotool search --title QEMU

Then use the windowsize option to set the window to an exact size. The window ID in my example is 90177539.

xdotool windowsize 90177539 1151 863

Following the one pixel less workaround, you would use this command for a 1024x768 guest.

xdotool windowsize 90177539 1023 767

Now the guest screen is sharp and crisp again.

Wednesday, May 5, 2010

Crank up the throttle on DD transfers

This is a quick one but I want to write it down just because I always seem to forget it if I don't use DD for a while.

DD's default block size (bs) is 512 bytes!

That's fine for some small work but you'll be waiting around for hours if your trying to shovel large disk images around.

Kick DD into high gear by raising the bs value to 32k and be done in minutes.

dd if=/dev/loop0 of=/dev/sdb bs=32k

In the example above, I've mounted a KVM disk image locally and an iSCSI lun as /dev/sdb. With the default 512b, DD was only able capitalize my 1g network at 5Mb/s. Switching to a 32k block size it utilized 40Mb/s.

Monday, May 3, 2010

Use Clonezilla for physical disk to iSCSI volume transfer

For the last few nights, I've been playing around with open-iscsi on Debian, Ubuntu and Windows 2008. Getting things up and running was fairly straight forward thanks to all of the helpful blogs and howtos people have posted. What I found missing was how one moves a Linux installation from a physical or virtual disk to an iSCSI volume. The little I found about the subject involved physically mounting the source disk to the iSCSI host or performing some tricky PXE boot magic to run the Linux distribution's installer. I find both of these methods inelegant and limited.

Tonight I came at it again. This time with my favorite FOSS disk imaging tool, Clonezilla!. The wonderful team behind it didn't skimp out and included the open-iscsi packages.


Download and burn a copy of the latest Ubuntu version of Clonezilla.

Boot from the Clonezilla live CDROM. Select all of the regional configuration options you require.

Stop when you get the the ncurses prompt to begin using Clonezilla or use the console. Press +F2 to switch the second tty console. This will let you work with the tools and setup a connection to your iSCSI share.

Get some networking configured otherwise you aren't going to be able to connect to the LUN.
sudo dhclient eth0

Now edit the iscsid.conf file.
sudo vi /etc/iscsi/iscsid.conf

Look for the node.startup property and set it to automatic.

Now start the open-iscsi daemon.
sudo /etc/init.d/open-iscsi start

Use the following command to query the iSCSI target for LUNs.
iscsiadm -m discovery -t sendtargets -p IP_OF_YOUR_TARGET

Here's an example of what mine looked like:
user@karmic:~$ sudo iscsiadm -m discovery -t st -p localhost,1 iqn.2007-10.local.server-1:storage.lun0

Now I can connect using the following:
iscsiadm -m node -T iqn.2007-10.local.server-1:storage.lun0 -p -l

Now check the /var/log/messages for the newly created virtual SCSI device.
tail /var/log/messages

Now you can switch back to console #1 and continue with Clonezilla wizard. Select local disk to local disk when prompted for which mode to use.

Create Cisco VPN on Ubuntu Karmic/Lucid

It is very easy to setup a Cisco VPN on Ubuntu. I used the following instructions to get my corporate tunnels running. This tutorial assumes you have already acquired a .pcf file from your network IT staff.


Install the vpnc package and any required dependencies:
sudo apt-get install vpnc

Open your vpn pcf configuration file with your favorite text editor.
vim corporatenet.pcf

It will looking something like this:

Note the values for Host, GroupName and enc_GroupPwd. You'll need these to create your vpnc configuration file.

sudo vim /etc/vpnc/corpnet.conf

Make your configuration file look like this. Just make sure to change the fictional CorpNet values with your own.

IPSec gateway
IPSec ID CorpNet
IPSec obfuscated secret C555E3A4BE82FF0001601A38260A92D93FF5693A482367E117EF8697CBED681C5FDD7F2AE0DEEA4B37DBBB21434189A46D8955F11916040A
Xauth username YOURUSERNAME
Xauth password YOURPASSWORD

It's important to note the obfuscated option in the group password. Most of the examples and howtos I've seen on the Net leave this out because they were written several years ago before VPNC supported Cisco encrypted passwords. The older guides required you to de-crypt the Cisco string. This isn't necessary anymore with Karmic and Lucid releases.

Thursday, April 29, 2010

Run Windows virtual machines on Ubuntu/Debian desktop with KVM

Both at home and at work, I use Ubuntu as my operating system. There are times when I'm forced use Windows for some reason and there are several solutions for host Windows OS virtual machines on an Ubuntu laptop. Several years ago, I used what I most understood, VMware's workstation offering for Linux. Later when Virtualbox-ose (open source edition) caught up with VMware's features and hosted from Ubuntu's repositories, I switched to it.

These days, I'm much more technically adept with FOSS virtualization technologies and made the switch to using Linux KVM on my newer machines which support Intel's VT and AMD's AMD-V acceleration. I don't have any Phoronix style detailed comparisons but KVM feels faster and lighter than Virtualbox or VMware.

Quick Setup

Install the qemu-kvm package
sudo apt-get install qemu-kvm

Create a directory to hold your virtual machines.
mkdir -p ~/VM/WinXP

Move to that directory and create a disk image file.
cd ~/VM/WinXP
qemu-img create -f raw windows_xp.img 12G

-f raw = creates raw IO driver format image (You could also use the qcow2 mode. It has more features but doesn't perform as fast as raw)
windows_xp.img = name of the image file
12G = The virtual disk size.

Now create a bash script using your favourite text editor. I like vim but you could just as easily use gedit from GNOME.

Here's how my script looks:
# Description: Launches Windows XP QEMU64
# Verion: 1
# Author: Clayton Kramer clayton.kramer @
# Modified: Fri 23 Apr 2010 11:43:35 AM EDT 

# Ubuntu Karmic tweek - Prepare audio to use Pulse driver instead of ALSA 
export QEMU_AUDIO_DRV=pa

# Launch Windows XP KVM
kvm  \
    -name "Windows XP Guest" \
    -m 1024 \
    -smp 1 \
    -localtime \
    -drive file=~/VM/WinXP/windows_xp.img,if=virtio,index=0,boot=on,cache=writeback \
    -drive file=~/ISO/windows_xp_sp2.iso,if=ide,media=cdrom,index=2 \
    -fda ~/ISO/viostor-31-03-2010-floppy.img \
    -net nic,model=virtio \
    -net user \
    -soundhw ac97 \
    -usb \
    -usbdevice tablet 

By default Ubuntu 9.10's qemu-kvm will use ALSA drivers which can lead to some choppy sound. You can change this behavior by setting the QEMU_AUDIO_DRV environmental variable to pa before launching the KVM.

I am using the VirtIO drivers in the script above. They improve the IO performance for Windows guests. Haydn Solomon provides some detailed instructions on setting them up in his KVM blog. I've decided to live a little dangerous and enabled the writeback option for the block driver.

After the Windows installation is complete you can ommit the virtual floppy disk device line.

You may also want to take note that my script also configures the paravirtualized network device. You'll need to get the latest driver for that from:

If you wanted wanted to get a Windows XP install going without using the VirtIO drivers you can use this compatibility script. It uses IDE for the IO controller bus and Intel e1000 driver for the NIC.

# Launch Windows XP KVM (compatibility)
kvm  \
    -name "Windows XP Guest" \
    -m 1024 \
    -smp 1 \
    -localtime \
    -drive file=~/VM/WinXP/windows_xp.img,if=ide,index=0,boot=on \
    -drive file=~/ISO/windows_xp_sp2.iso,if=ide,media=cdrom,index=2 \
    -net nic,model=e1000 \
    -net user \
    -soundhw ac97 \
    -usb \
    -usbdevice tablet 

Monday, April 26, 2010

How do you manage multiple Ubuntu desktops?

I spent a large part of my day at work trying to figure out how to replicate Windows style login scripts for our office Ubuntu desktops. This seemed like a straight forward problem that some amount of community effort would have already solved. It was so easy to setup Active Directory (AD) integration with Likewise-open so where is the howto on setting up automatic drive mapping on a Gnome desktop?

There are mass management tools like CFGengine and Puppet for server farms but where are the tools for running an office on Linux desktops? There are some proprietary offerings from Likewise (the contributers of Likewise-open) and Centrify that provide tools for integrating AD group policy objects (GPO) but they are geared toward fortune 2000 size companies. I'm looking for something beyond just being able to authenticate with a AD and connect to a CIFS share. If Linux and especially Ubuntu are ever going to really crack the desktop market, someone needs to launch a project to bridge this small enterprise gap.

Saturday, April 24, 2010

Integrate Ubuntu 9.10 Karmic's Samba With Microsoft Active Directory

Last week, I needed to setup a file server at work. Most of our back office server run Ubuntu except a few Microsoft Active Directory (AD) servers which control our workstations and user accounts. We make sure all of our Linux hosted services integrate with AD via Kerberos and LDAP.

Samba has been able to integrate with AD via winbind for a few years now. There are numerous postings on the net about how to do this. All of them a just a little different and many are a just a touch out of date for various distributions. Here's what I used to get an Ubuntu 9.10 server connected with our Microsoft Windows domain.

ACL Instructions

The normal install of Ubuntu and Debian support the standard Linux POSIX file system permissions. Access Control Lists (ACLs) provide a much more flexible way of specifying permissions on a file or other object than the standard Unix user/group/owner system. A good example you might deal with in production is the need to have "Domain Admins" and "HR" groups have write permission on a folder but the "Domain Users" only should have read access. That's not easy to do with standard POSIX.

Install the acl package:
sudo apt-get install acl

Now edit the partition that will hold your Samba shares so that it mounts with acl enabled. I typically create my shares in the /home/shares folder with my /home being mounted on its own volume.
vim /etc/fstab

/dev/mapper/vg0-home /home           ext4    acl,defaults        0       2

Please be careful when editing your fstab file. It's a good idea to make a backup of it first especially if you are making changes to the / "root" mount.

Some recommend a reboot at this point but you don't have to if you execute the following remount command.
mount -o remount,rw /dev/mapper/vg0-home

Kerberos Instructions

Install the kerberos packages:
sudo apt-get install ntp krb5-config krb5-user

The package installer will prompt you for Kerberos server information. Don't worry about those just enter something to satisfy it. You are going to replace the cumbersome default krb5.conf with a specific one for Active Directory authentication.

If you already have samba and winbind daemons installed an running, stop them now.
sudo service samba stop
sudo service winbind stop

Now let's setup the Kerberos configuration for authentication with Active Directory.
sudo mv /etc/krb5.conf /etc/krb5.orig
sudo vim /etc/krb5.conf

Copy the following text. Make sure to change SCHOOL.UNIVERSITY.EDU to your domain. Keep it in CAPS, though.
## /etc/krb5.conf
 default = FILE:/var/log/krb5.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmin.log

 default_realm = SCHOOL.UNIVERSITY.EDU
 dns_lookup_realm = false
 dns_lookup_kdc = false
 clock_skew = 300
 ticket_lifetime = 24h
 forwardable = yes

  default_domain = SCHOOL.UNIVERSITY.EDU

[domain_realm] = SCHOOL.UNIVERSITY.EDU

Note: In my example above, I've listed a secondary Kerberos server for authentication should the first domain controller be unavailable. You can add as many secondary kdc as you want. Remove this line if you only have one AD server.

Test to make sure the Kerberos connection before proceeding. This can save you some troubleshooting headaches later on.
kinit Administrator@SCHOOL.UNIVERSITY.EDU

The command should return clean and using klist should report a valid ticket good for 24 hours.

Now you can setup the Samba configuration.

Samba smb.conf for Active Directory

The default Samba config file is verbose with comments and easy to make a mistake. It is better just to make a backup copy of it and create a clean configuration.

sudo mv /etc/samba/smb.conf /etc/samba/smb.orig
sudo /etc/samba/smb.conf

There are almost countless examples on the net about how to configure you Samba file. Everyone's got a slightly different setup. I've settled on the following for production use.

Note: You will want to make sure to change SCHOOL.UNIVERSITY.EDU with your domain name.
        dos charset = UTF8
        display charset = UTF8
        workgroup = SCHOOL
        server string = %h
        security = ADS
        map to guest = Bad User
        null passwords = Yes
        obey pam restrictions = Yes
        pam password change = Yes
        password server = AD-CONTROLER1.SCHOOL.UNIVERSITY.EDU
        username map = /etc/samba/smbusers
        max log size = 10
        log file = /var/log/samba/log.%m
        unix extensions = No
        deadtime = 10
        socket options = TCP_NODELAY SO_KEEPALIVE SO_SNDBUF=65536 SO_RCVBUF=65536
        load printers = No
        disable spoolss = Yes
        dns proxy = No
        idmap uid = 10000-20000
        idmap gid = 10000-20000
        template shell = /bin/bash
        winbind separator = +
        winbind cache time = 3600
        winbind enum users = Yes
        winbind enum groups = Yes
        winbind refresh ticket = Yes
        create mask = 0777
        directory mask = 0777
        use sendfile = Yes
        delete veto files = Yes
        veto files = /.AppleDB/.AppleDouble/.AppleDesktop/:2eDS_Store/Network Trash Folder/Temporary It
        map hidden = Yes
        map system = Yes

        comment = School HR Server Share
        path = /home/shares/HR
        read only = No
        create mask = 0775
        valid users = @"HR-Dept"

Some key notes about this configuration:
  • socket options - The TCP_NODELAY makes noticeable improvement on file transfer speeds especially if you are using 1G NICs.
  • obey pam restrictions - This integrates your PAM authentication system.
  • veto files - Got Macs on your network? Keep those pesky .Apple file droppings off of your file server
  • valid users - Only members of the HR-Dept user group will have access to the HR file share.
Restart the Samba and Winbind services.
sudo /etc/init.d/winbind stop
sudo /etc/init.d/samba restart
sudo /etc/init.d/winbind start

Now you can join the Samba server to the Active Directory Domain.
sudo net ads join -U Administrator

You should see a message that the target domain was joined successfully.

Testing & Troubleshooting

Check you domain membership with the wbinfo -t command. This will validate that workstation trust account is working correctly:
sudo wbinfo -t

You should see your domain users with this command:
sudo wbinfo -u

The -g option should list your domain's groups.
sudo wbinfo -g

Configure System Security

Now modify the /etc/nsswitch.conf file so the system can start recognising your domain accounts.

vim /etc/nsswitch.conf

Append winbind after compat for passwd and group. Leave everything else alone.
passwd:         compat winbind
group:          compat winbind
shadow:         compat

hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

Edit these PAM config files.

sudo mv /etc/pam.d/common-account /etc/pam.d/common-account.orig
sudo vim /etc/pam.d/common-account

Copy the following.
account sufficient
account required

Now edit the common-auth file.
sudo cp /etc/pam.d/common-auth /etc/pam.d/common-auth.orig
sudo vim /etc/pam.d/common-auth

Now create a common-auth that looks like this
# /etc/pam.d/common-auth - authentication settings common to all services
# This file is included from other service-specific PAM config files,
# and should contain a list of the authentication modules that define
# the central authentication scheme for use on the system
# (e.g., /etc/shadow, LDAP, Kerberos, etc.).  The default is to use the
# traditional Unix authentication mechanisms.
# As of pam 1.0.1-6, this file is managed by pam-auth-update by default.
# To take advantage of this, it is recommended that you configure any
# local modules either before or after the default block, and use
# pam-auth-update to manage selection of other modules.  See
# pam-auth-update(8) for details.

# here are the per-package modules (the "Primary" block)
auth    [success=3 default=ignore] krb5_auth krb5_ccache_type=FILE
auth    [success=2 default=ignore] minimum_uid=1000
auth    [success=1 default=ignore] nullok_secure try_first_pass
# here's the fallback if no module succeeds
auth    requisite             
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
auth    required              
# and here are more per-package modules (the "Additional" block)
# end of pam-auth-update config

Setup the pam common-session file so new users to the system get created with a standard skel profile and home directory

sudo cp /etc/pam.d/common-session /etc/pam.d/common-session.orig
sudo vim /etc/pam.d/common-session

Your common-session should look like this:
# /etc/pam.d/common-session - session-related modules common to all services
# This file is included from other service-specific PAM config files,
# and should contain a list of modules that define tasks to be performed
# at the start and end of sessions of *any* kind (both interactive and
# non-interactive).
# As of pam 1.0.1-6, this file is managed by pam-auth-update by default.
# To take advantage of this, it is recommended that you configure any
# local modules either before or after the default block, and use
# pam-auth-update to manage selection of other modules.  See
# pam-auth-update(8) for details.

# here are the per-package modules (the "Primary" block)
session [default=1]           
# here's the fallback if no module succeeds
session requisite             
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
session required              
# and here are more per-package modules (the "Additional" block)
session optional               minimum_uid=1000
session required
session required umask=0022 skel=/etc/skel
# end of pam-auth-update config

Sudo'ers file

You can grant domain administrators elevated sudo permissions on the server by adding this line to your sudo configuration.

Open the sudo safe editor
sudo visudo

Add the following to the configuration:
# Allow "Domain Admins" from the domain "DOMAIN" to run all commands
%SCHOOL+Domain\ Admins  ALL=(ALL) ALL

You will want to replace SCHOOL with your domain name.


ACLs on Samba by Dustin Puryear
Join Samba 3 to Your Active Directory Domain - Carla Schroder
Active Directory Winbind Howto - Ubuntu Community Documentation