IT From All Angles

So a fairly trivial (but critical) aspect of using KVM is of course performing graceful shutdowns of your domains, without having to reach inside of the guest to perform the shutdown.  Now when it comes to turning off your guests you have two ways of proceeding (with virsh)…

# virsh destroy vmname

Which is a HARD power off of your guest.  This is the same as pulling the power cord from the wall, nothing is shutdown, but the noise stops just the same.  This method can result in data corruption if writes are interrupted.

# virsh shutdown vmname

Now in theory this will perform a graceful shutdown, or more importantly it sends a signal to the guest which results in the guest initiating a shutdown.  However this is not guaranteed, if your virtual machine configuration or guest OS is not configured to acknowledge or even receive these signals then nothing will happen.

Enable ACPI on the Virtual Hardware

Now the signal that is sent is an ACPI signal, this means that we need to configure the virtual machine to have acpi on the virtual processor.  This is accomplished by adding <acpi/> inside the features section of the virtual machines XML file.

This will print just the features block out of the XML file, it will not make any changes.

# awk '/<features>/,/<\/features>/' /etc/libvirt/qemu/vmname.xml
<features>
<acpi/>
<apic/>
<pae/>
</features>

Configure the Guest Operating System (Linux)

Linux is pretty simple, just make sure that acpid is installed.

Using APT

# apt-get install acpid

Using YUM

# yum install acpid

Configure the Guest Operating System (Windows)

Post Server 2000 Microsoft decided that by default servers should not respond to any idiot in a Datacenter with at least one finger.  Which is not an unreasonable assumption.  This had the unintended consequence of hamstringing virtualization, since obviously virtual machines don’t have the weakness of a power button which can be “bumped” by a one-fingered idiot.

I have seen different behavior based on the version of the OS you use, as I assume Microsoft has tried to make the behavior a bit more friendly and workable.  Basically as I have noticed.  Windows 2003 has to be logged on fully to have a virsh shutdown be successful.  Windows 2008 has to have the console open and the user has to have pressed CTRL-ALT-DEL, but not necessarily logged in.  Windows 2008 R2 simply needs the console open.

Regardless if you’d like to disable this protection on your VMs there is a registry change which can be made, of course this can also be made via a group policy.

HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\Shutdownwithoutlogon

0=disabled
1=enabled

This of course needs to be done on all of your Windows guests.

Now let me preface this post by saying…

I am writing this article from my professional experiences, inside of a software company which provided enterprise support for its products, as well as from the perspective of the Enterprise Customer.  I am intentionally discarding all experiences with support as a home user, and even as a small business owner.  This is specific to Enterprise Support, consumer support plays by different rules.

Value Your Customers Time And Investments

Your customers have invested in your products, once a customer buys a product they are no longer customers, they are partners.  Your partners are extremely intelligent people who chose your product because of that aforementioned intelligence (if this is not true then why do you sell it?).  With regards to their investments your partner’s employees are also an investment, and if you waste their employee’s time, you are robbing your partner of productivity which will NEVER be regained.

This is real money folks.  If your partner has someone making $50/hour on a phone call with support and they spend 1 hour trying to resolve the issue this means that not only is your customer losing the $50 that they are paying them, but there is another invisible $50 which is lost due to the lost productivity.  Your customer is willing to deal with this IF you keep the waste to a minimum.

Don’t Make Support A Profit Center

Support should never be a profit center, once an organization switches support from being a cost center (something that costs the company money) to a profit center (something that makes the company money) the company will start to make decisions which will increase profitability and alienate customers.  This will result in more marketing dollars being spent to neutralize the ill-effects of your support organization.  Don’t misunderstand me, I don’t have a problem with paying for support, and I am not advocating that it be free or even cheap.  Ultimately the support organization should NEARLY pay for itself (unless of course there is some large recall required), what should not happen EVER, is customers should never be pressured to put a failed drive back in a system to see if it would rebuild.  This sort of thing might be common practice, but it shows the customer that we don’t value their data and their business.

Market The Organization Not The Product

Products come and products go…  A good support organization will actually pull in its own Enterprise Customers, I would submit that a portion of your marketing dollars could actually be spent on making critical changes in your support organization.  This approach is invaluable when your organization is small and flexible, and the products down sell themselves.

Once the changes are made and you have a phenomenal support organization then you should spend a little bit of your marketing budget on advertising your support organization, since from an Enterprise Customer perspective it is all about risk mitigation and cost containment.  Highlight this, show the world the changes that you have made.

Build Your Organization For Success

If you are a US-based company, you had better make sure that if you use any overseas support that they speak english really well (this also goes for really deep accents from anywhere).  I don’t have a problem if the support center is in India or Mexico or Singapore or wherever,  but if we cannot communicate clearly then forget it.  It won’t matter how good the support was all I will remember is straining to understand what you are saying.  Remember that when I call you, I am already having a bad day at best, don’t make it worse.  Your employees should always provide attentive customer service, it should be prompt and respectful.  However they should also be technically adept and there should be a clear and logical path of escalations.  If someone needs a script then let them go work for your competitor, your organization will be much better off.

These four components will create a support organization which will actually help sell your product, instead of giving customers a reason to complain about your company.

Let me preface this article by stating that this issue is NOT a ZFS issue or a Solaris issue.  Previously I documented a issue in which the Crucial M4 SSD would “hesitate” causing ZFS to see it as faulted.  This was promptly fixed in v0002 of their firmware.  That issue is documented here.  This issue is a bit more aggravating and has not been fully resolved.

The issue is that after a drive has been operating for 5184 hours it will cease to respond to the system.  After a reboot the drive will work again, although the problem will repeat itself after an hour.  This will continue to happen until a fix is implemented in the firmware.

Firmware v0309 has been released which will resolve the issue IF you are not using a SAS expander.  If you are using a SAS expander (like that found in the Dell Poweredge R515 for example) then you have nothing you can do but wait for a new version of the firmware which will address this issue AND work with SAS expanders.  If you are not using a SAS expander then you should install this firmware as soon as possible whether you are seeing the issue or not.

Now with regards to ZFS, if you are using these devices unpatched as a ZIL then you need to either patch this immediately or if you are behind a SAS expander then you have no choice but to replace these SSDs with a different model, deconfigure them as a ZIL.  With a ZIL this error can cause real data loss.  If you are using these drives as a cache device (L2ARC) then the risk is much lower, you basically run the risk of losing your cache (which is lost and rebuilt on every reboot anyways) so no data loss there, so you could choose to operate with faulted cache devices until a fix is implemented.

http://www.crucial.com/support/firmware.aspx

Good luck and I hope you all dodged the bullet, or perhaps that the bullet hasn’t been fired at you yet.

UPDATE

If you are using SAS expanders with this SSD, you can install the SSD in a machine without a SAS expander and then update to firmware 0309 and reinstall the updated drive back into the SAS expander.  Apparently this issue was with the actual firmware update process and the SAS expander, not the firmware and the SAS expander.  But don’t take my word for it contact support and talk to them about your situation.

So I am on the ground in Dallas and trying to get my head around some of the technologies that we use in my new environment.  One of those being the Xen kernel.  I have never actually used Xen before, though I have used xenserver a bit.  Anyways so my first foray into the Xen world is trying to get OpenIndiana (successor to OpenSolaris) running on it, and see what sort of performance issues it might have.  We are hoping to be able to leverage its CIFS and vscan functionality to provide some security for a secured network file server.

I am by no means an expert in Xen, so you might notice some minor mistakes.  These instructions worked in my environment and should work in your environment as well, and the performance was actually quite good.  I am making the following assumptions.

• You are using a Linux Dom0, this might not matter, but for the purposes of this exercise we will pretend that it does.
• You are using LVM to back your DomU.
• You are going to run OpenIndiana, if you are trying to run something different (Solaris, Nexenta, SmartOS, etc) then there will be other nuances to get those particular distributions working.
• You are using XM configuration files directly and not using libvirt to try and simplify the configuration (I love libvirt, but could not get it working for these configurations).
• You do have libvirt installed, so that we can take advantage of virsh console.

Prepare the File System

Make directories for your Xen DomU kernels, and for the OpenIndiana cd mount.

# mkdir /etc/xen/kernels/oi151a; mkdir /mnt/oi151a

Create the Logical Volume for your OpenIndiana DomU.

# lvcreate -L40G yourvgnamehere -n vm_openindiana_boot

Mount the OpenIndiana ISO.

# mount -o loop,ro /mnt/scratch/oi151a.iso /mnt./oi151a

Copy the Kernel and Boot Archive out of the CD.

# cp /mnt/oi151a/platform/i86pc/amd64/boot_archive /etc/xen/kernels/oi151a/
# cp /mnt/oi151a/platform/i86xpv/kernel/amd64/unix /etc/xen/kernels/oi151a/

Unmount the OpenIndiana CD.

# umount /mnt/oi151a

Create the DomU Configurations

You actually need 2 configurations, you have to boot your LiveCD installer differently then the installed system.

The first is the build configuration, in other words the one that you can boot the LiveCD and run the installer for OpenIndiana.  This configuration requires that the ISO image be the first disk, we also must use the pygrub boot loader and finally we configure the console and livemode with extra parameters.

# cat /etc/xen/vm/openindiana-build
disk = [ 'file:/mnt/scratch/oi151a.iso,xvdc:cdrom,r', 'phy:yourvgnamehere/vm_openindiana_boot,xvda,w' ]
vcpus = 1
memory = 1024
name = "openindiana"
#kernel = "/etc/xen/kernels/oi151a/unix"
#ramdisk = "/etc/xen/kernels/oi151a/boot_archive"
#extra = "/platform/i86xpv/kernel/amd64/unix -B console=ttya,livemode=text"
bootloader = "/usr/bin/pygrub"
on_shutdown = "destroy"
on_reboot = "restart"
on_crash = "destroy"
vif = [ 'mac=aa:00:0a:4c:00:31, bridge=br0' ]

The second is the running configuration, which will actually boot of the disk, and persist your changes across reboots.

This one no longer needs to pygrub bootloader, console and livemode, but instead we need to point to the rpool (system zfs filesystem) and the boot path, so that it can find the file systems.

# cat /etc/xen/vm/openindiana-run
disk = [ 'phy:yourvgnamehere/vm_openindiana_boot,xvda,w' ]
vcpus = 1
memory = 1024
name = "openindiana"
kernel = "/etc/xen/kernels/oi151a/unix"
ramdisk = "/etc/xen/kernels/oi151a/boot_archive"
extra = "/platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/ROOT/openindiana,bootpath='/xpvd/xdf@51712:a'"
on_shutdown = "destroy"
on_reboot = "restart"
on_crash = "destroy"
vif = [ 'mac=aa:00:0a:4c:00:31, bridge=br0' ]

Boot to OpenIndiana LiveCD

Here is where everything starts to get a bit crazy.  So firstly we need to create the VM.

# xm create openindiana-build

Now we need to connect to the VM via console.  Keep in mind that when you need to close the console, you can use ^] to close it.  Default username is jack with a password of jack.

# virsh console openindiana

OpenIndiana Build oi_151a 64-bit (illumos f342d051b376)
SunOS Release 5.11 - Copyright 1983-2010 Oracle and/or its affiliates.
All rights reserved. Use is subject to license terms.
WARNING: No randomness provider enabled for /dev/random. Use cryptoadm(1M) to enable a provider.
Hostname: openindiana
Remounting root read/write
Probing for device nodes ...
Preparing live image for use
Done mounting Live image
USB keyboard
1. Albanian                      25. Latin-American
2. Arabic                        26. Lithuanian
3. Belarusian                    27. Latvian
4. Belgian                       28. Macedonian
5. Brazilian                     29. Malta_UK
6. Bulgarian                     30. Malta_US
7. Canadian-Bilingual            31. Norwegian
8. Croatian                      32. Polish
9. Czech                         33. Portuguese
10. Danish                        34. Romanian
11. Dutch                         35. Russian
12. Dvorak                        36. Serbia-And-Montenegro
13. Estonian                      37. Slovak
14. Finnish                       38. Slovenian
15. French                        39. Spanish
16. French-Canadian               40. Swedish
17. Hungarian                     41. Swiss-French
18. German                        42. Swiss-German
19. Greek                         43. Traditional-Chinese
20. Icelandic                     44. TurkishF
21. Italian                       45. TurkishQ
22. Japanese-type6                46. UK-English
23. Japanese                      47. US-English
24. Korean
To select the keyboard layout, enter a number [default 47]:

1. Arabic                        12. Hungarian
2. Catalan                       13. Indonesian
3. Chinese - Simplified          14. Italian
4. Chinese - Traditional         15. Japanese
5. Czech                         16. Korean
6. Dutch                         17. Polish
7. English                       18. Portuguese - Brazil
8. French                        19. Russian
9. German                        20. Slovak
10. Greek                         21. Spanish
11. Hebrew                        22. Swedish
To select the language you wish to use, enter a number [default is 7]:
User selected: English
Configuring devices.

openindiana console login: Jan 10 16:06:51 openindiana nwamd[66]: 10: start_dhcp: ipadm_create_addr failed for xnf0: Address object already exists
Jan 10 16:06:54 openindiana gdm-simple-slave[2150]: CRITICAL: file gdm-session-direct.c: line 2167: assertion `value != NULL' failed
Jan 10 16:06:56 openindiana gnome-keyring-daemon[2301]: couldn't allocate secure memory to keep passwords and or keys from being written to the disk

openindiana console login: jack
Password:
Last login: Tue Jan 10 16:06:54 on console
OpenIndiana (powered by illumos)    SunOS 5.11    oi_151a    September 2011

Configure VNC in the LiveCD Session

In my environment we do not have DHCP on this VLAN, so I have to reconfigure networking to use a static IP address.  I have a separate article here.  But we will need to configure VNC, so that we can launch the gui install program.

$ mkdir .vnc
$ cp .Xclients .vnc/startup
$ vncserver

You will require a password to access your desktops.

Password:
Verify:
xauth:    creating new authority file /jack/.Xauthority

New 'openindiana:1 ()' desktop is openindiana:1

Starting applications specified in /jack./.vnc/startup
Log file is /jack/.vnc/openindiana:1.log
$

Now simply use a VNC client on your workstation to connect to openindiana:1 via VNC.  Depending on your local environment you might need to use the IP address instead of the name.  Also please keep in mind that if you execute the VNC server as root (by doing a sudo -s) then you will be logged on as root.  Which means you won’t have an icon from which to install OpenIndiana, you can still execute it manually…  However at some point your screen could lock and you will be unable to unlock it.  So long story short execute it as jack.

Install OpenIndiana

Now everything should be setup in such a way that you are able to actually perform the installation of OpenIndiana.  So go ahead and do it.

Switch to the Running Configuration

So once we are done, just shutdown the machine.  Once it is safely shutdown we can proceed.

Destroy the DomU which is configured for the LiveCD.

# xm destroy openindiana

Create the DomU which is configured to boot from the disk.

# xm create openindiana-run
Using config file "./openindiana-run".
Started domain openindiana

Now once it is up and running you can go about your business of configuring your DomU services.

I have been working on creating a bunch of debootstrap images lately, and I ended up doing a whole bunch of work creating tarballs and did it wrong.  So I took the time to sort out an easy way to fix them all with two commands.

The Core of My Problem

So what started this is I forgot to “stand” in the directory I was archiving, as such all of my images ended up having a top level directory named lenny32 (insert appropriate distribution and architecture here) which isn’t very helpful when you are extracting it to the root of a file system.

Here is the command I used to originally create my archive.

# tar -czf lenny32.tgz lenny32/

Here is the command I should have used.

# tar -czf lenny32.tgz -C lenny32/ .

The -C switch tells tar to cd into the directory before performing the archive.

Now before we get started lets make sure that you are have 2 separate working directories, one for .tgz files and one for the source files.

Extracting All of the Archives

So the first part of the process is re-extracting all of the archives, you can optionally just re-archive your original source files provided that they are still accessible.  This assumes that you have created an unfixed directory, in which we will be locating our source files.

# for a in `ls -l *.tgz`; do echo -n $a; tar -xzf $a -C unfixed; echo "     done."; done

Recreating All of the Archives

Now since we have extracted everything we end up with a directory which holds a bunch of subdirectories for each of our archives.  Now we will turn each directory into its own archive.  Make sure there are no extra directories in here.

# for a in `ls`; do echo -n $a; tar -czf $a.tgz -C $a .; echo "      done."; done

Now once you have done this you will have your source files in the same directory as your fixed tarballs.  This can be resolved by doing a move.

# mv *.tgz ../fixed

There you go.  This is really rather simple, but it isn’t something I use frequently so I wanted to write it down.  Now when you extract one of these archives make sure you are “standing” in an empty directory OR use the -C option to have tar stand in that empty directory for you, otherwise you will dirty up that directory.

# tar -xzf lenny32.tgz -C lenny32/

Enjoy!