IT From All Angles

Let me preface this article by stating that this issue is NOT a ZFS issue or a Solaris issue.  Previously I documented a issue in which the Crucial M4 SSD would “hesitate” causing ZFS to see it as faulted.  This was promptly fixed in v0002 of their firmware.  That issue is documented here.  This issue is a bit more aggravating and has not been fully resolved.

The issue is that after a drive has been operating for 5184 hours it will cease to respond to the system.  After a reboot the drive will work again, although the problem will repeat itself after an hour.  This will continue to happen until a fix is implemented in the firmware.

Firmware v0309 has been released which will resolve the issue IF you are not using a SAS expander.  If you are using a SAS expander (like that found in the Dell Poweredge R515 for example) then you have nothing you can do but wait for a new version of the firmware which will address this issue AND work with SAS expanders.  If you are not using a SAS expander then you should install this firmware as soon as possible whether you are seeing the issue or not.

Now with regards to ZFS, if you are using these devices unpatched as a ZIL then you need to either patch this immediately or if you are behind a SAS expander then you have no choice but to replace these SSDs with a different model, deconfigure them as a ZIL.  With a ZIL this error can cause real data loss.  If you are using these drives as a cache device (L2ARC) then the risk is much lower, you basically run the risk of losing your cache (which is lost and rebuilt on every reboot anyways) so no data loss there, so you could choose to operate with faulted cache devices until a fix is implemented.

http://www.crucial.com/support/firmware.aspx

Good luck and I hope you all dodged the bullet, or perhaps that the bullet hasn’t been fired at you yet.

So I am on the ground in Dallas and trying to get my head around some of the technologies that we use in my new environment.  One of those being the Xen kernel.  I have never actually used Xen before, though I have used xenserver a bit.  Anyways so my first foray into the Xen world is trying to get OpenIndiana (successor to OpenSolaris) running on it, and see what sort of performance issues it might have.  We are hoping to be able to leverage its CIFS and vscan functionality to provide some security for a secured network file server.

I am by no means an expert in Xen, so you might notice some minor mistakes.  These instructions worked in my environment and should work in your environment as well, and the performance was actually quite good.  I am making the following assumptions.

• You are using a Linux Dom0, this might not matter, but for the purposes of this exercise we will pretend that it does.
• You are using LVM to back your DomU.
• You are going to run OpenIndiana, if you are trying to run something different (Solaris, Nexenta, SmartOS, etc) then there will be other nuances to get those particular distributions working.
• You are using XM configuration files directly and not using libvirt to try and simplify the configuration (I love libvirt, but could not get it working for these configurations).
• You do have libvirt installed, so that we can take advantage of virsh console.

Prepare the File System

Make directories for your Xen DomU kernels, and for the OpenIndiana cd mount.

# mkdir /etc/xen/kernels/oi151a; mkdir /mnt/oi151a

Create the Logical Volume for your OpenIndiana DomU.

# lvcreate -L40G yourvgnamehere -n vm_openindiana_boot

Mount the OpenIndiana ISO.

# mount -o loop,ro /mnt/scratch/oi151a.iso /mnt./oi151a

Copy the Kernel and Boot Archive out of the CD.

# cp /mnt/oi151a/platform/i86pc/amd64/boot_archive /etc/xen/kernels/oi151a/
# cp /mnt/oi151a/platform/i86xpv/kernel/amd64/unix /etc/xen/kernels/oi151a/

Unmount the OpenIndiana CD.

# umount /mnt/oi151a

Create the DomU Configurations

You actually need 2 configurations, you have to boot your LiveCD installer differently then the installed system.

The first is the build configuration, in other words the one that you can boot the LiveCD and run the installer for OpenIndiana.  This configuration requires that the ISO image be the first disk, we also must use the pygrub boot loader and finally we configure the console and livemode with extra parameters.

# cat /etc/xen/vm/openindiana-build
disk = [ 'file:/mnt/scratch/oi151a.iso,xvdc:cdrom,r', 'phy:yourvgnamehere/vm_openindiana_boot,xvda,w' ]
vcpus = 1
memory = 1024
name = "openindiana"
#kernel = "/etc/xen/kernels/oi151a/unix"
#ramdisk = "/etc/xen/kernels/oi151a/boot_archive"
#extra = "/platform/i86xpv/kernel/amd64/unix -B console=ttya,livemode=text"
bootloader = "/usr/bin/pygrub"
on_shutdown = "destroy"
on_reboot = "restart"
on_crash = "destroy"
vif = [ 'mac=aa:00:0a:4c:00:31, bridge=br0' ]

The second is the running configuration, which will actually boot of the disk, and persist your changes across reboots.

This one no longer needs to pygrub bootloader, console and livemode, but instead we need to point to the rpool (system zfs filesystem) and the boot path, so that it can find the file systems.

# cat /etc/xen/vm/openindiana-run
disk = [ 'phy:yourvgnamehere/vm_openindiana_boot,xvda,w' ]
vcpus = 1
memory = 1024
name = "openindiana"
kernel = "/etc/xen/kernels/oi151a/unix"
ramdisk = "/etc/xen/kernels/oi151a/boot_archive"
extra = "/platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/ROOT/openindiana,bootpath='/xpvd/xdf@51712:a'"
on_shutdown = "destroy"
on_reboot = "restart"
on_crash = "destroy"
vif = [ 'mac=aa:00:0a:4c:00:31, bridge=br0' ]

Boot to OpenIndiana LiveCD

Here is where everything starts to get a bit crazy.  So firstly we need to create the VM.

# xm create openindiana-build

Now we need to connect to the VM via console.  Keep in mind that when you need to close the console, you can use ^] to close it.  Default username is jack with a password of jack.

# virsh console openindiana

OpenIndiana Build oi_151a 64-bit (illumos f342d051b376)
SunOS Release 5.11 - Copyright 1983-2010 Oracle and/or its affiliates.
All rights reserved. Use is subject to license terms.
WARNING: No randomness provider enabled for /dev/random. Use cryptoadm(1M) to enable a provider.
Hostname: openindiana
Remounting root read/write
Probing for device nodes ...
Preparing live image for use
Done mounting Live image
USB keyboard
1. Albanian                      25. Latin-American
2. Arabic                        26. Lithuanian
3. Belarusian                    27. Latvian
4. Belgian                       28. Macedonian
5. Brazilian                     29. Malta_UK
6. Bulgarian                     30. Malta_US
7. Canadian-Bilingual            31. Norwegian
8. Croatian                      32. Polish
9. Czech                         33. Portuguese
10. Danish                        34. Romanian
11. Dutch                         35. Russian
12. Dvorak                        36. Serbia-And-Montenegro
13. Estonian                      37. Slovak
14. Finnish                       38. Slovenian
15. French                        39. Spanish
16. French-Canadian               40. Swedish
17. Hungarian                     41. Swiss-French
18. German                        42. Swiss-German
19. Greek                         43. Traditional-Chinese
20. Icelandic                     44. TurkishF
21. Italian                       45. TurkishQ
22. Japanese-type6                46. UK-English
23. Japanese                      47. US-English
24. Korean
To select the keyboard layout, enter a number [default 47]:

1. Arabic                        12. Hungarian
2. Catalan                       13. Indonesian
3. Chinese - Simplified          14. Italian
4. Chinese - Traditional         15. Japanese
5. Czech                         16. Korean
6. Dutch                         17. Polish
7. English                       18. Portuguese - Brazil
8. French                        19. Russian
9. German                        20. Slovak
10. Greek                         21. Spanish
11. Hebrew                        22. Swedish
To select the language you wish to use, enter a number [default is 7]:
User selected: English
Configuring devices.

openindiana console login: Jan 10 16:06:51 openindiana nwamd[66]: 10: start_dhcp: ipadm_create_addr failed for xnf0: Address object already exists
Jan 10 16:06:54 openindiana gdm-simple-slave[2150]: CRITICAL: file gdm-session-direct.c: line 2167: assertion `value != NULL' failed
Jan 10 16:06:56 openindiana gnome-keyring-daemon[2301]: couldn't allocate secure memory to keep passwords and or keys from being written to the disk

openindiana console login: jack
Password:
Last login: Tue Jan 10 16:06:54 on console
OpenIndiana (powered by illumos)    SunOS 5.11    oi_151a    September 2011

Configure VNC in the LiveCD Session

In my environment we do not have DHCP on this VLAN, so I have to reconfigure networking to use a static IP address.  I have a separate article here.  But we will need to configure VNC, so that we can launch the gui install program.

$ mkdir .vnc
$ cp .Xclients .vnc/startup
$ vncserver

You will require a password to access your desktops.

Password:
Verify:
xauth:    creating new authority file /jack/.Xauthority

New 'openindiana:1 ()' desktop is openindiana:1

Starting applications specified in /jack./.vnc/startup
Log file is /jack/.vnc/openindiana:1.log
$

Now simply use a VNC client on your workstation to connect to openindiana:1 via VNC.  Depending on your local environment you might need to use the IP address instead of the name.  Also please keep in mind that if you execute the VNC server as root (by doing a sudo -s) then you will be logged on as root.  Which means you won’t have an icon from which to install OpenIndiana, you can still execute it manually…  However at some point your screen could lock and you will be unable to unlock it.  So long story short execute it as jack.

Install OpenIndiana

Now everything should be setup in such a way that you are able to actually perform the installation of OpenIndiana.  So go ahead and do it.

Switch to the Running Configuration

So once we are done, just shutdown the machine.  Once it is safely shutdown we can proceed.

Destroy the DomU which is configured for the LiveCD.

# xm destroy openindiana

Create the DomU which is configured to boot from the disk.

# xm create openindiana-run
Using config file "./openindiana-run".
Started domain openindiana

Now once it is up and running you can go about your business of configuring your DomU services.

I am still in the process of moving to Texas, however one major accomplishment has been completed the migration of a lot of my services down to Texas.  I am still making preparations for my email system to be migrated, however everything else has been migrated and this blog is now online in Texas.  I took this opportunity to perform some upgrades as well (I had the latest ones on hold since I was 2500 miles away from my servers – I always get a bit jumpy in those situations).

I have been working on creating a bunch of debootstrap images lately, and I ended up doing a whole bunch of work creating tarballs and did it wrong.  So I took the time to sort out an easy way to fix them all with two commands.

The Core of My Problem

So what started this is I forgot to “stand” in the directory I was archiving, as such all of my images ended up having a top level directory named lenny32 (insert appropriate distribution and architecture here) which isn’t very helpful when you are extracting it to the root of a file system.

Here is the command I used to originally create my archive.

# tar -czf lenny32.tgz lenny32/

Here is the command I should have used.

# tar -czf lenny32.tgz -C lenny32/ .

The -C switch tells tar to cd into the directory before performing the archive.

Now before we get started lets make sure that you are have 2 separate working directories, one for .tgz files and one for the source files.

Extracting All of the Archives

So the first part of the process is re-extracting all of the archives, you can optionally just re-archive your original source files provided that they are still accessible.  This assumes that you have created an unfixed directory, in which we will be locating our source files.

# for a in `ls -l *.tgz`; do echo -n $a; tar -xzf $a -C unfixed; echo "     done."; done

Recreating All of the Archives

Now since we have extracted everything we end up with a directory which holds a bunch of subdirectories for each of our archives.  Now we will turn each directory into its own archive.  Make sure there are no extra directories in here.

# for a in `ls`; do echo -n $a; tar -czf $a.tgz -C $a .; echo "      done."; done

Now once you have done this you will have your source files in the same directory as your fixed tarballs.  This can be resolved by doing a move.

# mv *.tgz ../fixed

There you go.  This is really rather simple, but it isn’t something I use frequently so I wanted to write it down.  Now when you extract one of these archives make sure you are “standing” in an empty directory OR use the -C option to have tar stand in that empty directory for you, otherwise you will dirty up that directory.

# tar -xzf lenny32.tgz -C lenny32/

Enjoy!

I have been located in the Greater Seattle Area (less-than-affectionately-referred-to-as her) for the past 7 years and frankly I have had a bit of a love-hate relationship with her, in all honesty though it has been more of a HATE-love relationship.  Now lets be honest I LOVE the tech job market in Seattle, and I have family here which makes life a bit easier.  That is really where my hate relationship starts.  I hate the traffic (it is not bad it is just idiotic – I can only deal with so much passive-aggressive – just pick one), I hate the housing market (why should anyone buy a condo for 300K anywhere – especially in Seattle), I hate the weather (do you know how hard it is to wake up in the morning when the sun can’t even be bothered to rise?).  Trust me I could go on.

I was recently offered and have decided to accept a position in Dallas, TX and as such I will be relocating there for 2012.  My new position will allow me to continue with some of my favorite technologies (ZFS), and start learning some new ones with Xen being the most interesting for me.

How Will This Affect My Customers?

By now you should have received an email from me providing some more detail around how this will affect your individual situation.  If you have not please contact me and I can resend.  But generally, I will continue to be able to provide remote support, with new hardware being shipped through me for configuration, and online/telephone support for existing hardware.

How Will This Affect My Readers?

Well in theory there could be some downtime associated with this site (while I move from one server to another or while we wait for DNS changes to propogate).  I would suspect that the actual downtime with the site itself will be minimal, though updates will be scant until I am fully moved and operational in Texas.  I will also probably be slow when it comes to moderating comments and responding to emails, though I should be able to respond to emails more quickly than comments.