The Basic Concepts

Now why is a clustered file system important?  So basically if you have the need to have a shared volume between two hosts, you can provision the disk to both machines, and everything could appear to work, however in the event that writes ever happened to the same areas of the disk at the same time you will end up with data corruption.  Now the key here is that you need a way to track locks from multiple nodes.  This is called a Distributed Locking Manager or DLM.  Now to get this DLM functionality working then it will create a cluster.  Valid cluster nodes can then mount the disk and interact with it as a normal disk.  So as part of OCFS2 we have two file systems which are created /sys/kernel/config and /dlm the prior is used for the cluster configurations, and the latter is for the distributed lock manager

Requirements

OCFS2 has been in the mainline Linux kernel for years, so it is widely available, though if you compile your own kernels then you will need to include support in your kernel.  Other than that all you need is the userland configuration tools to interact with it.

Install OCFS2 Tools

# yum install ocfs2-tools

Load and Online the O2CB Service

# service o2cb load
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading stack plugin "o2cb": OK
Loading filesystem "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK

# service o2cb online
Setting cluster stack "o2cb": OK
Checking O2CB cluster configuration : Failed

Notice that when we online o2cb, that it fails at checking the O2CB cluster configuration.  This is expected.  It is due to not having a cluster configuration to check at this point.

Create the OCFS2 Cluster Configuration

Now we need to create the /etc/ocfs2/cluster.conf.  This can be done with o2cb_ctl or manually.  Though it is considerably easier with o2cb_ctl.

# o2cb_ctl -C -n prdcluster -t cluster -a name=prdcluster

Here we are naming our cluster prdcluster.  The cluster itself doesn’t know anything about nodes until we add them in the next step.

Add Nodes to the OCFS2 Cluster Configuration

Create an entry for each node, using the below command.  We will need the IP of the nodes, the port, the cluster name we defined before and the host name of each node.

# o2cb_ctl -C -n ocfs01 -t node -a number=0 -a ip_address=172.16.88.131 -a ip_port=11111 -a cluster=prdcluster
# o2cb_ctl -C -n ocfs02 -t node -a number=1 -a ip_address=172.16.88.132 -a ip_port=11111 -a cluster=prdcluster

The IP Address and Port are used for the Cluster heartbeat.  The node name is used to verify a cluster member when attempting to join the cluster.  The node name needs to match the systems host name.

Review the OCFS2 Cluster Configuration

Now we can take a peek at the cluster.conf which our o2cb_ctl command created.

# cat /etc/ocfs2/cluster.conf
node:
name = ocfs01
cluster = prdcluster
number = 0
ip_address = 172.16.88.131
ip_port = 11111

node:
name = ocfs02
cluster = prdcluster
number = 1
ip_address = 172.16.88.132
ip_port = 11111

cluster:
name = prdcluster
heartbeat_mode = local
node_count = 2

Configure the O2CB Service

In order to have the cluster start with the correct information we need to update the o2cb service and include the name of our cluster.

# service o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets ('[]').  Hitting
<ENTER> without typing an answer will keep that current value.  Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [n]: y
Cluster stack backing O2CB [o2cb]:
Cluster to start on boot (Enter "none" to clear) [ocfs2]: prdcluster
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Setting cluster stack "o2cb": OK
Registering O2CB cluster "prdcluster": OK
Setting O2CB cluster timeouts : OK

Offline and Online the O2CB Service

To ensure that everything is working as we expect, I like to offline and online the service.

# service o2cb offline
Clean userdlm domains: OK
Stopping O2CB cluster prdcluster: Unregistering O2CB cluster "prdcluster": OK

We just want to watch that it is unregistering and registering the correct cluster, in this case the prdcluster.

# service o2cb online
Setting cluster stack "o2cb": OK
Registering O2CB cluster "prdcluster": OK
Setting O2CB cluster timeouts : OK

Repeat for All Nodes

All of the above actions need to be done on all nodes in the cluster, with no variations.  Once all nodes are Registering O2CB cluster “prdcluster”: OK then you can move on.

Format Our Shared Disk

This part is no different from any other format, keep in mind that once you have formatted the disk on one cluster node, it does not need to be done on the other node.

# mkfs.ocfs2 /dev/xvdb
mkfs.ocfs2 1.8.0
Cluster stack: classic o2cb
Label:
Features: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super xattr indexed-dirs refcount discontig-bg
Block size: 4096 (12 bits)
Cluster size: 4096 (12 bits)
Volume size: 53687091200 (13107200 clusters) (13107200 blocks)
Cluster groups: 407 (tail covers 11264 clusters, rest cover 32256 clusters)
Extent allocator size: 8388608 (2 groups)
Journal size: 268435456
Node slots: 8
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 3 block(s)
Formatting Journals: done
Growing extent allocator: done
Formatting slot map: done
Formatting quota files: done
Writing lost+found: done
mkfs.ocfs2 successful

Mount Our OCFS2 Volume

You can either use a manual issuance of the mount command, or you can create an entry in the /etc/fstab

# mount -t ocfs2 /dev/xvdb /d01/share

# cat /etc/fstab

#
 # /etc/fstab
 # Created by anaconda on Wed Feb 27 13:44:01 2013
 #
 # Accessible filesystems, by reference, are maintained under '/dev/disk'
 # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
 #
 /dev/mapper/vg_system-lv_root /                       ext4    defaults        1 1
 UUID=4b397e61-7954-40e9-943f-8385e46d263d /boot                   ext4    defaults        1 2
 /dev/mapper/vg_system-lv_swap swap                    swap    defaults        0 0
 tmpfs                   /dev/shm                tmpfs   defaults        0 0
 devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
 sysfs                   /sys                    sysfs   defaults        0 0
 proc                    /proc                   proc    defaults        0 0
 /dev/xvdb        /d01/share        ocfs2    defaults    1 1

Then mount our entry from the /etc/fstab.

# mount /d01/share

Mounts will need to be configured on all cluster nodes.

Check Our Mounts

Once we have mounted our devices we need to ensure that they are showing up correctly.

# mount
 /dev/mapper/vg_system-lv_root on / type ext4 (rw)
 proc on /proc type proc (rw)
 sysfs on /sys type sysfs (rw)
 devpts on /dev/pts type devpts (rw,gid=5,mode=620)
 tmpfs on /dev/shm type tmpfs (rw)
 /dev/xvda1 on /boot type ext4 (rw)
 none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
 sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
 configfs on /sys/kernel/config type configfs (rw)
 ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
 /dev/xvdb on /d01/share type ocfs2 (rw,_netdev,heartbeat=local)

Notice that /d01/share is mounted as ocfs2, and that it is mounted with rw, _netdev, heartbeat=local.  These are the expected options (these are gathered from the previous configuration).

Check Service Status

Finally we can check the status on the o2cb service and we can see information about our cluster, heartbeat and the various other mounts that are needed to maintain the cluster (configfs, and ocfs2_dlmfs).

# service o2cb status
 Driver for "configfs": Loaded
 Filesystem "configfs": Mounted
 Stack glue driver: Loaded
 Stack plugin "o2cb": Loaded
 Driver for "ocfs2_dlmfs": Loaded
 Filesystem "ocfs2_dlmfs": Mounted
 Checking O2CB cluster "prdcluster": Online
 Heartbeat dead threshold: 31
 Network idle timeout: 30000
 Network keepalive delay: 2000
 Network reconnect delay: 2000
 Heartbeat mode: Local
 Checking O2CB heartbeat: Active

# mysql ovs  -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'

As we can see it is attempting and failing to use /var/lib/mysql/mysql.sock as the connection.  So lets take a look at the process and see if it has any clues.

# ps -ef | grep mysql
oracle    2234  1778  1 Jan23 ?        00:47:09 /usr/sbin/mysqld --defaults-file=/u01/app/oracle/mysql/data/my.cnf --basedir=/usr --datadir=/u01/app/oracle/mysql/data --plugin-dir=/usr/lib64/mysql/plugin --user=oracle --log-error=/u01/app/oracle/mysql/data/mysqld.err --pid-file=/u01/app/oracle/mysql/data/mysqld.pid --socket=/u01/app/oracle/mysql/data/mysqld.sock --port=49500

Above we see a couple of key pieces of information.  We now know that the socket is /u01/app/oracle/mysql/data/mysqld.sock and we also see that our configuration file is /u01/app/oracle/mysql/data/my.cnf.  So based on this new socket we can attempt to connect to mysql again.

# mysql ovs -S /u01/app/oracle/mysql/data/mysqld.sock -u root -p
Enter password:
mysql>

Now we are connected to the backend, here comes the bad news.  The database is completely worthless, they are still using longblobs for everything.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| ovs                |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)

Use the ovs database so we can look at its content.

mysql> use ovs;
Database changed

Next we will show all tables so that we can get an idea of what the schema looks like.

mysql> show tables;
+--------------------------------+
| Tables_in_ovs                  |
+--------------------------------+
| Mgr_AbcStore                   |
| Mgr_AccessManager              |
| Mgr_ActionEngineProperties     |
| Mgr_ActionManager              |
| Mgr_ArchiveManager             |
| Mgr_BackupManager              |
| Mgr_BalancerControl            |
| Mgr_BindingMismatchEvent       |
| Mgr_BondPort                   |
| Mgr_BusinessManager            |
| Mgr_Cluster                    |
| Mgr_Coherence                  |
| Mgr_ControlDomain              |
| Mgr_CpuCompatibilityGroup      |
| Mgr_CreateStatisticLog         |
| Mgr_CreatedEvent               |
| Mgr_DeletedEvent               |
| Mgr_DiscoverEngineProperties   |
| Mgr_DiscoverManager            |
| Mgr_EthernetNetwork            |
| Mgr_EthernetPort               |
| Mgr_EventEngineProperties      |
| Mgr_EventLog                   |
| Mgr_EventManager               |
| Mgr_FibreChannelStorageArray   |
| Mgr_FileManager                |
| Mgr_FileSystemMount            |
| Mgr_FileSystemPlugin           |
| Mgr_Foundry                    |
| Mgr_HashMap                    |
| Mgr_InformationalEvent         |
| Mgr_InternalJob                |
| Mgr_InternalPort               |
| Mgr_InternalSystemLog          |
| Mgr_InternalTaggingObject      |
| Mgr_IscsiStorageArray          |
| Mgr_IscsiStorageInitiator      |
| Mgr_Iterator                   |
| Mgr_JobConstructingEvent       |
| Mgr_JobDoneEvent               |
| Mgr_JobRunningEvent            |
| Mgr_LinkedList                 |
| Mgr_LocalFileServer            |
| Mgr_LocalFileSystem            |
| Mgr_LocalStorageArray          |
| Mgr_LocalStorageInitiator      |
| Mgr_LocalStoragePath           |
| Mgr_LogEngineProperties        |
| Mgr_LogManager                 |
| Mgr_LogStore                   |
| Mgr_ModelEngineProperties      |
| Mgr_ModelManager               |
| Mgr_NetworkFileServer          |
| Mgr_NetworkFileSystem          |
| Mgr_NetworkSelectionManager    |
| Mgr_ObjectChangeEvent          |
| Mgr_ObjectCheckerTask          |
| Mgr_OdofManager                |
| Mgr_OvfAssembly                |
| Mgr_PathDownEvent              |
| Mgr_PathUpEvent                |
| Mgr_PerfManager                |
| Mgr_PortDownEvent              |
| Mgr_PortUpEvent                |
| Mgr_Processor                  |
| Mgr_Properties                 |
| Mgr_QueuedJobCreateEvent       |
| Mgr_QueuedServerUpdateNtpServe |
| Mgr_QueuedServerYumRepositoryU |
| Mgr_RasEngineProperties        |
| Mgr_RasManager                 |
| Mgr_RefreshRepoFileSystemsTask |
| Mgr_Repository                 |
| Mgr_RestoreManager             |
| Mgr_RoleService                |
| Mgr_RootStatisticLog           |
| Mgr_RulesEngineProperties      |
| Mgr_RulesManager               |
| Mgr_SchedulableTaskProperties  |
| Mgr_Server                     |
| Mgr_ServerClusterStateDownEven |
| Mgr_ServerDefaultInfo          |
| Mgr_ServerDisconnectErrorEvent |
| Mgr_ServerDiscoverScanEvent    |
| Mgr_ServerNotification         |
| Mgr_ServerOfflineEvent         |
| Mgr_ServerOutofDateEvent       |
| Mgr_ServerPool                 |
| Mgr_ServerPoolMasterMissingEve |
| Mgr_ServerRunningEvent         |
| Mgr_ServerSelectionManager     |
| Mgr_ServerStartingEvent        |
| Mgr_ServerStoppedEvent         |
| Mgr_ServerUserMissingEvent     |
| Mgr_ServerVersionMismatchWarni |
| Mgr_ServerYumRepositoryInforma |
| Mgr_ServerYumUpdateCheckingEve |
| Mgr_SeverityChangeEvent        |
| Mgr_StatisticManager           |
| Mgr_StatisticSubjectLog        |
| Mgr_StatisticTypeLog           |
| Mgr_StatsIntervalAdjusterTask  |
| Mgr_StorageArrayPlugin         |
| Mgr_StorageDeviceUpEvent       |
| Mgr_StorageElement             |
| Mgr_StorageSelectionManager    |
| Mgr_Tag                        |
| Mgr_TaskEngineProperties       |
| Mgr_TaskManager                |
| Mgr_TreeMap                    |
| Mgr_TreeStore                  |
| Mgr_User                       |
| Mgr_UserAccount                |
| Mgr_UserStore                  |
| Mgr_VirtualCdrom               |
| Mgr_VirtualDisk                |
| Mgr_VirtualMachine             |
| Mgr_VirtualMachineCfgFile      |
| Mgr_VirtualMachineDisconnectEr |
| Mgr_VirtualMachineRunningEvent |
| Mgr_VirtualMachineStartingEven |
| Mgr_VirtualMachineStoppedEvent |
| Mgr_VirtualMachineStoppingEven |
| Mgr_VirtualMachineSuspendedEve |
| Mgr_VirtualMachineTemplate     |
| Mgr_VmApiMessages              |
| Mgr_VmCloneDefinition          |
| Mgr_VmCloneNetworkMapping      |
| Mgr_VmCloneStorageMapping      |
| Mgr_VmDiskMapping              |
| Mgr_VmSelectionManager         |
| Mgr_Vnic                       |
| Mgr_VnicManager                |
| Mgr_VnicManagerProperties      |
| Mgr_VolumeGroup                |
| Mgr_XenHypervisor              |
| Mgr_YumRepoOutofDateEvent      |
| Mgr_YumUpdateCheckerTask       |
| Odof_id_to_type                |
| Odof_not_tabled                |
| Odof_sys_properties            |
| Odof_type_to_class             |
| WL_LLR_ADMINSERVER             |
+--------------------------------+
143 rows in set (0.00 sec)

Now lets look at the columns of the Mgr_VirtualMachine table.

mysql> describe Mgr_VirtualMachine;
+--------+------------+------+-----+---------+-------+
| Field  | Type       | Null | Key | Default | Extra |
+--------+------------+------+-----+---------+-------+
| m_id   | bigint(20) | NO   | PRI | 0       |       |
| m_data | longblob   | YES  |     | NULL    |       |
+--------+------------+------+-----+---------+-------+
2 rows in set (0.00 sec)

Now lets look at the columns of the Mgr_Server table.

mysql> describe Mgr_Server;
+--------+------------+------+-----+---------+-------+
| Field  | Type       | Null | Key | Default | Extra |
+--------+------------+------+-----+---------+-------+
| m_id   | bigint(20) | NO   | PRI | 0       |       |
| m_data | longblob   | YES  |     | NULL    |       |
+--------+------------+------+-----+---------+-------+
2 rows in set (0.00 sec)

Here is a command to pull the whole schema, and every single table has two tables, m_id and m_data with the m_data being longblog.

mysqldump --no-data ovs -S /u01/app/oracle/mysql/data/mysqld.sock -u root -p

Searching for the MySQL Package

Now when we search using the below command, you will notice that we are using the parameter -r this tells it to use the remote repository in addition to the local repository, this allows us to find software that we do not have installed on the machine.

# pkg search -r mysql
INDEX       ACTION VALUE                                                                   PACKAGE
pkg.summary set    A MySQL database adapter for the Python programming language            pkg:/library/python-2/python-mysql-26@1.2.2-0.175.1.0.0.11.0
pkg.summary set    Apache Portable Runtime Utility (APR-util) 1.3 DBD Driver for MySQL 5.0 pkg:/library/apr-util-13/dbd-mysql@1.3.9-0.175.1.0.0.24.0
pkg.summary set    MySQL Database Management System (Base)                                 pkg:/database/mysql-common@0.5.11-0.175.1.0.0.24.0
pkg.summary set    MySQL extension module for PHP                                          pkg:/web/php-53/extension/php-mysql@5.3.14-0.175.1.0.0.24.0
pkg.summary set    MySQL extension module for PHP                                          pkg:/web/php-52/extension/php-mysql@5.2.17-0.175.1.0.0.24.0
pkg.summary set    MySQL 5.1 Database Management System                                    pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
pkg.summary set    MySQL 5.1 libraries                                                     pkg:/database/mysql-51/library@5.1.37-0.175.1.0.0.24.0
pkg.summary set    MySQL 5.1 tests                                                         pkg:/database/mysql-51/tests@5.1.37-0.175.1.0.0.24.0
basename    file   usr/mysql/5.1/bin/amd64/mysql                                           pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
basename    file   usr/mysql/5.1/bin/mysql                                                 pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
basename    file   usr/mysql/5.1/bin/sparcv9/mysql                                         pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
name        group  mysql                                                                   pkg:/database/mysql-common@0.5.11-0.175.1.0.0.24.0
basename    link   usr/bin/mysql                                                           pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
name        user   mysql                                                                   pkg:/database/mysql-common@0.5.11-0.175.1.0.0.24.0
basename    dir    etc/mysql                                                               pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
basename    dir    usr/mysql                                                               pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
basename    dir    usr/mysql/5.1/include/mysql                                             pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
basename    dir    usr/mysql/5.1/share/mysql                                               pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
basename    dir    var/mysql                                                               pkg:/database/mysql-51@5.1.37-0.175.1.0.0.24.0
basename    dir    usr/mysql                                                               pkg:/database/mysql-51/library@5.1.37-0.175.1.0.0.24.0
basename    dir    usr/mysql/5.1/lib/amd64/mysql                                           pkg:/database/mysql-51/library@5.1.37-0.175.1.0.0.24.0
basename    dir    usr/mysql/5.1/lib/mysql                                                 pkg:/database/mysql-51/library@5.1.37-0.175.1.0.0.24.0
basename    dir    usr/mysql/5.1/lib/sparcv9/mysql                                         pkg:/database/mysql-51/library@5.1.37-0.175.1.0.0.24.0
basename    dir    usr/mysql                                                               pkg:/database/mysql-51/tests@5.1.37-0.175.1.0.0.24.0

Now in the output we are looking for a pkg.summary which is the software we are looking for, in our case pkg:/database/mysql-51 or simply mysql-51.

Once we think we have the right package I like to do a pkg info to make sure that it is what I expect, again here we want to look against remote repositories as well with the -r parameter.

# pkg info -r mysql-51
Name: database/mysql-51
Summary: MySQL 5.1 Database Management System
Category: Development/Databases
State: Not installed
Publisher: solaris
Version: 5.1.37
Build Release: 5.11
Branch: 0.175.1.0.0.24.0
Packaging Date: September  4, 2012 05:09:22 PM
Size: 147.23 MB
FMRI: pkg://solaris/database/mysql-51@5.1.37,5.11-0.175.1.0.0.24.0:20120904T170922Z

Install the MySQL Package

Here we can install MySQL 5.1 via the IPS repositories.

# pkg install mysql-51
Packages to install:  2
Create boot environment: No
Create backup boot environment: No
Services to change:  2

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                                2/2       252/252    52.2/52.2 16.3M/s

PHASE                                          ITEMS
Installing new actions                       343/343
Updating package state database                 Done
Updating image state                            Done
Creating fast lookup database                   Done 

Enable the MySQL Service

Now lets take a look at the service.  We can see that the service is installed but disabled.

# svcs -a | grep mysql
disabled       10:28:40 svc:/application/database/mysql:version_51

Enable the service.

# svcadm enable mysql
# svcs -a | grep mysql
online         10:30:26 svc:/application/database/mysql:version_51

Connect to MySQL

# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.37 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Please note this is not a secure MySQL configuration.  You will need to secure this before use.

Connect to MySQL

# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 5.1.37 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

Show the Available Databases

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| TST               |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)

Use the TST Database

Our example lives on the TST database.

mysql> use TST;
Database changed

List All Tables on TST

mysql> show tables;
+-------------------+
| Tables_in_TST     |
+-------------------+
| customer          |
| product           |
+-------------------+
2 rows in set (0.00 sec)

Describe the Customer Table

mysql> describe customer;
+-------------+-----------------------+------+-----+---------+----------------+
| Field       | Type                  | Null | Key | Default | Extra          |
+-------------+-----------------------+------+-----+---------+----------------+
| customer_id | mediumint(8) unsigned | NO   | PRI | NULL    | auto_increment |
| first_name  | varchar(30)           | NO   |     | NULL    |                |
| last_name   | varchar(30)           | NO   |     | NULL    |                |
| email       | varchar(30)           | NO   |     | NULL    |                |
+-------------+-----------------------+------+-----+---------+----------------+
4 rows in set (0.01 sec)

Describe the Product Table

mysql> describe product;
+--------------+--------------+------+-----+---------+-------+
| Field        | Type         | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| product_id   | mediumint(8) | YES  |     | NULL    |       |
| product_name | varchar(100) | YES  |     | NULL    |       |
+--------------+--------------+------+-----+---------+-------+
2 rows in set (0.00 sec)

Now just wash rinse and repeat for all tables…  OR

The Easy Way

No sense in killing ourselves.  Lets use a tool that does the heavy lifting for us.

# mysqldump --no-data TST -u root
-- MySQL dump 10.13  Distrib 5.1.37, for pc-solaris2.11 (i386)
--
-- Host: localhost    Database: TST
-- ------------------------------------------------------
-- Server version       5.1.37

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Table structure for table `customer`
--

DROP TABLE IF EXISTS `customer`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `customer` (
`customer_id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`first_name` varchar(30) NOT NULL,
`last_name` varchar(30) NOT NULL,
`email` varchar(30) NOT NULL,
PRIMARY KEY (`customer_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `product`
--

DROP TABLE IF EXISTS `product`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `product` (
`product_id` mediumint(8) DEFAULT NULL,
`product_name` varchar(100) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

-- Dump completed on 2013-02-05 16:12:23

Now we can spend our time interpreting and understanding the output instead of trying to recurse through every table in the database.

I have built in a few parameters.  We need to provide the OVM 3.x Manager server name, additionally we can provide the port, and username.  Both of which have default values.  Additionally I have added a dry run parameter which will give you the chance to see what the output would be.  Finally there is a verbose parameter which is self explanatory.

Please keep in mind that in OVM 3.x vdisk names are simple metadata.  This doesn’t change any names on disk.  But this does help when you have taken the time to name your vdisks when creating the VMs but have had to re-manage a hypervisor, as in that scenario you end up being able to rediscover the VMs themselves, and their names, but the vdisks will come up as named their UUID.

Please keep in mind that this script will make multiple SSH connections to the ovmcli, as such you will want to use keys to streamline this authentication.  I have an article covering that here.

Name       : rename-vdisks.sh
Version   :  1.2.1
MD5        :  c1a0cf9935a4cdcfd12f3eda71bfdd5d
SHA256  :  bd96192e20e5371b0124eec1c8372c97dbf8469b2c5538400461b82d7f43f5e2
URL         :  http://source.allanglesit.net/pub/rename-vdisks.sh

#!/bin/bash
# chkconfig:
# description:
#
#: Script Name    : rename-vdisks.sh
#: Version    : 1.2.1
#: Author    : Matthew Mattoon - http://blog.allanglesit.com
#: Date Created    : January 26, 2013
#: Date Updated    : February 20, 2013
#: Description    : Renames all OVM 3.x virtual disks to a standard naming convention.
#: Examples    : rename-vdisks.sh -m OVMMANAGER -p PORT -u USER -v
#:         : rename-vdisks.sh -m ovmserver.localdomain -v

usage()
{
cat << EOF
usage: $0 options

This script allows you to rename all OVM 3.x virtual disks to follow a consistent standard.

Standard:    vmname_system.img (slot 0)
vmname_dataX.img (slot 1-x where X is the slot number)
vmname1_vmname2_dataX.img (where a disk is shared between 2 VMs)

OPTIONS:
-h    Show this message
-m    OVM 3.x Manager Server (required).
-p    OVM 3.x Manager Port (default: 10000).
-u    OVM 3.x Manager User (default: admin).
-d   Dry Run.  Script will make no changes, but instead provide the output of an actual run.
-v   Verbose flag.
EOF
}

while getopts "hm:pudv" OPTION
do
case $OPTION in
h) usage; exit 1;;
m) ovmmgr=$OPTARG;;
p) ovmport=$OPTARG;;
u) ovmuser=$OPTARG;;
d) dryrun=1;;
v) verbose=1;;
?) usage; exit 1;;
esac
done

if [[ -z $ovmmgr ]]
then
usage
exit 1
fi

echo ""

if [[ -z $ovmuser ]]
then
ovmuser=admin
echo "Using default OVM User [ $ovmuser ]"
fi

if [[ -z $ovmport ]]
then
ovmport=10000
echo "Using default OVM Port [ $ovmport ]"
fi

if [ "$dryrun" = "1" ]
then
echo "Using Dry Run Option"
fi

if [ "$verbose" = "1" ]
then
echo "Using Verbose Option"
fi

echo ""
echo ""
echo `sed -n 3p $0 | sed 's/#://'`
echo `sed -n 5p $0 | sed 's/#://'`
echo `sed -n 4p $0 | sed 's/#://'`
echo ""
echo ""

for i in `ssh -p $ovmport $ovmuser@$ovmmgr "list vmdiskmapping" | grep -v 'OVM>\|Command:\|Status:\|Time:\|Data:' | sed 's/  /:/g'`
do
id=`echo $i | cut -d ":" -f 3`
echo "Examining $id  "
read name id slot emdev vdisk vm <<<$(ssh -p $ovmport $ovmuser@$ovmmgr "show vmdiskmapping id=$id" | grep -v 'OVM>\|Command:\|Status:\|Time:\|Data:' | sed 's/Emulated Block Device/EmulatedBlockDevice/' | sed 's/Virtual Disk Id/VirtualDiskId/' | sed 's/Vm Id/VmId/' | sed 's/  //' | sed 's/ = /=/g' | sed 's/  \[/:\[/g')
name=`echo $name | cut -d = -f 2`
id=`echo $id | cut -d = -f 2`
slot=`echo $slot | cut -d = -f 2`
emdev=`echo $emdev | cut -d = -f 2`
vdiskid=`echo $vdisk | cut -d = -f 2 | cut -d ":" -f 1`
vdiskname=`echo $vdisk | cut -d = -f 2 | cut -d ":" -f 2 | sed 's/\[\|\]//g'`
vm=`echo $vm | cut -d = -f 2 | cut -d ":" -f 2 | sed 's/\[\|\]//g'`
if [ "$slot" != "0" ]
then
slotname=data"$slot".img
else
slotname=system.img
fi
newvdiskname="$vm"_"$slotname"

imginfo=( `ssh -p $ovmport $ovmuser@$ovmmgr "show virtualdisk id=$vdiskid" | grep -v 'OVM>\|Command:\|Status:\|Time:\|Data:' | sed 's/Max (GiB)/Max(GiB)/' | sed 's/Used (GiB)/Used(GiB)/' | sed 's/Repository Id/RepositoryId/' | sed 's/Vm /Vm/' | sed 's/  //' | sed 's/ = /=/g' | sed 's/  \[/:\[/g'` )
imgname=`printf "%s\n" "${imginfo[0]}" | cut -d = -f 2`
imgid=`printf "%s\n" "${imginfo[1]}" | cut -d = -f 2`
imgmaxsize=`printf "%s\n" "${imginfo[2]}" | cut -d = -f 2`
imgusedsize=`printf "%s\n" "${imginfo[3]}" | cut -d = -f 2`
imgshareable=`printf "%s\n" "${imginfo[4]}" | cut -d = -f 2`
imgrepoid=`printf "%s\n" "${imginfo[5]}" | cut -d = -f 2 | cut -d : -f 1`
imgreponame=`printf "%s\n" "${imginfo[5]}" | cut -d = -f 2 | cut -d : -f 2 | sed 's/\[\|\]//g'`
if [ "$verbose" = "1" ]
then
echo "  VM Name         :  $vm"
echo "  Disk Id         :  $vdiskid"
echo "  Disk Name       :  $vdiskname"
echo "  Slot Number     :  $slot"
echo "  Shareable       :  $imgshareable"
fi
if [ -n "$vdiskid" -o  "$vdiskid" != *".iso" ]
then
if [ "$imgshareable" = "Yes"  ]
then
for item in `printf "%s\n" "${imginfo[@]}" | grep Vm | cut -d : -f 2 | sed 's/\[\|\]//g'`
do
shareddiskvmnames+="$item"_
done
newvdiskname="${shareddiskvmnames}${slotname}"
unset shareddiskvmnames
fi
if [ "$vdiskname" != "$newvdiskname" ]
then
if [ "$verbose" = "1" ]
then
echo "  Disk Name       :  $vdiskname"
echo "  New Disk Name   :  $newvdiskname"
fi
echo "Renaming [ $vdiskname ] to follow Standard [ $newvdiskname ]  "
if [ "$dryrun" = "1" ]
then
echo ""
else
ssh -p $ovmport $ovmuser@$ovmmgr "edit virtualdisk id=$vdiskid name=$newvdiskname" | grep "Status:"
fi
echo ""
else
echo "Name [ $vdiskname ] follows Standard [ $newvdiskname ]"
echo ""
fi
else
echo "Virtual CDROM Detected.  Rename not Required."
echo ""
fi
done

Today we are going over a fairly simple script.  I like to use all lower case in my VM naming conventions.  So I wrote up a little script to check every name in the environment and then rename it if it is not already lower case.  Then once I completed this I decided to change it to be a toggle, so that those of you who prefer upper case would not feel left out.  Basically what we are doing, is simply performing a few calls to the ovmcli to collect the information necessary to make our changes.  Then we perform the changes via a separate call.  Prior to making the edit call, we compare the actual value with the requested value in order to see if it is already what we wanted.

I have built a few parameters into the script, we can select the case, upper or lower, we can also select the OVM 3.x Manager Server, and the username and port.  The username and port are built with default values of admin and 10000.

Please keep in mind that this script will make multiple SSH connections to the ovmcli, as such you will want to use keys to streamline this authentication.  I have an article covering that here.

Name       : changecase-vmname.sh
Version   :  1.0.1
MD5        :  5006c9ff20b3301e66ac503f417b98ee
SHA256  :  bfff98b74c54f1fec3ec540a22ca4b1377c9f4b25e0b90ddbb722eeb64bf6bc8
URL         :  http://source.allanglesit.net/pub/changecase-vmname.sh

#!/bin/bash
# chkconfig:
# description:
#
#: Script Name    : changecase-vmname.sh
#: Version    : 1.0
#: Author    : Matthew Mattoon - http://blog.allanglesit.com
#: Date Created    : January 26, 2013
#: Date Updated    : February 20, 2013
#: Description    : Changes case of all Oracle VM 3.x VMs to either upper or lower case.
#: Examples    : changecase-vmname.sh -c CASE -m OVMMANAGER -p PORT -u USER
#:         : changecase-vmname.sh -c lower -m ovmserver.localdomain

usage()
{
cat << EOF
usage: $0 options

This script allows you to change case on all OVM 3.x VMs to either uppercase or lowercase.

OPTIONS:
-h    Show this message
-c    The preferred case for all VMs (required). upper or lower
-m    OVM 3.x Manager Server (required).
-p    OVM 3.x Manager Port (default: 10000).
-u    OVM 3.x Manager User (default: admin).
EOF
}

while getopts "hc:m:pu" OPTION
do
case $OPTION in
h) usage; exit 1;;
c) case=$OPTARG;;
m) ovmmgr=$OPTARG;;
p) ovmport=$OPTARG;;
u) ovmuser=$OPTARG;;
?) usage; exit 1;;
esac
done

if [[ -z $case ]] || [[ -z $ovmmgr ]]
then
usage
exit 1
fi

if [ $case != "lower" -a $case != "l" -a $case != "upper" -a $case != "u" ]
then
usage
exit 1
fi

if [[ -z $ovmuser ]]
then
ovmuser=admin
fi

if [[ -z $ovmport ]]
then
ovmport=10000
fi

if [ $case = "lower" -o $case = "l" ]
then
trcase="'[:upper:]' '[:lower:]'"
fi

if [ $case = "upper" -o $case = "u" ]
then
trcase="'[:lower:]' '[:upper:]'"
fi

for i in `ssh -p $ovmport $ovmuser@$ovmmgr "list vm" | grep -v 'OVM>\|Command:\|Status:\|Time:\|Data:' | sed 's/  /:/g'`
do
id=`echo $i | cut -d ":" -f 3`
echo -n "Examining $id...  "
vmname=`ssh -p $ovmport $ovmuser@$ovmmgr "show vm id=$id" | grep -v 'OVM>\|Command:\|Status:\|Time:\|Data:' | grep "Name" | sed 's/ = /=/' | cut -d = -f 2`
newvmname=`echo $vmname | tr $trcase`
if [ $vmname != $newvmname ]
then
echo "Renaming $vmname to $newvmname"
ssh -p $ovmport $ovmuser@$ovmmgr edit vm id=$id name="$newvmname" | grep "Status:"
else
echo "Rename not required"
fi
done

Allow A Specific Command Execution as Root (The Real Sudo)

This first policy example is what sudo is for.  This is where you want individuals to perform a specific action as root (for example restart a service which is known to be flaky instead of calling you at 2AM).  These commands can be specific down to parameters and everything.  Of course once the sudo policy is in place you could even go a step further and create aliases in their user profile so that they can execute simpler commands.  For example “/etc/init.d/network restart” can be aliased to “internet-reboot” or something equally simple/ridiculous.

# cat /etc/sudoers
# Specific Command Execution
Defaults logfile=/var/log/sudo.log
User_Alias ROOTCOMMAND = user1, user2
Cmnd_Alias REBOOT = /sbin/reboot
Cmnd_Alias SERVICE = /etc/init.d/network restart
ROOTCOMMAND ALL=NOPASSWD: REBOOT, SERVICE

One other small thing I did in this example.  Since this has a VERY restrictive policy and we are only allowing to possible commands to be run I have removed the password requirement. I do not allow this on a wider policy.  This comes back to the saying:

Quick, Easy, Secure – pick two.

Allow Root Access with Some Disallowances

This policy is a much safer policy for individuals who are not fully trusted, for example Junior Sys Admins.  This policy closes most of the holes which would allow them to inadvertently or intentionally lock out other individuals from the machine, the key difference between this and the above policy is that it requires all commands be executed via sudo, instead of being able to spawn a shell as root, and bypass the logging.

# cat /etc/sudoers
# Root with Restrictions
Defaults logfile=/var/log/sudo.log
User_Alias ROOTRESTRICTED = user1, user2
Cmnd_Alias SHELLS = /bin/sh, /bin/bash, /bin/ash, /bin/bsh, /bin/tcsh, /bin/csh, /bin/ksh, /bin/ksh93
Cmnd_Alias SU = /bin/su, /usr/bin/vncserver
Cmnd_Alias PASSWD = /bin/passwd, /usr/sbin/useradd, /usr/sbin/userdel, /usr/sbin/usermod
Cmnd_Alias VISUDO = /usr/sbin/visudo
ROOTRESTRICTED ALL = ALL, !SHELLS, !SU, !PASSWD, !VISUDO

Allow Unrestricted Root Access

This is the basics of a policy that I use on my workstations.  This also fits very well on servers where you have a requirement to not login to the root account except in an emergency.

This policy has NO restrictions on it.  As such the only people whom you will want to grant this policy to are the same folks you would provide your root password to.  This policy also allows the direct invocation of a shell as root, which gives you a root shell, which after that nothing is logged in the sudo.log.

# cat /etc/sudoers
# Root Equivalent
Defaults logfile=/var/log/sudo.log
User_Alias ROOTEQUIVALENT = user1, user2
ROOTRESTRICTED ALL = ALL

Switch Users Using Sudo

Here is another little tidbit I like to use on systems which require multiple system accounts to run multiple applications.  Instead of having to track and know each user and password combination (or having someone else have to do the same) I like to enable users to use sudo to invoke su to switch to the specific user.  This of course means that they will not be prompted to enter a password for the user that they are logging into but they will be logging that entry into sudo.

# cat /etc/sudoers
# Switch User Without Additional Password Management
Defaults logfile=/var/log/sudo.log
User_Alias TSTEBSUSER = bob, bill
Cmnd_Alias SUTSTEBS = /bin/su tstebs, /bin/su - tstebs
TSTEBSUSER ALL = SUTSTEBS
User_Alias TSTWLSUSER = bob, mark, jon
Cmnd_Alias SUTSTWLS = /bin/su tstebs, /bin/su - tstebs
TSTWLSUSER ALL = SUTSTWLS

Above we can see that we have two environments to be managed our TSTWLS and our TSTEBS environment.  Each of these environments is executed and managed using the tstwls and tstebs users respectively.  We can also see that according to the policy we have a user named bob with access to both environments.  The TSTEBS environment also has a user named bill who has access to the environment.  While on TSTWLS we additionally have mark, and jon who have access to the environment.

These are the most common use cases I have run into for using sudo.  There are many more policy restrictions (noexec – disallow the execution of scripts) which are detailed in the man pages for sudo.

Prior to sudo whenever you wanted to run a command as a specific user then you would su (switch user) to gain access to a shell for that user.  This of course required that you knew the password for that user, and it would spawn a new shell after successful authentication.  The major drawback to this was that in order to allow a user to do something, you had to allow the user to do anything.  This sort of configuration also makes it fairly trivial for an authorized user to perform unauthorized actions, such as changing the root password and locking everyone out.

Basically sudo is a setuid binary which allows any user to invoke it as root.  Here is where it starts to get a bit tricky.  Any user can invoke sudo as root.  However that doesn’t mean that any user can invoke anything with sudo as root.  Sudo itself is invoked as root, and then the policy processing begins and sudo determines if it is going to do what you are requesting (if it is in policy).

Remember Su-Do

If you remember nothing else about sudo (as a user) then remember this.  Sudo allows you to “su and do” without having to “su and do”.  This of course means that the below actions are identical in terms of executing the action.

$ su -
Password:
# whoami
root

$ sudo whoami
[sudo] password for oracle:
root

The only real difference in the above is the password that we enter.  In the first example we need to enter the root password.  In the second we enter the password of the user we are logged in as.

Sudo Usage

A great example to illustrate the functionality of sudo is the simple “id” this command simply lists your security contexts.  When we execute id

$ id
uid=1000(matthew) gid=1000(matthew) groups=1000(matthew),10(wheel),18(dialout),1001(vboxusers)

Above we can see that we are logged in with the user matthew with the uid of 1000.  You also see the groups that I am a member of.

$ sudo id
[sudo] password for matthew:
uid=0(root) gid=0(root) groups=0(root)

Now when we insert sudo in front of the id command we notice that it prompts for a password, specifically the password of the user invoking sudo (in this case matthew).  After successful authentication then we see that we actually get the id information for the root user.

There is another major way that sudo can be used.  And that is to achieve a root shell.

$ sudo -s
[sudo] password for matthew:
#

Above we see that we now have a persistent root shell.  The big takeaway from this usage is that you lose all of your sudo logging.  The only thing that is logged is the execution of the shell (this will be the users shell defined in the /etc/passwd – usually /bin/bash).

# more /var/log/sudo.log
Jan 22 16:04:09 : matthew : TTY=pts/8 ; PWD=/home/matthew ; USER=root ;
COMMAND=/bin/bash

Sudo Logging

One of the biggest benefits of sudo is the ability to log what individuals are doing on a system, and thus having the ability to forensically resolve problems based on the actions that were taken just prior to the problem.  Here is an example of what the output looks like.

# more /var/log/sudo.log
Jan 22 13:25:13 : user1 : TTY=pts/0 ; PWD=/home/user1 ; USER=root ;
COMMAND=/etc/init.d/network restart
Jan 22 13:25:34 : user1 : command not allowed ; TTY=pts/0 ; PWD=/home/user1 ;
USER=root ; COMMAND=/etc/init.d/network status

Now we can see that we have a successful execution of /etc/init.d/network restart on line 1.  This took place at Jan 22 13:25:13 local time.  We also know that it was invoked by the oracle user, while the oracle user was standing in /home/oracle (this is important for anything executed using a relative path).  We also then see the user that the command is executed as (which in this case is root).

The second entry we have is an attempt by the oracle user to execute /etc/init.d/network status.  This was denied because it is outside the allowance of the sudo policy.

List Effective Sudo Policy

You won’t always be in an environment where you will be able to push for changes to your sudo policies, however it doesn’t mean that you can’t look at the effective ruleset to see what you have in effect.  To list the policies as they are in effect against your user.

$ sudo -l
Matching Defaults entries for matthew on this host:
requiretty, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR
LS_COLORS", env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE",
env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES",
env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME
LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin, logfile=/var/log/sudo.log

User matthew may run the following commands on this host:
(ALL) ALL

If you are building policies and you want to test the effective policies for a specific user.  You can invoke it like so to list the effective policy.

# sudo -U matthew -l
Matching Defaults entries for matthew on this host:
requiretty, !visiblepw, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC
KDEDIR LS_COLORS MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE LC_COLLATE
LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER
LC_TELEPHONE LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
logfile=/var/log/sudo.log

Runas and Command-specific defaults for matthew:

User matthew may run the following commands on this host:
(root) ALL, (root) !/bin/sh, !/bin/bash, !/bin/bash2, !/bin/ash, !/bin/bsh,
!/bin/tcsh, !/bin/csh, !/bin/ksh, !/bin/ksh93, (root) !/bin/su, !/usr/bin/vncserver,
(root) !/bin/passwd, !/usr/sbin/useradd, !/usr/sbin/userdel, !/usr/sbin/usermod,
(root) !/usr/sbin/visudo

That wraps up Part One.  In Part Two we will look at specific policy examples and how they would be used.

Connect to OVM CLI with Password

$ ssh -p 10000 admin@ovmmgr.allanglesit.com
admin@ovmmgr.allanglesit.com's password:
OVM>

Authorize Keys for OVM CLI

To allow key based authentication to OVM CLI simply add your public key on your OVM Manager Server to /home/oracle/.ssh/ovmcli_authorized_keys.  For our environment we already had everyone who needed access in the authorized_keys for root.

# cat /root/.ssh/authorized_keys >> /home/oracle/.ssh/ovmcli_authorized_keys

However if you wanted to do it remotely using the public key file directly from your client.

# cat .ssh/id_rsa.pub | ssh root@ovmmgr.allanglesit.com "cat >> /home/oracle/.ssh/ovmcli_authorized_keys"

Connect to OVM CLI with Public Keys

Now our connection is without that pesky password prompt.

$ ssh -p 10000 admin@ovmmgr.allanglesit.com
OVM>

The Way It Is

From the hypervisor if we use native Xen commands to look at all of the instantiated VMs.

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
0004fb000006000002a00feb1f52cae4           155  8192     2     -b----  39176.1
0004fb00000600000b5b1f8673c451be           171  8192     2     -b---- 1007276.5
0004fb000006000026a40b6fb53fe2c3           135 16384     2     -b---- 169716.7
0004fb0000060000506f591d6877c284           194  3072     1     -b----  16108.4
0004fb00000600007052ad7b040749ea           176  8192     2     -b---- 1726118.2
0004fb0000060000a07b66923b0f68a4           138  4096     2     -b---- 2092394.0
0004fb0000060000abfcf710ebd640df           101  2048     2     -b---- 540488.4
0004fb0000060000bf16256b99159fc6           143 16384     2     -b---- 3972112.7
0004fb0000060000e3f234f6c1d26448           173  8192     2     -b---- 852019.6
0004fb0000060000e602b0e0969dba07            92  2048     2     -b---- 234244.0
0004fb0000060000f9205e11159e5459           175 12288     2     r----- 4145289.3
Domain-0                                     0  2486    24     r----- 4455022.6

Now since under the covers we are using Xen that means that somewhere there is a vm.cfg file (could be named anything, but for now they are sticking with vm.cfg).

# cat vm.cfg
vif = ['mac=00:21:f6:00:00:07,bridge=172.16.88.0']
OVM_simple_name = 'PRD-IM'
guest_os_type = 'linux'
disk = ['file:/OVS/Repositories/0004fb00000300005527ce584481a013/VirtualDisks/0004fb000012000052e7181e3458aba7.img,xvda,w']
vncunused = '1'
uuid = '0004fb00-0006-0000-506f-591d6877c284'
on_reboot = 'restart'
boot = 'cn'
cpu_weight = 27500
memory = 3072
cpu_cap = 0
maxvcpus = 2
OVM_high_availability = False
vnclisten = '127.0.0.1'
vnc = '1'
OVM_description = 'Keste IM - Prod'
on_poweroff = 'destroy'
on_crash = 'restart'
maxmem = 3072
name = '0004fb0000060000506f591d6877c284'
builder = 'hvm'
vcpus = 1
keymap = 'en-us'
OVM_os_type = 'Oracle Linux 6'
OVM_cpu_compat_group = ''
OVM_domain_type = 'xen_hvm'

But finding it can be tricky.  All repositories are stored in UUID form under /OVS/Respositories.  Then inside of each repo you have multiple folders, Assemblies, ISOs, Templates, VirtualDisks, VirtualMachines.  They are pretty self explanatory, but vm.cfgs go under VirtualMachines, they then refer to virtual disk images in the VirtualDisks folders.  Here is an example for the vm.cfg we looked at above.

# pwd
/OVS/Repositories/0004fb00000300005527ce584481a013/VirtualMachines/0004fb0000060000506f591d6877c284

This breaks down like this…

/OVS/Repositories/ [UUID of REPO] /VirtualMachines/ [UUID of VM] /vm.cfg

The Problem

If we lose the manager, then we lose all ability to manage the environment without heavy navigation of the file system, or if we discover a hypervisor which was previously managed by the same OVM Manager UUID then we end up with a bunch of Orphaned VMs.  You will notice that not only is there no meaningful names, there is also no meaningful information about the VMs, I assure that not all of my VMs have 256MB of RAM with a single CPU.

Figure 1-1: Orphaned VMs

The Solution

Now the crux of this problem is that while we have discovered the Hypervisor we have not discovered the underlying repository.

Figure 1-2: Our Server Pool and Hypervisor

So first thing we need to do is right-click on the hypervisor and “Rescan Physical Disks” this should complete fairly quickly.  This will allow OVM Manager to be aware of the Physical Disk on which we have our repository with our running VMs.  Disregard the warning icon, that is due to having an older version of the hypervisor on that machine (3.1.1)

Figure 1-3: Rescan Physical Disks Job

Here we simply monitor the job until completion.  After it is complete we right-click on the hypervisor and “Rediscover Server” this allows the OVM Manager to look at the physical disk that it just found out about to see if there is a repository on it.

Figure 1-4: Rediscover Server Job

Monitor until the job completes.

Figure 1-5: New Repository Visible

Now we see a UUID for a new repository under the Repositories tab.  Right-click and Refresh.

Figure 1-6: Repository Refresh Job

Monitor until the job completes.

Figure 1-7: Fixed VMs

Now our VMs show the correct information.

A Few Notes

Any names that you assigned VirtualDisks are gone, they are back to UUID, sorry.  Also if you had any machines that are stopped when you perform this action you will not see them on the hypervisor.  They are in the Unassigned Virtual Machines folder.  Click and drag to re-assign them.  Or use the Migrate button.

About HDDR

Basically HDDR is a non-profit organization which want to help people afflicted by disasters, they have been all over the world working in disaster areas trying to help people clean up and rebuild their lives.  At the center of this ministry is Albert Knight, he is a contractor with a passion for helping people.  He coordinates volunteer teams of all skill levels to complete the work that is being done under the organization.  While HDDR is a Christian Organization they do not require volunteers have an affiliation with a specific church or even religion.  People of all faiths and lifestyles are encouraged to get involved.  Another thing that HDDR does differently is they really try to defray some of the costs of your participation in the relief efforts.  They are able to provide housing and food for groups, leaving you with only the costs associated with getting there.

About Al

We came to work, and to be honest Al didn’t expect alot from us, all he really wanted was 8AM – 4PM so he was surprised when our crew went back out after dinner (most nights) to finish up some jobs.  But the really important thing for you to understand about Al, is that he really burns for this ministry.  While this is in his backyard.  This is by no means without sacrifice for him.  Al has a home in Philadelphia, PA but during the week he is living with the volunteers in Atlantic City, NJ so that they can get an early start.

Lessons Learned

Al runs a really tight ship.  We lead a team of 9 to New Jersey from Dallas, TX.  This was a 25-27 hour drive.  We had two pickups one of which was packed to the gills with tools and gear.  Looking back we brought too much.  Al already had most everything that was needed.  That said I would recommend that everyone who plans on going forget all of the tools that they think they will need (Al has all of those) and instead focus on having a basic toolkit for each member of your team.

• Tool Belt / Leathers
• Hammer
• Tape Measure
• Carpenters Pencils (at least 3)
• Utility Knife or Break-Away Razor
• 1 each Phillips and Standard Screwdriver
• Work Boots (steel toe)

Now had we packed the above we would have needed one storage box which could have held all of these, instead of half of a pick-up bed.  And honestly the only other things that we used were things that Al already had (skil saw, nail gun, miter saw) but we used ours because the pickup was 10 feet closer than the trailer.

How Can I Help?

Oh here it comes, the part where he tries to pry money out of my wallet…  Honestly though, HDDR and Al don’t want your money.  But they do need your skilled or unskilled labor.  They really can work with all skill levels, and they will put you to work at a level where you feel comfortable and challenged, so you won’t just be picking up garbage.  He takes the time to show you what to do, and then he lets you run and do it.  Now all that said if you are not able to give of your time but you’d like to give of your resources instead they will be more than happy to receive that donation and will make sure that it is maximized.  You have a couple of options…

1. Buy a shirt or sweater on the HDDR website
2. Make a cash donation, or a donation of specific supplies and gift cards
3. HDDR is working on getting PayPal set up so that they can streamline the donation process, in the interim if you’d like to make a donation to them through my site using the link on the bottom of this post please feel free to do so.  Note: if you choose to do so please understand that I will forward the money to HDDR as quickly as possible (I would guess they should be able to have it inside of a week and a half based on clearing times) and I encourage you to coordinate with HDDR to ensure that the money is received (and that you get all appropriate tax receipts).

The Videos

We had some guys with us who are creative geniuses who were able to capture quite a bit of great footage and edit it together into daily videos.

Things I Tried

• Updating the Windows 7 Print Driver
• Updating the Printers Firmware
• Deleting the Printer and Recreating it in Window

None of the above “fixed” the problem.  However I was able to find a solution.  The problem apparently was Bi-Directional Support.

1. Click on Start
2. Select Devices and Printers
3. Right Click on the Symptomatic Printer
4. Select Printer Properties
5. Click on the Ports Tab
6. Uncheck Enable Bidirectional Support
7. Click Apply and Re-Test

Install Crashplan Application

#mv CrashPlan_2010-03-08_Solaris.tar.gz /var/spool/pkg/

#tar -xzvf CrashPlan_3.4.1_Solaris.tar.gz

#/usr/sbin/pkgadd

Create a Service Bundle

# svcbundle -o crashplanengine.xml -s service-name=application/crashplanengine -s model=daemon -s start-method="/opt/sfw/crashplan/bin/CrashPlanEngine start

Import the Service Bundle

# mv crashplanengine.xml /lib/svc/manifest/site/

# svcadm restart manifest-import

Check the Status of our Service

# svcs -a | grep crashplan
online         14:38:53 svc:/application/crashplanengine:default

Since I have done this my Crashplan service has been much more stable and my backups have been completing more reliably.

REF: http://support.crashplan.com/doku.php/getting_started/installing_crashplan

Enter the Service Management Facility (SMF) which is built into Solaris starting in Solaris 10.  Basically the idea is that starting and stopping services is similar enough from service to service, that why should we be writing/modifying scripts from service to service.  Why not simply tell it what command to run when starting and then the SMF does the heavy lifting.  The biggest benefit of this is that the SMF can work with other Solaris components to detect the failure of a service and then automatically start it again, making these services more resilient.  This is accomplished by simply acknowledging that services are not simply user processes and thus should not be treated as such.

Now as an example I have moved my mercurial repositories from Linux to a Solaris zone, mercurial has a functionality which can share repositories via HTTP.  This can be done ad-hoc by simply executing the following command.

Determine the Command to Execute

$ hg serve

This will share via HTTP on port 8000.  Now this isn’t entirely how I want it to operate so lets start by creating a configuration file to change some options.

root@source:~# more /etc/mercurial/hgpub.config
[paths]
hg/ = /rpool/repo/hg/pub/*

[web]
allow_archive = gz

So now to reference the above configuration file we need to modify the command we will execute.

$ hg serve --webdir-conf /etc/mercurial/hgpub.conf

Now we are getting closer.  Another thing I like to do is define the port number, even though I am using the default of 8000.

$ hg serve --webdir-conf /etc/mercurial/hgpub.conf -p 8000

Now we need to use the daemon mode flag to start this in the background and not the foreground.

$ hg serve --webdir-conf /etc/mercurial/hgpub.conf -p 8000 -d

A final test will reveal that we are ready to turn this process into a service.  Kill the service that you manually started.

Create a SMF Bundle

SMF Bundles are XML files which will tell the SMF what to do and when.  You can either copy one that is for another service and modify it to suit or you can use svcbundle to create one specific to your requirements.

# svcbundle -o hgpub.xml -s service-name=application/hgpub -s model=daemon -s start-method="/usr/bin/hg serve --webdir-conf /etc/mercurial/hgpub.config -p 8000 -d"

Inspect and Modify the SMF Bundle

# cat hgpub.xml
<?xml version="1.0" ?>
<!DOCTYPE service_bundle
SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
Manifest created by svcbundle (2013-Jan-04 21:10:40-0600)
-->
<service_bundle type="manifest" name="application/hgpub">
<service version="1" type="service" name="application/hgpub">
<!--
The following dependency keeps us from starting until the
multi-user milestone is reached.
-->
<dependency restart_on="none" type="service"
name="multi_user_dependency" grouping="require_all">
<service_fmri value="svc:/milestone/multi-user"/>
</dependency>
<exec_method timeout_seconds="60" type="method" name="start"
exec="/usr/bin/hg serve --webdir-conf /etc/mercurial/hgpub.config -p 8000 -d"
/>
<!--
The exec attribute below can be changed to a command that SMF
should execute to stop the service.  See smf_method(5) for more
details.
-->
<exec_method timeout_seconds="60" type="method" name="stop"
exec=":kill"/>
<!--
The exec attribute below can be changed to a command that SMF
should execute when the service is refreshed.  Services are
typically refreshed when their properties are changed in the
SMF repository.  See smf_method(5) for more details.  It is
common to retain the value of :true which means that SMF will
take no action when the service is refreshed.  Alternatively,
you may wish to provide a method to reread the SMF repository
and act on any configuration changes.
-->
<exec_method timeout_seconds="60" type="method" name="refresh"
exec=":true"/>
<!--
We do not need a duration property group, because contract is
the default.  Search for duration in svc.startd(1M).
-->
<instance enabled="true" name="default"/>
<template>
<common_name>
<loctext xml:lang="C">
<!--
Replace this comment with a short name for the
service.
-->
</loctext>
</common_name>
<description>
<loctext xml:lang="C">
<!--
Replace this comment with a brief description of
the service
-->
</loctext>
</description>
</template>
</service>
</service_bundle>

Here we can add a service common name and description to make our service a little easier to use.

1# cat hg-pub.xml
<?xml version="1.0" ?>
<!DOCTYPE service_bundle
SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
Manifest created by svcbundle (2013-Jan-11 11:43:52-0600)
-->
<service_bundle type="manifest" name="application/hgpub">
<service version="1" type="service" name="application/hgpub">
<!--
The following dependency keeps us from starting until the
multi-user milestone is reached.
-->
<dependency restart_on="none" type="service"
name="multi_user_dependency" grouping="require_all">
<service_fmri value="svc:/milestone/multi-user"/>
</dependency>
<exec_method timeout_seconds="60" type="method" name="start"
exec="/usr/bin/hg serve --webdir-conf /etc/mercurial/hgpub.config -p 8000 -d"
/>
<!--
The exec attribute below can be changed to a command that SMF
should execute to stop the service.  See smf_method(5) for more
details.
-->
<exec_method timeout_seconds="60" type="method" name="stop"
exec=":kill"/>
<!--
The exec attribute below can be changed to a command that SMF
should execute when the service is refreshed.  Services are
typically refreshed when their properties are changed in the
SMF repository.  See smf_method(5) for more details.  It is
common to retain the value of :true which means that SMF will
take no action when the service is refreshed.  Alternatively,
you may wish to provide a method to reread the SMF repository
and act on any configuration changes.
-->
<exec_method timeout_seconds="60" type="method" name="refresh"
exec=":true"/>
<!--
We do not need a duration property group, because contract is
the default.  Search for duration in svc.startd(1M).
-->
<instance enabled="true" name="default"/>
<template>
<common_name>
<loctext xml:lang="C">
hgpub
</loctext>
</common_name>
<description>
<loctext xml:lang="C">
Mercurial Web Server for Public Repositories
</loctext>
</description>
</template>
</service>
</service_bundle>

Import the Service Bundle

# mv hgpub.xml /lib/svc/manifest/site

Now just restart the manifest-import service to pick up the service.

# svcadm restart manifest-import

Now we can check for unhealthy services by using the below command.

# svcs -xv
svc:/system/manifest-import:default (service manifest import)
State: offline since January 11, 2013 12:02:15 PM CST
Reason: Start method is running.
See: http://support.oracle.com/msg/SMF-8000-C4
See: man -M /usr/share/man -s 5 smf_bootstrap
See: /var/svc/log/system-manifest-import:default.log
Impact: This service is not running.

Above shows you what you will see if you check the services to quickly, you will see the manifest-import service itself in a non-running state, this is due to the import.  Give it a couple seconds and try again and it should clear.

I also saw errors when I did the following.

• Created Services with port conflicts
• Forgot to put the service into daemon mode in the command (use & or nohup for utilities/servers which do not have a daemon mode parameter)
• Created Services with duplicate names

Below is an example of the output that I got when I forgot to put the service into daemon mode.  The log file mentioned gives much more information on what is actually happening, in this case it is starting it, waiting, and then killing it, over and over again.

# svcs -xv
svc:/application/hgpub:default (hgpub)
State: offline* transitioning to online since January 11, 2013 11:58:49 AM CST
Reason: Start method is running.
See: http://support.oracle.com/msg/SMF-8000-C4
See: /var/svc/log/application-hgpub:default.log
Impact: This service is not running.

If you make a mistake and want to start over then you need to

1. Disable the service (svcadm disable hgpub)
2. Delete the service (svccfg delete hgpub)
3. Delete the service bundle in /lib/svc/manifest/site/ (rm /lib/svc/manifest/site/hgpub.xml)
4. Restart the manifest-import service (svcadm restart manifest-import) - I think a refresh would work here as well, but I have not tested that

REF: http://mercurial.selenic.com/wiki/hgserve

I own both a Google Galaxy Nexus (the phone) and a Google Nexus 7 (the tablet). Both of these devices have served me well in the time I had them. Until the Over-the-Air update for Android 4.2 on my Galaxy Nexus. This really ruined my day. Basically the long and short of it is that the battery took a nose dive after installing this update. Immediately I went from a full day of battery life (only charging it while I slept) to approx 3 hours of battery life. As soon as that happened I changed my email checking settings to manual, so that it would not constantly poll, I also disabled wifi, bluetooth etc. However there was a little gem of information in the Battery Usage reports. Basically it was reporting that ~85% of my battery was being used by “Exchange Services” no longer was the screen the biggest consumer of my juice.

After disabling all of these “extra” services my battery report changed a bit, basically Exchange Services went to ~81%. Still flatly unacceptable. Especially considering that this would only give approximately 5 hours of battery time.

The Solution

Delete your ActiveSync Mail Account, and re-add it. Before you do this make sure you look over your Sync settings so you know what you are Syncing (email, calendars, contacts) so that you can reconfigure your Sync settings appropriately.

Now I am back to once a day charging except on the heaviest of conference call days.

REF: http://code.google.com/p/android/issues/detail?id=39728

Libvirt Versions

Here is the libvirt version present in Ubuntu 12.10.

# libvirtd --version
libvirtd (libvirt) 0.9.13

Here is the libvirt version present in Ubuntu 12.04.

# libvirtd --version
libvirtd (libvirt) 0.9.8

So as we can see above this guide will only work on Ubuntu 12.10 and not on 12.04 or older.  For Ubuntu 12.04, please refer to my older article available here.

Install Updates and Prerequisite Software

Check for updates, and install some commonly used hypervisor utilities along with the openvswitch and kvm stacks.  Note that below I am installing the openvswitch-brcompat package.  I do not think that you need this, however I did install it in my environment in case I was going to need to fail back to the brcompat route, and I haven’t had the time to validate it without that package installed.  But it is not being used in this configuration.

# apt-get install aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv openvswitch-controller openvswitch-brcompat openvswitch-switch

Destroy the Default Libvirt Bridge (virbr0)

We want to delete the default libvirt interface which is created by default to be used for guests (I think it is NAT – I never use it).

# virsh net-destroy default
# virsh net-autostart --disable default

Stop Libvirt and Qemu

This will prevent libvirt from bringing up the legacy bridge.

# service libvirt-bin stop
# service qemu-kvm stop

Purge Ebtables from System

Ebtables was needed when we were using VLANs on bridge-utils, we no longer need this for openvswitch.

# aptitude purge ebtables

Restart the Openvswitch Services

#service openvswitch-switch restart
#service openvswitch-controller restart

Configure Interfaces

Now this configuration is a single interface which will be used both for management of the hypervisor as well as the bridge.  There are three key things to note.  We need to set the physical interface to “manual” (instead of “dhcp” or “static”).  We also need to perform a quick up and down of the physical interface in order to be able to initialize the bridge on it.  Finally we need to setup the configuration of an interface which will connect up to the bridge that we define, this becomes the management interface for the OS.  In this example we are simply using DHCP.  This will show us if it is working quickly, but it can also add some complexity if things are not working, please keep in mind that you might want to just use a static IP address to eliminate that as an issue.

# cat /etc/network/interfaces
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto ovsbr0p1
iface ovsbr0p1 inet dhcp

Configure our Network

Configuring the network will be a three step process.  First we define the bridge which I am naming ovsbr0 unlike legacy bridges we don’t need to stick to brX as a naming convention.

# ovs-vsctl add-br ovsbr0

Second we connect our bridge with its uplink which in this case is eth0.

# ovs-vsctl add-port ovsbr0 eth0

Finally we create a port for our hypervisor to connect up to the bridge and out the physical interface.

# ovs-vsctl add-port ovsbr0 ovsbr0p1 -- set interface ovsbr0p1 type=internal

After all of that we should end up with something like below.

# ovs-vsctl show
ed1986fc-830e-4f66-9ce7-92fad0787bb8
Bridge "ovsbr0"
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "eth0"
Interface "eth0"
ovs_version: "1.4.3"

Reboot

# reboot

Adjusting Service Sleeps

If your networking seems to hang on boot up.

In /etc/failsafe.conf we need to adjust the sleeps.

Change this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 40
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 59
$PLYMOUTH message --text="Booting system without full network configuration..." || :

To this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Booting system without full network configuration..." || :

Obviously if you end up needing this change, make sure you reboot again to ensure that it resolved the issue.

Define VM Configuration

I am not going to go into details on how to create a guest from XML, but if you need some help on that refer to my previous article here.

Basically below we have a snippet from a guests XML configuration which configures a single network interface.  You can edit an existing machine configuration using “virsh edit vmnamehere”

<interface type='bridge'>
<mac address='52:54:00:0f:b4:7a'/>
<source bridge='ovsbr0'/>
<virtualport type='openvswitch'/>
<model type='virtio'/>
</interface>

After you have merged the changes in, libvirt will go create a port on Openvswitch for this guest.  Which we can see below as vnet0.

# ovs-vsctl show
ed1986fc-830e-4f66-9ce7-92fad0787bb8
Bridge "ovsbr0"
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "vnet0"
Interface "vnet0"
Port "eth0"
Interface "eth0"
ovs_version: "1.4.3"

Additionally we can see that in our VM configuration libvirt has added more details, the target dev and parameters interfaceid.  This is how the VM knows which port on the Openvswitch to connect to.

<interface type='bridge'>
<mac address='52:54:00:0f:b4:7a'/>
<source bridge='ovsbr0'/>
<virtualport type='openvswitch'>
<parameters interfaceid='a78ac510-2442-1326-c60f-d76c1e830bd6'/>
</virtualport>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

That is all there is to it.  When you bring up your guest. you will have networking exactly as you expected.  But now you open yourself up to a world of possibilities with Openvswitch, such as port mirroring, QoS, LACP and many more.

Usability Comparison to ZFS

To give you an idea of what I would expect lets look at how it works in ZFS.

In ZFS you configure a pool of disks, where we have disk0-4 and want to name the pool tank.

# zpool create tank mirror disk0 disk1 mirror disk2 disk3

Then you configure file systems on top of that, in this case fs1 on tank with a size of 10G.

# zfs create 10G tank/fs1

Regardless of which camp you are in the above commands are simple to understand and can be easily modified to suit my use case.

Interpretation of What is Being Done

Now as far as I can tell Microsoft has integrated VHD files into the solution in order to expose this functionality.  I don’t have a technical problem with this, however at this point there is a lot of manual work that has to be done in order to get to the point where I have a usable file system.  So you create your pool with your disks (this is no different from ZFS, however in ZFS this is where your RAID is performed.  Whereas in the Storage Pool/Spaces implementation they are doing that on the file system level (I suspect with software RAID on top of VHD files).  Once the pool is up and configured you create a Virtual Disk, where you will assign your resiliency (RAID level), then initialize the Virtual Disk, create a partition on the Virtual Disk, and finally format the Virtual Disk.  Now I am not saying that you shouldn’t have to technically do all of these things, however for this to have the seamless experience that they are trying to go for, they will need to put a bit more polish on this.

Find Our Physical Disks

As you can see we have a 25GB disk for our OS, which has the “CanPool” as False, this is because it is the system disk, and it is already being used.  We also have 2 10GB disks and 6 5GB disks.  I am doing this testing on my Virtualbox Windows 2012, so the disks are simple to scale out.  So the goal is to get them all in the same pool.  But in order to be more thorough we will be creating the pool with ONLY the 5GB disks, and then adding the 10GB disk, just to prove that we can.

PS> Get-PhysicalDisk

FriendlyName        CanPool             OperationalStatus   HealthStatus        Usage                              Size
------------        -------             -----------------   ------------        -----                              ----
PhysicalDisk0       False               OK                  Healthy             Auto-Select                       25 GB
PhysicalDisk1       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk2       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk3       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk4       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk5       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk6       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk7       True                OK                  Healthy             Auto-Select                       10 GB
PhysicalDisk8       True                OK                  Healthy             Auto-Select                       10 GB

Assign our Disks to a Variable

PS> $disk = Get-PhysicalDisk | where {$_.size -eq "5GB"}

PS> $disk

FriendlyName        CanPool             OperationalStatus   HealthStatus        Usage                              Size
------------        -------             -----------------   ------------        -----                              ----
PhysicalDisk1       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk2       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk3       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk4       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk5       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk6       True                OK                  Healthy             Auto-Select                        5 GB

Check the Name of the Storage SubSystem

I am assuming that you could wildcard this to your PC name or to Storage Spaces, but I didn’t want to document it that way since I am not sure how it would behave if you have multiple pools.

PS> Get-StorageSubSystem

FriendlyName                            HealthStatus                            OperationalStatus
------------                            ------------                            -----------------
Storage Spaces on WIN-PTP6J8CKQNG       Healthy                                 OK

Create the New Storage Pool

Here we are going to combine the variable we assigned earlier, with the name of the Storage SubSystem to create our new pool.  Then we will check the pools on the machine.  You might have noticed this before, but there is a Primordial pool, which holds disks which have not been assigned.  Sounds like someone at Microsoft has a sense of humor.  It is the goo from which your pool rises.

PS> New-StoragePool -StorageSubSystemFriendlyName "Storage Spaces on WIN-PTP6J8CKQNG" -FriendlyName pool0 -PhysicalDisks $disk

FriendlyName            OperationalStatus       HealthStatus            IsPrimordial            IsReadOnly
------------            -----------------       ------------            ------------            ----------
pool0                   OK                      Healthy                 False                   False

PS> Get-StoragePool

FriendlyName            OperationalStatus       HealthStatus            IsPrimordial            IsReadOnly
------------            -----------------       ------------            ------------            ----------
Primordial              OK                      Healthy                 True                    False
pool0                   OK                      Healthy                 False                   False

Add Additional Disks to the Pool

Assign the new disks to a variable, the same way we did before.

PS> $newdisk = Get-PhysicalDisk | where {$_.size -eq "10GB"}

PS> $newdisk

FriendlyName        CanPool             OperationalStatus   HealthStatus        Usage                              Size
------------        -------             -----------------   ------------        -----                              ----
PhysicalDisk7       True                OK                  Healthy             Auto-Select                       10 GB
PhysicalDisk8       True                OK                  Healthy             Auto-Select                       10 GB

Here we use the Add-PhysicalDisks cmdlet to add them to the Pool.  Now from a user experience this command is hard to find.  It should probably be a Modify-StoragePool or Add-StoragePoolMembers or similar.

PS> Add-PhysicalDisk -PhysicalDisks $newdisk -StoragePoolFriendlyName pool0

Create Virtual Disk (vdisk0)

This is pretty straight forward.  We are just creating a disk.

PS>  New-VirtualDisk -StoragePoolFriendlyName pool0 -FriendlyName vdisk0 -ResiliencySettingName mirror -ProvisioningType Thin -Size 30GB

FriendlyName        ResiliencySettingNa OperationalStatus   HealthStatus        IsManualAttach                     Size
me
------------        ------------------- -----------------   ------------        --------------                     ----
vdisk0              Mirror              OK                  Healthy             False                             30 GB

Initialize Disk (vdisk0)

The next step is to initialize it.

PS> Get-VirtualDisk vdisk0 | Initialize-Disk -PartitionStyle GPT

Create and Initialize Virtual Disk (vdisk1)

I combined the above two steps into one, via some powershell magic.  Functionally the same things are happening though.

PS>  New-VirtualDisk -StoragePoolFriendlyName pool0 -FriendlyName vdisk1 -ResiliencySettingName mirror -ProvisioningType Thin -Size 30GB | Initialize-Disk -PartitionStyle GPT

Partition and Format the Virtual Disk

Here is where it gets interesting.  I have combined everything into one step, but we have two distinct actions, the creation of the partition, and the formatting of the volume that is created (via the partitioning).  In the formatting, you have your choice of file system (of course the FAT’s but also NTFS and ReFS which is the interesting one).  If you are logged into the GUI this step will also result in a Windows Explorer pop-up asking if you want to format the disk, you can cancel that you are formatting via the command line.

PS>  get-disk | where {$_.uniqueid -eq ((Get-VirtualDisk vdisk0).UniqueId)} | New-Partition -UseMaximumSize -AssignDriveLetter | Format-Volume -FileSystem NTFS

Confirm
Are you sure you want to perform this action?
Warning, all data on the volume will be lost!
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

DriveLetter       FileSystemLabel  FileSystem       DriveType        HealthStatus        SizeRemaining             Size
-----------       ---------------  ----------       ---------        ------------        -------------             ----
E                                  NTFS             Fixed            Healthy                  29.78 GB         29.87 GB

Also if you’d like to do the above without having to confirm your action, pass Format-Volume with -Confirm:$false to avoid that confirmation.

PS>  get-disk | where {$_.uniqueid -eq ((Get-VirtualDisk vdisk0).UniqueId)} | New-Partition -UseMaximumSize -AssignDriveLetter | Format-Volume -FileSystem NTFS -Confirm:$false

Well there you have it.  All and all I think it is an interesting concept, though it definitely needs some work around integrating the components.

Well the good news is that it appears that Microsoft has decided to reverse course with this in Windows Server 2012.  Which is most definitely a good thing for users.

So while I don’t plan on going too deep into Windows 2012 I thought this would be worth a quick run through.

Figure 1-1:  Above you will find the Local Server view of the Server Manager tool.  You will notice on the left near the center we have our 2 ethernet connections.  In my case these are inside of Virtualbox VM and these are NAT connections.  But directly above that you see “Nic Teaming: Disabled”

Figure 1-2:  Above you will notice the NIC Teaming Details screen.  You get here by clicking on Disabled in Figure 1-1.  Notice in the bottom left we have no Teams configured, and in the bottom right we have 2 adapters available for teaming.

Figure 1-3:  Here we simply highlight the available interfaces, then in the TASKS menu select Add to New Team.

Figure 1-4:  Assign a Team Name.  Here I am using team0.

Figure 1-5:  Now we have a working Team which contains the two Ethernet Adapters.

Figure 1-6:  Just for giggles I switched the second Ethernet Adapter in Virtualbox from “NAT” to “Not Attached” to simulate network failure, and above is what I got.

Now I don’t like to do anything that I can’t do from the command line, frankly that is mostly because that is far easier to document then having to worry about screenshots.  So lets do the same thing with Powershell.

PS> New-NetLbfoTeam -Name team0 -TeamMembers (Get-NetAdapter | where {$_.name -match "Ethernet"}).name

Confirm
Are you sure you want to perform this action?
Creates Team:'team0' with TeamMembers:{'Ethernet 2', 'Ethernet'}, TeamNicName:'team0', TeamingMode:'SwitchIndependent'
and LoadBalancingAlgorithm:'TransportPorts'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

Name                   : team0
Members                : {Ethernet 2, Ethernet}
TeamNics               : team0
TeamingMode            : SwitchIndependent
LoadBalancingAlgorithm : TransportPorts
Status                 : Down

As we can see from the above output we can choose the Teaming Mode and Load Balancing Algorithm.  For the Teaming Mode you can use Switch Indepent, Static, and LACP.  The latter two options require the switch to support and/or be configured to allow that configuration to work.  You have your choice of Load Balancing Algorithms (IP Address, MAC Address, Transport Ports, and Hyper-V Port) the first three use the IP/MAC/Port from the source and destination to create a hash and then balance the traffic appropriately, the Hyper-V Port doesn’t do any hashing as far as I can tell, it simply splits the Hyper-V Ports over the interfaces.

We can also see that the status is “down” but fear not.  Give it about 15 seconds and then look at the team again.

PS> Get-NetLbfoTeam

Name                   : team0
Members                : {Ethernet 2, Ethernet}
TeamNics               : team0
TeamingMode            : SwitchIndependent
LoadBalancingAlgorithm : TransportPorts
Status                 : Up

This will give you network based teaming, there are also some additional options to allow you to use LACP and other algorithms.  For me this is a big change, but the more impactful change is the change in Network Connection names.  Now they seem to be using Ethernet followed by Ethernet 2 and so on.  Which for me is far superior to Local Area Connection followed by Local Area Connection 2 and so on.  I think it would have been better to use Ethernet0 or Ethernet1 just to give you consistent numbering, through multiple interfaces.  But I will take what I can get.

Please keep in mind that this series of changes is very high risk.  Please make sure you (1) understand fully what you are doing (2) understand fully why you are doing it (3) have backups of the data if it is of any importance to you.  If you do something stupid (even if I tell you to do it I will not be held responsible for any consequences – scared yet?).

Check our Current Configuration

Below you will see the 4 Logical Volumes, 1 Volume Group and 1 Physical Volume that we have.

# lvs
LV        VG      Attr   LSize   Origin Snap%  Move Log Copy%  Convert
lv_root  LocalVG -wi-ao 100.00G
lv_swap LocalVG -wi-ao  24.00G
lv_data1  LocalVG -wi-ao  30.00G
lv_data2  LocalVG -wi-a-  30.00G

You will see here we have ~94GB of free space in our Volume Group.  Now in this particular situation we actually have ~300GB+ due to a change in the RAID configuration (or in a more common scenario an expansion of a virtual hard disk file).

#vgs
VG      #PV #LV #SN Attr   VSize   VFree
LocalVG   1   4   0 wz--n- 278.74G 94.74G

Here is where the issue ultimately lies.  Since this OS was installed using LVM on top of a disk partition instead of the whole disk (there is no other option I am afraid when it comes to root installations).  We will need to expand the partition in order to see this on the LVM Physical Volume.

# pvs
PV         VG      Fmt  Attr PSize   PFree
/dev/sda2  LocalVG lvm2 a-   278.74G 94.74G

Make our Change Plan

Now in order to “extend” the partition we actually have to delete the partition and then recreate it.  The important part of this is that we want to ensure that when we are creating our new partition that we start it on the exact same block (we will end it on a further block, but we need to start in the right place).  Don’t try and get smart and relocate the partition in addition to extending it.  That would classify as stupid.

# fdisk -l /dev/sda

Disk /dev/sda: 598.8 GB, 598879502336 bytes
255 heads, 63 sectors/track, 72809 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          16      128488+  83  Linux
/dev/sda2              17       36404   292286610   8e  Linux LVM

From the above output we can see a few things…

1) The disk is ~598GB
2) The last cylinder on the disk is 72809
3) The last cylinder on our (/dev/sda2) partition is 36404
4) The first cylinder on our (/dev/sda2) partition is 17
5) The type of the partition is 8e which means Linux LVM.

So this will consist of 3 changes to the partition changes.

1) Delete /dev/sda2
2) Create /dev/sda2 starting on the 17th cylinder and ending on the 72,809th cylinder (the last cylinder).
3) Set the type of /dev/sda2 to 8e.

Implement our Change Plan

Now that we know what we have to do, and we have written down all of these key details.  Keep in mind I actually mean written down on paper, if you sustain a power outage during the change you will not have a file system or a terminal window to reference the pre-existing configuration and you will need to boot to a live-cd to rebuild your partition table from your notes.

# fdisk /dev/sda

The number of cylinders for this disk is set to 72809.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 598.8 GB, 598879502336 bytes
255 heads, 63 sectors/track, 72809 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          16      128488+  83  Linux
/dev/sda2              17       36404   292286610   8e  Linux LVM

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Command action
e   extended
p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (17-72809, default 17):
Using default value 17
Last cylinder or +size or +sizeM or +sizeK (17-72809, default 72809):
Using default value 72809

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sda: 598.8 GB, 598879502336 bytes
255 heads, 63 sectors/track, 72809 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          16      128488+  83  Linux
/dev/sda2              17       72809   584709772+  8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Reboot

So hopefully you noticed that after I was done, I printed the configuration and double checked my work against my plan to ensure that I had it right.  Next we must reboot in order to re-read your partition table.

# reboot

Broadcast message from root (pts/0) (Fri Aug 10 11:25:58 2012):

The system is going down for reboot NOW!

Resize the LVM Physical Volume

OK so now that we have extended the partition and rebooted, LVM will show the same characteristics as before we extended the partition.  The next step is to perform a pvresize so that LVM will re-examine the disk characteristics of the physical volume, and then take advantage of any new space.

# pvs
PV         VG      Fmt  Attr PSize   PFree
/dev/sda2  LocalVG lvm2 a-   278.74G 94.74G

# pvresize /dev/sda2
Physical volume "/dev/sda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

# vgs
VG      #PV #LV #SN Attr   VSize   VFree
LocalVG   1   4   0 wz--n- 557.62G 373.62G

Enjoy

That was it now, we have managed to extend the underlying partition which is the LVM Physical Volume.  This has enabled us to seamlessly extend our Volume Group, and now the full disk is being used in our environment.

Solaris Virtualization: Using Logical Domains on Solaris 11 Part One
Solaris Virtualization: Using Logical Domains on Solaris 11 Part Two

Before we start the domain, we need to bind the associated resources to the domain.  This will also show us the port that the console is running on, so that we can connect to it.

root@t4:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    4     4G       0.8%  2h 44m
t4-g1            inactive   ------          8     4G
root@t4:~# ldm bind-domain t4-g1
root@t4:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    4     4G       3.0%  2h 45m
t4-g1            bound      ------  5000    8     4G    

In the above example we can see that once we “bind-domain” this is binding specific hardware resources to the domain.  In this case we can now see that we have a port for the console connections.  For this domain it is 5000.  So if we telnet to port 5000 on the localhost we will end up connected to the console.  I personally like to connect prior to starting the domain, because I prefer to watch the boot up process, so that I know everything is working properly, this means I will have a separate ssh session which I use for telnet.

root@t4:~# telnet localhost 5000
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to t4.
Escape character is '^]'.

Connecting to console "t4-g1" in group "t4-g1" ....
Press ~? for control options ..

Start the domain.

root@t4:~# ldm start-domain t4-g1
LDom t4-g1 started

Observe boot process.  During the below boot process we can see a couple of things.  Most importantly we see that the boot fails, with a “Can’t open boot device” this is expected as we are attempting to boot from disk0 which has not been installed yet.

SPARC T4-1, No Keyboard
Copyright (c) 1998, 2011, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.33.1, 4096 MB memory available, Serial #83496026.
Ethernet address 0:14:4f:fa:c:5a, Host ID: 84fa0c5a.

Boot device: disk0  File and args:
Bad magic number in disk label
ERROR: /virtual-devices@100/channel-devices@200/disk@0: Can't open disk label package

ERROR: boot-read fail

Evaluating:

Can't open boot device

{0} ok

We will be dumped to the ok prompt for the domain, from here we can simply select to boot from the iso that you defined earlier in the domain setup.  In my case I named the device s11.iso, so if you do a devalias you would see s11.iso as an alias to a device.

{0} ok boot s11.iso

Just a quick note, the escape character for telnet is listed as ‘^]’ in the beginning of the telnet session, this is CTRL + ] once you have returned the the telnet> prompt you can type quit.

Now in my environment I went through the install of Solaris 11, since it was the disk I had readily available, you can additionally use Soalris 10, or OpenSolaris if you so desire.

Solaris Virtualization: Using Logical Domains on Solaris 11 Part One

Create the Logical Domain

root@t4:~# ldm add-domain t4-g1

Configure Devices

We will need to add devices to our newly created domain.  Here we will add virtual CPUs, memory, and a network device.

root@t4:~# ldm add-vcpu 8 t4-g1
root@t4:~# ldm add-memory 4G t4-g1
root@t4:~# ldm add-vnet vnet1 primary-vsw0 t4-g1

Configure Storage Devices

Add the installation media to our domain using an iso image which we added to the Virtual Disk Service service which we created in Part One.

root@t4:~# ldm add-vdisk s11.iso solaris11_media@primary-vds0 t4-g1

Create a couple of ZFS volumes for use by the domain, we are using the -p option to create the parent file system with the name of the logical domain for organization.

root@t4:~# zfs create -p -V 20G rpool/ldoms/t4-g1/disk0
root@t4:~# zfs create -p -V 20G rpool/ldoms/t4-g1/disk1

Add the ZFS volumes to the Virtual Disk Server service (primary-vds0).

root@t4:~# ldm add-vdsdev /dev/zvol/dsk/rpool/ldoms/t4-g1/disk0 t4-g1-disk0@primary-vds0
root@t4:~# ldm add-vdsdev /dev/zvol/dsk/rpool/ldoms/t4-g1/disk1 t4-g1-disk1@primary-vds0

Configure our domain with our disk storage devices (disk0 and disk1) from the Virtual Disk Server service.

root@t4:~# ldm add-vdisk disk0 t4-g1-disk0@primary-vds0 t4-g1
root@t4:~# ldm add-vdisk disk1 t4-g1-disk1@primary-vds0 t4-g1

Configure Boot Devices

We want our domain to auto-boot and use disk0 as the primary boot device.

root@t4:~# ldm set-var auto-boot\?=true t4-g1
root@t4:~# ldm set-var boot-device=disk0 t4-g1

Set the Host ID

root@t4:~# ldm set-domain hostid=85e92821 t4-g1

Additionally the hostid option can be passed in the add-domain command when you originally create the domain.

Tomorrow we will go through the process of connecting to the console of the Logical Domain and then install an operating system.

This series is going to focus on the basic administration of a Logical Domains environment using the basic command line tools of the Solaris 11 Operating System.  In Part One we will go over what it takes to configure a Solaris 11 installation to be a Logical Domains hypervisor.  In Part Two we will document defining and configuring Logical Domains.  In Part Three we will discuss connecting to the Logical Domain in order to perform an installation of the operating system.

A little bit about my environment.  I am using a Oracle Sun SPARC T4-1 which has a single socket eight-core 2.85GHz SPARC T4 Processor.  Each processor core on this model has 8 threads, giving us a total of 64 processor threads.  Lets take a look at our environment shall we?

Check Operating System Version

This is the released version of Solaris 11 for SPARC, which of course means it can only be used on the SPARC processor architecture.

root@t4:~# uname -a
SunOS t4 5.11 11.0 sun4v sparc sun4v

View Logical Domains Software Version

The release version of Solaris 11 for SPARC comes with Logical Domains 2.1.  If you have purchased support and configured access to the repositories then pkg update will give you access to Logical Domains 2.2.

root@t4:~# ldm -V

Logical Domains Manager (v 2.1.0.4)
Hypervisor control protocol v 1.7
Using Hypervisor MD v 1.3

System PROM:
Hostconfig      v. 1.1.1        @(#)Hostconfig 1.1.1 2011/08/03 23:04
Hypervisor      v. 1.10.1.      @(#)Hypervisor 1.10.1.b 2011/09/12 09:56
OpenBoot        v. 4.33.1       @(#)OpenBoot 4.33.1 2011/08/03 10:34

View Existing Logical Domains

Here we see the specifics of our physical machine.  The primary domain has all of the resources and before we will be able to allocate resources to other Logical domains we will need to remove some from the primary domain.

root@t4:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-c--  UART    64    32256M   0.0%  4d 23h 30m

Create Services for Control Domain

Here we are creating the Virtual Console Concentrator service.  This is what allows us to connect to the console of the domains for installation and reconfiguration.

root@t4:~# ldm add-vcc port-range=5000-5100 primary-vcc0 primary

This is the Virtual Disk Server service.  This is used to hand out disk devices to the domains.

root@t4:~# ldm add-vds primary-vds0 primary

This is the Virtual Switch service.  It provides networking to the domains.

root@t4:~# ldm add-vsw net-dev=net0 primary-vsw0 primary

Lets review our changes.

root@t4:~# ldm list-services primary
VCC
NAME             LDOM             PORT-RANGE
primary-vcc0     primary          5000-5100

VSW
NAME             LDOM             MAC               NET-DEV   ID   DEVICE     LINKPROP   DEFAULT-VLAN-ID PVID VID                  MTU   MODE   INTER-VNET-LINK
primary-vsw0     primary          00:14:4f:fb:bd:bd net0      0    switch@0              1               1                         1500         on

VDS
NAME             LDOM             VOLUME         OPTIONS          MPGROUP        DEVICE
primary-vds0     primary  

Configure the Control Domain

Look at the existing domains.  By default we have primary, which is essentially the host operating system.

root@t4:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-c--  UART    64    32256M   0.0%  5d 53m

Reduce the number of vCPU available to the control domain.

root@t4:~# ldm set-vcpu 4 primary

Reduce the amount of RAM available to the control domain.

root@t4:~# ldm set-memory 4G primary

Reconfigure primary domain to reflect changes.

root@t4:~# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

Create alternate configuration file.

root@t4:~# ldm add-config primary-config
root@t4:~# ldm list-config
factory-default
primary-config [current]

Enable the Virtual Network Terminal Server Daemon.

root@t4:~# svcadm enable vntsd
root@t4:~# svcs | grep vntsd
online         16:40:01 svc:/ldoms/vntsd:default

Reboot.

root@t4:~# reboot

After a reboot, the primary domain reflects the new memory and vCPU configuration.

root@t4:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    4     4G       0.4%  1h 4m

Create Install Media Repository

Create a ZFS file system to hold the iso files.  The -p will create the parent file system /rpool/ldoms as well.

root@t4:~# zfs create -p rpool/ldoms/iso

Add the Solaris 11 and Solaris 10 iso files to the Virtual Disk Server service.

root@t4:~# ldm add-vdsdev /rpool/ldoms/iso/oracle-solaris-11-11.11-sparc.iso solaris11_media@primary-vds0
root@t4:~# ldm add-vdsdev /rpool/ldoms/iso/oracle-solaris-10-8.11-sparc.iso solaris10_media@primary-vds0

Now we have a working environments to support the Logical Domains that we will be adding tomorrow in part two.

Create the Service Script

Below you will see the script I created.  This script will test for the presence of a volume group, and if present on invocation (boot up or manual) then it will mount each logical volume defined in the /etc/fstab of the present volume group.

Name       :  custom_mount.sh
Version   :  1.0.3
MD5        :  0b8cf3b0b5c9d3c16e4384fde88e8659
SHA256  :  877ef66ab464e71172986fa9cd5510f0ad6752f05a414af703878d7e058c845b
URL         :  http://source.allanglesit.net/pub/custom_mount.sh

#!/bin/bash
# chkconfig: 345 85 15
# description: Will automatically mount a removable device if present.
#
#: Script Name    : custom_mount.sh
#: Version    : 1.0.3
#: Author    : Matthew Mattoon - http://blog.allanglesit.com
#: Date Created    : August 15, 2012
#: Date Updated    : February 20, 2013
#: Description    : Automount Removable Logical Volumes Script.
#: Examples    : custom_mount.sh ACTION
#:         : custom_mount.sh start

vgtest="vg_data"
vgs=`vgs | grep $vgtest`

case "$1" in
start)
if [ -n "$vgs" ]
then
echo "Logical Volume Group: $vgtest present."
mounts=`cat /etc/fstab | grep $vgtest | grep noauto | tr '\t' ' ' | tr -s ' ' | cut -d " " -f 2`
for mount in $mounts
do
if [ -z "`mount | grep $mount`" ]
then
echo "Mounting $mount file system..."
mount $mount
else
echo "File system $mount is already mounted..."
fi
done
else
echo "Logical Volume Group: $vgtest not present."
exit 1
fi
;;
stop)
if [ -n "$vgs" ]
then
echo "Logical Volume Group: $vgtest present."
mounts=`cat /etc/fstab | grep $vgtest | grep noauto | tr '\t' ' ' | tr -s ' ' | cut -d " " -f 2`
for mount in $mounts
do
if [ -n "`mount | grep $mount`" ]
then
echo "Unmounting $mount file system..."
umount $mount
else
echo "File system $mount is already unmounted..."
fi
done
else
echo "Logical Volume Group: $vgtest not present."
exit 1
fi
;;
*)
echo "Usage: $0 start"
exit 1
esac

Create the Service to Auto Start

Based on my /etc/init.d/custom_mount script it will start in run levels 3, 4, and 5.

# chkconfig --add custom_mount

{0} ok

This is the OK Prompt, some systems will display ok> and this is how I will refer to it in subsequent examples.  So lets dig in.

What is OpenBoot PROM?

The OpenBoot PROM (OK Prompt) is a boot monitor which allows us to interact with the boot environment, we can use it to pass different boot parameters.  We can use it to select a boot device, boot into various modes (verbose, single user, reconfigure).  There is much more to it then this, look up Open Firmware for more information.

Why am I Stuck at the OK Prompt?

Now in my situation, the system was not even attempting booting up.  That is because on this particular system auto_boot was disabled.  This can be seen by printing the environment…

ok> printenv
Variable Name           Value                          Default Value

...
auto-boot?              false                          true
...

Now in my case that is fine.  I don’t want this to boot automatically, but I still need to boot it, but before we do lets check our boot order.

Configuring Boot Devices

ok> printenv boot-device
boot-device =           disk net

We can change this easily enough if it is incorrect.  For example if we wanted to add a cdrom, but after the disk.

ok> setenv boot-device disk cdrom net

ok> printenv boot-device
boot-device =           disk cdrom net

But I don’t want to have a cdrom as part of the boot order, so instead I would rather boot up the device as an override.

Manually Select a Boot Device

ok> boot cdrom

This can also be done against any other device which is bootable, for example another disk.  But first lets find out what exactly we have for disks.

View Disks

ok> show-disks
a) /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/storage@2/disk
b) /pci@400/pci@2/pci@0/pci@4/scsi@0/disk
c) /pci@400/pci@1/pci@0/pci@4/scsi@0/disk
d) /iscsi-hba/disk
q) NO SELECTION
Enter Selection, q to quit: q

So we can see here we have 2 disks:

/pci@400/pci@1/pci@0/pci@4/scsi@0/disk
/pci@400/pci@2/pci@0/pci@4/scsi@0/disk

Now with this information, we still haven’t made the connection between our boot-device which is “disk cdrom net” in order to define these names we need to look at devalias.

Just Call me Disk.  A Lesson in aliases.

Devalias allows us to define names for our devices, which can be more easily referred to.  Below is trimmed for brevity.

ok> devalias
...
net0                     /pci@400/pci@2/pci@0/pci@6/network@0
net                      /pci@400/pci@2/pci@0/pci@6/network@0
...
disk4                    /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p0
cdrom                    /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p6
...
disk0                    /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0
disk                     /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0
...

Now when we look at this output we notice a few things.

1) @p0 on disks – I believe this refers to partitions on the disk.  p0 being the whole disk.
2) Multiple aliases to the same underlying device, case in point disk and disk0 as well as net and net0.

So we can create aliases to denote a primary and a mirror disk on our two disks.

ok> devalias primary /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0
ok> devalias mirror /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p0
ok> devalias
mirror                   /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p0
primary                  /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0
...

Important to note that when you create an alias using devalias it doesn’t get committed, and will disappear after a reboot…  If you need a persistent alias, use nvalias.

ok> nvalias primary /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0
ok> nvalias mirror /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0

Then of course we can change our boot-device to reflect this.

ok> setenv boot-device primary mirror

Now at some point you might need to remove an alias.

ok> nvunalias mirror
nvunalias primary

To finalize the removal, we need to reset the machine.

ok> reset-all

There is much more to the OK prompt, but this ought to get you started.  Enjoy.

Configure SSH to Permit Root Logins

# vi /etc/ssh/sshd_config

Here we want to find “PermitRootLogin” and flip it to yes.

# svcadm restart ssh

Then restart SSH.  SSH will now accept root as a login account, however root is still set as a role.

# vi /etc/user_attr

Here we want to find the line which is for the user root, and remove the last portion of the line which says “type=role”.

That is it.  You can now login directly on the console or via SSH using the root account and password you set during installation.  Of course you can reset the password using passwd as well if need be.

Configure Oracle Linux Repositories

You can use Oracle’s Public Yum server.  This provides free software updates and a package source for Oracle Linux.  Keep in mind, these updates do not require a support contract and thus do not come with any support.  If you need support you can buy it from the Oracle Store.

# cd /etc/yum.repos.d/
# wget http://public-yum.oracle.com/public-yum-ol6.repo

Install Latest Updates

I installed all of the latest updates, it came out to be around 120 updates.  And it brought me up to the RHEL-Compatible Kernel 2.6.32-220.17.1.el6 and UEK 2.6.32-300.25.1.el6uek.

#yum update

Configure Ethernet Bridge

# cat ifcfg-br0
DEVICE=br0
TYPE=Bridge
NM_CONTROLLED=no
ONBOOT=yes
BOOTPROTO=dhcp

# cat ifcfg-eth1
DEVICE=eth1
NM_CONTROLLED=no
ONBOOT=yes
BRIDGE=br0

# /etc/init.d/network restart
Shutting down interface eth1:  bridge br0 does not exist!
[  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth1:                                [  OK  ]
Bringing up interface br0:
Determining IP information for br0... done.
[  OK  ]

Install Required Software

The only things we really have to install is kvm, and libvirt if you want to use libvirt to manage it.

# yum install kvm libvirt

I also include the following software in my KVM builds as they add a lot of flexibility (vim and pv are not even hypervisor related but I use them extensively to edit xml and move disk images).

# yum install python-virtinst virt-top virt-manager virt-v2v virt-viewer vim pv

Once you have performed the install everything required for KVM is ready to go.  Next we need to define a VM, this can be done with virt-manager.  Or by using one of my previously documented methods.

http://blog.allanglesit.com/2011/03/kvm-guests-manipulating-libvirt-xml-for-guest-creation/

http://blog.allanglesit.com/2011/03/kvm-guests-using-virt-install-to-install-vms-from-a-cd-or-iso-imag/

The important thing to notice here is snv_151a, which means that it is Solaris 11 Express.

# uname -a
SunOS storage01 5.11 snv_151a i86pc i386 i86pc Solaris

# cat /etc/release
Oracle Solaris 11 Express snv_151a X86
Copyright (c) 2010, Oracle and/or its affiliates.  All rights reserved.
Assembled 04 November 2010

Display the Current IPS Repositories

This will show you the repositories configured on your system.

# pkg publisher
PUBLISHER                             TYPE     STATUS   URI
solaris                               origin   online   http://pkg.oracle.com/solaris/release/

Configure the IPS Repository

If you don’t already have the repository configured you will need to configure it.

# pkg set-publisher -g http://pkg.oracle.com/solaris/release solaris

Update Packages

# pkg update
WARNING: pkg(5) appears to be out of date, and should be updated before
running update.  Please update pkg(5) using 'pfexec pkg install
pkg:/package/pkg' and then retry the update.

As it mentions we need to install the updated version of pkg to preceed with update installation.

# pkg install pkg
Packages to update:     5
Create boot environment:   Yes
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  5/5     485/485      3.5/3.5

PHASE                                        ACTIONS
Removal Phase                                   1/58
Warning - directory /tmp/tmpw15nh0/usr/share/man/cat1m not empty or not expected during operation - contents preserved in /tmp/tmpw15nh0/var/pkg/lost+found/usr/share/man/cat1m-20120528T202140Z
Removal Phase                                  58/58
Install Phase                                199/199
Update Phase                                 465/465

PHASE                                          ITEMS
Package State Update Phase                     10/10
Package Cache Update Phase                       5/5
Image State Update Phase                         2/2

A clone of solaris exists and has been updated and activated.
On the next boot the Boot Environment solaris-1 will be mounted on '/'.
Reboot when ready to switch to this updated BE.

Now we need to reboot into the new boot environment.

# reboot

Now lets run our updates.

# pkg update
Packages to remove: 272
Packages to install: 207
Packages to update: 577
Create boot environment: Yes
Create backup boot environment:  No

DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                              1056/1056 68642/68642 1042.2/1042.2

PHASE                                        ACTIONS
Removal Phase                            29017/29017
Install Phase                            72725/72725
Update Phase                             72453/72453

PHASE                                          ITEMS
Package State Update Phase                 1633/1633
Package Cache Update Phase                   849/849
Image State Update Phase                         2/2

A clone of solaris-1 exists and has been updated and activated.
On the next boot the Boot Environment solaris-2 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

The following unexpected or editable files and directories were
salvaged while executing the requested package operation; they
have been moved to the displayed location in the image:

var/saf -> /tmp/tmpe27V3b/var/pkg/lost+found/var/saf-20120528T233642Z
var/sadm/system/logs -> /tmp/tmpe27V3b/var/pkg/lost+found/var/sadm/system/logs-20120528T233642Z
var/run -> /tmp/tmpe27V3b/var/pkg/lost+found/var/run-20120528T233642Z
etc/saf/zsmon -> /tmp/tmpe27V3b/var/pkg/lost+found/etc/saf/zsmon-20120528T233642Z
etc/saf -> /tmp/tmpe27V3b/var/pkg/lost+found/etc/saf-20120528T233642Z

---------------------------------------------------------------------------
NOTE: Please review release notes posted at:

http://www.oracle.com/pls/topic/lookup?ctx=E23824&id=SERNS

---------------------------------------------------------------------------

Another reboot to get into the new boot environment.

# reboot

Check Current Version

The “11.0″ is what we are looking for.  You are now on the release version Solaris 11.

# uname -a
SunOS storage01 5.11 11.0 i86pc i386 i86pc Solaris

# cat /etc/release
Oracle Solaris 11 11/11 X86
Copyright (c) 1983, 2011, Oracle and/or its affiliates.  All rights reserved.
Assembled 18 October 2011

METHOD ONE

This option is the simplest, we simply echo the line into the file as we want it to appear, then based on the >> it is added as the last line in the file.  This is fine, however this does not provide us with a way of inserting tabs, it is simply whitespace.

echo "nfsserver.allanglesit.com:/nfsshare      /mnt/nfs nfs rw,bg,hard,nointr,rsize=131072,wsize=131072,tcp,vers=3" >> /etc/fstab

METHOD TWO

Here we are incorporating the \t special character which is representative of the [tab] key.  With this method we must call echo with the -e parameter which tells it that we plan on adding in special characters and we want echo to translate them instead of treating them as text.  This method has a huge problem…  It is completely unreadable.

echo -e "nfsserver.allanglesit.com:/nfsshare\t/mnt/nfs\tnfs\trw,bg,hard,nointr,rsize=131072,wsize=131072,tcp,vers=3" >> /etc/fstab

METHOD THREE

This is the best of both worlds.  The functionality is there, but we are achieving it with a little more effort, which results in some cleaner code.  And to make it even more legible we can add comments after each line to indicate what each line is doing, as I have done with each tab line.  Another thing to notice, is that we are calling echo with -n to prevent it from creating the standard newline at the end of the entry.  This will allow us to keep pumping data onto the same line.  However we want to not call the -n on the last entry on the line.  Otherwise we will add more data to this same line if we do subsequent redirects into the same file.

echo -n "nfsserver.allanglesit.com:/nfsshare" >> /etc/fstab
echo -e -n "\t" >> /etc/fstab    #INSERT TAB
echo -n "/mnt/nfs" >> /etc/fstab
echo -e -n "\t" >> /etc/fstab    #INSERT TAB
echo -n "nfs" >> /etc/fstab
echo -e -n "\t" >> /etc/fstab    #INSERT TAB
echo "rw,bg,hard,nointr,rsize=131072,wsize=131072,tcp,vers=3" >> /etc/fstab

Of course any of these methods can be adapted in whatever way you need to fit your use case, this isn’t only helpful for /etc/fstab.  That is just what I was using it for.

Fdisk will produce output similar to this…

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


WARNING: The size of this disk is 2.3 TB (2290828181504 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).

Use Parted to Create a Partition Using the Whole Disk

Parted is able to handle large volumes which are not able to be modified with fdisk.  Here we are going to launch parted, and then create the partition with a size of 1 -1 which means that it will use it all until it gets to the end.  You can additionally specify the number in megabytes, so 1 1024 would start at 1 and go until 1024 giving you a 1GB partition.

# parted /dev/sdb
(parted) mkpart primary ext3 1 -1

Review the created partition.

(parted) print

Model: DELL PERC H700 (scsi)
Disk /dev/sdb: 2291GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
1      17.4kB  2291GB  2291GB               primary       

Set the LVM flag.

(parted) set 1 lvm on    

Review our changes to our partition.

(parted) print

Model: DELL PERC H700 (scsi)
Disk /dev/sdb: 2291GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
1      17.4kB  2291GB  2291GB               primary  lvm
(parted) quit

Initialize the Physical Volume for LVM

The rest of this is pretty standard, and already covered in my article Linux LVM2: Flexible Local Storage Management in case you’d like more details.

# pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created

Create the LVM Volume Group

# vgcreate VolGroup01 /dev/sdb1
Volume group "VolGroup01" successfully created
# vgs
VG         #PV #LV #SN Attr   VSize  VFree
VolGroup00   1   2   0 wz--n- 99.88G    0
VolGroup01   1   0   0 wz--n-  2.08T 2.08T

Well now we have a Volume Group (VolGroup01) which can be used to create Logical Volumes from.

View Existing DNS Client Configuration

# svccfg -s network/dns/client listprop config
config                      application
config/value_authorization astring     solaris.smf.value.name-service.dns.client
config/domain              astring     test.local
config/nameserver          net_address 10.0.0.152

Update Existing DNS Client Configuration

Here we will update our name servers.  In this case we are replacing the original with two different addresses.

# svccfg -s network/dns/client setprop config/nameserver = net_address: "(10.0.0.141 10.0.0.142)"

Here we are changing the domain to b.test.local.

# svccfg -s network/dns/client setprop config/domain = astring: b.test.local

And we are defining a previously undefined setting for the search domains, we are including test.local and b.test.local.

# svccfg -s network/dns/client setprop config/search = astring: '("test.local" "b.test.local")'

Here we are defining our name resolution order.

# svccfg -s name-service/switch setprop config/ipnodes = astring: '("files dns")'
# svccfg -s name-service/switch setprop config/host = astring: '("files dns")'

Review Changed DNS Client Configuration

# svccfg -s network/dns/client listprop config
config                      application
config/value_authorization astring     solaris.smf.value.name-service.dns.client
config/domain              astring     b.test.local
config/nameserver          net_address 10.0.0.141 10.0.0.142
config/search              astring     "test.local" "b.test.local"

Review Changed Name Service Configuration

# svccfg -s name-service/switch listprop config
config                      application
config/default             astring     files
config/value_authorization astring     solaris.smf.value.name-service.switch
config/printer             astring     "user files"
config/ipnodes             astring     "files dns"
config/host                astring     "files dns"

Export DNS Client Configuration

This command will build an /etc/resolv.conf based on your settings above.

# svcadm enable dns/client
# nscfg export svc:/network/dns/client:default

# cat /etc/resolv.conf

#
# Copyright (c) 2012, Oracle and/or its affiliates.  All rights reserved.
#

#
# _AUTOGENERATED_FROM_SMF_V1_
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
#     DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
# See resolv.conf(4) for details.

domain  b.test.local
search  test.local b.test.local
nameserver  10.0.0.141
nameserver  10.0.0.142

If you manually edit the /etc/resolv.conf then your changes will be lost on a restart of the network/dns/client service or a reboot, as the warning says.

Export Name Service Configurations

# svcadm refresh name-service/switch

# cat /etc/nsswitch.conf

#
# _AUTOGENERATED_FROM_SMF_V1_
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
#   DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
# See nsswitch.conf(4) for details.

passwd: files
group:  files
hosts:  files dns
ipnodes:        files dns
networks:       files
protocols:      files
rpc:    files
ethers: files
netmasks:       files
bootparams:     files
publickey:      files
netgroup:       files
automount:      files
aliases:        files
services:       files
printers:       user files
project:        files
auth_attr:      files
prof_attr:      files
tnrhtp: files
tnrhdb: files
sudoers:        files

An Extra Trick

Now if you can’t be bothered to do things the new way they also put in an import mechanism, whereby you can take advantage of your existing knowledge and simply import your modified configuration files into the SMF to manage them going forward.

So modify up your /etc/resolv.conf and your /etc/nsswitch.conf and then import them with nscfg.

# nscfg import -f name-service/switch:default
# nscfg import -f dns/client:default

UPDATE
December 30, 2012

I have modifed my commands above to account for the mistakes that noxxsan and Mikel brought forth in the comments.  Sorry for the delay.  I additionally added the import trick.

View Hostname in SMF

This will read back the configuration out of the SMF.

# svccfg -s system/identity:node listprop config
config           application
config/nodename astring     solaris11box
config/loopback astring     solaris11box

Change Hostname in SMF

We have two separate settings, which need to be updated.

# svccfg -s system/identity:node setprop config/nodename = astring: newname
# svccfg -s system/identity:node setprop config/loopback = astring: newname

Review the Changes

# svccfg -s system/identity:node listprop config
config           application
config/nodename astring     newname
config/loopback astring     newname

Restart the Service to Reflect Changes

The system is now aware of the change, however it doesn’t use the change until the next reboot.  We can accelerate this by restarting the system/identity:node service.

# svccfg -s system/identity:node refresh
# svcadm restart system/identity:node

This will then show a system message reflecting the new hostname, to see the change in your shell prompt, you can spin off a child bash shell.  You can also use the hostname command to view the current hostname.

UPDATEDecember 30, 2012

Included an update based on Mark’s comment below to account for the refresh which needs to take place.

In this article I will be going over fairly vanilla configurations of zones.  I will not be going into branded zones, which will allow you to run a different operating environment from the global zone.  This article will be large enough to not need the additional complexity of those tasks.  I will however have follow up articles as time permits, which will take deeper dives into this aspect of Solaris.

List Zones

By default we will only list running zones, also it is important to note that, a standard installation creates a “global” zone, this is the operating system that you are used to interacting with, all others are considered “non-global” zones (or just zones).  The key thing to remember is that all non-global zones are part of the global zone, but the reverse is not true.

# zoneadm list

We can also use the following parameter to list only installed zones.

# zoneadm list -i

The final state that they can be in is configured, this means that the configuration exists but they may not be installed yet.  This is the fullest list of zones.

# zoneadm list -c

Any of the above lists can be passed with a verbose parameter, this will give us the ID, Name, Status, Path, Brand, and IP.

# zoneadm list -v

List Specific Zone Configuration

# zonecfg -z testzone info

Zone Configuration

Here is a quick and dirty way to configure a new zone.

# zonecfg -z zone001 "create;set zonepath=/export/zones/zone001"

For more complex configurations you will need to enter the zonecfg prompt.

# zonecfg -z zone001
zone001: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:zone001> create
create: Using system default template 'SYSdefault'
zonecfg:zone001> set zonepath=/export/zones/zone001
zonecfg:zone001> add attr
zonecfg:zone001:attr> set name=comment
zonecfg:zone001:attr> set type=string
zonecfg:zone001:attr> set value="this is a comment"
zonecfg:zone001:attr> end
zonecfg:zone001> add net
zonecfg:zone001:net> set physical=zone001vnic0
zonecfg:zone001:net> end
zonecfg:zone001> add dataset
zonecfg:zone001:dataset> set name=rpool/zonedata
zonecfg:zone001:dataset> end
zonecfg:zone001> verify
zonecfg:zone001> commit
zonecfg:zone001> exit

At the end we performed a verify to look for any problems with out configuration.  If we for example forgot to define a zonepath, then we would receive a notice similar to this…

 zonepath cannot be empty.
zone001: Required resource missing

This can be resolved by providing the missing zonepath.

Create Zone Dataset

The install process will create the ZFS dataset for the zonepath, however since we have defined an additional dataset then we must pre-create that dataset.

# zfs create rpool/zonedata

Create Virtual Networking Card

Solaris 11 using exclusive networking by default, meaning, one zone per device.  This allows you to limit traffic to a particular zone.  Additionally there is a shared-type which you will assign the zone to use the physical network card.  When using the exclusive method we must create a vnic.

# dladm create-vnic -l net0 zone001vnic0

Installation of a Zone

When doing a default install it will use the IPS repositories from the host, so make sure that your host connectivity is worked out.

# zoneadm -z zone001 install
Progress being logged to /var/log/zones/zoneadm.20120520T172618Z.zone001.install
Image: Preparing at /export/zones/zone001/root.

Install Log: /system/volatile/install.1980/install_log
AI Manifest: /tmp/manifest.xml.zuaa2d
SC Profile: /usr/share/auto_install/sc_profiles/enable_sci.xml
Zonename: zone001
Installation: Starting ...

Creating IPS image
Installing packages from:
solaris
origin:  http://pkg.oracle.com/solaris/release/
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                              167/167 32062/32062  175.8/175.8

PHASE                                        ACTIONS
Install Phase                            44313/44313

PHASE                                          ITEMS
Package State Update Phase                   167/167
Image State Update Phase                         2/2
Installation: Succeeded

Note: Man pages can be obtained by installing pkg:/system/manual

done.

Done: Installation completed in 668.825 seconds.

Next Steps: Boot the zone, then log into the zone console (zlogin -C)

to complete the configuration process.

Log saved in non-global zone as /export/zones/zone001/root/var/log/zones/zoneadm.20120520T172618Z.zone001.install

Renaming a Zone

It is almost certain that you will at some point need to change the name of a zone.

zonecfg -z testzone "set zonename=newname'

Connect to the Console of the Zone

On first start up after install you will want to use two shells, and connect to the console from one first, then using the other shell, boot the zone.

zlogin -C zone001
[Connected to zone 'zone001' console]

[NOTICE: Zone booting up]

After booting up you should see something like this…

System Configuration Tool

System Configuration Tool enables you to specify the following
configuration parameters for your newly-installed Oracle Solaris 11
system:
- network, time zone, user and root accounts, name services

System Configuration Tool produces an SMF profile file in
/system/volatile/scit_profile.xml.

How to navigate through this tool:
- Use the function keys listed at the bottom of each screen to move
from screen to screen and to perform other operations.
- Use the up/down arrow keys to change the selection or to move
between input fields.
- If your keyboard does not have function keys, or they do not
respond, press ESC; the legend at the bottom of the screen will
change to show the ESC keys for navigation and other functions.

F2_Continue  F6_Help  F9_Quit                                                

Follow this through the initial setup.  I set my computer name to zone001 to match the zone name, I also set the network connection to be configured Automatically.  After the configuration is complete you should see something like this.

Exiting System Configuration Tool. Log is available at:
/var/tmp/install/sysconfig.log
Hostname: zone001
zone001 console login:

To disconnect we can use the ~. keystroke.  That is tilde + period.

Start/Boot a Zone

zoneadm -z zone001 boot

Reboot a Zone

zoneadm -z zone001 reboot

Shutdown a Zone

zoneadm -z zone001 shutdown

Halt a Zone

This is a forced power off of the zone, equivalent to a power unplug.

zoneadm -z zone001 halt

UPDATE
December 30, 2012

One minor thing I have noticed, is that when connecting to a Solaris machine via ssh and then disconnecting from a zone console connection using the ~. (tilde and period) hot key you will actually disconnect not only from the console, but also from the SSH session on your machine.  To avoid this second disconnection and only disconnect from the zone console session instead use the ~~. (tilde and tilde and period) hot key.  This will leave your SSH session intact, and allow you to change between zone consoles.

UPDATE
January 2, 2012

When using the method outlined in my article “SSH Hop Through Multiple Hosts” and the ~~. hot key you will find that your SSH connection is still disconnected.  For this use case simple append one more tilde.  So if you are connecting from Machine A through Machine B to Machine C, then you will need ~~~. (tilde and tilde and tilde and period) or one for each machine (including the SSH client and server.

1)  I am far more effective with Linux, due to the nature of the shell, and tools which I am familiar with (vim, dd, really take your pick they are everywhere).
2)  I personally prefer Ubuntu 12.04 (though they nearly lost me with some of the early iterations of Ubuntu – 12.04 has it mostly sorted), my primary reason for liking Ubuntu is because of the apt package manager.

But I digress.  When I jumped into this I ran into a rather serious issue for me.  I was unable to launch the console of my VMs on Oracle VM Manager (3.1.1).  Initially it seemed that all I might need was the appropriate version of a Java JRE.  Although it turned out to be a bit more complicated than that.

Install OpenJDK 6 JRE

This is the core of what we need to do.  Basically we are trying to launch a Java application, and without it we are unable to carry on.

# apt-get install openjdk-6-jre

However this gets us to a point where we can launch the Java applications, but not actually get console to our system.  For that there is another little bit of software that Oracle has, the Oracle VM Console application.

Download Oracle VM Console Application

Here we have a 64 bit and 32 bit rpm files, download whichever one fits your architecture.

http://oss.oracle.com/oraclevm/manager/RPMS/ovm-console-1.0.0-2.i386.rpm

http://oss.oracle.com/oraclevm/manager/RPMS/ovm-console-1.0.0-2.x86_64.rpm

So in my case

# wget http://oss.oracle.com/oraclevm/manager/RPMS/ovm-console-1.0.0-2.x86_64.rpm

Also it might be wise to check the parent directory to see if newer versions are released.

http://oss.oracle.com/oraclevm/manager/RPMS/

Now of course we still have to deal with the fact that we are running Ubuntu which doesn’t know what to do with RPMs.  Enter Alien.

Install RPM Conversion Tools

Alien will convert a RPM into a deb file which can be used via dpkg (the underlying system to apt).  So basically we put in an .rpm and get back a .deb.  Simple enough?

# apt-get install alien dpkg-dev debhelper build-essential

Perform Conversion and Installation

So now we just need to feed alien our .rpm and it couldn’t be easier…

# alien ovm-console-1.0.0-2.x86_64.rpm
Warning: Skipping conversion of scripts in package ovm-console: postinst
Warning: Use the --scripts parameter to include the scripts.
ovm-console_1.0.0-3_amd64.deb generated

And now to install our .deb…

# dpkg -i ovm-console_1.0.0-3_amd64.deb
Selecting previously unselected package ovm-console.
(Reading database ... 175033 files and directories currently installed.)
Unpacking ovm-console (from ovm-console_1.0.0-3_amd64.deb) ...
Setting up ovm-console (1.0.0-3) ...

Now just try and launch the console and see what happens.

The same steps allowed it to work on both Firefox and Chromium.

During my setup I misplaced the password for the agent.  This resulted in my management server being unable to successfully authenticate with the hypervisor in order to complete the discovery process.

Once you have successfully discovered the hypervisor you can change the agent password at the server pool level, which will allow you to maintain different agent passwords for different logical areas.

In my environment I am using OVM 3.1.1.

# ovs-agent-passwd oracle
Password: <yourpasshere>
Again: <yourpasshere>

After changing the password, discovery went off without a hitch, of course I could have just as easily misplaced it again, and I would have been right back at square one.

Today I will be documenting the steps to install Oracle VM Manager 3.1.1.  Oracle VM (OVM) is a Xen-based Virtualization Technology.  An OVM Deployment consists of the OVM Server (the bare metal hypervisor) and the OVM Manager, the application stack which manages the afforementioned hypervisors (organized into Server Pools).

The OVM Server installation is really no different from a standard Oracle Linux, or even RHEL install, so I am not going to bother writing that one up unless something changes architecturally.

Now to get started I have pre-provisioned a Oracle Linux Server (5.6) machine which will accept the OVM Manager application.

# cat /etc/issue
Oracle Linux Server release 5.6
Kernel \r on an \m

Download Media

Download Oracle VM Manager 3.1.1 from http://edelivery.oracle.com/oraclevm

Mount Media

I have an NFS mount which holds my media, but the process is the same if you are using local storage.  We want to mount the iso image file into a folder (/mnt in my case)

# mount -o loop /nfsmount/OracleVM-Manager-3.1.1.iso /mnt
# cd /mnt
# ls
components       EULA     oracle-validated.params  runInstaller.sh  TRANS.TBL
createOracle.sh  LICENSE  ovmm-installer.bsx       runUpgrader.sh   upgrade

Prepare Machine for Installation

Oracle VM Manager includes a script which does most of the dirty work for us…  This will create the users/groups, folders, and iptables rules needed by OVM.  If you are going to be using an external firewall, then these are the ports you will need to open on your firewall (bold denotes client to management server communications – remainder are management server to hypervisor).

tcp: 7001-7002, 15901, 54321-5432

udp: 123

Luckily though all we need to do is execute the below script…

# ./createOracle.sh
Adding group 'oinstall' with gid '54321' ...
Adding group 'dba'
Adding user 'oracle' with user id '54321', initial login group 'dba', supplementary group 'oinstall' and  home directory '/home/oracle' ...
Changing ownership of '/home/oracle' to oracle:dba
Creating user 'oracle' succeeded ...
Verifying user 'oracle' OS prerequisites for Oracle VM Manager ...
oracle  soft    nofile          8192
oracle  hard    nofile          8192
oracle  soft    nproc           4096
oracle  hard    nproc           4096
oracle  soft    core            unlimited
oracle  hard    core            unlimited
Setting  user 'oracle' OS limits for Oracle VM Manager ...
Altered file /etc/security/limits.conf
Original file backed up at /etc/security/limits.conf.orabackup
Verifying & setting of user limits succeeded ...
Modifying iptables for OVM
Adding rules to enable access to:
7001  : Oracle VM Manager http
7002  : Oracle VM Manager https
15901 : Oracle VM Manager VM console proxy
54321 : Oracle VM Manager core
54322 : Oracle VM Manager core via SSL
123 : NTP
Saving firewall rules to /etc/sysconfig/iptables:          [  OK  ]
Rules added.

The documentation mentions libaio, bc, and unzip as prerequsites.  My install already included this, so installing them was redundant.

# yum install libaio bc unzip
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
ULN support will be disabled.
Setting up Install Process
Package libaio-0.3.106-5.x86_64 already installed and latest version
Package libaio-0.3.106-5.i386 already installed and latest version
Package bc-1.06-21.x86_64 already installed and latest version
Package unzip-5.52-3.0.1.el5.x86_64 already installed and latest version
Nothing to do.

Install the Oracle VM Manager Software

Here we simply execute the installer script and follow the prompts.  Notice the first prompt asks us the installation type.  I am using Demo, this will install the XE database, and configure OVM Manager to use that.  This is not a support configuration.  If you plan on using this stack in production you will need to install using option two, which will not install a database but rather will require that you connect to a pre-created database, on either the local or a remote server.

# ./runInstaller.sh

Oracle VM Manager Release 3.1.1 Installer

Oracle VM Manager Installer log file:
/tmp/install-2012-05-09-162921.log

Please select an installation type:
1: Demo
2: Production
3: Uninstall
4: Help

Select Number (1-4): 1

Starting demo installation ...

Just some warnings about the XE database.  If you are installing this for production, make sure you heed this warning.

The Demo installation type will use an XE database.  The usage of XE is for *demo purposes only* and is not supported for production. Please *do not* plan to start with XE and migrate to a supported version of the database as this may not be possible. For production environments or any long term usage please use the "Production" option with an SE or EE database.
1: Continue
2: Abort

Select Number (1-2): 1

Now we are going to enter the password to be used for all purposes.  You must have a password with only alpha-numeric characters, no special characters here, you also must have both upper and lower case, with a length of 8.

Verifying installation prerequisites ...

One password is used for all users created and used during the installation.
Enter a password for all logins used during the installation: <enter password here>
Enter a password for all logins used during the installation (confirm): <re-enter password here>

Now that we are ready to begin, this is your last chance to back out.

Verifying configuration ...

Start installing the configured components:
1: Continue
2: Abort

Select Number (1-2): 1

Beginning the installation, now we just wait.

Step 1 of 9 : Database ...
Installing Database ...
Retrieving Oracle Database 11g XE ...
Installing Oracle Database 11g XE ...
Configuring Oracle Database 11g XE ...

Step 2 of 9 : Java ...
Installing Java ...

I ran into a problem here.  I used a password which I believe met all of the requirements, 1 upper-case, 1 number, 8 lower-case, no special characters.  However the install failed here with a failed authentication error.  I suspected this had something to do with differing password requirements for various components.  I tried re-running the installation with a 8 digit password, that met the requirements, but it was having a problem since the Database and Java were already installed.  I got around this by choosing the uninstall option when first executing the install script.  Then the subsequent run with the 8 digit-only password worked an example of a password which meets the requirements would be Gpdf5098.

Step 3 of 9 : Database Schema ...
Creating database schema 'ovs' ...

Step 4 of 9 : WebLogic ...
Retrieving Oracle WebLogic Server 11g ...
Installing Oracle WebLogic Server 11g ...

Step 5 of 9 : ADF ...
Retrieving Oracle Application Development Framework (ADF) ...
Unzipping Oracle ADF ...
Installing Oracle ADF ...
Installing Oracle ADF Patch...

Step 6 of 9 : Oracle VM  ...
Retrieving Oracle VM Manager Application ...
Extracting Oracle VM Manager Application ...
Installing Oracle VM Manager Core ...

Step 7 of 9 : Domain creation ...
Creating Oracle WebLogic Server domain ...
Starting Oracle WebLogic Server 11g ...
Configuring data source 'OVMDS' ...
Creating Oracle VM Manager user 'admin' ...

Step 8 of 9 : Deploy ...
Deploying Oracle VM Manager Core container ...
Deploying Oracle VM Manager UI Console ...
Deploying Oracle VM Manager Help ...
Enabling HTTPS ...
Granting ovm-admin role to user 'admin' ...

Step 9 of 9 : Oracle VM Manager Shell ...
Retrieving Oracle VM Manager Shell & API ...
Extracting Oracle VM Manager Shell & API ...
Installing Oracle VM Manager Shell & API ...

Retrieving Oracle VM Manager Upgrade tool ...
Extracting Oracle VM Manager Upgrade tool ...
Installing Oracle VM Manager Upgrade tool ...
Copying Oracle VM Manager shell to '/usr/bin/ovm_shell.sh' ...
Installing ovm_admin.sh in '/u01/app/oracle/ovm-manager-3/bin' ...
Installing ovm_upgrade.sh in '/u01/app/oracle/ovm-manager-3/bin' ...
Enabling Oracle VM Manager service ...
Shutting down Oracle VM Manager instance ...
Restarting Oracle VM Manager instance ...
Waiting 15 seconds for the application to initialize ...
Oracle VM Manager is running ...
Oracle VM Manager installed.

Please wait while WebLogic configures the applications... This can take up to 5 minutes.

Well the installation is nearly done.  It is doing the last step of the installation, before giving you the information you need for accessing the GUI.

Installation Summary
--------------------
Database configuration:
Database host name          : localhost
Database instance name (SID): XE
Database listener port      : 1521
Application Express port    : 8080
Oracle VM Manager schema    : ovs

Weblogic Server configuration:
Administration username     : weblogic

Oracle VM Manager configuration:
Username                    : admin
Core management port        : 54321
UUID                        : 0004fb00000100004ac53f7c774415a7


Passwords:
There are no default passwords for any users. The passwords to use for Oracle VM Manager, Oracle Database 11g XE, and Oracle WebLogic Server have been set by you during this installation. In the case of a default install, all passwords are the same.

Oracle VM Manager UI:

http://ovmmgr.allanglesit.com:7001/ovm/console


https://ovmmgr.allanglesit.com:7002/ovm/console

Log in with the user 'admin', and the password you set during the installation.

Please note that you need to install tightvnc-java on this computer to access a virtual machine's console.

For more information about Oracle Virtualization, please visit:

http://www.oracle.com/virtualization/

Oracle VM Manager installation complete.

Please remove configuration file /tmp/ovm_configD62uIU.

So now with the completion of the install we can see that the site is available at both an http and an https site running on ports 7001 and 7002 respectively.

Now that we have laid this foundation we can start to get into some of the neat processes around using this particular stack.

ref: Oracle VM Installation and Upgrade Guide for Release 3.1.1 – http://docs.oracle.com/cd/E27300_01/E27308/E27308.pdf

If I have a file..

# cat /etc/file.txt
blah
blah
teststring
blah
blah

And I want to replace “teststring” with “productionvalue” then I could use sed.

# sed -i 's/teststring/productionvalue/g' /etc/file.txt

# cat /etc/file.txt
blah
blah
productionvalue
blah
blah

This is all well and good, but recently I found a better way…

Using perl, not only can we make the change, but we also create a backup file of the original configuration.

# perl -pi.orig -e 's/teststring/productionvalue/g' /etc/file.txt

So now lets take a look at our original file with the .orig extension…

# cat /etc/file.txt.orig
blah
blah
teststring
blah
blah

And finally our changed file…

# cat /etc/file.txt
blah
blah
productionvalue
blah
blah

This is certainly a more useful way of making programmatic changes while retaining the ability to quickly recover from programmatic errors…

For my situation this was a server that pulls data from other servers in order to optimize getting that data to tape.  So we had 2 VGs, one for the system and one for the data we wanted to move.  We planned on simply performing a rsync to move the data itself, but the volumes had to be pre-staged.

Here is the full script.  Nothing fancy, but very effective.

for line in `ssh root@oldserver.domain 'lvs --noheadings -o lv_name,lv_size,vg_name --separator , | tr -d " " | grep oldserver-vg | grep -v unneeded-lv'`
do
lvname=`echo $line | cut -d , -f 1`
lvsize=`echo $line | cut -d , -f 2`
lvcreate -L${lvsize} newserver-vg -n ${lvname}
mkfs.reiserfs -q /dev/newserver-vg/${lvname}
mkdir /backups/${lvname}
done

Gather Existing Configuration

Lets break it down…  The core of this script is gathering and formatting the existing LV configuration.

lvs --noheadings -o lv_name,lv_size,vg_name --separator , | tr -d " " | grep oldserver-vg | grep -v unneeded-lv

You will notice we are using lvs to show all of our logical volumes, but we are also telling it that we don’t want the headings to be displayed, and we only want the lv_name, lv_size, and vg_name columes to be displayed.  Additionally we want to use a comma as the separator, which makes our output easier to parse.  This gives us the core of the data we need, however we have some white space to remove, so we just pipe it through tr to get rid of that.  Now we just want to trim out some of the extra data from our output using grep.  We have two statements, one where we only include anything that contains the string oldserver-vg, and the other where we specifically exclude (with the -v) something that is not needed but previously included in our case unneeded-lv.  This would also work if you had a vgname which matched more than it should have.

Build “For” Loop

So now that we have a command to gather our original configuration, we need to wrap it in a for loop, and run it through SSH.

for line in `ssh root@oldserver.domain 'lvs --noheadings -o lv_name,lv_size,vg_name --separator , | tr -d " " | grep oldserver-vg | grep -v unneeded-lv'`

The way a “for” loop works is that you define a variable which represents each item in an array of data.  In our case our array of data is coming from another command (the boundaries of this command is delineated by the back-ticks) which happens to be an SSH command to the original server where the LVM configuration is intact (the boundaries of this command is delineated by the single-quotes).

Inside the “For” Loop

This is simple, as it goes through each line of output it will do everything between do and done.  Once it reaches done it goes back and increments to the next ${item} and starts again.  When there are no more $item then the for loop is complete, and we move on to the next part of the script.  In this particular script there is nothing after the “For” loop so the script will now be complete and your prompt will be returned.

Inside the “For” Loop…  Define Variables

Here we simply define our variables based on the content of $line.

lvname=`echo $line | cut -d , -f 1`

lvsize=`echo $line | cut -d , -f 2`

The core of each of these commands is the cut command.  The -d option sets the deliminator to “,” the -f option sets the field to the field which contains the data we need.  Which gives us the appropriate variables to actually do stuff.  So lets do it…

Inside the “For” Loop…  Create the Logical Volumes, Format, and Make a Mounting Directory

Now we actually do work.  If you are running this code I recommend you test it in your environment first, by inserting some debugging code instead of doing this stuff on your first run.  Unless you don’t care about the machine.  Make sure you look for pre-existing file systems and what not, if you accidently format your root you won’t be very happy…  Moving on.

lvcreate -L${lvsize} newserver-vg -n ${lvname}

Here we are simply creating the LVs.  You will need to hardcode the VG Name on your server which is being built up.

mkfs.reiserfs -q /dev/newserver-vg/${lvname}

Here we are formatting the file systems as we create them.  This particular environment prefers reiser, use your choice.

mkdir /backups/${lvname}

And finally to make the directories which will have the LVs mounted into.

Final Thoughts

The final step is to copy over the /etc/fstab and update the VG names in it from oldserver-vg to newserver-vg.  Which can be done with sed.  You could also manually mount the file systems as part of the script, or create the fstab by hand.

This is specifically to interact with VMs on ESX 5.  There is also esxcli which will allow you to make hypervisor level changes such as configuring a vSwitch, the esxcli is similar to a switch in how it behaves but it is beyond the scope of this article.

List All VMs on the Host

# vim-cmd vmsvc/getallvms

Get Information for a Specific VM

# vim-cmd vmsvc/get.guest 30

Get Configuration for a Specific VM

# vim-cmd vmsvc/get.config 30

Get Summary for a Specific VM

# vim-cmd vmsvc/get.summary 30

Get Current Power State of a Specific VM

# vim-cmd vmsvc/power.getstate 30

Power On a Specific VM

# vim-cmd vmsvc/power.on 30

Power Off a Specific VM (Hard)

# vim-cmd vmsvc/power.off 30

Shutdown a Specific VM

# vim-cmd vmsvc/power.shutdown 30

Reboot a Specific VM

# vim-cmd vmsvc/power.reset 30

List a Specific VM’s Snapshots

# vim-cmd vmsvc/get.snapshot 30

Unregister a VM from a ESX Host

# vim-cmd vmsvc/unregister 30

Register a VM on a ESX Host

# vim-cmd solo/registervm path/to/.vmx

Create Directory Structure

Here we are just creating the directories into which we will be installing the operating systems into.

# mkdir -p /mnt/build/amd64 /mnt/build/i386
# cd /mnt/build/amd64; mkdir ubuntu_10.04 ubuntu_10.10 ubuntu_11.04 ubuntu_11.10 debian_5 debian_6
# cd /mnt/build/i386; mkdir ubuntu_10.04 ubuntu_10.10 ubuntu_11.04 ubuntu_11.10 debian_5 debian_6

Install Required Software

# apt-get install debootstrap

Install Operating Systems into the Folders

Ubuntu Lucid

http://us.archive.ubuntu.com

# debootstrap --arch=i386 --variant=minbase lucid /mnt/build/i386/ubuntu_10.04/ http://us.archive.ubuntu.com/ubuntu/
# debootstrap --arch=amd64 --variant=minbase lucid /mnt/build/amd64/ubuntu_10.04/ http://us.archive.ubuntu.com/ubuntu/

Ubuntu Maverick

# debootstrap --arch=i386 --variant=minbase maverick /mnt/build/i386/ubuntu_10.10/ http://us.archive.ubuntu.com/ubuntu/
# debootstrap --arch=amd64 --variant=minbase maverick /mnt/build/amd64/ubuntu_10.10/ http://us.archive.ubuntu.com/ubuntu/

Ubuntu Natty

# debootstrap --arch=i386 --variant=minbase natty /mnt/build/i386/ubuntu_11.04/ http://us.archive.ubuntu.com/ubuntu/
# debootstrap --arch=amd64 --variant=minbase natty /mnt/build/amd64/ubuntu_11.04/ http://us.archive.ubuntu.com/ubuntu/

Ubuntu Oneiric

# debootstrap --arch=i386 --variant=minbase oneiric /mnt/build/i386/ubuntu_11.10/ http://us.archive.ubuntu.com/ubuntu/
# debootstrap --arch=amd64 --variant=minbase oneiric /mnt/build/amd64/ubuntu_11.10/ http://us.archive.ubuntu.com/ubuntu/

Debian Lenny

# debootstrap --arch=i386 --variant=minbase lenny /mnt/build/i386/debian_5/ http://ftp.us.debian.org/debian/
# debootstrap --arch=amd64 --variant=minbase lenny /mnt/build/amd64/debian_5/ http://ftp.us.debian.org/debian/

Debian Squeeze

# debootstrap --arch=i386 --variant=minbase squeeze /mnt/build/i386/debian_6/ http://ftp.us.debian.org/debian/
# debootstrap --arch=amd64 --variant=minbase squeeze /mnt/build/amd64/debian_6/ http://ftp.us.debian.org/debian/

Enter the Newly Installed Environments using Chroot

Chroot or CHange ROOT allows us to enter a given folder and have that be our new operating environment.  It becomes a completely isolated enviornment with access only to the files and utilities which already exist within that operating environment.  You do not need to enter a chroot to modify files, but you will need to enter a chroot to install software packages.

# mount -o bind /proc /mnt/build/i386/ubuntu_10.04/proc/
# chroot /mnt/build/i386/ubuntu_10.04

When you are ready to exit your environment and move to the next one, then break the chroot with exit, and then unmount proc.

Exiting the Chroot (not yet)

Before exiting make sure you make the appropriate modifications in the next section.

# exit
# unmount /mnt/build/i386/ubuntu_10.04

You will then need to enter the other installed templates so that you can get them sorted out as well.

Make Modifications to Ubuntu Templates

When using apt-get to install software I noticed these errors on Ubuntu.

   LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"

GPG error: http://ftp.utexas.edu lucid Release: Could not execute '/usr/bin/gpgv' to verify signature (is gpgv installed?)

The first error is simply a problem with the language pack.  We can fix it by running the following.

# apt-get install --reinstall language-pack-en

The second error didn’t occur on oneiric, but occurred on all older versions of Ubuntu I tried.  It can be fixed by running the following.

# apt-get install gpgv

In addition to running those commands, I also run the following as part of my preparations.

# apt-get update && apt-get upgrade
# apt-get install vim aptitude

If you have any other modifications to make on your templates now is the time to do it.

Make Modifications to Debian Templates

On Debian we need to fix than locale error we received on ubuntu.

The problem…

    LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"

Although the fix is different.

# apt-get install locales
# sed -i '/en_US.UTF-8/s/^#//' /etc/locale.gen
# locale-gen en_US.UTF-8

I also run the following.

# apt-get update && apt-get upgrade
# apt-get install vim aptitude

Create the Final Archives

Now we are going to tar gzip these things into there own little files so that we can simply extract as needed.  These commands will do the whole batch, instead of having to type each one manually.

cd /mnt/build/i386/; for a in `ls`; do echo -n $a; tar -czf "/mnt/build/"$a"_i386.tgz" -C $a .; echo "      done."; done

cd /mnt/build/amd64/; for a in `ls`; do echo -n $a; tar -czf "/mnt/build/"$a"_amd64.tgz" -C $a .; echo "      done."; done

Well that pretty much sorts it out for us.  Now we end up with a bunch of tgz files which can be extracted into a file system and used for whatever purpose we need to use them (lxc, openvz, vserver).

This article doesn’t work on Ubuntu 12.10, to use Ubuntu 12.10 please refer to my new article here.

Openvswitch is a better way of managing your virtual networking stacks for KVM and Xen.  This can be used instead of the bridge-utils package, and has the ability to use VLANs, LACP, QoS, sFlow, and others.  In this article we will be using Ubuntu 12.04 (beta1) to configure Openvswitch to hand out networking interfaces to KVM guests.

Please keep in mind that at the time of this writing Ubuntu 12.04 is beta, which means it is not fully baked for production.  Though it is looking very stable.

Install Updates and Prerequisite Software

Check for updates, and install some commonly used hypervisor utilities along with the openvswitch and kvm stacks.

# apt-get update && apt-get dist-upgrade
# apt-get install aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv openvswitch-controller openvswitch-brcompat openvswitch-switch openvswitch-datapath-source

Destroy the Default Libvirt Bridge (virbr0)

We want to delete the default libvirt interface which is created by default to be used for guests (I think it is NAT – I never use it).

# virsh net-destroy default
# virsh net-autostart --disable default

Stop Libvirt and Qemu

This will prevent libvirt from bringing up the legacy bridge.

# service libvirt-bin stop
# service qemu-kvm stop

Enable Openvswitch for Brcompat

# vi /etc/default/openvswitch-switch
BRCOMPAT=yes

Purge Ebtables from System

Ebtables was needed when we were using VLANs on bridge-utils, we no longer need this for openvswitch.

# aptitude purge ebtables

Fix DKMS Module Build

When we originally installed openvswitch, you might have noticed an error talking about the module not available for your kernel, if you didn’t notice it you can see the same error by trying to start the openvswitch-switch service.

# service openvswitch-switch start

If you receive the module error, then we will need to build the openvswitch-datapath module.

# module-assistant auto-install openvswitch-datapath

Note that after installing new kernels you will most likely need to repeat this step, to rebuild the module.

Restart the Openvswitch Services

# service openvswitch-switch restart
# service openvswitch-controller restart

Reboot

# reboot

Check Configuration

Based on your reboot one of two things happened…  Your modules came back up loaded and ready for business, though more likely they didn’t come up.

Here is an example of a working configuration.  If you get this then move on to configuration of the switch itself.

# lsmod | grep brcom
brcompat_mod 13512 0
openvswitch_mod 83993 2 brcompat_mod
# service openvswitch-switch status
ovsdb-server is running with pid 1885
ovs-vswitchd is running with pid 1908
ovs-brcompatd is running with pid 2096

# lsmod | grep brcom
# service openvswitch-switch status
ovsdb-server is running with pid 1885
ovs-vswitchd is running with pid 1908
ovs-brcompatd is running with pid 2096

Ensuring the Modules Load at Boot

The easy way to fix this is to restart the openvswitch-switch service whenever you reboot your hypervisor.

# service openvswitch-switch restart
* Killing ovs-brcompatd (2096)
* Killing ovs-vswitchd (1908)
* Killing ovsdb-server (1885)
* Starting ovsdb-server
* Configuring Open vSwitch system IDs
* Starting ovs-vswitchd
* Starting ovs-brcompatd
* iptables already has a rule for gre, not explicitly enabling

But that might not be ideal for your situation.  As an alternative we can try and address the timing issue which is causing the whole issue in the first place.  Basically what is happening is that upstart is starting openvswitch too late, and it is failing.  If we adjust the waits in /etc/init/failsafe.conf

We want to change this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 40
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 59
$PLYMOUTH message --text="Booting system without full network configuration..." || :

To this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Booting system without full network configuration..." || :

Now after a reboot you shouldn’t have this problem with the modules not loading.

Reboot

# reboot

Check Modules

# lsmod | grep brcom
brcompat_mod           13512  0
openvswitch_mod        83993  1 brcompat_mod

Design our Network Layout

In our example we will use 2 physical interfaces, one of which (eth0) we will consider our management interface, it will have a static IP address and will be “the hypervisor” the other one (eth1) will be the uplink for our Openvswitch.  Now eth1 will have an untagged fake bridge (br0) which will be the parent for our other fake bridge (br10) which will be tagged 10.  If we had more to add they would still use br0 as the parent and then be tagged on the subordinate fake bridge.  Another important thing to keep in mind is that your uplinked switch port must support and be configured to allow your devices to tag for these other VLANs.

Configure Openvswitch Fake Bridges

# ovs-vsctl add-br br0
# ovs-vsctl add-port br0 eth1
# ovs-vsctl add-br br10 br0 10
# ovs-vsctl list-br
br0
br10

Create Dummy Interface Entries

We need to edit /etc/network/interfaces to add some entries

# cat /etc/network/interfaces
auto eth0
iface eth0 inet static
address 10.0.0.141
netmask 255.255.255.0
gateway 10.0.0.1

auto eth1
iface eth1 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto br0
iface br0 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto br10
iface br10 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

These bring up the interfaces with No IP address, now if you run ifconfig -a they should be visible.

Once you have a guest up on br0 or br10 you will see a  “vnet0” interface.

The Problem

During the initial setup via the console…  I selected bnx0 as the primary and set it to use DHCP, after a bit of chewing I received the error below…

IOError: can not safely set interface bnx0 via DHCP.

After the wizard I went into the shell to look at the configuration.

nmc@myhost:/$ show network interface bnx0
==== Interface: bnx0 ====
bnx0: flages=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 5
inet 0.0.0.0 netmask 00000000 broadcast 255.255.255.255
ether e8:9a:8f:be:25:ab

The only real important points to note from this output is that the IP address is 0.0.0.0, which is expected since it threw an error during the initial configuration.  Additionally the MTU is set for 1496.

Configure Static IP with MTU, Gateway, and Nameserver

So perhaps the problem is specific with DHCP, so I go through the process of configuring the IP address statically.

nmc@myhost:/$ setup network interface bnx0
Option ? static
bnx0 IP address        : 10.0.8.41
bnx0 netmask            : 255.255.255.0
bnx0 mtu                    : 1500
Gateway IP address : 10.0.8.1
Name Server #1         : 8.8.8.8
Name Server #2         :
Name Server #3         :
SystemCallError: failed to configure bnx0 with ip 10.0.8.41 netmask 255.255.255.0 mtu 1500 broadcast + up: ifconfig: setifmtu: SIOCSLIFMTU: bnx0: invalid argument

Configure Static IP with MTU and Gateway

So the error above says invalid argument, so lets remove stuff from the bottom up, no more name server.

nmc@myhost:/$ setup network interface bnx0

Option ? static
bnx0 IP address        : 10.0.8.41
bnx0 netmask            : 255.255.255.0
bnx0 mtu                    : 1500
Gateway IP address : 10.0.8.1
Name Server #1         :
Name Server #2         :
Name Server #3         :
SystemCallError: add net default: gateway 10.0.8.1: Network is unreachable 

Configure Static IP with MTU

So above we have an issue with being unable to reach the gateway so Nexenta won’t accept the configuration, fine lets remove that.

nmc@myhost:/$ setup network interface bnx0

Option ? static
bnx0 IP address        : 10.0.8.41
bnx0 netmask            : 255.255.255.0
bnx0 mtu                    : 1500
Gateway IP address :
Name Server #1         :
Name Server #2         :
Name Server #3         :
OK. 

Check Interface Configuration for MTU

So above this responds with an OK.  As in alright everything worked, but not so fast, we haven’t done anything to fix it and by definition our configuration would still be broken, no gateway and no name server.  So lets look at our configuration.

nmc@myhost:/$ show network interface bnx0
==== Interface: bnx0 ====
bnx0: flages=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 5
inet 10.0.8.41 netmask ffffff00 broadcast 10.0.8.255
ether e8:9a:8f:be:25:ab

Configure Static IP with MTU of 1496

Interesting, so it didn’t actually accept our MTU of 1500, like it said it did.  Well lets see what happens if we play along and try and set the MTU to 1496 – since Nexenta is so adamant that is the correct way.

nmc@myhost:/$ setup network interface bnx0

Option ? static
bnx0 IP address        : 10.0.8.41
bnx0 netmask            : 255.255.255.0
bnx0 mtu                    : 1496
Invalid mtu '1496' please enter value in range [1500:8982]

How contradictory, apparently Nexenta will only accept 1500-8982 as a valid MTU.  Looking at our configuration we can see that nothing changed, and our configuration is still broken.

nmc@myhost:/$ show network interface bnx0
==== Interface: bnx0 ====
bnx0: flages=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 5
inet 10.0.8.41 netmask ffffff00 broadcast 10.0.8.255
ether e8:9a:8f:be:25:ab

The Fix

Now the problem here is that the Broadcom NICs are using an invalid MTU setting and Nexenta is not smart enough to properly handle it, so we need to tell the bnx driver how to use the Broadcom NICs.

Enter expert mode, this allows us to use an actual command shell, instead of the incredibly limited NMC.

nmc@myhost:/# option expert_mode = 1

Now that we are in expert mode, we can do silly things like launch bash.  I have no idea why we need to say not-bash to launch bash, but we do.

nmc@myhost:/# !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute?  (y/n)

Below is a little sed snippet which will allow you to uncomment the line without actually entering the file yourself.

nmc@myhost:/# sed -i '/mtu=/s/^#//' /kernel/drv/bnx.conf

Quick explanation on what this sed command does, -i tells it to make changes “in-place” or directly on the file in question, without this the file is sent to stdout where the change is made.  The apostrophes annotate the full search string, separated by forward slashes.  The first section /mtu=/ is our search string, so basically any line that matches this search string will have the next part run against it.  The second section s/^#// is actually two sections, but it is a substitution section so we will treat it as one, since wherever it finds “^#” it will replace with “” so what is “^#” simple, it is a comment which occurs as the first character in a line.  This simply removes that comment.

A simpler but much more verbose method would be to substitute the whole line (with the comment) for the whole line without the comment.

nmc@myhost:/# sed -i 's/#mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;/mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;/g' /kernel/drv/bnx.conf

Or if you’d prefer to handle the change manually what we are doing is uncommenting the following line from /kernel/drv/bnx.conf

#mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;

Verify the change, then reboot.

# cat /kernel/drv/bnx.conf | grep mtu=
mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;

# reboot

After the reboot, we can finally appropriately configure our networking.

nmc@myhost:/$ setup network interface bnx0
Option ? static
bnx0 IP address        : 10.0.8.41
bnx0 netmask            : 255.255.255.0
bnx0 mtu                    : 1500
Gateway IP address : 10.0.8.1
Name Server #1         : 8.8.8.8
Name Server #2         :
Name Server #3         :
Enabling bnx0 as 10.0.8.41/255.255.255.0 mtu 1500 ... OK.

Bottom Line

Once I implemented this change I was able to successfully configure my networking with either DHCP or static, with the appropriate MTU for my purpose.  I have no idea why this is required, if a simple uncommenting of the line is required then why would Nexenta as a Solution Builder perform that uncommenting for me to spare me from the trouble of doing it myself.  I personally prefer “pure” ZFS solutions without the pretty interface, especially considering that I spent more time trying to learn how to use the Nexenta Management Console (command line) then I should have had to based on my experience.  Add into this the fact that I have to do low level changes like this just to get basic networking so that I can run through the wizard.

verbal-bsod

I fought the original P2V battle with NT many years ago (and emerged bloody but victorious), I don’t remember the exact process we went through but it wound its way through every virtualization product to get it to its final resting place of Hyper-V (or so we thought).  The purpose of this article is to help you get NT running on Linux KVM, in my case from a Hyper-V VM, though a “fresh” install should work as well.  For those of you looking to go from physical, when I did it years ago I think I used VMWare Converter to create an image, and then the VirtualBox tools to convert it to vdi, and then to vhd (fully allocated).  Then once that was done got it running in VirtualBox before dragging it to Hyper-V.

Final, Working XML

<domain type='kvm'>
<name>ntmachine</name>
<description>NT Machine</description>
<memory>524288</memory>
<currentMemory>524288</currentMemory>
<vcpu>1</vcpu>
<os>
<type arch='x86_64' machine='pc-0.14'>hvm</type>
<boot dev='hd'/>
</os>
<cpu match='exact'>
<model>pentium</model>
</cpu>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/usdckvm202/vm_newmfg_boot'/>
<target dev='hda' bus='ide'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
</disk>
<controller type='ide' index='0'>
</controller>
<interface type='bridge'>
<mac address='00:1d:d8:b7:1c:5f'/>
<source bridge='br0'/>
<model type='ne2k_pci'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'/>
<video>
<model type='vga' vram='9216' heads='1'/>
</video>
</devices>
</domain>

Now in the above there are a couple of key areas I want to point out.  The CPU, Network Cards, and Disks are key.

CPU

We need to virtualize a specific processor to make NT happy, in this case the predominant processor of the era was the pentium.

<cpu match='exact'>
<model>pentium</model>
</cpu>

Network Cards

In order to have working networking we must emulate a network device which actually existed, the ne2k_pci NIC was very common and has a driver included in the NT Server CD the RTL80292.

<interface type='bridge'>
<mac address='00:1d:d8:b7:1c:5f'/>
<source bridge='br0'/>
<model type='ne2k_pci'/>
</interface>

Disks

Here we just need to use IDE, no VirtIO (obviously).  Also here I am using a Logical Volume instead of a raw disk image file, simply because that is how I like to run my production environments, if you plan on using a file, then adjust your XML so that you are referencing a file.

<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/usdckvm202/vm_newmfg_boot'/>
<target dev='hda' bus='ide'/>
</disk>

Other Notes

I also removed anything extraneous that I thought could cause problems, everything in the <features> tag (acpi, apic, pae) I also pulled the memballoon stuff.

Examine the Pool and the Existing Free Space and Capacity

Notice that we get two different results with regard to FreeSpace.  This is because the “pool show” command will deduct the snap-reserves of all allocated volumes and thus show us the actual usable space for provisioning a new volume.  Alternatively “member show -poolinfo” will leave the snapshot reserve space unless it is actually written, so basically we have our 363GB + 2TB in reserves – actual space used by snapshots (in this case 551GB).

SANGroup00> pool show
Name                 Default Members Volumes Capacity   FreeSpace
-------------------- ------- ------- ------- ---------- ----------
default              true    1       2       4.35TB     363.22GB

SANGroup00> member show -poolinfo
Name       Status  Version    Disks Capacity   FreeSpace  Connections Pool
---------- ------- ---------- ----- ---------- ---------- ----------- -------
EQL00      online  V5.1.1 (R1 12    4.35TB     1.89TB     24          default
85010)     

Show Existing Configuration of the First Volume

Now in the below output we are looking at the configuration of Volume00.  We want to notice, the snap-reserve, snap-reserve-available, and snapshots field.  You will see that in my example we have a 100% reserve (which is default) and that we have 54% of that snap-reserve available.  This means that we have snapshots in the snapshots field, and that we are using some of that space (which means part of it is no longer “reserved”).

SANGroup00> volume select Volume00
SANGroup00(volume_Volume00)> show
_____________________________ Volume Information ______________________________
Name: Volume00                 Size: 1TB
VolReserve: 1TB                        VolReserveInUse: 492.7GB
ReplReserveInUse: 0MB                  iSCSI Alias: Volume00
iSCSI Name:                            ActualMembers: 1
iqn.2001-05.com.equallogic:8-cb2b76- Snap-Warn: 10%
a51cd9059-188000000244f04f-volume00  Snap-Depletion: delete-oldest
Description:
Snap-Reserve: 100%                     Snap-Reserve-Avail: 54% (551.86GB)
Permission: read-write                 DesiredStatus: online
Status: online                         Connections: 8
Snapshots: 1                           Bind:
Type: not-replicated                   ReplicationReserveSpace: 0MB
Replicas: 0                            ReplicationPartner:
Pool: default                          Transmitted-Data: 1011.65GB
Received-Data: 3.15TB                  Pref-Raid-Policy: none
Pref-Raid-Policy-Status: none          Thin-Provision: disabled
Thin-Min-Reserve: 0% (0MB)             Thin-Growth-Warn: 0% (0MB)
Thin-Growth-Max: 0% (0MB)              ReplicationTxData: 0MB
MultiHostAccess: enabled               iSNS-Discovery: disabled
Replica-Volume-Reserve: 0MB            Thin-Clone: N
Template: N                            NAS File System: N
Administrator:                         Thin-Warn-Mode: offline
_______________________________________________________________________________

Remove the Snapshot from the First Volume

The snapshot we have will no longer be needed so we are getting rid of that now.

SANGroup00(volume_Volume00)> snapshot show
Name                        Permission Status      Schedule Connections
--------------------------- ---------- ----------- -------- -----------
Volume00-2012-01-04 read-write offline              0
-20:20:43.3.1

SANGroup00(volume_Volume00)> snapshot delete Volume00-2012-01-04-20:20:43.3.1
Do you want to delete the snapshot? (y/n) [n]y

Snapshot deletion succeeded.

Modify the Existing Snap-Reserve of the First Volume

I am removing the snap-reserve in its entirety.  If you simply want to decrease the space available for snapshots use an appropriate percentage.

SANGroup00(volume_Volume00)> snap-reserve 0%

Review Configuration of First Volume

Make sure everything looks right before moving on.

SANGroup00(volume_Volume00)> show
_____________________________ Volume Information ______________________________
Name: Volume00                 Size: 1TB
VolReserve: 1TB                        VolReserveInUse: 492.77GB
ReplReserveInUse: 0MB                  iSCSI Alias: Volume00
iSCSI Name:                            ActualMembers: 1
iqn.2001-05.com.equallogic:8-cb2b76- Snap-Warn: 10%
a51cd9059-188000000244f04f-volume00  Snap-Depletion: delete-oldest
Description:
Snap-Reserve: 0%                       Snap-Reserve-Avail: 0% (0MB)
Permission: read-write                 DesiredStatus: online
Status: online                         Connections: 8
Snapshots: 0                           Bind:
Type: not-replicated                   ReplicationReserveSpace: 0MB
Replicas: 0                            ReplicationPartner:
Pool: default                          Transmitted-Data: 1015.44GB
Received-Data: 3.15TB                  Pref-Raid-Policy: none
Pref-Raid-Policy-Status: none          Thin-Provision: disabled
Thin-Min-Reserve: 0% (0MB)             Thin-Growth-Warn: 0% (0MB)
Thin-Growth-Max: 0% (0MB)              ReplicationTxData: 0MB
MultiHostAccess: enabled               iSNS-Discovery: disabled
Replica-Volume-Reserve: 0MB            Thin-Clone: N
Template: N                            NAS File System: N
Administrator:                         Thin-Warn-Mode: offline
_______________________________________________________________________________

Show Existing Configuration for the Second Volume

Check our snap-reserve, snap-reserve-avail, and snapshots.

SANGroup00> volume select Volume01
SANGroup00(volume_Volume01)> show
_____________________________ Volume Information ______________________________
Name: Volume01                        Size: 1TB
VolReserve: 1TB                        VolReserveInUse: 875.07GB
ReplReserveInUse: 0MB                  iSCSI Alias: TestDS
iSCSI Name:                            ActualMembers: 1
iqn.2001-05.com.equallogic:8-cb2b76- Snap-Warn: 10%
8cecd9059-2b20000000b4ec1a-volume01  Snap-Depletion: delete-oldest
Description:                           Snap-Reserve: 100%
Snap-Reserve-Avail: 100% (1TB)         Permission: read-write
DesiredStatus: online                  Status: online
Connections: 8                         Snapshots: 0
Bind:                                  Type: not-replicated
ReplicationReserveSpace: 0MB           Replicas: 0
ReplicationPartner:                    Pool: default
Transmitted-Data: 5.46TB               Received-Data: 7.77TB
Pref-Raid-Policy: none                 Pref-Raid-Policy-Status: none
Thin-Provision: disabled               Thin-Min-Reserve: 0% (0MB)
Thin-Growth-Warn: 0% (0MB)             Thin-Growth-Max: 0% (0MB)
ReplicationTxData: 0MB                 MultiHostAccess: enabled
iSNS-Discovery: enabled                Replica-Volume-Reserve: 0MB
Thin-Clone: N                          Template: N
NAS File System: N                     Administrator:
Thin-Warn-Mode: offline
_______________________________________________________________________________

Modify the Existing Snap-Reserve for Second Volume

Again simply removing the snap-reserve.

SANGroup00(volume_Volume01)> snap-reserve 0%

Review Configuration of Second Volume

Sanity check our changes to the second volume.

SANGroup00(volume_Volume01)> show
_____________________________ Volume Information ______________________________
Name: Volume01                        Size: 1TB
VolReserve: 1TB                        VolReserveInUse: 875.07GB
ReplReserveInUse: 0MB                  iSCSI Alias: TestDS
iSCSI Name:                            ActualMembers: 1
iqn.2001-05.com.equallogic:8-cb2b76- Snap-Warn: 10%
8cecd9059-2b20000000b4ec1a-volume01  Snap-Depletion: delete-oldest
Description:                           Snap-Reserve: 0%
Snap-Reserve-Avail: 0% (0MB)           Permission: read-write
DesiredStatus: online                  Status: online
Connections: 8                         Snapshots: 0
Bind:                                  Type: not-replicated
ReplicationReserveSpace: 0MB           Replicas: 0
ReplicationPartner:                    Pool: default
Transmitted-Data: 5.46TB               Received-Data: 7.77TB
Pref-Raid-Policy: none                 Pref-Raid-Policy-Status: none
Thin-Provision: disabled               Thin-Min-Reserve: 0% (0MB)
Thin-Growth-Warn: 0% (0MB)             Thin-Growth-Max: 0% (0MB)
ReplicationTxData: 0MB                 MultiHostAccess: enabled
iSNS-Discovery: enabled                Replica-Volume-Reserve: 0MB
Thin-Clone: N                          Template: N
NAS File System: N                     Administrator:
Thin-Warn-Mode: offline
_______________________________________________________________________________

Examine the Pool and the New Free Space and Capacity

Now we should notice a significantly different FreeSpace indications on both commands, and they should match if you disabled all snap-reserves on all volumes.

SANGroup00> pool show
Name                 Default Members Volumes Capacity   FreeSpace
-------------------- ------- ------- ------- ---------- ----------
default              true    1       2       4.35TB     2.35TB

SANGroup00> member show -poolinfo
Name       Status  Version    Disks Capacity   FreeSpace  Connections Pool
---------- ------- ---------- ----- ---------- ---------- ----------- -------
EQL00      online  V5.1.1 (R1 12    4.35TB     2.35TB     24          default
85010)  

Configure the Default Snap-Reserve

Of course if you don’t already have a bunch of volumes on which to make this change you can always just change the default behavior.  Then all subsequently created volumes will have a snap-reserve of whatever you have picked.

SANGroup00> grpparams def-snap-reserve 0%

So firstly what are hot corners, and why do I hate them?  Hot corners is the upper left hand corner of your screen (0,0) the idea is that if you move your mouse into the upper left hand corner then you must want the dash to come up.  However it frustrates me, because sometimes when I am typing or cussing or both, I will inadvertently bump my mouse and then I end up in the dash.

The file we need to change is /usr/share/gnome-shell/js/ui/layout.js

Here is the chunk of the file we are interested in:

        this._corner = new Clutter.Rectangle({ name: 'hot-corner',
width: 1,
height: 1,
opacity: 0,
reactive: true });

We want to change that to:

        this._corner = new Clutter.Rectangle({ name: 'hot-corner',
width: 1,
height: 1,
opacity: 0,
reactive: false });

Notice the only change is that we are making this not reactive.

Now after a reboot the hot corners should no longer be an issue.

UPDATE 19 FEB 2013

A little cleaner way to accomplish this same way, without having to open up a text editor would be to use awk…  I find myself doing this often due to updates that overwrite my configuration.

# awk '/this._corner = new Clutter.Rectangle/,/ });/ { sub(/reactive: true/, "reactive: false");} { print }' /usr/share/gnome-shell/js/ui/layout.js > /usr/share/gnome-shell/js/ui/layout.js.tmp; mv /usr/share/gnome-shell/js/ui/layout.js /usr/share/gnome-shell/js/ui/layout.js.orig; mv /usr/share/gnome-shell/js/ui/layout.js.tmp /usr/share/gnome-shell/js/ui/layout.js

Now what exactly are we doing?

Step 1 – Look for a block of text.

Find block starting with “this._corner = new Clutter.Rectangle” and ending with ” });”

Step 2 – Substitute within that block only.

“reactive: true” will become “reactive: false” only within the block.  There are other instances of reactive: true which we don’t want to modify.

Step 3 – Output the modified file into a temporary file.

“> /usr/share/gnome-shell/js/ui/layout.js.tmp”

Step 4 – Rearrange the files

“mv /usr/share/gnome-shell/js/ui/layout.js /usr/share/gnome-shell/js/ui/layout.js.orig; mv /usr/share/gnome-shell/js/ui/layout.js.tmp /usr/share/gnome-shell/js/ui/layout.js”

This will leave you with an active configuration which will not have reactive hot corners.

# awk '/this._corner = new Clutter.Rectangle/,/ });/' /usr/share/gnome-shell/js/ui/layout.js
this._corner = new Clutter.Rectangle({ name: 'hot-corner',
width: 1,
height: 1,
opacity: 0,
reactive: false });

As well as a backup of the original configuration file.  That can be put back into service if need be.

# awk '/this._corner = new Clutter.Rectangle/,/ });/' /usr/share/gnome-shell/js/ui/layout.js.orig
this._corner = new Clutter.Rectangle({ name: 'hot-corner',
width: 1,
height: 1,
opacity: 0,
reactive: true });

Create New LAG

(switch01)# configure
(switch01) (Config)# port-channel esxhost00-vmnet
(switch01) (Config)# exit

Show New LAG

(switch01) #show port-channel

Logical Interface Group Id Port-Channel Name Link State Mbr Ports Active Ports
----------------- -------- ----------------- ---------- --------- ------------
0/1/11            11       esxhost00-vmnet   Down       

The above command gives us the Logical Interface, which we will need.  But if we look at it with a different command we can see the problem.

(switch01) #show port-channel all

Port-                  Link
Log.       Channel            Adm. Trap  STP           Mbr      Port    Port
Intf        Name        Link  Mode Mode  Mode   Type   Ports    Speed   Active
------ --------------- ------ ---- ---- ------ ------- ------ --------- ------
lag 11 esxhost00-vmnet Up     En.  En.  En.    Dynamic

The problem here is that by default we create LAGs in dynamic mode.  ESX requires static mode.

Set the LACP Mode as Static

(switch01)# configure
(switch01) (Config)#interface 0/1/11
(switch01) (Interface 0/1/11)#port-channel static
(switch01) (Interface 0/1/11)#exit

Down the Switchport for the Standby Vmnic

(switch01)# configure
(switch01) (Config)#interface 2/0/2
(switch01) (Interface 2/0/2)#shutdown
(switch01) (Interface 2/0/2)#exit

Review Existing ESX Portgroup Configuration

# esxcli network vswitch standard portgroup policy failover get --portgroup-name="VM Network"
Load Balancing: srcport
Network Failure Detection: link
Notify Switches: true
Failback: true
Active Adapters: vmnic4
Standby Adapters: vmnic6
Unused Adapters:
Override Vswitch Load Balancing: false
Override Vswitch Network Failure Detection: false
Override Vswitch Notify Switches: false
Override Vswitch Failback: false
Override Vswitch Uplinks: true
# esxcli network vswitch standard portgroup policy failover get --portgroup-name="Management Network"
Load Balancing: srcport
Network Failure Detection: link
Notify Switches: true
Failback: true
Active Adapters: vmnic4
Standby Adapters: vmnic6
Unused Adapters:
Override Vswitch Load Balancing: false
Override Vswitch Network Failure Detection: false
Override Vswitch Notify Switches: false
Override Vswitch Failback: false
Override Vswitch Uplinks: true

Change Existing ESX Portgroup Configuration

# esxcli network vswitch standard portgroup policy failover set --portgroup-name="VM Network" --load-balancing=iphash --failure-detection=link --notify-switches true --failback false --active-uplinks=vmnic4,vmnic6
# esxcli network vswitch standard portgroup policy failover set --portgroup-name="Management Network" --load-balancing=iphash --failure-detection=link --notify-switches true --failback false --active-uplinks=vmnic4,vmnic6

Review New ESX Portgroup Configuration

# esxcli network vswitch standard portgroup policy failover get --portgroup-name="VM Network"
Load Balancing: iphash
Network Failure Detection: link
Notify Switches: true
Failback: false
Active Adapters: vmnic4, vmnic6
Standby Adapters:
Unused Adapters:
Override Vswitch Load Balancing: true
Override Vswitch Network Failure Detection: true
Override Vswitch Notify Switches: true
Override Vswitch Failback: true
Override Vswitch Uplinks: true
# esxcli network vswitch standard portgroup policy failover get --portgroup-name="Management Network"
Load Balancing: iphash
Network Failure Detection: link
Notify Switches: true
Failback: false
Active Adapters: vmnic4, vmnic6
Standby Adapters:
Unused Adapters:
Override Vswitch Load Balancing: true
Override Vswitch Network Failure Detection: true
Override Vswitch Notify Switches: true
Override Vswitch Failback: true
Override Vswitch Uplinks: true

Add Both Switchports to the LAG

(switch01)# configure
(switch01) (Config)#interface 2/0/3
(switch01) (Interface 2/0/3)#addport 0/1/11
(switch01) (Interface 2/0/3)#exit
(switch01) (Config)#interface 2/0/2
(switch01) (Interface 2/0/2)#addport 0/1/11
(switch01) (Interface 2/0/2)#exit
(switch01) (Config)#exit

Review LAG Configuration

(switch01) #show port-channel all

Port-                  Link
Log.       Channel            Adm. Trap  STP           Mbr      Port    Port
Intf        Name        Link  Mode Mode  Mode   Type   Ports    Speed   Active
------ --------------- ------ ---- ---- ------ ------- ------ --------- ------
lag 11 esxhost00-vmnet Up     En.  En.  En.    Static  2/0/3 Auto      True
2/0/2 Auto      False 

Verify ESX Connectivity

So now that we have a LAG configured and operational albeit only on one port, the traffic will either work or not.  So if you have connectivity then you should be able to safely enable the disabled port (in my case 2/0/2) and not have any problems with your traffic.  Now if you do have problems with connectivity, then you probably have a problem with something like VLANs not being configured correctly on the LAG, we ran into that when we put this into production.

Up the Switchport for the Standby Vmnic

(switch01) #configure
(switch01) (Config)#interface 2/0/2
(switch01) (Interface 2/0/2)#no shutdown
(switch01) (Interface 2/0/2)#exit
(switch01) (Config)#exit

Review Final LAG Configuration

(switch01) #show port-channel all

Port-                  Link
Log.       Channel            Adm. Trap  STP           Mbr      Port    Port
Intf        Name        Link  Mode Mode  Mode   Type   Ports    Speed   Active
------ --------------- ------ ---- ---- ------ ------- ------ --------- ------
lag 11 vdihost00-vmnet Up     En.  En.  En.    Static  2/0/3 Auto      True
2/0/2 Auto      True   

Now notice here that we are showing the LAGs interface up with both ports Active, so assuming that you still have basic connectivity to each of your vswitches then you should be good to go.

The core of this issue is that in the previous generation there was a 10g module and a separate stacking module.  With the current version they decided to simply have one module that is 10g or stacking, and you can set the mode to toggle between them, this of course is good for Dell, because it requires them to maintain smaller inventories, and also it is good for you because if your needs change you can re-purpose the same module for a completely different use-case.

I am not a “switch” guy, I find the stuff boring in most cases, however this was just an interesting enough issue with a simple enough fix that I decided to document the fix.  This article assumes that you know how to connect to the console of your switch and that you are familiar with the basics of configuring a switch.

Enter Enable Mode

> enable 

View Current Stack Configuration

# show stack-port
Configured  Running
Stack      Stack      Link       Link
Unit    Interface       Mode       Mode       Status     Speed (Gb/s)
---- ---------------- ---------- ---------- ------------ ------------
1    xg1              Ethernet   Ethernet   Link Down    Unknown
1    xg2              Ethernet   Ethernet   Link Down    Unknown
1    xg3              Ethernet   Ethernet   Link Down    Unknown
1    xg4              Ethernet   Ethernet   Link Down    Unknown    

Enter Configuration Mode

# config

Enter Stack Configuration Mode

(config)# stack

Change From Ethernet To Stack Mode

(config-stack)# stack-port 1/xg1 stack
(config-stack)# stack-port 1/xg2 stack
(config-stack)# exit

Verify New Stack Configuration

# show stack-port
Configured  Running
Stack      Stack      Link       Link
Unit    Interface       Mode       Mode       Status     Speed (Gb/s)
---- ---------------- ---------- ---------- ------------ ------------
1    xg1              Stack   Ethernet   Link Down    Unknown
1    xg2              Stack   Ethernet   Link Down    Unknown
1    xg3              Ethernet   Ethernet   Link Down    Unknown
1    xg4              Ethernet   Ethernet   Link Down    Unknown    

Once you are done with one wash, rinse, and repeat on the other switch(es), but one last thing to notice.  Here that we are showing two different modes, “configured stack mode” and “running stack mode” this is reflecting that we have made the configuration changes necessary to flip the modules over, but that the machine has not be rebooted so that it can reload its configuration.  So now when you have some downtime reboot the switches.  The first one that comes up will be the master and the second one will end up with a default configuration.  In our case the only configuration specific to our environment is jumbo frames.  Which can be configured switch wide.

Enter Enable Mode

> en

Enter Configuration Mode

# configure

Enter Interface Configuration Mode For All Ethernet Interfaces

(config)# interface range ethernet all

Enable Jumbo Frames

(config-if)# mtu 9216
(config-if)# exit

Well that does it.  You now have a stacked Dell PowerConnect with Jumbo Frames enabled.

Identify the Initiator IQN

First we need to identify the IQN of the iSCSI initiator, this is necessary to configure the security on the iSCSI target.

# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:c0ca4ce904ff.4f45b957
Initiator node alias: openindiana
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS Access: disabled
Tunable Parameters (Default/Configured):
Session Login Response Time: 60/-
Maximum Connection Retry Time: 180/-
Login Retry Time Interval: 60/-
Configured Sessions: 1

Configure iSCSI Static Discovery

Here we configure the type of discovery that we want to use, in our case we are using static discovery.

# iscsiadm modify discovery --static enable

Here we add the actual discovery parameters, the IQN of the target, as well as the IP which we will connect to to create the connection.

# iscsiadm add static-config iqn.2010-09.org.openindiana:02:1a7a530f-8508-4736-f269-d6363a8cb5e6,10.0.0.21:3260

View Available LUNs

Now with everything working on both sides of the connection we will end up seeing specifics about our connection, and our available LUNs

# iscsiadm list target -vS
Target: iqn.2010-09.org.openindiana:02:1a7a530f-8508-4736-f269-d6363a8cb5e6
Alias: -
TPGT: 1
ISID: 4000002a0000
Connections: 1
CID: 0
IP address (Local): 10.0.0.22:59749
IP address (Peer): 10.0.0.21:3260
Discovery Method: Static
Login Parameters (Negotiated):
Data Sequence In Order: yes
Data PDU In Order: yes
Default Time To Retain: 20
Default Time To Wait: 2
Error Recovery Level: 0
First Burst Length: 65536
Immediate Data: yes
Initial Ready To Transfer (R2T): yes
Max Burst Length: 262144
Max Outstanding R2T: 1
Max Receive Data Segment Length: 32768
Max Connections: 1
Header Digest: NONE
Data Digest: NONE

LUN: 0
Vendor:  OI
Product: COMSTAR
OS Device Name: /dev/rdsk/c4t600144F0B7EA490000004F471B750001d0s2

Notice LUN 0 in the above output.  This is our LUN, remember if you expose multiple LUNs then you will see multiple LUNs here, this is not yet a usable disk for that we will need to use the format utility to identify the disk name.

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c3t0d0 <ATA-WDCWD5003ABYX-1-1S02 cyl 60798 alt 2 hd 255 sec 63>
/pci@0,0/pci1028,4dd@1f,2/disk@0,0
1. c4t600144F0B7EA490000004F471B750001d0 <OI-COMSTAR-1.0 cyl 1303 alt 2 hd 255 sec 63>
/scsi_vhci/disk@g600144f0b7ea490000004f471b750001

We now have a completed iSCSI session, and we have a usable volume on our client side.

I did my testing on OpenIndiana 151a, which had up to date packages as of Feb 23, 2012.  The process should be largely similar if not the same on Solaris 11 Express and Solaris 11 GA.

Install Prequisite Packages

# pkg install network/iscsi/target

Start SCSI Target Mode Framework Service

# svcadm enable svc:/system/stmf:default

Start iSCSI Target Service

# svcadm enable svc:/network/iscsi/target:default

Display the Status of the SCSI Target Service

The ALUA Status is what allows a LUN to be served via FC and iSCSI at the same time.  We do not require or want it.

# stmfadm list-state
Operational Status: online
Config Status     : initialized
ALUA Status       : disabled
ALUA Node         : 0

Create a File System (Volume) to be Provisioned

Before we can present a LUN then we need to create a ZFS file system as a volume so that it can be used as a block device.

# zfs create -V 10G rpool/testvol

Notice the lack of a mount point, this is due to provisioning the file system as a volume.

# zfs list rpool/testvol
NAME            USED  AVAIL  REFER  MOUNTPOINT
rpool/testvol  10.3G   436G    16K  -

Create a LUN for the Volume

So we have created a volume, now we need to make the STMF aware of it, so that it can hand it out as a block device, and thus it can be used.

# sbdadm create-lu /dev/zvol/rdsk/rpool/testvol
Created the following LU:

GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f0b7ea490000004f471b750001  10737418240          /dev/zvol/rdsk/rpool/testvol

We will need the path and the name of the LUN, which is available from the create-lu command (as GUID and SOURCE), however stmfadm gives us a more readable format.

# stmfadm list-lu -v
LU Name: 600144F0B7EA490000004F471B750001
Operational Status: Online
Provider Name     : sbd
Alias             : /dev/zvol/rdsk/rpool/testvol
View Entry Count  : 0
Data File         : /dev/zvol/rdsk/rpool/testvol
Meta File         : not set
Size              : 10737418240
Block Size        : 512
Management URL    : not set
Vendor ID         : OI
Product ID        : COMSTAR
Serial Num        : not set
Write Protect     : Disabled
Writeback Cache   : Enabled
Access State      : Active

Create an iSCSI Qualified Name (iqn) Target

This simply creates the target iqn for the machine we will be serving out iSCSI volumes from.

# itadm create-target
Target iqn.2010-09.org.openindiana:02:1a7a530f-8508-4736-f269-d6363a8cb5e6 successfully created

Put the New Target in Offline Mode

In order to add the target to the target group we will be creating (to control storage views) then we must offline our target first.

# stmfadm offline-target iqn.2010-09.org.openindiana:02:1a7a530f-8508-4736-f269-d6363a8cb5e6

Create a Target Group

This the group which will control the target side of the storage connection, so in other words it is on your array, and I like to name it as such, so if your array were named “array001″ then I would name your target group that.

# stmfadm create-tg array001

Add the Target to the Target Group

# stmfadm add-tg-member -g array001 iqn.2010-09.org.openindiana:02:1a7a530f-8508-4736-f269-d6363a8cb5e6

Review the Target Group Configuration

# stmfadm list-tg -v
Target Group: array001
Member: iqn.2010-09.org.openindiana:02:1a7a530f-8508-4736-f269-d6363a8cb5e6

Create a Host Group

This is the group which will identify where the traffic is coming from, so for organization I like to have it named to reflect the server name of the initiator side of the connection.  So if your server was “server001″ then I would name your host group to match that, this will help keep things nice and tidy and make your views much more legible.

# stmfadm create-hg server001

Add the Host to the Host Group

Here you will need to get the iqn of the client side of the storage connector and insert it into this command.

# stmfadm add-hg-member -g server001 iqn.1986-03.com.sun:01:c0ca4ce904ff.4f45b957

Create a View

A view is what ties together everything, we take the target group, the host group, and the LUN and put them in a view.  If the connection does not meet the first two criteria then it does not get to see the LUN.  This is important in the event that you have multiple servers accessing your array (which you will).  You don’t want to end up with multiple servers writing to the same LUNs at the same time without certain precautions being taken first, so we want to block that behavior.

# stmfadm add-view -t array001 -h server001 600144F0B7EA490000004F471B750001

Create a Target Portal Group

The Target Portal Group is what the initator actually connects to in order to find the storage.  Here we need to use the storage IP on array001.

# itadm create-tpg array001portal 10.0.0.21:3260

Enable Static Mode Discovery

# devfsadm -i iscsi

At this point once you have configured your iSCSI initiator on the client side then you should be able to see your iSCSI block device.

That wraps it up.  Put a neat little bow on it and send it on its way.

A few things to remember with iSCSI and your ZFS box.  You will need a mirrored ZIL (mirrored because you don’t want to lose data) to counteract the performance penalties associated with synchronous writes.  The most important thing is that iSCSI is a protocol that is suited towards flexibility but not performance.  If you need performance you should be considering fibre channel, remember a good chunk of the cost comes from the fabric, if you already have the fabric for another SAN you can easily connect another array into the existing fabric, remember it is Storage Area NETWORK.

Factor One – Use Case

The use case is the single most important factor, this basically tells you how you plan on using the data.  Is this going to serve files (NFS or CIFS), will you be using it to serve block devices (Fibre Channel or iSCSI), or perhaps you will be using this as a consolidation point of multiple sources of data for use in a backup scenario.  The answer to this question will primarily determine if you are using a ZIL or Cache device or if you are using spares.  This of course will steal slots (where disks could go) so you will want to factor that into your larger capacity planning.

Physical Capacity

Obviously the machine that you buy will have a limit to how many disks you can fit in it.  After that you can plan on adding expansion chassis.  You need to make sure you understand how your zpools will be formatted and the types of disks that you will be using so that you can begin to see how much data you will be able to squeeze onto the spindles.  One easy thing to overlook is the need for drive bays for other purposes (spares, log, cache, system drives, etc).  If you plan on expanding with an additional chassis, will you use up all of your expansion slots with various cards (10Gb, FibreChannel, NICs, SSD, etc).  Also when considering disks you need to keep in mind that capacity is not the only metric to consider.  You also need to keep in mind that your workload will determine how performant the solution must be to meet requirements, for example a backup server will not require the same performance as a storage head which is handing out block devices which have Operating Systems installed on them (read: Virtual Machine environments).

Spare Devices

Depending on your use case you might need to consider allocating some drives as spares to allow you the time to procure and/or replace failed drives.  This will simply protect you from additional failure(s).  If you have the space I like to have 1 spare per 12-15 acting as a spare.  This really straight forward, just make sure that as part of any large system that the company accepts the risks involved with your decision, if they are not willing to accept the risks then provide them with an option which will protect them from that risk.  They will then either pay for the lower risk system or accept the risk of the cheaper system.

Cache Devices

Cache devices (L2ARC) is a really quick and cheap way to increase your read iops on a system wide basis.  Basically as data is read off of ZFS it is read through the cache device, then when subsequent read requests for the data come in then they can be served directly off of the cache, this works in combination with the ARC which is where all of your memory goes in ZFS.  Now a cache device is really necessary if you are using deduplication.  When you enable deduplication all subsequently written files are indexed into the dedup table, before a write can be committed it must be compared to items that are already in the dedup table (to see if anything else is the same – thus if it should actually be written to disk or if a pointer to the previously existing file will exist) initially this will not pose a problem as your dedupe table will be relatively small and it can be stored in the 25% of the ARC that it is relegated to.  However at some point you will run out of space in the ARC and if you do not have a L2ARC then your dedup table will be swapped which of course means that it will be holding up writes to disk, in other words it will be slow.  Since a cache device’s purpose is to speed up reads, then you want to look for a SSD which is slanted towards better performance on the read side.  I like Crucial M4 for this purpose, but please keep in mind they had a serious firmware bug in a previous version so make sure you end up with v0309 on the disk and update to it if you don’t.

Log Devices

Log devices (ZFS Intent Log) is a really cool way to speed up technologies which utilize synchronous writes.  Synchronous writes are used to ensure that data is committed to the disk before additional data is sent, it essentially serializes the connection.  Most commonly you will find synchronous writes in databases, NFS and iSCSI.  So if you are planning on using one of these technologies then you will want to consider using a mirrored ZIL device.  Now I am sure someone out there is saying, “Whoa mirrored?  But that takes up two slots man…” and you would be correct, however keep in mind that if you have a failure of your ZIL, you have actual data committed to it which has not been committed to disk (read: data loss).  As such you really ought to have a mirrored ZIL or none at all, you can also stack multiple mirrors as a ZIL if you had a higher synchronous writes workload.  Since log devices are meant to improve writes (albeit a certain type of write) then you will want to look for a SSD which is slanted towards better write performance.  I tend to like OCZ Vertex 3 for this purpose.

Factor Two – Data Growth

Now we get into actually sizing data.  When you try to project growth you need to make sure that (1) you take use an appropriately sized sample set of data (2) factor in any relevant business or technical factors which could have skewed your sample.  I like to size my storage based off of backup size.  Simply grab the size of your full backups for the data in question, if you are going to use this to serve block devices collecting this data becomes a bit more complex, because now you are talking more of a machine sprawl type of growth which can be harder to account for.  Here is are a few formulas to get you started (make sure you are using the same metric, i.e. B, KB, MB, GB, TB).

Calculate Weekly Growth

(full_week1 - full_week0) = growth_week1
(full_week2 - full_week1) = growth_week2

Calculate Weekly Growth Rate

(growth_week1 / full_week0) = growth_rate_week1

Average Weekly Growth

(growth_week1 + growth_week2 + growth_week3 + growth_week4) / number_of_weeks = average_weekly_growth

Average Weekly Growth Rate

(growth_rate_week1 + growth_rate_week2 + growth_rate_week3 + growth_rate_week4) / number_of_weeks = average_weekly_growth_rate

Annualized Weekly Growth

(average_weekly_growth * 53) = annual_growth

So for example if we start with this example…

Notice that in this example we have a couple of points where I saw it fit to make notes.  One was when our dataset decreased and the other was when it grew sharper than the trend up to that point.  I made these notes because I wanted to point out that if you had a very large deviation then you might want to make some sort of adjustment so as to not skew your results.  In this case both of these happened because of regular course of business so they should be legitimately considered as part of the growth curve.  However if you had some sort of failure in your backups that you were already accounting for that data in another way you would not want to double count it if it came back into the backups in the middle of your sample.

Weekly Growth

week1 (105 - 100) = 5
week2 (120 - 105) = 15
week3 (122 - 120) = 2
week4 (122 - 122) = 0
week5 (118 - 122) = -4
week6 (130 - 118) = 12
week7 (135 - 130) = 5
week8 (136 - 135) = 1

Weekly Growth Rate

week1 (5 / 100) = .05 = 5%
week2 (15 / 105) = .143 = 14.3%
week3 (2 / 120) = .017 = 1.7%
week4 (0 / 122) = 0 = 0%
week5 (-4 / 122) = -.033 = -3.3%
week6 (12 / 118) = .102 = 10.2%
week7 (5 / 130) = .039 = 3.9%
week8 (1 / 135) = .007 = .7%

Average Weekly Growth

(5 + 15 + 2 + 0 + -4 + 12 + 5 + 1) / 8 = 4.5

Average Weekly Growth Rate

(5 + 14.3 + 1.7 + 0 + -3.3 + 10.2 + 3.9 + .7) / 8 = 4.1%

Annualized Growth

(4.5 * 53) = 238.5

Now as you can see from this example the numbers add up quickly, in a year we have easily doubled our dataset, now why is this important?  Basically most businesses perform budgeting at the beginning of the year and as such expenses need to be planned out, and any expansion that you do perform will need to _at least_ last through the year, either way you at least need to know how long you can reasonably expect to be able to use this hardware in its capacity before having to look for upgrades.  Your project will be generally regarded as successful if it meets the technical requirements with minimal involvement from the business (having to ask for more money), especially if you can tell them that this storage will need to be expanded in x number of months.

Factor Three – Data Churn

Churn is freaking scary when it comes to calculating capacity requirements.  Churn is the amount of data change, now some churn is actually growth so we will want to keep that in mind when we are performing our analysis, however churn is the amount of data that changes in a given week. It is amazing the levels of churn that some companies have.  This is especially disconcerting if you plan on utilizing snapshots or send/receive as a form of backup.  Now we still use backups to calculate churn, however instead we will use our mid-week backups instead of our fulls.  Now if you are using incremental backups then you will total all of those during the week to get your dataset size.  If you are using differentials you can calculate this using the final differential in the week.

Now the tricky thing about churn is that growth is churn, but churn is not growth.  So after we calculate our growth and our churn we will subtract our growth from our churn to get our actual churn, otherwise we will be double counting our growth.  I don’t bother doing this until we are talking about the averages and the annualized numbers.  You will notice that these formulas are largely the same as the growth formulas, just remember her that you need to calculate these against your aggregated incrementals or your final differential to get valid numbers.

Calculate Weekly Churn (differentials)

diff_week1 = churn_week1

Calculate Weekly Churn (incrementals)

(inc1_week1 + inc2_week1 + inc3_week1 + inc4_week1 + inc5_week1) = churn_week1

Calculate Weekly Churn Rate

(churn_week1 / full_week0) = churn_rate_week1

Average Weekly Churn

(churn_week1 + churn_week2 + churn_week3 + churn_week4) / number_of_weeks = average_weekly_churn

Average Weekly Churn Rate

(churn_rate_week1 + churn_rate_week2 + churn_rate_week3 + churn_rate_week4) / number_of_weeks = average_weekly_churn_rate

Annualized Weekly Churn

(average_churn_growth * 53) = annual_churn

Expanding on our previous example, if we had churn rates that looked like this…

Weekly Churn Rate

week1 (8 / 100) = .08 = 8%
week2 (6 / 105) = .057 = 5.7%
week3 (5 / 120) = .042 = 4.2%
week4 (2 / 122) = .016 = 1.6%
week5 (2 / 122) = .016 = 1.6%
week6 (15 / 118) = .127 = 12.7%
week7 (6 / 130) = .046 = 4.6%
week8 (1 / 135) = .007 = .7%

Average Weekly Churn

(8 + 6 + 5 + 2 + 2 + 15 + 6 + 1) / 8 = 5.6

Now this is only partly correct, we still need to account for the growth we have already included  by subtracting the average growth from our average churn.

(5.6 - 4.5) = 1.1

Giving us a growth-adjusted weekly average churn of 1.1.

Average Weekly Churn Rate

(8 + 5.7 + 4.2 + 1.6 + 1.6 + 12.7 + 4.6 + .7) / 8 = 4.9%

Again we must remove our growth.

(4.9 - 4.5) = 0.4

Giving us a growth-adjusted weekly average churn of 0.4%

Annualized Churn

(1.1 * 53) = 58.3

Now that we have calculated our storage needs we can start to work out our final configurations based on the goals of our project.  In our example case we have learned that based on our current dataset (136) and our annualized growth (58.3), which means that in 1 year our dataset will increase by 30% which means that if we had a goal of engineering a system which would not need any upgrades (based on current use cases) in the first two years then you would need a minimum size of 252.6.

Now one final consideration when sizing ZFS is pool capacity, ZFS uses copy on write to turn random writes into sequential writes, this is very good for performance, however when a pool exceeds 80% ZFS will not be able to do this as well and as such your writes will become semi-random (since parts of files will have to be written in non-sequential blocks that happen to be free).  So we should also make sure that we have a 20% ceiling on our projections so that we can ensure the same level of performance throughout the systems life.  And with that we have a final number of 303.1.

Now please keep in mind, these formulas will allow you to work through your own sizing exercise, but I am not suggesting that you have a growth rate of 4.5% and a churn rate of 1.1% this was merely an example, you will need to use your own numbers to come up with projections which are applicable to your scenario.  Also you will notice that in my example I did not use a size metric (MB, GB, TB, PB) I did this intentionally so as to not confuse you these formulas will work regardless of your metric, just ensure you use the same metric in all of your calculations.

Happy Sizing!