IT From All Angles

Today I am revisiting my previous post on Openvswitch on Ubuntu 12.04.  Things have changed since then.  Previously Openvswitch was relatively new and as such the userland tools (with libvirt being the one I use) didn’t support it yet, so while you could have used Openvswitch by executing the kvm processes for each VM manually.  Since this was a predictable problem the Openvswitch guys created a brcompat module, which would take the place of the bridge_utils functionality and expose Openvswitch bridges as legacy bridges.  This allowed the existing userland which already had the legacy bridge functionality builtin to continue to operate the same way.  Anyways as of libvirt 0.9.11 the native Openvswitch functionality is present and supported.  As such I am updating my documentation to include this alternate and preferred method of operation.

Libvirt Versions

Here is the libvirt version present in Ubuntu 12.10.

# libvirtd --version
libvirtd (libvirt) 0.9.13

Here is the libvirt version present in Ubuntu 12.04.

# libvirtd --version
libvirtd (libvirt) 0.9.8

So as we can see above this guide will only work on Ubuntu 12.10 and not on 12.04 or older.  For Ubuntu 12.04, please refer to my older article available here.

Install Updates and Prerequisite Software

Check for updates, and install some commonly used hypervisor utilities along with the openvswitch and kvm stacks.  Note that below I am installing the openvswitch-brcompat package.  I do not think that you need this, however I did install it in my environment in case I was going to need to fail back to the brcompat route, and I haven’t had the time to validate it without that package installed.  But it is not being used in this configuration.

# apt-get install aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv openvswitch-controller openvswitch-brcompat openvswitch-switch

Destroy the Default Libvirt Bridge (virbr0)

We want to delete the default libvirt interface which is created by default to be used for guests (I think it is NAT – I never use it).

# virsh net-destroy default
# virsh net-autostart --disable default

Stop Libvirt and Qemu

This will prevent libvirt from bringing up the legacy bridge.

# service libvirt-bin stop
# service qemu-kvm stop

Purge Ebtables from System

Ebtables was needed when we were using VLANs on bridge-utils, we no longer need this for openvswitch.

# aptitude purge ebtables

Restart the Openvswitch Services

#service openvswitch-switch restart
#service openvswitch-controller restart

Configure Interfaces

Now this configuration is a single interface which will be used both for management of the hypervisor as well as the bridge.  There are three key things to note.  We need to set the physical interface to “manual” (instead of “dhcp” or “static”).  We also need to perform a quick up and down of the physical interface in order to be able to initialize the bridge on it.  Finally we need to setup the configuration of an interface which will connect up to the bridge that we define, this becomes the management interface for the OS.  In this example we are simply using DHCP.  This will show us if it is working quickly, but it can also add some complexity if things are not working, please keep in mind that you might want to just use a static IP address to eliminate that as an issue.

# cat /etc/network/interfaces
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto ovsbr0p1
iface ovsbr0p1 inet dhcp

Configure our Network

Configuring the network will be a three step process.  First we define the bridge which I am naming ovsbr0 unlike legacy bridges we don’t need to stick to brX as a naming convention.

# ovs-vsctl add-br ovsbr0

Second we connect our bridge with its uplink which in this case is eth0.

# ovs-vsctl add-port ovsbr0 eth0

Finally we create a port for our hypervisor to connect up to the bridge and out the physical interface.

# ovs-vsctl add-port ovsbr0 ovsbr0p1 -- set interface ovsbr0p1 type=internal

After all of that we should end up with something like below.

# ovs-vsctl show
ed1986fc-830e-4f66-9ce7-92fad0787bb8
Bridge "ovsbr0"
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "eth0"
Interface "eth0"
ovs_version: "1.4.3"

Reboot

# reboot

Adjusting Service Sleeps

If your networking seems to hang on boot up.

In /etc/failsafe.conf we need to adjust the sleeps.

Change this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 40
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 59
$PLYMOUTH message --text="Booting system without full network configuration..." || :

To this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Booting system without full network configuration..." || :

Obviously if you end up needing this change, make sure you reboot again to ensure that it resolved the issue.

Define VM Configuration

I am not going to go into details on how to create a guest from XML, but if you need some help on that refer to my previous article here.

Basically below we have a snippet from a guests XML configuration which configures a single network interface.  You can edit an existing machine configuration using “virsh edit vmnamehere”

<interface type='bridge'>
<mac address='52:54:00:0f:b4:7a'/>
<source bridge='ovsbr0'/>
<virtualport type='openvswitch'/>
<model type='virtio'/>
</interface>

After you have merged the changes in, libvirt will go create a port on Openvswitch for this guest.  Which we can see below as vnet0.

# ovs-vsctl show
ed1986fc-830e-4f66-9ce7-92fad0787bb8
Bridge "ovsbr0"
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "vnet0"
Interface "vnet0"
Port "eth0"
Interface "eth0"
ovs_version: "1.4.3"

Additionally we can see that in our VM configuration libvirt has added more details, the target dev and parameters interfaceid.  This is how the VM knows which port on the Openvswitch to connect to.

<interface type='bridge'>
<mac address='52:54:00:0f:b4:7a'/>
<source bridge='ovsbr0'/>
<virtualport type='openvswitch'>
<parameters interfaceid='a78ac510-2442-1326-c60f-d76c1e830bd6'/>
</virtualport>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

That is all there is to it.  When you bring up your guest. you will have networking exactly as you expected.  But now you open yourself up to a world of possibilities with Openvswitch, such as port mirroring, QoS, LACP and many more.

Windows Server 2012 includes a new storage model.  It follows in the footsteps of ZFS and btrfs, in that it uses a pooled disk model, and attempts to combine all of the functionality of RAID, LVM, and the file system.  Now this looks like a 1.0 from the perspective of integration, however I am quite impressed by the implementation from the usability perspective.  While it is not as polished and integrated the flow is consistent and when using the GUI it is very simple to actually configure your storage.  That said this feels like multiple layers.

Usability Comparison to ZFS

To give you an idea of what I would expect lets look at how it works in ZFS.

In ZFS you configure a pool of disks, where we have disk0-4 and want to name the pool tank.

# zpool create tank mirror disk0 disk1 mirror disk2 disk3

Then you configure file systems on top of that, in this case fs1 on tank with a size of 10G.

# zfs create 10G tank/fs1

Regardless of which camp you are in the above commands are simple to understand and can be easily modified to suit my use case.

Interpretation of What is Being Done

Now as far as I can tell Microsoft has integrated VHD files into the solution in order to expose this functionality.  I don’t have a technical problem with this, however at this point there is a lot of manual work that has to be done in order to get to the point where I have a usable file system.  So you create your pool with your disks (this is no different from ZFS, however in ZFS this is where your RAID is performed.  Whereas in the Storage Pool/Spaces implementation they are doing that on the file system level (I suspect with software RAID on top of VHD files).  Once the pool is up and configured you create a Virtual Disk, where you will assign your resiliency (RAID level), then initialize the Virtual Disk, create a partition on the Virtual Disk, and finally format the Virtual Disk.  Now I am not saying that you shouldn’t have to technically do all of these things, however for this to have the seamless experience that they are trying to go for, they will need to put a bit more polish on this.

Find Our Physical Disks

As you can see we have a 25GB disk for our OS, which has the “CanPool” as False, this is because it is the system disk, and it is already being used.  We also have 2 10GB disks and 6 5GB disks.  I am doing this testing on my Virtualbox Windows 2012, so the disks are simple to scale out.  So the goal is to get them all in the same pool.  But in order to be more thorough we will be creating the pool with ONLY the 5GB disks, and then adding the 10GB disk, just to prove that we can.

PS> Get-PhysicalDisk

FriendlyName        CanPool             OperationalStatus   HealthStatus        Usage                              Size
------------        -------             -----------------   ------------        -----                              ----
PhysicalDisk0       False               OK                  Healthy             Auto-Select                       25 GB
PhysicalDisk1       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk2       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk3       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk4       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk5       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk6       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk7       True                OK                  Healthy             Auto-Select                       10 GB
PhysicalDisk8       True                OK                  Healthy             Auto-Select                       10 GB

Assign our Disks to a Variable

PS> $disk = Get-PhysicalDisk | where {$_.size -eq "5GB"}

PS> $disk

FriendlyName        CanPool             OperationalStatus   HealthStatus        Usage                              Size
------------        -------             -----------------   ------------        -----                              ----
PhysicalDisk1       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk2       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk3       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk4       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk5       True                OK                  Healthy             Auto-Select                        5 GB
PhysicalDisk6       True                OK                  Healthy             Auto-Select                        5 GB

Check the Name of the Storage SubSystem

I am assuming that you could wildcard this to your PC name or to Storage Spaces, but I didn’t want to document it that way since I am not sure how it would behave if you have multiple pools.

PS> Get-StorageSubSystem

FriendlyName                            HealthStatus                            OperationalStatus
------------                            ------------                            -----------------
Storage Spaces on WIN-PTP6J8CKQNG       Healthy                                 OK

Create the New Storage Pool

Here we are going to combine the variable we assigned earlier, with the name of the Storage SubSystem to create our new pool.  Then we will check the pools on the machine.  You might have noticed this before, but there is a Primordial pool, which holds disks which have not been assigned.  Sounds like someone at Microsoft has a sense of humor.  It is the goo from which your pool rises.

PS> New-StoragePool -StorageSubSystemFriendlyName "Storage Spaces on WIN-PTP6J8CKQNG" -FriendlyName pool0 -PhysicalDisks $disk

FriendlyName            OperationalStatus       HealthStatus            IsPrimordial            IsReadOnly
------------            -----------------       ------------            ------------            ----------
pool0                   OK                      Healthy                 False                   False

PS> Get-StoragePool

FriendlyName            OperationalStatus       HealthStatus            IsPrimordial            IsReadOnly
------------            -----------------       ------------            ------------            ----------
Primordial              OK                      Healthy                 True                    False
pool0                   OK                      Healthy                 False                   False

Add Additional Disks to the Pool

Assign the new disks to a variable, the same way we did before.

PS> $newdisk = Get-PhysicalDisk | where {$_.size -eq "10GB"}

PS> $newdisk

FriendlyName        CanPool             OperationalStatus   HealthStatus        Usage                              Size
------------        -------             -----------------   ------------        -----                              ----
PhysicalDisk7       True                OK                  Healthy             Auto-Select                       10 GB
PhysicalDisk8       True                OK                  Healthy             Auto-Select                       10 GB

Here we use the Add-PhysicalDisks cmdlet to add them to the Pool.  Now from a user experience this command is hard to find.  It should probably be a Modify-StoragePool or Add-StoragePoolMembers or similar.

PS> Add-PhysicalDisk -PhysicalDisks $newdisk -StoragePoolFriendlyName pool0

Create Virtual Disk (vdisk0)

This is pretty straight forward.  We are just creating a disk.

PS>  New-VirtualDisk -StoragePoolFriendlyName pool0 -FriendlyName vdisk0 -ResiliencySettingName mirror -ProvisioningType Thin -Size 30GB

FriendlyName        ResiliencySettingNa OperationalStatus   HealthStatus        IsManualAttach                     Size
me
------------        ------------------- -----------------   ------------        --------------                     ----
vdisk0              Mirror              OK                  Healthy             False                             30 GB

Initialize Disk (vdisk0)

The next step is to initialize it.

PS> Get-VirtualDisk vdisk0 | Initialize-Disk -PartitionStyle GPT

Create and Initialize Virtual Disk (vdisk1)

I combined the above two steps into one, via some powershell magic.  Functionally the same things are happening though.

PS>  New-VirtualDisk -StoragePoolFriendlyName pool0 -FriendlyName vdisk1 -ResiliencySettingName mirror -ProvisioningType Thin -Size 30GB | Initialize-Disk -PartitionStyle GPT

Partition and Format the Virtual Disk

Here is where it gets interesting.  I have combined everything into one step, but we have two distinct actions, the creation of the partition, and the formatting of the volume that is created (via the partitioning).  In the formatting, you have your choice of file system (of course the FAT’s but also NTFS and ReFS which is the interesting one).  If you are logged into the GUI this step will also result in a Windows Explorer pop-up asking if you want to format the disk, you can cancel that you are formatting via the command line.

PS>  get-disk | where {$_.uniqueid -eq ((Get-VirtualDisk vdisk0).UniqueId)} | New-Partition -UseMaximumSize -AssignDriveLetter | Format-Volume -FileSystem NTFS

Confirm
Are you sure you want to perform this action?
Warning, all data on the volume will be lost!
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

DriveLetter       FileSystemLabel  FileSystem       DriveType        HealthStatus        SizeRemaining             Size
-----------       ---------------  ----------       ---------        ------------        -------------             ----
E                                  NTFS             Fixed            Healthy                  29.78 GB         29.87 GB

Also if you’d like to do the above without having to confirm your action, pass Format-Volume with -Confirm:$false to avoid that confirmation.

PS>  get-disk | where {$_.uniqueid -eq ((Get-VirtualDisk vdisk0).UniqueId)} | New-Partition -UseMaximumSize -AssignDriveLetter | Format-Volume -FileSystem NTFS -Confirm:$false

Well there you have it.  All and all I think it is an interesting concept, though it definitely needs some work around integrating the components.

I don’t tend to use a lot of Windows in recent years, but if you look at my past articles I used to do quite a bit around Hyper-V, DFS, Exchange, and various other Windows Services.  One thing that was always frustrating is the inconsistent experience around NIC teaming.  Basically Teaming allows you to gain throughput and resiliency by having multiple physical NICs listen over the same logical NIC.  The reason of this inconsistency was that teaming was always provided by the hardware/driver manufacturer so features and configurations would be implemented separately.

Well the good news is that it appears that Microsoft has decided to reverse course with this in Windows Server 2012.  Which is most definitely a good thing for users.

So while I don’t plan on going too deep into Windows 2012 I thought this would be worth a quick run through.

Figure 1-1:  Above you will find the Local Server view of the Server Manager tool.  You will notice on the left near the center we have our 2 ethernet connections.  In my case these are inside of Virtualbox VM and these are NAT connections.  But directly above that you see “Nic Teaming: Disabled”

Figure 1-2:  Above you will notice the NIC Teaming Details screen.  You get here by clicking on Disabled in Figure 1-1.  Notice in the bottom left we have no Teams configured, and in the bottom right we have 2 adapters available for teaming.

Figure 1-3:  Here we simply highlight the available interfaces, then in the TASKS menu select Add to New Team.

Figure 1-4:  Assign a Team Name.  Here I am using team0.

Figure 1-5:  Now we have a working Team which contains the two Ethernet Adapters.

Figure 1-6:  Just for giggles I switched the second Ethernet Adapter in Virtualbox from “NAT” to “Not Attached” to simulate network failure, and above is what I got.

Now I don’t like to do anything that I can’t do from the command line, frankly that is mostly because that is far easier to document then having to worry about screenshots.  So lets do the same thing with Powershell.

PS> New-NetLbfoTeam -Name team0 -TeamMembers (Get-NetAdapter | where {$_.name -match "Ethernet"}).name

Confirm
Are you sure you want to perform this action?
Creates Team:'team0' with TeamMembers:{'Ethernet 2', 'Ethernet'}, TeamNicName:'team0', TeamingMode:'SwitchIndependent'
and LoadBalancingAlgorithm:'TransportPorts'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

Name                   : team0
Members                : {Ethernet 2, Ethernet}
TeamNics               : team0
TeamingMode            : SwitchIndependent
LoadBalancingAlgorithm : TransportPorts
Status                 : Down

As we can see from the above output we can choose the Teaming Mode and Load Balancing Algorithm.  For the Teaming Mode you can use Switch Indepent, Static, and LACP.  The latter two options require the switch to support and/or be configured to allow that configuration to work.  You have your choice of Load Balancing Algorithms (IP Address, MAC Address, Transport Ports, and Hyper-V Port) the first three use the IP/MAC/Port from the source and destination to create a hash and then balance the traffic appropriately, the Hyper-V Port doesn’t do any hashing as far as I can tell, it simply splits the Hyper-V Ports over the interfaces.

We can also see that the status is “down” but fear not.  Give it about 15 seconds and then look at the team again.

PS> Get-NetLbfoTeam

Name                   : team0
Members                : {Ethernet 2, Ethernet}
TeamNics               : team0
TeamingMode            : SwitchIndependent
LoadBalancingAlgorithm : TransportPorts
Status                 : Up

This will give you network based teaming, there are also some additional options to allow you to use LACP and other algorithms.  For me this is a big change, but the more impactful change is the change in Network Connection names.  Now they seem to be using Ethernet followed by Ethernet 2 and so on.  Which for me is far superior to Local Area Connection followed by Local Area Connection 2 and so on.  I think it would have been better to use Ethernet0 or Ethernet1 just to give you consistent numbering, through multiple interfaces.  But I will take what I can get.

Logical Volume Manager makes the dynamic expansion of file systems dead stupid simple.  However there is a weakness, if you are using a partitioned file system as your Physical Volume (PV) then you will end up needing to expand the file system if you ever need to grow the actual storage.  This can be avoided by using the actual physical device /dev/sda as the PV, however if this is the same volume where your /boot partition lives then you will have no choice but to partition it out since /boot cannot reside on a Logical Volume (LV).  The other way you can avoid this is by simply creating a second partition on the same disk and adding it as a new PV to the existing Volume Group (VG).  This is preferable as there is far less room for error than in this procedure, but I personally find that method a little more confusing to inherit then a straight forward single partition made an LV.

Please keep in mind that this series of changes is very high risk.  Please make sure you (1) understand fully what you are doing (2) understand fully why you are doing it (3) have backups of the data if it is of any importance to you.  If you do something stupid (even if I tell you to do it I will not be held responsible for any consequences – scared yet?).

Check our Current Configuration

Below you will see the 4 Logical Volumes, 1 Volume Group and 1 Physical Volume that we have.

# lvs
LV        VG      Attr   LSize   Origin Snap%  Move Log Copy%  Convert
lv_root  LocalVG -wi-ao 100.00G
lv_swap LocalVG -wi-ao  24.00G
lv_data1  LocalVG -wi-ao  30.00G
lv_data2  LocalVG -wi-a-  30.00G

You will see here we have ~94GB of free space in our Volume Group.  Now in this particular situation we actually have ~300GB+ due to a change in the RAID configuration (or in a more common scenario an expansion of a virtual hard disk file).

#vgs
VG      #PV #LV #SN Attr   VSize   VFree
LocalVG   1   4   0 wz--n- 278.74G 94.74G

Here is where the issue ultimately lies.  Since this OS was installed using LVM on top of a disk partition instead of the whole disk (there is no other option I am afraid when it comes to root installations).  We will need to expand the partition in order to see this on the LVM Physical Volume.

# pvs
PV         VG      Fmt  Attr PSize   PFree
/dev/sda2  LocalVG lvm2 a-   278.74G 94.74G

Make our Change Plan

Now in order to “extend” the partition we actually have to delete the partition and then recreate it.  The important part of this is that we want to ensure that when we are creating our new partition that we start it on the exact same block (we will end it on a further block, but we need to start in the right place).  Don’t try and get smart and relocate the partition in addition to extending it.  That would classify as stupid.

# fdisk -l /dev/sda

Disk /dev/sda: 598.8 GB, 598879502336 bytes
255 heads, 63 sectors/track, 72809 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          16      128488+  83  Linux
/dev/sda2              17       36404   292286610   8e  Linux LVM

From the above output we can see a few things…

1) The disk is ~598GB
2) The last cylinder on the disk is 72809
3) The last cylinder on our (/dev/sda2) partition is 36404
4) The first cylinder on our (/dev/sda2) partition is 17
5) The type of the partition is 8e which means Linux LVM.

So this will consist of 3 changes to the partition changes.

1) Delete /dev/sda2
2) Create /dev/sda2 starting on the 17th cylinder and ending on the 72,809th cylinder (the last cylinder).
3) Set the type of /dev/sda2 to 8e.

Implement our Change Plan

Now that we know what we have to do, and we have written down all of these key details.  Keep in mind I actually mean written down on paper, if you sustain a power outage during the change you will not have a file system or a terminal window to reference the pre-existing configuration and you will need to boot to a live-cd to rebuild your partition table from your notes.

# fdisk /dev/sda

The number of cylinders for this disk is set to 72809.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 598.8 GB, 598879502336 bytes
255 heads, 63 sectors/track, 72809 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          16      128488+  83  Linux
/dev/sda2              17       36404   292286610   8e  Linux LVM

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Command action
e   extended
p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (17-72809, default 17):
Using default value 17
Last cylinder or +size or +sizeM or +sizeK (17-72809, default 72809):
Using default value 72809

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sda: 598.8 GB, 598879502336 bytes
255 heads, 63 sectors/track, 72809 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          16      128488+  83  Linux
/dev/sda2              17       72809   584709772+  8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Reboot

So hopefully you noticed that after I was done, I printed the configuration and double checked my work against my plan to ensure that I had it right.  Next we must reboot in order to re-read your partition table.

# reboot

Broadcast message from root (pts/0) (Fri Aug 10 11:25:58 2012):

The system is going down for reboot NOW!

Resize the LVM Physical Volume

OK so now that we have extended the partition and rebooted, LVM will show the same characteristics as before we extended the partition.  The next step is to perform a pvresize so that LVM will re-examine the disk characteristics of the physical volume, and then take advantage of any new space.

# pvs
PV         VG      Fmt  Attr PSize   PFree
/dev/sda2  LocalVG lvm2 a-   278.74G 94.74G

# pvresize /dev/sda2
Physical volume "/dev/sda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

# vgs
VG      #PV #LV #SN Attr   VSize   VFree
LocalVG   1   4   0 wz--n- 557.62G 373.62G

Enjoy

That was it now, we have managed to extend the underlying partition which is the LVM Physical Volume.  This has enabled us to seamlessly extend our Volume Group, and now the full disk is being used in our environment.

In our previous articles we built the foundation of our Logical Domains environment, which then enabled us to create a Logical Domain in part two.  Now in the final article in this series we will be connecting to and performing the installation of the operating system.

Solaris Virtualization: Using Logical Domains on Solaris 11 Part One
Solaris Virtualization: Using Logical Domains on Solaris 11 Part Two

Before we start the domain, we need to bind the associated resources to the domain.  This will also show us the port that the console is running on, so that we can connect to it.

root@t4:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    4     4G       0.8%  2h 44m
t4-g1            inactive   ------          8     4G
root@t4:~# ldm bind-domain t4-g1
root@t4:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    4     4G       3.0%  2h 45m
t4-g1            bound      ------  5000    8     4G    

In the above example we can see that once we “bind-domain” this is binding specific hardware resources to the domain.  In this case we can now see that we have a port for the console connections.  For this domain it is 5000.  So if we telnet to port 5000 on the localhost we will end up connected to the console.  I personally like to connect prior to starting the domain, because I prefer to watch the boot up process, so that I know everything is working properly, this means I will have a separate ssh session which I use for telnet.

root@t4:~# telnet localhost 5000
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to t4.
Escape character is '^]'.

Connecting to console "t4-g1" in group "t4-g1" ....
Press ~? for control options ..

Start the domain.

root@t4:~# ldm start-domain t4-g1
LDom t4-g1 started

Observe boot process.  During the below boot process we can see a couple of things.  Most importantly we see that the boot fails, with a “Can’t open boot device” this is expected as we are attempting to boot from disk0 which has not been installed yet.

SPARC T4-1, No Keyboard
Copyright (c) 1998, 2011, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.33.1, 4096 MB memory available, Serial #83496026.
Ethernet address 0:14:4f:fa:c:5a, Host ID: 84fa0c5a.

Boot device: disk0  File and args:
Bad magic number in disk label
ERROR: /virtual-devices@100/channel-devices@200/disk@0: Can't open disk label package

ERROR: boot-read fail

Evaluating:

Can't open boot device

{0} ok

We will be dumped to the ok prompt for the domain, from here we can simply select to boot from the iso that you defined earlier in the domain setup.  In my case I named the device s11.iso, so if you do a devalias you would see s11.iso as an alias to a device.

{0} ok boot s11.iso

Just a quick note, the escape character for telnet is listed as ‘^]’ in the beginning of the telnet session, this is CTRL + ] once you have returned the the telnet> prompt you can type quit.

Now in my environment I went through the install of Solaris 11, since it was the disk I had readily available, you can additionally use Soalris 10, or OpenSolaris if you so desire.