February 4th, 2013 | Tags: , , , , , ,

So I have been working with the OVM CLI lately to try and fill in some holes in my knowledge and provide for some easily reproducible procedures, usually around consistency of the environment when performing frequent tear downs and upgrades.  So expect to see more OVM related scripts going forward.

Today we are going over a fairly simple script.  I like to use all lower case in my VM naming conventions.  So I wrote up a little script to check every name in the environment and then rename it if it is not already lower case.  Then once I completed this I decided to change it to be a toggle, so that those of you who prefer upper case would not feel left out.  Basically what we are doing, is simply performing a few calls to the ovmcli to collect the information necessary to make our changes.  Then we perform the changes via a separate call.  Prior to making the edit call, we compare the actual value with the requested value in order to see if it is already what we wanted.

I have built a few parameters into the script, we can select the case, upper or lower, we can also select the OVM 3.x Manager Server, and the username and port.  The username and port are built with default values of admin and 10000.

Please keep in mind that this script will make multiple SSH connections to the ovmcli, as such you will want to use keys to streamline this authentication.  I have an article covering that here.

Name       : changecase-vmname.sh
Version   :  1.0.1
MD5        :  5006c9ff20b3301e66ac503f417b98ee
SHA256  :  bfff98b74c54f1fec3ec540a22ca4b1377c9f4b25e0b90ddbb722eeb64bf6bc8
URL         :  http://source.allanglesit.net/pub/changecase-vmname.sh

#!/bin/bash
# chkconfig:
# description:
#
#: Script Name    : changecase-vmname.sh
#: Version    : 1.0
#: Author    : Matthew Mattoon - http://blog.allanglesit.com
#: Date Created    : January 26, 2013
#: Date Updated    : February 20, 2013
#: Description    : Changes case of all Oracle VM 3.x VMs to either upper or lower case.
#: Examples    : changecase-vmname.sh -c CASE -m OVMMANAGER -p PORT -u USER
#:         : changecase-vmname.sh -c lower -m ovmserver.localdomain

usage()
{
cat << EOF
usage: $0 options

This script allows you to change case on all OVM 3.x VMs to either uppercase or lowercase.

OPTIONS:
-h    Show this message
-c    The preferred case for all VMs (required). upper or lower
-m    OVM 3.x Manager Server (required).
-p    OVM 3.x Manager Port (default: 10000).
-u    OVM 3.x Manager User (default: admin).
EOF
}

while getopts "hc:m:pu" OPTION
do
case $OPTION in
h) usage; exit 1;;
c) case=$OPTARG;;
m) ovmmgr=$OPTARG;;
p) ovmport=$OPTARG;;
u) ovmuser=$OPTARG;;
?) usage; exit 1;;
esac
done

if [[ -z $case ]] || [[ -z $ovmmgr ]]
then
usage
exit 1
fi

if [ $case != "lower" -a $case != "l" -a $case != "upper" -a $case != "u" ]
then
usage
exit 1
fi

if [[ -z $ovmuser ]]
then
ovmuser=admin
fi

if [[ -z $ovmport ]]
then
ovmport=10000
fi

if [ $case = "lower" -o $case = "l" ]
then
trcase="'[:upper:]' '[:lower:]'"
fi

if [ $case = "upper" -o $case = "u" ]
then
trcase="'[:lower:]' '[:upper:]'"
fi

for i in `ssh -p $ovmport $ovmuser@$ovmmgr "list vm" | grep -v 'OVM>\|Command:\|Status:\|Time:\|Data:' | sed 's/  /:/g'`
do
id=`echo $i | cut -d ":" -f 3`
echo -n "Examining $id...  "
vmname=`ssh -p $ovmport $ovmuser@$ovmmgr "show vm id=$id" | grep -v 'OVM>\|Command:\|Status:\|Time:\|Data:' | grep "Name" | sed 's/ = /=/' | cut -d = -f 2`
newvmname=`echo $vmname | tr $trcase`
if [ $vmname != $newvmname ]
then
echo "Renaming $vmname to $newvmname"
ssh -p $ovmport $ovmuser@$ovmmgr edit vm id=$id name="$newvmname" | grep "Status:"
else
echo "Rename not required"
fi
done

In Part One we went over the basics of sudo, what it is, why we use it, and how it is used properly.  In this article we are going to take it a step further and look at specific use cases for sudo.  The key thing to remember though, is that you have two ways you can use sudo, 1) to allow for program execution as root, a white-list  2) to restrict program execution as root, a black-list.  If you are doing the later there is no definitive way to restrict a program execution, so you should only use the latter with administrators who are trusted completely (the same users who would have access to the root password).

Allow A Specific Command Execution as Root (The Real Sudo)

This first policy example is what sudo is for.  This is where you want individuals to perform a specific action as root (for example restart a service which is known to be flaky instead of calling you at 2AM).  These commands can be specific down to parameters and everything.  Of course once the sudo policy is in place you could even go a step further and create aliases in their user profile so that they can execute simpler commands.  For example “/etc/init.d/network restart” can be aliased to “internet-reboot” or something equally simple/ridiculous.

# cat /etc/sudoers
# Specific Command Execution
Defaults logfile=/var/log/sudo.log
User_Alias ROOTCOMMAND = user1, user2
Cmnd_Alias REBOOT = /sbin/reboot
Cmnd_Alias SERVICE = /etc/init.d/network restart
ROOTCOMMAND ALL=NOPASSWD: REBOOT, SERVICE

One other small thing I did in this example.  Since this has a VERY restrictive policy and we are only allowing to possible commands to be run I have removed the password requirement. I do not allow this on a wider policy.  This comes back to the saying:

Quick, Easy, Secure – pick two.

Allow Root Access with Some Disallowances

This policy is a much safer policy for individuals who are not fully trusted, for example Junior Sys Admins.  This policy closes most of the holes which would allow them to inadvertently or intentionally lock out other individuals from the machine, the key difference between this and the above policy is that it requires all commands be executed via sudo, instead of being able to spawn a shell as root, and bypass the logging.

# cat /etc/sudoers
# Root with Restrictions
Defaults logfile=/var/log/sudo.log
User_Alias ROOTRESTRICTED = user1, user2
Cmnd_Alias SHELLS = /bin/sh, /bin/bash, /bin/ash, /bin/bsh, /bin/tcsh, /bin/csh, /bin/ksh, /bin/ksh93
Cmnd_Alias SU = /bin/su, /usr/bin/vncserver
Cmnd_Alias PASSWD = /bin/passwd, /usr/sbin/useradd, /usr/sbin/userdel, /usr/sbin/usermod
Cmnd_Alias VISUDO = /usr/sbin/visudo
ROOTRESTRICTED ALL = ALL, !SHELLS, !SU, !PASSWD, !VISUDO

Allow Unrestricted Root Access

This is the basics of a policy that I use on my workstations.  This also fits very well on servers where you have a requirement to not login to the root account except in an emergency.

This policy has NO restrictions on it.  As such the only people whom you will want to grant this policy to are the same folks you would provide your root password to.  This policy also allows the direct invocation of a shell as root, which gives you a root shell, which after that nothing is logged in the sudo.log.

# cat /etc/sudoers
# Root Equivalent
Defaults logfile=/var/log/sudo.log
User_Alias ROOTEQUIVALENT = user1, user2
ROOTRESTRICTED ALL = ALL

Switch Users Using Sudo

Here is another little tidbit I like to use on systems which require multiple system accounts to run multiple applications.  Instead of having to track and know each user and password combination (or having someone else have to do the same) I like to enable users to use sudo to invoke su to switch to the specific user.  This of course means that they will not be prompted to enter a password for the user that they are logging into but they will be logging that entry into sudo.

# cat /etc/sudoers
# Switch User Without Additional Password Management
Defaults logfile=/var/log/sudo.log
User_Alias TSTEBSUSER = bob, bill
Cmnd_Alias SUTSTEBS = /bin/su tstebs, /bin/su - tstebs
TSTEBSUSER ALL = SUTSTEBS
User_Alias TSTWLSUSER = bob, mark, jon
Cmnd_Alias SUTSTWLS = /bin/su tstebs, /bin/su - tstebs
TSTWLSUSER ALL = SUTSTWLS

Above we can see that we have two environments to be managed our TSTWLS and our TSTEBS environment.  Each of these environments is executed and managed using the tstwls and tstebs users respectively.  We can also see that according to the policy we have a user named bob with access to both environments.  The TSTEBS environment also has a user named bill who has access to the environment.  While on TSTWLS we additionally have mark, and jon who have access to the environment.

These are the most common use cases I have run into for using sudo.  There are many more policy restrictions (noexec – disallow the execution of scripts) which are detailed in the man pages for sudo.

What is Sudo?

Prior to sudo whenever you wanted to run a command as a specific user then you would su (switch user) to gain access to a shell for that user.  This of course required that you knew the password for that user, and it would spawn a new shell after successful authentication.  The major drawback to this was that in order to allow a user to do something, you had to allow the user to do anything.  This sort of configuration also makes it fairly trivial for an authorized user to perform unauthorized actions, such as changing the root password and locking everyone out.

Basically sudo is a setuid binary which allows any user to invoke it as root.  Here is where it starts to get a bit tricky.  Any user can invoke sudo as root.  However that doesn’t mean that any user can invoke anything with sudo as root.  Sudo itself is invoked as root, and then the policy processing begins and sudo determines if it is going to do what you are requesting (if it is in policy).

Remember Su-Do

If you remember nothing else about sudo (as a user) then remember this.  Sudo allows you to “su and do” without having to “su and do”.  This of course means that the below actions are identical in terms of executing the action.

$ su -
Password:
# whoami
root
$ sudo whoami
[sudo] password for oracle:
root

The only real difference in the above is the password that we enter.  In the first example we need to enter the root password.  In the second we enter the password of the user we are logged in as.

Sudo Usage

A great example to illustrate the functionality of sudo is the simple “id” this command simply lists your security contexts.  When we execute id

$ id
uid=1000(matthew) gid=1000(matthew) groups=1000(matthew),10(wheel),18(dialout),1001(vboxusers)

Above we can see that we are logged in with the user matthew with the uid of 1000.  You also see the groups that I am a member of.

$ sudo id
[sudo] password for matthew:
uid=0(root) gid=0(root) groups=0(root)

Now when we insert sudo in front of the id command we notice that it prompts for a password, specifically the password of the user invoking sudo (in this case matthew).  After successful authentication then we see that we actually get the id information for the root user.

There is another major way that sudo can be used.  And that is to achieve a root shell.

$ sudo -s
[sudo] password for matthew:
#

Above we see that we now have a persistent root shell.  The big takeaway from this usage is that you lose all of your sudo logging.  The only thing that is logged is the execution of the shell (this will be the users shell defined in the /etc/passwd – usually /bin/bash).

# more /var/log/sudo.log
Jan 22 16:04:09 : matthew : TTY=pts/8 ; PWD=/home/matthew ; USER=root ;
COMMAND=/bin/bash

Sudo Logging

One of the biggest benefits of sudo is the ability to log what individuals are doing on a system, and thus having the ability to forensically resolve problems based on the actions that were taken just prior to the problem.  Here is an example of what the output looks like.

# more /var/log/sudo.log
Jan 22 13:25:13 : user1 : TTY=pts/0 ; PWD=/home/user1 ; USER=root ;
COMMAND=/etc/init.d/network restart
Jan 22 13:25:34 : user1 : command not allowed ; TTY=pts/0 ; PWD=/home/user1 ;
USER=root ; COMMAND=/etc/init.d/network status

Now we can see that we have a successful execution of /etc/init.d/network restart on line 1.  This took place at Jan 22 13:25:13 local time.  We also know that it was invoked by the oracle user, while the oracle user was standing in /home/oracle (this is important for anything executed using a relative path).  We also then see the user that the command is executed as (which in this case is root).

The second entry we have is an attempt by the oracle user to execute /etc/init.d/network status.  This was denied because it is outside the allowance of the sudo policy.

List Effective Sudo Policy

You won’t always be in an environment where you will be able to push for changes to your sudo policies, however it doesn’t mean that you can’t look at the effective ruleset to see what you have in effect.  To list the policies as they are in effect against your user.

$ sudo -l
Matching Defaults entries for matthew on this host:
requiretty, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR
LS_COLORS", env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE",
env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES",
env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME
LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin, logfile=/var/log/sudo.log

User matthew may run the following commands on this host:
(ALL) ALL

If you are building policies and you want to test the effective policies for a specific user.  You can invoke it like so to list the effective policy.

# sudo -U matthew -l
Matching Defaults entries for matthew on this host:
requiretty, !visiblepw, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC
KDEDIR LS_COLORS MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE LC_COLLATE
LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER
LC_TELEPHONE LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
logfile=/var/log/sudo.log

Runas and Command-specific defaults for matthew:

User matthew may run the following commands on this host:
(root) ALL, (root) !/bin/sh, !/bin/bash, !/bin/bash2, !/bin/ash, !/bin/bsh,
!/bin/tcsh, !/bin/csh, !/bin/ksh, !/bin/ksh93, (root) !/bin/su, !/usr/bin/vncserver,
(root) !/bin/passwd, !/usr/sbin/useradd, !/usr/sbin/userdel, !/usr/sbin/usermod,
(root) !/usr/sbin/visudo

That wraps up Part One.  In Part Two we will look at specific policy examples and how they would be used.

January 29th, 2013 | Tags: , , , , ,

I have been spending some time learning the OVM 3 CLI since it was released in the earlier 3.x versions, however the biggest roadblock for me has always been that they only allow access via SSH on port 10000.  However now I know how to enable key based authentication for these connections opening this up for a much more robust scripting experience.

Connect to OVM CLI with Password

$ ssh -p 10000 admin@ovmmgr.allanglesit.com
admin@ovmmgr.allanglesit.com's password:
OVM>

Authorize Keys for OVM CLI

To allow key based authentication to OVM CLI simply add your public key on your OVM Manager Server to /home/oracle/.ssh/ovmcli_authorized_keys.  For our environment we already had everyone who needed access in the authorized_keys for root.

# cat /root/.ssh/authorized_keys >> /home/oracle/.ssh/ovmcli_authorized_keys

However if you wanted to do it remotely using the public key file directly from your client.

# cat .ssh/id_rsa.pub | ssh root@ovmmgr.allanglesit.com "cat >> /home/oracle/.ssh/ovmcli_authorized_keys"

Connect to OVM CLI with Public Keys

Now our connection is without that pesky password prompt.

$ ssh -p 10000 admin@ovmmgr.allanglesit.com
OVM>

One of my biggest complaints about Oracle VM 3 is that everything is abstracted at every level.  A VM name is simply meta data stored in a database (kind of) while everywhere that it matters the VM only exists as a UUID, lets look at some examples.

The Way It Is

From the hypervisor if we use native Xen commands to look at all of the instantiated VMs.

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
0004fb000006000002a00feb1f52cae4           155  8192     2     -b----  39176.1
0004fb00000600000b5b1f8673c451be           171  8192     2     -b---- 1007276.5
0004fb000006000026a40b6fb53fe2c3           135 16384     2     -b---- 169716.7
0004fb0000060000506f591d6877c284           194  3072     1     -b----  16108.4
0004fb00000600007052ad7b040749ea           176  8192     2     -b---- 1726118.2
0004fb0000060000a07b66923b0f68a4           138  4096     2     -b---- 2092394.0
0004fb0000060000abfcf710ebd640df           101  2048     2     -b---- 540488.4
0004fb0000060000bf16256b99159fc6           143 16384     2     -b---- 3972112.7
0004fb0000060000e3f234f6c1d26448           173  8192     2     -b---- 852019.6
0004fb0000060000e602b0e0969dba07            92  2048     2     -b---- 234244.0
0004fb0000060000f9205e11159e5459           175 12288     2     r----- 4145289.3
Domain-0                                     0  2486    24     r----- 4455022.6

Now since under the covers we are using Xen that means that somewhere there is a vm.cfg file (could be named anything, but for now they are sticking with vm.cfg).

# cat vm.cfg
vif = ['mac=00:21:f6:00:00:07,bridge=172.16.88.0']
OVM_simple_name = 'PRD-IM'
guest_os_type = 'linux'
disk = ['file:/OVS/Repositories/0004fb00000300005527ce584481a013/VirtualDisks/0004fb000012000052e7181e3458aba7.img,xvda,w']
vncunused = '1'
uuid = '0004fb00-0006-0000-506f-591d6877c284'
on_reboot = 'restart'
boot = 'cn'
cpu_weight = 27500
memory = 3072
cpu_cap = 0
maxvcpus = 2
OVM_high_availability = False
vnclisten = '127.0.0.1'
vnc = '1'
OVM_description = 'Keste IM - Prod'
on_poweroff = 'destroy'
on_crash = 'restart'
maxmem = 3072
name = '0004fb0000060000506f591d6877c284'
builder = 'hvm'
vcpus = 1
keymap = 'en-us'
OVM_os_type = 'Oracle Linux 6'
OVM_cpu_compat_group = ''
OVM_domain_type = 'xen_hvm'

But finding it can be tricky.  All repositories are stored in UUID form under /OVS/Respositories.  Then inside of each repo you have multiple folders, Assemblies, ISOs, Templates, VirtualDisks, VirtualMachines.  They are pretty self explanatory, but vm.cfgs go under VirtualMachines, they then refer to virtual disk images in the VirtualDisks folders.  Here is an example for the vm.cfg we looked at above.

# pwd
/OVS/Repositories/0004fb00000300005527ce584481a013/VirtualMachines/0004fb0000060000506f591d6877c284

This breaks down like this…

/OVS/Repositories/ [UUID of REPO] /VirtualMachines/ [UUID of VM] /vm.cfg

The Problem

If we lose the manager, then we lose all ability to manage the environment without heavy navigation of the file system, or if we discover a hypervisor which was previously managed by the same OVM Manager UUID then we end up with a bunch of Orphaned VMs.  You will notice that not only is there no meaningful names, there is also no meaningful information about the VMs, I assure that not all of my VMs have 256MB of RAM with a single CPU.

ovm3-orphaned-vms-001

Figure 1-1: Orphaned VMs

The Solution

Now the crux of this problem is that while we have discovered the Hypervisor we have not discovered the underlying repository.

 

ovm3-orphaned-vms-002

Figure 1-2: Our Server Pool and Hypervisor

So first thing we need to do is right-click on the hypervisor and “Rescan Physical Disks” this should complete fairly quickly.  This will allow OVM Manager to be aware of the Physical Disk on which we have our repository with our running VMs.  Disregard the warning icon, that is due to having an older version of the hypervisor on that machine (3.1.1)

ovm3-orphaned-vms-003

Figure 1-3: Rescan Physical Disks Job

Here we simply monitor the job until completion.  After it is complete we right-click on the hypervisor and “Rediscover Server” this allows the OVM Manager to look at the physical disk that it just found out about to see if there is a repository on it.

ovm3-orphaned-vms-004

Figure 1-4: Rediscover Server Job

Monitor until the job completes.

ovm3-orphaned-vms-006

Figure 1-5: New Repository Visible

Now we see a UUID for a new repository under the Repositories tab.  Right-click and Refresh.

ovm3-orphaned-vms-007

Figure 1-6: Repository Refresh Job

Monitor until the job completes.

ovm3-orphaned-vms-008

Figure 1-7: Fixed VMs

Now our VMs show the correct information.

A Few Notes

Any names that you assigned VirtualDisks are gone, they are back to UUID, sorry.  Also if you had any machines that are stopped when you perform this action you will not see them on the hypervisor.  They are in the Unassigned Virtual Machines folder.  Click and drag to re-assign them.  Or use the Migrate button.

Page 2 of 2312345...1020...Last »
TOP