September 24th, 2013 | Tags: , ,

ZFS gives us the ability to move data with ZFS send and receive.  This can be used to combine snapshots, create full or incremental backups, or replicate data between servers on your LAN or even over the WAN.

Local Backup with Send

This is very simple, simply perform a send of a snapshot then output it to the file, I have complicated it a bit to include a folder which is named off of the current date (as would be helpful with backups).

# zfs send tank/filesystem@now > /backup/`date +%m%d%Y`/tank-filesystem.zsnd

Keep in mind zfs send only makes the backups.  In order to restore them we will need to be familiar with zfs receive as well.

Local Restore with Receive

With the basic restore it is very easy, simply specify the file system to restore to, and then define the file to pull it from.

# zfs receive tank/restoredfilesystem < /backup/03152013/tank-filesystem.zsnd

Notice here we are doing redirection back into the ZFS receive command from the file, also important to note that

Local Copy of a File System – Full Clone

Occasionally you may have the need to perform a full copy clone.  With a normal zfs clone you will end up with a zero copy clone (copy-on-write), you will also have additional dependency when using zfs clones, which require a little extra caution.  If using zfs send and receive to make a copy, then there is no interdependency.

# zfs send tank/originalfilesystem@now | zfs receive tank/copyfilesystem

Here we are able to structure the command with  just a simple pipe to separate the send and receive statements.

Local Compressed Backup with Send

Perhaps you want to store your backups in a compressed form.  All we need to do is insert a gzip into the mix.

# zfs send tank/test@now | gzip > /backup/`date +%m%d%Y`/tank-originalfilesystem.zsnd.gz

This method combines a pipe with a redirect, and will give us a compressed file.

Local Compressed Restore with Receive

Of course backups don’t do any good for us if we cannot restore them.

# gunzip -c -d /backup/03152013/tank-originalfilesystem.zsnd.gz | zfs receive tank/copyfilesystem

Here we unzip the file, and pipe that into the zfs receive statement.

Local Encrypted (and Compressed) Backup with Send

Some workloads have serious security requirements.  This will encrypt and compress the contents of the file system into a backup file.  In this example I am using encrypt/decrypt to perform the encryption with aes, I also was unable to get the piping and redirection working without also injecting gzip as well.

# zfs send tank/test@now | encrypt -a aes | gzip > /backup/`date +%m%d%Y`/tank-originalfilesystem.zsnd.aes.gz
Enter passphrase:
Re-enter passphrase:

Since we aren’t using any keys with encrypt it will prompt us for a passphrase.  Remember when using encryption your data becomes worthless if you do not

Local Encrypted (and Compressed) Restore with Receive

Decrypting and restoring the backup file is pretty straight forward, unzip the file, pipe it through decrypt and pipe it into zfs.

# gunzip -c -d /backup/03152013/tank-originalfilesystem.zsnd.aes.gz | decrypt -a aes | zfs receive tank/copyfilesystem
Enter passphrase:

Use the passphrase that you used when encrypting the backup to decrypt it.

Remote Copy of a File System – Full Clone

Sometimes a full clone is needed from one server to another, perhaps for data migration.

# zfs send tank/originalfilesystem@now | ssh root@192.168.1.21 "zfs receive tank/copyfilesystem"
September 23rd, 2013 | Tags: , , , ,

In Solaris 11.1 Oracle decided to change the way that they stored filesystem share information.  I personally loved the change, but that was without understanding what the change really was.  For more information on how to enable NFS exports see my previous article here.

Setup Our Environment

Everything we are doing will be on some test file systems, with no data.  So lets put everything in place.

# zfs create tank/fs1
# zfs set share=name=tank_fs1,path=/tank/fs1,prot=nfs,sec=sys,rw=@10.0.0.15 tank/fs1
name=tank_fs1,path=/tank/fs1,prot=nfs,sec=sys,rw=@10.0.0.15

Now lets create our child filesystem, and export it the same way.

# zfs create tank/fs1/sub1
# zfs set share=name=tank_fs1_sub1,path=/tank/fs1/sub1,prot=nfs,sec=sys,rw=@10.0.0.15 tank/fs1/sub1
name=tank_fs1_sub1,path=/tank/fs1/sub1,prot=nfs,sec=sys,rw=@10.0.0.15

Now this will simulate a situation that I was in.  I had originally exported everything at a higher level (fs1) and then as the dataset grew it became necessary to export it at the child level (sub1) in order to have more granular control.  So now we have two exports, and only need sub1.

Disable Unused NFS Export

Since fs1 is no longer needed lets disable its export.

# zfs set share.nfs=off tank/fs1

Lets take a look at fs1 to see if our configuration is still active.

# zfs get share tank/fs1

OK it is empty.  But since I don’t need this any more I want to make sure that there are no remaining artifacts.  So lets re-enable it and confirm that the configuration is gone.

 # zfs set share.nfs=on tank/fs1
# zfs get share tank/fs1
NAME      PROPERTY  VALUE  SOURCE
tank/fs1  share     name=tank_fs1,path=/tank/fs1,prot=nfs,sec=sys,rw=@10.0.0.15  local

OK, so our configuration is still there, but it is disabled by share.nfs=off.  It must be kept elsewhere in the file system.

FYI, this is included in the ZFS metadata, so if you were to export you zpool and take it to another system, it would come over to the other system.

Explore ZFS Properties

Lets disable fs1 and start poking around.

# zfs set share.nfs=off tank/fs1

First thing to do is a get all.  This will list all of the properties of the filesystem.  I trimmed it back to only display the one that I found interesting.

# zfs get all tank/fs1
tank/fs1  share.*               ...                    local

OK so lets look a little bit closer at this share.* property.

# zfs get share.* tank/fs1
bad property list: invalid property specifier 'share.*', use 'share.all'
For more info, run: zfs help get

So share.* is not a valid property, it means that there are sub-properties, apparently share.all will display the sub-properties.

# zfs get share.all tank/fs1
NAME      PROPERTY         VALUE      SOURCE
tank/fs1  share.auto       off        local
tank/fs1  share.autoname              default
tank/fs1  share.desc                  default
tank/fs1  share.nfs        off        local
tank/fs1  share.nfs.*      ...        default
tank/fs1  share.point      /tank/fs1  default
tank/fs1  share.protocols             -
tank/fs1  share.smb        off        default
tank/fs1  share.smb.*      ...        default

Oh boy, now we are getting somewhere, here is our share.nfs, and we can also see that share.nfs has sub-properties as well.  Also notice that smb is also stored here in the same fashion.

# zfs get share.nfs.all tank/fs1
NAME      PROPERTY             VALUE  SOURCE
tank/fs1  share.nfs.aclok      off    default
tank/fs1  share.nfs.anon              default
tank/fs1  share.nfs.charset.*  ...    default
tank/fs1  share.nfs.cksum             default
tank/fs1  share.nfs.index             default
tank/fs1  share.nfs.log               default
tank/fs1  share.nfs.noaclfab   off    default
tank/fs1  share.nfs.nosub      off    default
tank/fs1  share.nfs.nosuid     off    default
tank/fs1  share.nfs.sec               default
tank/fs1  share.nfs.sec.*      ...    default

Under share.nfs.all we have a lot of NFS parameters, the most important at this point is share.nfs.sec.all, so lets look there.

# zfs get share.nfs.sec.all tank/fs1
NAME      PROPERTY                 VALUE  SOURCE
tank/fs1  share.nfs.sec.default.*  ...    default
tank/fs1  share.nfs.sec.dh.*       ...    default
tank/fs1  share.nfs.sec.krb5.*     ...    default
tank/fs1  share.nfs.sec.krb5i.*    ...    default
tank/fs1  share.nfs.sec.krb5p.*    ...    default
tank/fs1  share.nfs.sec.none.*     ...    default
tank/fs1  share.nfs.sec.sys.*      ...    default

We are getting warmer.  Now we have the share.nfs.sec.sys.all to take a look at.

# zfs get share.nfs.sec.sys.all tank/fs1
NAME      PROPERTY                        VALUE  SOURCE
tank/fs1  share.nfs.sec.sys.none                 default
tank/fs1  share.nfs.sec.sys.ro                   default
tank/fs1  share.nfs.sec.sys.root                 default
tank/fs1  share.nfs.sec.sys.root_mapping         default
tank/fs1  share.nfs.sec.sys.rw                   default

OK well we have run out of levels to go deeper, but I still don’t see anything.  Perhaps if we call the property by name?

# zfs get share.nfs.sec.sys.rw tank/fs1
NAME      PROPERTY              VALUE  SOURCE
tank/fs1  share.nfs.sec.sys.rw         default

So that is empty as well.  Perhaps that has something to do with the export being disabled?  Lets confirm our export is disabled.

# zfs get share.nfs tank/fs1
NAME      PROPERTY   VALUE  SOURCE
tank/fs1  share.nfs  off    local

Doesn’t get any more disabled than that. Well lets check the sub1 file system.

# zfs get share.nfs tank/fs1/sub1
NAME           PROPERTY   VALUE  SOURCE
tank/fs1/sub1  share.nfs  on     local

So we can see that it is enabled, lets check the share property to see what our whole string is.

# zfs get share tank/fs1/sub1
NAME           PROPERTY  VALUE  SOURCE
tank/fs1/sub1  share     name=tank_fs1_sub1,path=/tank/fs1/sub1,prot=nfs,sec=sys,rw=@10.0.0.15  local

Based on that string we should have share.nfs.sec.sys.rw = 10.0.0.15.

# zfs get share.nfs.sec.sys.rw tank/fs1/sub1
NAME           PROPERTY              VALUE  SOURCE
tank/fs1/sub1  share.nfs.sec.sys.rw         default

OK well that was unexpected.  Property is empty on the enabled export file system as well.  We must be missing something.

Back to the Drawing Board

Out of pure desperation and a fair bit of luck.  I was attempting to lookup the help command on zfs get, and accidentally typed list instead.  But look at what I found.

# zfs help list
usage:
list [-rH][-d max] [-o property[,...]] [-t type[,...]] [-s property] ...
[-S property] ... [filesystem|volume|snapshot|share] ...

There is a share file system type.  Certainly that must have something to do with everything I am seeing?

# zfs list -t share
NAME                                                                   USED  AVAIL  REFER  MOUNTPOINT
tank/fs1%tank_fs1                                                         -      -      -  /tank/fs1
tank/fs1/sub1%tank_fs1_sub1                                               -      -      -  /tank/fs1/sub1

Clearly we are onto something here.  We have a share file system type for both our enabled and our disabled NFS export.  Lets see what kind of properties these new file system types have.

ZFS File System Share Type

# zfs get all tank/fs1%tank_fs1
NAME               PROPERTY    VALUE                  SOURCE
tank/fs1%tank_fs1  creation    Sat Sep 21 10:09 2013  -
tank/fs1%tank_fs1  mountpoint  /tank/fs1              -
tank/fs1%tank_fs1  share.*     ...                    local
tank/fs1%tank_fs1  zoned       off                    default

Well there is a share.* so that must be what we are looking for.  Lets try our previous command against this new file system and take a look at what we get.

# zfs get share.nfs.sec.sys.all tank/fs1%tank_fs1
NAME               PROPERTY                        VALUE       SOURCE
tank/fs1%tank_fs1  share.nfs.sec.sys.none                      default
tank/fs1%tank_fs1  share.nfs.sec.sys.ro                        default
tank/fs1%tank_fs1  share.nfs.sec.sys.root                      default
tank/fs1%tank_fs1  share.nfs.sec.sys.root_mapping              default
tank/fs1%tank_fs1  share.nfs.sec.sys.rw            @10.0.0.15  local

Well there is that elusive read write statement.

Updating NFS Security Statements

Now lets see what happens when we change it.

# zfs set share.nfs.sec.sys.rw=@10.0.0.15:@10.0.0.16 tank/fs1%tank_fs1

Here we have added a second IP address to be allowed to read and write, on our disabled NFS export.

# zfs get share.nfs.sec.sys.all tank/fs1%tank_fs1
NAME               PROPERTY                        VALUE                  SOURCE
tank/fs1%tank_fs1  share.nfs.sec.sys.none                                 default
tank/fs1%tank_fs1  share.nfs.sec.sys.ro                                   default
tank/fs1%tank_fs1  share.nfs.sec.sys.root                                 default
tank/fs1%tank_fs1  share.nfs.sec.sys.root_mapping                         default
tank/fs1%tank_fs1  share.nfs.sec.sys.rw            @10.0.0.15:@10.0.0.16  local

The property is successfully updated on the share file system type.  Lets see if it also enabled the NFS export.

# zfs get share tank/fs1

Still disabled, that is perfect.  Now lets see what happens on an enabled NFS export.

# zfs set share.nfs.sec.sys.ro=@10.0.0.15:@10.0.0.16 tank/fs1/sub1%tank_fs1_sub1

Now we have added 2 new IPs to have read only access to this export.  Lets check its effect.

# zfs get share.nfs.sec.sys.all tank/fs1/sub1%tank_fs1_sub1
NAME                         PROPERTY                        VALUE                  SOURCE
tank/fs1/sub1%tank_fs1_sub1  share.nfs.sec.sys.none                                 default
tank/fs1/sub1%tank_fs1_sub1  share.nfs.sec.sys.ro            @10.0.0.15:@10.0.0.16  local
tank/fs1/sub1%tank_fs1_sub1  share.nfs.sec.sys.root                                 default
tank/fs1/sub1%tank_fs1_sub1  share.nfs.sec.sys.root_mapping                         default
tank/fs1/sub1%tank_fs1_sub1  share.nfs.sec.sys.rw            @10.0.0.15             local

The properties were successfully updated.  Lets check the file system to see if the share properties were brought over.

# zfs get share tank/fs1/sub1
NAME           PROPERTY  VALUE  SOURCE
tank/fs1/sub1  share     name=tank_fs1_sub1,path=/tank/fs1/sub1,prot=nfs,sec=sys,ro=@10.0.0.15:@10.0.0.16,rw=@10.0.0.15  local

That looks perfect, and lets ask NFS what it knows.

# showmount -e
export list for server1:
/tank/fs1/sub1                      @10.0.0.15,@10.0.0.16,@10.0.0.15

Wonderful!  That is exactly what we wanted.  Now one final thing.

Removing NFS Export Configuration Permanently

Now since there is this whole different type of file system to store the share configuration, removing it becomes incredibly easy.

# zfs destroy tank/fs1%tank_fs1

That pretty much wraps up everything.

September 10th, 2013 | Tags: , , ,

In this article we are going to go over some basics of using IPMP in Solaris.  We will discuss this on the latest versions of Solaris 10 and 11 on the SPARC platform, these instructions should also work on the x86 platform, but will be dependent on the listed version numbers, as the networking stack has changed alot of the past few minor versions of Solaris 10 and 11.

Solaris 11.1

Create an IPMP Group

The first step is to create an IPMP group, in this case named ipmp1.

solaris11~# ipadm create-ipmp ipmp1

Add Interfaces to an IPMP Group

Once the group is created we can add interfaces to it, in this case net1 and net2.

solaris11~# ipadm add-ipmp -i net1 -i net2 ipmp1

Create an IPMP Group and Add Interfaces

We can also combine the above steps so that we create the group and add the interfaces in one step.

solaris11~# ipadm create-ipmp -i net1 -i net2 ipmp1

Set Interface as Standby in an IPMP Group

In the event that you prefer to have a standby interface you can change an interface to be a standby, by default all interfaces are active.

solaris11~# ipadm set-ifprop -p standby=yes net2

Remove Interfaces from an IPMP Group

Prior to deleting an IPMP group you must remove all interfaces first.

solaris11~# ipadm remove-ipmp -i net1 -i net2 ipmp1

Delete an IPMP Group

Now we can delete the IPMP group.

solaris11~# ipadm delete-ipmp

Assign an Address

After we created everything then we can assign an address.  We will need an address for the IPMP group interface, in addition if we want to use probe-based IPMP then we will need to assign them on the underlying interfaces as well, otherwise you will be using the link-based version of IPMP.

Here is an example for probe-based IPMP.

solaris11~# ipadm create-addr -T static -a 192.168.1.1/24 ipmp1/v4
solaris11~# ipadm create-addr -T static -a 192.168.1.2/24 net1/v4
solaris11~# ipadm create-addr -T static -a 192.168.1.3/24 net2/v4

Here is an example for link-based IPMP.

solaris11~# ipadm create-addr -T static -a 192.168.1.1/24 ipmp1/v4

Multipath Configuration File

If you want to be able to control some of the specifics of multipath behavior, like failback for example, you can do so using the mpathd configuration file.

solaris11~# grep -v "^#" /etc/default/mpathd
FAILURE_DETECTION_TIME=10000
FAILBACK=yes
TRACK_INTERFACES_ONLY_WITH_GROUPS=yes

That completes our configuration on Solaris 11.1.

Solaris 10 1/13

The Solaris 10 version of these instructions are very similar to the way you would configure an IP on a normal interface, but with some additional parameters.

Interface Configuration Files (Active-Active) Link-based IPMP

In Solaris 10, we control everything with the interface configration files, named hostname.interfacename where the interface name is net1 and net2 in this environment, please change your link names or configuration file names to match your server.

solaris10~# cat /etc/hostname.net1
machinehostname netmask + broadcast + group ipmp1 up
solaris10~# cat /etc/hostname.net2
group ipmp1 up

Interface Configuration Files (Active-Active) Probe-based IPMP

solaris10~# cat /etc/hostname.net1
machinehostname-net1 depracated -failover netmask + broadcast + group ipmp1 up addif machinehostname netmask + broadcast + up
solaris10~# cat /etc/hostname.net2
machinehostname-net2 depracated -failover netmask + broadcast + group ipmp1 up

Interface Configuration Files (Active-Standby) Link-based IPMP

The only time that I see a value in an Active-Standby configuration is when you have to guarantee a level of service, even in a failed state.  In my environments the increased performance is of a much higher value than the non-reduced workload.

solaris10~# cat /etc/hostname.net1
machinehostname netmask + broadcast + group ipmp1 up
solaris10~# cat /etc/hostname.net2
group ipmp1 standby up

Interface Configuration Files (Active-Standby) Probe-based IPMP

solaris10~# cat /etc/hostname.net1
machinehostname-net1 depracated -failover netmask + broadcast + group ipmp1 up addif machinehostname netmask + broadcast + up
solaris10~# cat /etc/hostname.net2
machinehostname-net2 depracated -failover netmask + broadcast + group ipmp1 standby up

Hosts File

solaris10~# grep machinehostname /etc/hosts
192.168.1.1      machinehostname.localdomain machinehostname
192.168.1.2      machinehostname-net1.localdomain machinehostname-net1 # only needed for probe-based ipmp
192.168.1.3      machinehostname-net2.localdomain machinehostname-net2 # only needed for probe-based ipmp

Netmasks Configuration File

solaris10~# grep 192.168.1.0 /etc/netmasks
192.168.1.0       255.255.255.0

Multipath Configuration File

solaris10~# grep -v "^#" /etc/default/mpathd
FAILURE_DETECTION_TIME=10000
FAILBACK=yes
TRACK_INTERFACES_ONLY_WITH_GROUPS=yes
September 9th, 2013 | Tags: , , ,

So with ZFS you really have two ways of configuring your NFS exports, you can do it the old fashioned way (read: /etc/dfs/dfstab and share command).  Or the super-duper easy method…  Use ZFS metadata, and allow ZFS to configure the export for you.  The reason why you might choose this approach is simple, if you don’t want to have to worry about configs, and especially if you need this export to be tied to the filesystem regardless of where you put the file system from system to system.  Pause for effect.  Your NFS exports can be TIED TO THE VOLUME AND NOT THE SERVER.  So if you have a USB drive that you are using to move data on, it can immediately share the data on another server when the pool is imported, same is true for LUNs.  This adds a lot of flexibility.

Read-Write for Everyone

solaris11~# zfs set share.nfs=on zpool/fs1
solaris11~# zfs set share=name=fs1,path=/zpool/fs1,prot=nfs zpool/fs1

 

solaris10~# zfs set sharenfs=on zpool/fs1

Read-Write for a Single IP

solaris11~# zfs set share.nfs=on zpool/fs1
solaris11~# zfs set share=name=fs1,path=/zpool/fs1,prot=nfs,sec=sys,rw=@192.168.1.1 zpool/fs1

 

solaris10~# zfs set sharenfs=sec=sys,rw=@192.168.1.1/24 zpool/fs1

Read-Write and Root for Multiple IPs

solaris11~# zfs set share.nfs=on zpool/fs1
solaris11~# zfs set share=name=fs1,path=/zpool/fs1,prot=nfs,sec=sys,rw=@192.168.1.1:@192.168.1.2,root=@192.168.1.1:@192.168.1.2 zpool/fs1

 

solaris10~# zfs set sharenfs=sec=sys,rw=@192.168.1.1/24:@192.168.1.2/24,root=@192.168.1.1/24:@192.168.1.2/24 zpool/fs1

That pretty much takes care of it.  Feel free to mix and match.  One thing to keep in mind.  If you specify it the old way and the new way, then the ZFS settings will take precedence.

August 27th, 2013 | Tags: ,

Generally I don’t use my blog as a platform to find potential applicants for my employer, however at this point we are looking very seriously at some new candidates to fill out our Systems Team.

This is a job that is very much “Systems” focused.  At Keste we define Systems as everything from the hardware up to the applications, this includes but is not limited to Servers, Storage, Networking and Virtualization.  Because this is a very broad area, we expect folks to have expertise in multiple areas and most importantly to have the capabilities to learn and broaden their knowledge where they have weaknesses.

The other thing you must know about this job is that it is a consultant position, this means that you will need to travel (we say 50% – however it has been lower than that in my experience), and additionally you will need to be comfortable presenting in front of customers at all levels (technical, business, users, etc).  So if you have poor social skills then you would have to have some pretty phenomenal skills which will raise the team around you to support your weaknesses, but that wouldn’t necessarily be a deal breaker.

So if you think that you are a fit for the role please check the link below for the most up to date job posting, and apply there.

https://keste-openhire.silkroad.com/epostings/index.cfm?fuseaction=app.jobinfo&jobid=316&source=ONLINE&JobOwner=992276&company_id=16374&version=1&byBusinessUnit=NULL&bycountry=0&bystate=0&byRegion=&bylocation=&keywords=systems%20engineer&byCat=&proximityCountry=&postalCode=&radiusDistance=&isKilometers=&tosearch=yes

Page 1 of 2512345...1020...Last »
TOP