September 23rd, 2013 | Tags: , , , ,

In Solaris 11.1 Oracle decided to change the way that they stored filesystem share information.  I personally loved the change, but that was without understanding what the change really was.  For more information on how to enable NFS exports see my previous article here.

Setup Our Environment

Everything we are doing will be on some test file systems, with no data.  So lets put everything in place.

# zfs create tank/fs1
# zfs set share=name=tank_fs1,path=/tank/fs1,prot=nfs,sec=sys,rw=@ tank/fs1

Now lets create our child filesystem, and export it the same way.

# zfs create tank/fs1/sub1
# zfs set share=name=tank_fs1_sub1,path=/tank/fs1/sub1,prot=nfs,sec=sys,rw=@ tank/fs1/sub1

Now this will simulate a situation that I was in.  I had originally exported everything at a higher level (fs1) and then as the dataset grew it became necessary to export it at the child level (sub1) in order to have more granular control.  So now we have two exports, and only need sub1.

Disable Unused NFS Export

Since fs1 is no longer needed lets disable its export.

# zfs set share.nfs=off tank/fs1

Lets take a look at fs1 to see if our configuration is still active.

# zfs get share tank/fs1

OK it is empty.  But since I don’t need this any more I want to make sure that there are no remaining artifacts.  So lets re-enable it and confirm that the configuration is gone.

 # zfs set share.nfs=on tank/fs1
# zfs get share tank/fs1
tank/fs1  share     name=tank_fs1,path=/tank/fs1,prot=nfs,sec=sys,rw=@  local

OK, so our configuration is still there, but it is disabled by share.nfs=off.  It must be kept elsewhere in the file system.

FYI, this is included in the ZFS metadata, so if you were to export you zpool and take it to another system, it would come over to the other system.

Explore ZFS Properties

Lets disable fs1 and start poking around.

# zfs set share.nfs=off tank/fs1

First thing to do is a get all.  This will list all of the properties of the filesystem.  I trimmed it back to only display the one that I found interesting.

# zfs get all tank/fs1
tank/fs1  share.*               ...                    local

OK so lets look a little bit closer at this share.* property.

# zfs get share.* tank/fs1
bad property list: invalid property specifier 'share.*', use 'share.all'
For more info, run: zfs help get

So share.* is not a valid property, it means that there are sub-properties, apparently share.all will display the sub-properties.

# zfs get share.all tank/fs1
tank/fs1       off        local
tank/fs1  share.autoname              default
tank/fs1  share.desc                  default
tank/fs1  share.nfs        off        local
tank/fs1  share.nfs.*      ...        default
tank/fs1  share.point      /tank/fs1  default
tank/fs1  share.protocols             -
tank/fs1  share.smb        off        default
tank/fs1  share.smb.*      ...        default

Oh boy, now we are getting somewhere, here is our share.nfs, and we can also see that share.nfs has sub-properties as well.  Also notice that smb is also stored here in the same fashion.

# zfs get share.nfs.all tank/fs1
tank/fs1  share.nfs.aclok      off    default
tank/fs1  share.nfs.anon              default
tank/fs1  share.nfs.charset.*  ...    default
tank/fs1  share.nfs.cksum             default
tank/fs1  share.nfs.index             default
tank/fs1  share.nfs.log               default
tank/fs1  share.nfs.noaclfab   off    default
tank/fs1  share.nfs.nosub      off    default
tank/fs1  share.nfs.nosuid     off    default
tank/fs1  share.nfs.sec               default
tank/fs1  share.nfs.sec.*      ...    default

Under share.nfs.all we have a lot of NFS parameters, the most important at this point is share.nfs.sec.all, so lets look there.

# zfs get share.nfs.sec.all tank/fs1
NAME      PROPERTY                 VALUE  SOURCE
tank/fs1  share.nfs.sec.default.*  ...    default
tank/fs1  share.nfs.sec.dh.*       ...    default
tank/fs1  share.nfs.sec.krb5.*     ...    default
tank/fs1  share.nfs.sec.krb5i.*    ...    default
tank/fs1  share.nfs.sec.krb5p.*    ...    default
tank/fs1  share.nfs.sec.none.*     ...    default
tank/fs1  share.nfs.sec.sys.*      ...    default

We are getting warmer.  Now we have the share.nfs.sec.sys.all to take a look at.

# zfs get share.nfs.sec.sys.all tank/fs1
NAME      PROPERTY                        VALUE  SOURCE
tank/fs1  share.nfs.sec.sys.none                 default
tank/fs1                   default
tank/fs1  share.nfs.sec.sys.root                 default
tank/fs1  share.nfs.sec.sys.root_mapping         default
tank/fs1                   default

OK well we have run out of levels to go deeper, but I still don’t see anything.  Perhaps if we call the property by name?

# zfs get tank/fs1
tank/fs1         default

So that is empty as well.  Perhaps that has something to do with the export being disabled?  Lets confirm our export is disabled.

# zfs get share.nfs tank/fs1
tank/fs1  share.nfs  off    local

Doesn’t get any more disabled than that. Well lets check the sub1 file system.

# zfs get share.nfs tank/fs1/sub1
tank/fs1/sub1  share.nfs  on     local

So we can see that it is enabled, lets check the share property to see what our whole string is.

# zfs get share tank/fs1/sub1
tank/fs1/sub1  share     name=tank_fs1_sub1,path=/tank/fs1/sub1,prot=nfs,sec=sys,rw=@  local

Based on that string we should have =

# zfs get tank/fs1/sub1
NAME           PROPERTY              VALUE  SOURCE
tank/fs1/sub1         default

OK well that was unexpected.  Property is empty on the enabled export file system as well.  We must be missing something.

Back to the Drawing Board

Out of pure desperation and a fair bit of luck.  I was attempting to lookup the help command on zfs get, and accidentally typed list instead.  But look at what I found.

# zfs help list
list [-rH][-d max] [-o property[,...]] [-t type[,...]] [-s property] ...
[-S property] ... [filesystem|volume|snapshot|share] ...

There is a share file system type.  Certainly that must have something to do with everything I am seeing?

# zfs list -t share
NAME                                                                   USED  AVAIL  REFER  MOUNTPOINT
tank/fs1%tank_fs1                                                         -      -      -  /tank/fs1
tank/fs1/sub1%tank_fs1_sub1                                               -      -      -  /tank/fs1/sub1

Clearly we are onto something here.  We have a share file system type for both our enabled and our disabled NFS export.  Lets see what kind of properties these new file system types have.

ZFS File System Share Type

# zfs get all tank/fs1%tank_fs1
NAME               PROPERTY    VALUE                  SOURCE
tank/fs1%tank_fs1  creation    Sat Sep 21 10:09 2013  -
tank/fs1%tank_fs1  mountpoint  /tank/fs1              -
tank/fs1%tank_fs1  share.*     ...                    local
tank/fs1%tank_fs1  zoned       off                    default

Well there is a share.* so that must be what we are looking for.  Lets try our previous command against this new file system and take a look at what we get.

# zfs get share.nfs.sec.sys.all tank/fs1%tank_fs1
NAME               PROPERTY                        VALUE       SOURCE
tank/fs1%tank_fs1  share.nfs.sec.sys.none                      default
tank/fs1%tank_fs1                        default
tank/fs1%tank_fs1  share.nfs.sec.sys.root                      default
tank/fs1%tank_fs1  share.nfs.sec.sys.root_mapping              default
tank/fs1%tank_fs1            @  local

Well there is that elusive read write statement.

Updating NFS Security Statements

Now lets see what happens when we change it.

# zfs set tank/fs1%tank_fs1

Here we have added a second IP address to be allowed to read and write, on our disabled NFS export.

# zfs get share.nfs.sec.sys.all tank/fs1%tank_fs1
NAME               PROPERTY                        VALUE                  SOURCE
tank/fs1%tank_fs1  share.nfs.sec.sys.none                                 default
tank/fs1%tank_fs1                                   default
tank/fs1%tank_fs1  share.nfs.sec.sys.root                                 default
tank/fs1%tank_fs1  share.nfs.sec.sys.root_mapping                         default
tank/fs1%tank_fs1            @  local

The property is successfully updated on the share file system type.  Lets see if it also enabled the NFS export.

# zfs get share tank/fs1

Still disabled, that is perfect.  Now lets see what happens on an enabled NFS export.

# zfs set tank/fs1/sub1%tank_fs1_sub1

Now we have added 2 new IPs to have read only access to this export.  Lets check its effect.

# zfs get share.nfs.sec.sys.all tank/fs1/sub1%tank_fs1_sub1
NAME                         PROPERTY                        VALUE                  SOURCE
tank/fs1/sub1%tank_fs1_sub1  share.nfs.sec.sys.none                                 default
tank/fs1/sub1%tank_fs1_sub1            @  local
tank/fs1/sub1%tank_fs1_sub1  share.nfs.sec.sys.root                                 default
tank/fs1/sub1%tank_fs1_sub1  share.nfs.sec.sys.root_mapping                         default
tank/fs1/sub1%tank_fs1_sub1            @             local

The properties were successfully updated.  Lets check the file system to see if the share properties were brought over.

# zfs get share tank/fs1/sub1
tank/fs1/sub1  share     name=tank_fs1_sub1,path=/tank/fs1/sub1,prot=nfs,sec=sys,ro=@,rw=@  local

That looks perfect, and lets ask NFS what it knows.

# showmount -e
export list for server1:
/tank/fs1/sub1                      @,@,@

Wonderful!  That is exactly what we wanted.  Now one final thing.

Removing NFS Export Configuration Permanently

Now since there is this whole different type of file system to store the share configuration, removing it becomes incredibly easy.

# zfs destroy tank/fs1%tank_fs1

That pretty much wraps up everything.

September 10th, 2013 | Tags: , , ,

In this article we are going to go over some basics of using IPMP in Solaris.  We will discuss this on the latest versions of Solaris 10 and 11 on the SPARC platform, these instructions should also work on the x86 platform, but will be dependent on the listed version numbers, as the networking stack has changed alot of the past few minor versions of Solaris 10 and 11.

Solaris 11.1

Create an IPMP Group

The first step is to create an IPMP group, in this case named ipmp1.

solaris11~# ipadm create-ipmp ipmp1

Add Interfaces to an IPMP Group

Once the group is created we can add interfaces to it, in this case net1 and net2.

solaris11~# ipadm add-ipmp -i net1 -i net2 ipmp1

Create an IPMP Group and Add Interfaces

We can also combine the above steps so that we create the group and add the interfaces in one step.

solaris11~# ipadm create-ipmp -i net1 -i net2 ipmp1

Set Interface as Standby in an IPMP Group

In the event that you prefer to have a standby interface you can change an interface to be a standby, by default all interfaces are active.

solaris11~# ipadm set-ifprop -p standby=yes net2

Remove Interfaces from an IPMP Group

Prior to deleting an IPMP group you must remove all interfaces first.

solaris11~# ipadm remove-ipmp -i net1 -i net2 ipmp1

Delete an IPMP Group

Now we can delete the IPMP group.

solaris11~# ipadm delete-ipmp

Assign an Address

After we created everything then we can assign an address.  We will need an address for the IPMP group interface, in addition if we want to use probe-based IPMP then we will need to assign them on the underlying interfaces as well, otherwise you will be using the link-based version of IPMP.

Here is an example for probe-based IPMP.

solaris11~# ipadm create-addr -T static -a ipmp1/v4
solaris11~# ipadm create-addr -T static -a net1/v4
solaris11~# ipadm create-addr -T static -a net2/v4

Here is an example for link-based IPMP.

solaris11~# ipadm create-addr -T static -a ipmp1/v4

Multipath Configuration File

If you want to be able to control some of the specifics of multipath behavior, like failback for example, you can do so using the mpathd configuration file.

solaris11~# grep -v "^#" /etc/default/mpathd

That completes our configuration on Solaris 11.1.

Solaris 10 1/13

The Solaris 10 version of these instructions are very similar to the way you would configure an IP on a normal interface, but with some additional parameters.

Interface Configuration Files (Active-Active) Link-based IPMP

In Solaris 10, we control everything with the interface configration files, named hostname.interfacename where the interface name is net1 and net2 in this environment, please change your link names or configuration file names to match your server.

solaris10~# cat /etc/hostname.net1
machinehostname netmask + broadcast + group ipmp1 up
solaris10~# cat /etc/hostname.net2
group ipmp1 up

Interface Configuration Files (Active-Active) Probe-based IPMP

solaris10~# cat /etc/hostname.net1
machinehostname-net1 depracated -failover netmask + broadcast + group ipmp1 up addif machinehostname netmask + broadcast + up
solaris10~# cat /etc/hostname.net2
machinehostname-net2 depracated -failover netmask + broadcast + group ipmp1 up

Interface Configuration Files (Active-Standby) Link-based IPMP

The only time that I see a value in an Active-Standby configuration is when you have to guarantee a level of service, even in a failed state.  In my environments the increased performance is of a much higher value than the non-reduced workload.

solaris10~# cat /etc/hostname.net1
machinehostname netmask + broadcast + group ipmp1 up
solaris10~# cat /etc/hostname.net2
group ipmp1 standby up

Interface Configuration Files (Active-Standby) Probe-based IPMP

solaris10~# cat /etc/hostname.net1
machinehostname-net1 depracated -failover netmask + broadcast + group ipmp1 up addif machinehostname netmask + broadcast + up
solaris10~# cat /etc/hostname.net2
machinehostname-net2 depracated -failover netmask + broadcast + group ipmp1 standby up

Hosts File

solaris10~# grep machinehostname /etc/hosts      machinehostname.localdomain machinehostname      machinehostname-net1.localdomain machinehostname-net1 # only needed for probe-based ipmp      machinehostname-net2.localdomain machinehostname-net2 # only needed for probe-based ipmp

Netmasks Configuration File

solaris10~# grep /etc/netmasks

Multipath Configuration File

solaris10~# grep -v "^#" /etc/default/mpathd
Comments Off
September 9th, 2013 | Tags: , , ,

So with ZFS you really have two ways of configuring your NFS exports, you can do it the old fashioned way (read: /etc/dfs/dfstab and share command).  Or the super-duper easy method…  Use ZFS metadata, and allow ZFS to configure the export for you.  The reason why you might choose this approach is simple, if you don’t want to have to worry about configs, and especially if you need this export to be tied to the filesystem regardless of where you put the file system from system to system.  Pause for effect.  Your NFS exports can be TIED TO THE VOLUME AND NOT THE SERVER.  So if you have a USB drive that you are using to move data on, it can immediately share the data on another server when the pool is imported, same is true for LUNs.  This adds a lot of flexibility.

Read-Write for Everyone

solaris11~# zfs set share.nfs=on zpool/fs1
solaris11~# zfs set share=name=fs1,path=/zpool/fs1,prot=nfs zpool/fs1


solaris10~# zfs set sharenfs=on zpool/fs1

Read-Write for a Single IP

solaris11~# zfs set share.nfs=on zpool/fs1
solaris11~# zfs set share=name=fs1,path=/zpool/fs1,prot=nfs,sec=sys,rw=@ zpool/fs1


solaris10~# zfs set sharenfs=sec=sys,rw=@ zpool/fs1

Read-Write and Root for Multiple IPs

solaris11~# zfs set share.nfs=on zpool/fs1
solaris11~# zfs set share=name=fs1,path=/zpool/fs1,prot=nfs,sec=sys,rw=@,root=@ zpool/fs1


solaris10~# zfs set sharenfs=sec=sys,rw=@,root=@ zpool/fs1

That pretty much takes care of it.  Feel free to mix and match.  One thing to keep in mind.  If you specify it the old way and the new way, then the ZFS settings will take precedence.

Comments Off
August 27th, 2013 | Tags: ,

Generally I don’t use my blog as a platform to find potential applicants for my employer, however at this point we are looking very seriously at some new candidates to fill out our Systems Team.

This is a job that is very much “Systems” focused.  At Keste we define Systems as everything from the hardware up to the applications, this includes but is not limited to Servers, Storage, Networking and Virtualization.  Because this is a very broad area, we expect folks to have expertise in multiple areas and most importantly to have the capabilities to learn and broaden their knowledge where they have weaknesses.

The other thing you must know about this job is that it is a consultant position, this means that you will need to travel (we say 50% – however it has been lower than that in my experience), and additionally you will need to be comfortable presenting in front of customers at all levels (technical, business, users, etc).  So if you have poor social skills then you would have to have some pretty phenomenal skills which will raise the team around you to support your weaknesses, but that wouldn’t necessarily be a deal breaker.

So if you think that you are a fit for the role please check the link below for the most up to date job posting, and apply there.

Comments Off
July 16th, 2013 | Tags: , , , , ,

Today we will be configuring LACP to interconnect some switch stacks as part of a network upgrade.  Please keep in mind that part of configuring the LAG will result in the ports losing connectivity, so while it is not necessary to use a console cable, please keep in mind how you are connecting to the switch and if you will interrupt that connection by any of your actions, and plan accordingly.

This environment is a small environment, which consists of a stacked pair of Dell Powerconnect 6248 as the core.  Which uses 2 10Gb connections (per pair) to connect to 2 stacked pairs of Dell Powerconnect 5548 (one for servers and one for users), the user 5548 stack also has 2 10 Gb connections which connect to a third Dell Powerconnect 5548 stack which is in an IDF for some more users.

Configuring LACP

So each vendor and in some cases each model of switch can do LACP slightly differently, but the basics of it are this, if the LACP protocol configurations match on both side of the connection then the LAG is successfully created.  One thing I want to make clear is that while the configuration of the protocol has to match that doesn’t mean that you execute the same commands to get there, or that the applicable running-configs will match.  Each vendor and in some cases switch firmware has its own way of configuring LACP, it is the underlying configuration which matters NOT the visible configuration.  In most cases the same terminology is used which can simplify the configuration.

Powerconnect 6248 LAG Configuration

Now this section will need to be done on the 6248 side.  I have tested this configuration with 55xx, but not with another 62xx.

Configure Port Channel (LAG)

> enable# configure
(config)# interface port-channel 2
(config-if)# description "Downlink - stack-5548-user-1"

Really all we need on the Port Channel is a description so that we know what it is.  However if you have any specific port configs then you will put them here and NOT on the port itself.  So for example below I am setting the pvid as 10.

(config-if)# switchport mode general
(config-if)# switchport general pvid 10
(config-if)# switchport general allowed vlan add 10
(config-if)# exit

Configure Port Members

> enable# configure
(config)# interface ethernet 1/xg4
(config-if)# description "stack-5548-user-1 te1/0/2"
(config-if)# channel group 2 mode auto
(config-if)# exit
(config)# interface ethernet 2/xg4
(config-if)# description "stack-5548-user-1 te2/0/2"
(config-if)# channel group 2 mode auto
(config-if)# exit

Powerconnect 5548 LAG Configuration

This section should be done on the 5548 side.  I have tested this with both the 62xx and other 55xx switches.

Configure Port Channel (LAG)

> enable# configure
(config)# interface port-channel 1
(config-if)# description "Uplink - stack-6248-core"

Really all we need on the Port Channel is a description so that we know what it is.  However if you have any specific port configs then you will put them here and NOT on the port itself.  So for example below I am setting the pvid as 10.

(config-if)# switchport mode general
(config-if)# switchport general pvid 10
(config-if)# switchport general allowed vlan add 10
(config-if)# exit

Configure Port Members

> enable# configure
(config)# interface tengigabitethernet 1/0/2
(config-if)# description "stack-6248-core 1/xg4"
(config-if)# channel group 1 mode auto
(config-if)# exit
(config)# interface tengigabitethernet 2/0/2
(config-if)# description "stack-6248-core 2/xg4"
(config-if)# channel group 1 mode auto
(config-if)# exit

Check Status

That about wraps it up the only thing left to do is check the status of our configured LAG.

Powerconnect 62xx

#show interfaces port-channel 2

Channel   Ports                         Hash Algorithm Type
-------   ----------------------------- -------------------
ch2       Active: 1/xg4, 2/xg4          3

Hash Algorithm Type
1 - Source MAC, VLAN, EtherType, source module and port Id
2 - Destination MAC, VLAN, EtherType, source module and port Id
3 - Source IP and source TCP/UDP port
4 - Destination IP and destination TCP/UDP port
5 - Source/Destination MAC, VLAN, EtherType, source MODID/port
6 - Source/Destination IP and source/destination TCP/UDP port

Powerconnect 55xx

# show interfaces port-channel 1

Load balancing: src-dst-mac.

Gathering information...
Channel  Ports
-------  -----
Po1      Active: te1/0/1-2

So we can see that both switches are showing the LAG as Active, and showing the correct memberships.  Notice that the hashing algorithm is different.  That is due to the 6248 being a L3 switch and the 5548 being L2.  Depending on the type of traffic it will negotiate that algorithm.

Comments Off
Page 4 of 28« First...23456...1020...Last »