Solaris 11: Network Configuration Advanced

Solaris 11: Network Configuration Advanced

In my previous post we went over the basics of configuring a network interface in Solaris 11, without using the Network Auto Magic (NWAM) which is enabled by default.  In this article we will go over some of the more advanced features which can be leveraged including VLANs, aggregation groups, and jumbo frames.

Configure VLAN Tagging

Create a VLAN, specify the VLAN id with the “-v”, then specify the interface with the “-l”, and finally create a name for the tagged interface (in this case user0 since this particular system is the user VLAN, if you have more descriptive names of your VLANs you can use them here).

# dladm create-vlan -v 20 -l bge0 user0
# dladm show-vlan
LINK            VID      OVER         FLAGS
user0           20       bge0         -----
# ipadm show-if
IFNAME     STATE    CURRENT      PERSISTENT
lo0        ok       -m-v------46 ---
bge0       ok       bm--------46 -46
user0      ok       bm--------46 -46

For simplicity I will delete the IP configuration on the untagged interface.

# ipadm delete-addr bge0/v4

As we did in our previous article you can now create an IP configuration on top of the new tagged interface (which in this case is DHCP).

# ipadm create-addr -T dhcp user0/v4

Keep in mind if you change the VLAN you will most likely need to change your default route.  The below options “-fp” f will flush meaning delete all current routes, while the p will make the new settings persistent.  In this case afterwards we will only have 192.168.100.1 as a default gateway.

# route -fp add default 192.168.100.1

Delete a VLAN

If you have used the VLAN before you will need to “unwind” the configuration before deleting the VLAN.

# ipadm delete-addr user0/v4
# ipadm delete-if user0

Now that this is done you can delete the VLAN.

# dladm delete-vlan user0

Create an Aggregation Group

With this command we create a new aggregation group and assign interfaces to the aggregation group.  If the interface is already in use then you will need to delete the interface before adding it to an aggregation group.

# dladm create-aggr -l bnx0 -l bnx1 aggr0

Now we can view the details of our aggregation group.

# dladm show-aggr
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr0           L4       auto                 off           short       -----
# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
bnx0        phys      1500   up     --         --
bnx1        phys      1500   up       --         --
aggr0       aggr      1500   up  --         bnx0, bnx1

Now we add an IP configuration against the aggregated interface.

# ipadm create-addr -T static -a 192.168.100.172/24 aggr0/v4

Modify an Aggregration Group

If you need to add or remove an interface from an aggregation group then the following commands will allow you to do that.

# dladm add-aggr -l bnx2 aggr0
# dladm remove-aggr -l bnx1 aggr0

You can also adjust the LACP Policy using the below command.  Where L4 is L2, L3, L4, or any combination of them based on the desired behavior.

# dladm modify-aggr -P L4 aggr0

The LACP mode can be configured using the below command where active is either auto, active, or passive.  Additionally if configuring active mode you must also configure a timer value of short or long, this option is not needed for auto or passive.

# dladm modify-aggr -L active -T short aggr0

Delete Aggregation Group

Delete the IP configuration from the aggregation group

# ipadm delete-addr aggr0/v4

Delete the Aggregated Interface

# ipadm delete-if aggr0

Delete the Aggregation Group

# dladm delete-aggr aggr0

Enable Jumbo Frames

Basically Jumbo Frames allow the system to reduce the network overhead by combining more data into a single TCP frame, this is analogous to renting a box truck when you move into a new house.  If you had to use your Prius to move, you would spend much more time waiting to finish the process, as well as expending more resources.  Now Jumbo Frames doesn’t mean that it will always help.  If we step back to our analogy of moving into a new house, if you all of your stuff amounts to a single suitcase then renting a moving truck doesn’t do anything to make your trip more efficient.  So if you are not send large amounts of data then Jumbo Frames will not help you, however if you are working on a storage network and even with some file sharing you will get a bonus.  Also in order for Jumbo Frames to work, both sides of the communication must support it or it will not use the higher MTU, as well as all devices along the way.

To display the current mtu of an interface

# dladm show-linkprop -p mtu bge0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
bge0         mtu             rw   1500           1500           1500 

To set the mtu to enable jumbo frames

# dladm set-linkprop -p mtu=9000 bge0

To set the mtu to not use jumbo frames

# dladm set-linkprop -p mtu=1500 bge0

 

UPDATE
September 16, 2011

In the comments of my article “Solaris 11: Network Configuration Basics” you will notice “Kristen” mentioned that the ipadm command has changed in newer builds of Solaris 11.  At the time she was using a newer build than I had available to me, so I could not verify her claim, however now I have verified this change against the Solaris 11 Early Adopter release snv_173.  So be prepared to make the following changes.

# ipadm create-if bge0
# ipadm delete-if bge0

Will now be

# ipadm create-ip bge0
# ipadm delete-ip bge0

The following were not changed:

  • ipadm enable-if
  • ipadm disable-if
  • ipadm show-if

12 thoughts on “Solaris 11: Network Configuration Advanced

  1. Otto123

    Hey,

    these two are really great! They are so good actually that I, being a political scientist, am able to configure a trunk on my HP Microserver with Solaris Express 11 and an extra Intel Dual Nic Card. Many thanks!

    What you could elaborate for the less clever ones like me – which mode for the trunk should be used on the switch (active or passive) and on SE11. Could not find an answer yet. I am using a Procurve 1800-24 which allows active and passive mode.

    Cheers,
    Otto

  2. matthew.mattoon Post author

    Hi Otto,

    I am by no means an expert. I work on the server side and as such a lot of the networking side is just “magic” that said I will explain what I know.

    First definitions.

    Trunk – A trunk is a logical configuration which allows for multiple VLANs to be sent over the same physical port.
    Aggregration Group (LAG) – Also called Ether-channel in the Cisco world, and also called a trunk in a lot of circles, although trunk is not really a good way to refer to it. This technology is when you combine the bandwidth of multiple interfaces into a logical interface which can be used to increase bandwidth and provide redundancy.

    That said… Keep in mind that when you create a LAG there has to be two parts to the LAG, the switch side and the NIC side.

    Active means that it will use all LAG members when sending traffic. Passive means that it will return traffic on the same interface it was received on, which as long as traffic is being sorted on the other end of the switch the LAG will function. Now how you configure it really depends on your environment. I personally like to keep my servers as vanilla as possible and allow the network configuration on the network side (frankly there is more than enough work on the server side to be adding more). Also in a lot of cases it is just flat out easier to do it on the switch. That said I would try and set up the configuration on the switch with the SE11 box using “auto”. Once you have a working configuration generate some traffic and unplug some LAG members and see what happens, you will also want to see what kind of speed you are getting from you LAG (an FTP test to a well connected host [read: better than your LAG] on your LAN works well for this – If you are trying to test a 4Gbps LAG on the switch you can hard code the ports for 100Mbps giving you a 400Mbps LAG for the purpose of testing speed to a 1Gbps FTP Server).

    Good Luck!

    -matt

  3. matthew.mattoon Post author

    Also I forgot to mention. In some cases you will have to fiddle with the settings to get them just right, so don’t be afraid to tinker. Once you have a working config then it is time to test and validate.

    -matt

  4. Otto

    Thanks again Matthew,

    have quite some time to test it today. The four HDDs in RaidZ result in around 200 mb/s read. Happy to get that on the LAN ;-)
    Will also have a look at the Jumbo Frames. But as I read so far on Solaris Express it does not make a big difference.

    Best regards from the Swiss boarder
    Christian

  5. Otto123

    Hi Matthew,

    so I started and most went ok:
    1) Went through your Basics
    2) Created DHCP for Onboard Broadcomchip which I renamed to broadcomdhcp1
    3) Made an aggregation of e1000g0 and e1000g1 (that is an extra NIC Card)
    4) Assigned static IP v4 for aggregation (called trunk1)

    However, now I am stuck:
    1) For funny reasons I get a trunk1/-a AND a trunk1/v4
    2) I can see the IP numbers in my net using a portscanner but can not ping on them
    3) No access to internet.
    4) No access to the SMB shares provided by this Solarisbox (this box works as a NAS in my private net)

    Output of:
    dladm show-phys and show-link an dipadm show-addr and netstat -r can be found here
    http://www.abload.de/img/configmicrose11ex7d.png
    Advice would be highly appreciated.

    The net is 192.168.20.1/24. That is a monowall that sits behind the router which connects to WAN.

    BEst
    Otto

    1. matthew.mattoon Post author

      Try the following…

      Delete the current address configurations
      # ipadm delete-addr trunk1/v4
      # ipadm delete-addr trunk1/_a
      # ipadm delete-addr broadcomdhcp1/dhcp
      # ipadm delete-addr broadcomdhcp1/?

      Create DHCP configuration on broadcomdhcp1
      # ipadm create-addr -T dhcp broadcomdhcp1/v4
      Create Static IP configuration on trunk1
      # ipadm create-addr -T static -a 192.168.20.88/24 trunk1/v4

      Also keep in mind that generally numbering begins with “0” as opposed to “1” so it might make more sense to have your interfaces named trunk0. Though I generally stay with the default names unless I am doing VLANs then I will name them with accordance with the purpose of the VLAN (user0, server0, printer0, dmz0, etc).

      My guess is that this should sort you out. However if it doesn’t I emailed you separately so you can respond back with more details.

      -matt

  6. Otto

    Wow. Many thanks Matthew! However, have to try tomorrow. Time for bed here. Will let you know outcome. Did not get a mail but can post here anyway.

  7. Otto123

    Hey folks,

    the solution can be found here:
    http://blogs.tulsalabs.com/?p=135&cpage=1#comment-655
    Hope to save some others a long evening

    Otto

    Just in case:
    root@snarf:~# ping 172.16.1.2
    ping: sendto Network is unreachable
    root@snarf:~# ipfstat -io
    block out log all
    pass out quick on lo0 all
    pass out quick proto udp from any to any port = bootps
    block in log all
    pass in quick on lo0 all
    pass in quick proto udp from any to any port = bootpc
    root@snarf:~# svcadm disable svc:/network/ipfilter
    root@snarf:~# ping 172.16.1.2
    172.16.1.2 is alive
    root@snarf:~# ipfstat -io
    empty list for ipfilter(out)
    empty list for ipfilter(in)

  8. Andypandy

    any ideas, trying to enable jumbo frames, and it states link busy, ive tried ifconfig down, unplumb, but does change mtu to 9216, and it states supported and rw

  9. matthew.mattoon Post author

    Hi Andy,

    What do you get when you run the following against you interface which you have tried to enable jumbo frames for (where bge0 is the name of your interface)?

    # dladm show-linkprop -p mtu bge0

    Also have you tried other mtu sizes than 9216, such as 9000?

    Also how did you configure your networking on Solaris did you follow the instructions in my basic article or are you following something else?

    -matt

  10. Andypandy

    I had configured “myway”, jumbo frames didn’t work, so re-did all the config, following your method.

    sorry, Matt, forgot to check back!

    root@san1:~# dladm show-linkprop -p mtu e1000g0
    LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
    e1000g0 mtu rw 1500 1500 1500-9216
    root@san1:~#