Solaris 11: Network Configuration Basics

Solaris 11: Network Configuration Basics

I have been doing research for my next big series “So You Want to Learn ZFS.”  This series is basically going to be a multi-part series of How-To’s which hopefully will give you the ability to build a file server (or even a SAN) based on ZFS if you so choose.  However there are a few things that I failed to take into account.

  1. Solaris 11 is so different from Solaris 10.
  2. Solaris 11 is so different from EVERYTHING else.

I figured that I’d be able to kind of gloss over the high points of how to get your system up and running and just dive right into the fun ZFS stuff.  So before we get into the good stuff there are some basics that we will need to go over first.  Today we will cover basic networking.

First off with the acquisition of Sun by Oracle last year the documentation is kind of scattered.  The most important place to know of is here.  I am sure Oracle will get this under control eventually.

Alright so what makes network configuration so difficult with Solaris 11?  Some things are much easier than they should be while others are just ridiculously difficult.  I personally attribute this to a tendency towards over-engineering on the part of Sun Engineers, everything is done in the most correct way.  Now this is not to say that Solaris is better than everything or that Sun hardware was better than anything else.  My basic point is that the most correct way is not always the best way.  I think that Sun’s over-engineering hurt them in the long run (which ultimately is why Oracle bought them and not the other way around).  However there is one area where I think the over-engineering paid off and the most correct way was actually the best way, this would be ZFS.  But I digress that will be for a later article.

If you install Solaris 11 Express then by default a service called Network Auto Magic (NWAM), which simplifies the process significantly, however if you look to do more advanced tasks such as aggregation then this won’t work for you.  NWAM is really very much the same as Network Manger, it can provide location based networking profiles and manage multiple types of interfaces (wireless and wired) seamlessly, although it may not be the best for a server configuration.

Disable Network Auto Magic

# svcadm disable network/physical:nwam
# svcadm enable network/physical:default

Once we have disabled NWAM we will lose all network connectivity and configurations.

View the Datalink Devices

Solaris 11 devices have many layers to their configuration, which makes advanced configurations much simpler however does complicate basic configurations. Basically the kernel is aware of the physical hardware and we can see this visibility with the first command.

# dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
bge0         Ethernet             unknown    1000   full      bge0

The second command gives us the ability to see the physical interface linked to a logical interface.  After disabling NWAM you will NOT have a logical interface linked to your physical device (in my case bge0) because of this you will see that the state of the data-link device is “unknown”.  Also it is important to note that the device names are based off of vendor bge = broadcom and they are incremented based on the number of devices in the machine.

# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
bge0        phys      1500   unknown  --         --

Also before we move on we will just take a look at our existing logical interfaces, the only one you should have after disabling NWAM is lo0 which is your loopback interface.

# ipadm show-if
IFNAME     STATE    CURRENT      PERSISTENT
lo0        ok       -m-v------46 ---

Create and Configure a Logical Interface

So the first step is creating a logical interface, then we can apply an IP configuration against it.  This will create a link from the logical interface to the physical interface, and will change the state to “up” from “unknown” that we saw before.

# ipadm create-if bge0
# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
bge0        phys      1500   up       --         --
# ipadm show-if
IFNAME     STATE    CURRENT      PERSISTENT
lo0        ok       -m-v------46 ---
bge0       down     bm--------46 -46

Now above we have successfully created the logical interface and we can now apply an IP configuration to it.  This is where it gets a bit tricky.  Notice below we are going to apply DHCP as the configuration, we will end up deleting this configuration and making it static, this way you also get the opportunity to learn how to change the configuration (which is really a delete and add).  We will go through the specifics of the ipadm create-addr command after we also go over the static command as well since they are very similar.

# ipadm create-addr -T dhcp bge0/v4
# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
bge0/v4           dhcp     ok           192.168.100.225/24
lo0/v6            static   ok           ::1/128

Now to delete the DHCP configuration from the logical interface so that we can make it static.

# ipadm delete-addr bge0/v4

And to create a static IP configuration on the logical interface.

ipadm create-addr -T static -a 192.168.100.200/24 bge0/v4
# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
bge0/v4           static   ok           192.168.100.200/24
lo0/v6            static   ok           ::1/128

Alright so as we can see these are the two commands to create the configurations.

# ipadm create-addr -T dhcp bge0/v4
# ipadm create-addr -T static -a 192.168.100.200/24 bge0/v4

Now the -T option defines the type of configuration static and dhcp are the most common options, -a is for the address on a static configuration and you will notice that we are not using the logical interface name (bge0), but instead a variation (bge0/v4).  This represents the version of the IP protocol the configuration is using.  So you can have a bge0/v6 and a bge0/v4.

Alright so you have successfully configured your network interfaces, however NWAM was doing more than just this, so you might not have full network connectivity yet.

Verify Full Network Configuration and Connectivity

Using some of the above commands we can review our configurations.

# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
bge0/v4           static   ok           192.168.100.200/24
lo0/v6            static   ok           ::1/128

Additionally we need to verify name resolution and routing in order to be confident in our configuration.

# netstat -r

Routing Table: IPv4
Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
solaris              solaris              UH        2          0 lo0
192.168.100.0        192.168.100.200      U         3          1 bge0

Routing Table: IPv6
Destination/Mask            Gateway                   Flags Ref   Use    If
--------------------------- --------------------------- ----- --- ------- -----
solaris                     solaris                     UH      2       4 lo0

Above will display the current routing table (which does not have a default route), ensure your default route is defined and correct.  If you need to create it use the below command.

# route -p add default 192.168.100.1
add net default: gateway 192.168.100.1
add persistent net default: gateway 192.168.100.1

Once it has been corrected it should look something like this, and you should be able to ping off-net.

# netstat -r

Routing Table: IPv4
Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
default              fw01.allanglesit.net UG        2      10466
solaris              solaris              UH        2         12 lo0
192.168.100.0        192.168.100.200      U         6       1810 bge0

Routing Table: IPv6
Destination/Mask            Gateway                   Flags Ref   Use    If
--------------------------- --------------------------- ----- --- ------- -----
solaris                     solaris                     UH      2     156 lo0

To verify DNS configuration check the /etc/resolv.conf and then verify the functionality with nslookup or dig.

# cat /etc/resolv.conf
domain  allanglesit.net
nameserver  192.168.100.22
nameserver  192.168.100.25
# dig www.google.com

Solaris additionally uses /etc/nsswitch.conf to tell the system what types of name resolution to use for different types of lookups.  When disabling NWAM (which was configuring /etc/nsswitch.conf for us) then we will have a hosts file only configuration, which means our system won’t attempt to use DNS on its own (nslookup and dig will work since they know to use DNS themselves, but things like Firefox, wget, samba, etc only look to the system for name resolution).

# cat /etc/nsswitch.conf
.
.
hosts:      files dns
ipnodes:    files dns
.
.

I trimmed the above file for brevity.

At this point you should have full network connectivity without using NWAM.  So now just reboot to ensure that your settings persist after a reboot.

For WAY more information…

http://download.oracle.com/docs/cd/E19963-01/pdf/821-1458.pdf

 

UPDATE
September 16, 2011

In the comments below you will notice “Kristen” mentioned that the ipadm command has changed in newer builds of Solaris 11.  At the time she was using a newer build than I had available to me, so I could not verify her claim, however now I have verified this change against the Solaris 11 Early Adopter release snv_173.  So be prepared to make the following changes.

# ipadm create-if bge0
# ipadm delete-if bge0

Will now be

# ipadm create-ip bge0
# ipadm delete-ip bge0

The following were not changed:

  • ipadm enable-if
  • ipadm disable-if
  • ipadm show-if

UPDATE
February 28, 2012

Another astute user “j.marcos” (comment below) pointed out another change in the GA version of Solaris 11

For Solaris 11, instead of disabling network/physical:nwam and enabling network/physical:default we control NWAM by setting the ncp mode to DefaultFixed

# netadm enable -p ncp DefaultFixed

If you wanted to re-enable NWAM then we can set the ncp mode back to Automatic

# netadm enable -p ncp Automatic

20 thoughts on “Solaris 11: Network Configuration Basics

  1. hotzen

    Thanks a LOT for that brief summary!
    The net is full of outdated, non-working, or NWAM-using information….

  2. Dragos

    Great job!

    Thanks! Other places are highly outdated.

    Btw: Is there a change for Solaris 11 to come this year?

    1. matthew.mattoon Post author

      Hi Dragos,

      I don’t work for Oracle, so I can’t speak authoritatively on the release date of Solaris 11. However based on the release quality of Solaris 11 Express 2010.11, I would think they would be close enough to polish it off before the end of 2011, otherwise they must have some pretty incredible features and I would guess 2 or more Express releases. But realistically “Solaris 11… Completing 2011″ what marketing department could pass that up?

      -matt

  3. Dragos

    Thanks Matthew,

    Can I ask you what documentation did you used to this tutorial and also for Solaris 11
    in general ?

    1. matthew.mattoon Post author

      Solaris has a ridiculous amount of official documentation, though it is a bear to get through. All of my documentation is performed off of tests that I have actually performed. In other words if I document a command it is because I have actually done it and verified the results on an actual machine.

      http://www.oracle.com/technetwork/server-storage/solaris11/documentation/productdocumentation-178191.html#sys

      You will also notice the link at the end of the article, which is a link to the networking document in the Solaris 11 Express (2010.11) Documentation Library.

      You can also use the man pages.

  4. kristen

    Thanks Matthew! This guide helped me immensely! One corrective note I have:

    At least on Solaris 11 build 166, the command: “ipadm create-if bge0″
    is now: “ipadm create-ip bge0″.

    note that the create-if has become create-ip.

    I don’t know if this is your typo, or a change by Oracle.

    1. matthew.mattoon Post author

      Hi Kristen,

      Thanks for the heads up…

      Everyone,

      I have not yet verified Kristen’s comment, when I do I will update the article accordingly. However if you are running a newer build of Solaris please keep in mind that this may have changed.

      -matt

  5. @teslim

    Hi Matthew,
    Please I am new to solaris, I installed the solaris 11 on a partition on my system. The problem I am having now is that the wireless network is not being identified but everything appear fine with the wired network.

    1. matthew.mattoon Post author

      Hi teslim,

      This article doesn’t configure any sort of wireless networking. For that you will need to dig into the Solaris Networking documentation or stick with NWAM (which handles it pretty well – given driver availability).

      -matt

  6. Danny

    I finally upgraded x86 Opensolaris -> Solaris 11 Express and finally Solaris 11. Quite a bit has changed since Express which I wasn’t really expecting (read the release notes!!) and concur with the lack of upto date information, especially when everyone refers to their flavour as Solaris 11 whether it was open, express or actually Solaris 11.

    One issue I’m left with is Power Management, I can’t enable speedstep in poll mode to prevent kernel usage being permanently 20-40% CPU as it bounces from one speedstep to another. Editing /etc/power.conf was the work around previously, not anymore, poweradm is the new command but lacks from what I can tell the finer detail I require to work around the issue. Disabling power management (#poweradm set administrative-authority=none)rectified kernel cpu usage at the cost of no speedstep. I also encounted strange behaviour around root as a role. In opensolaris and express, for me root was a user and could ssh remotely (I’m aware of the risks) upon upgrade I could login to the console but if the screen locked and I entered the correct pass it would say ‘roles can not login directly’ yet I already had! remote ssh wouldn’t work either although the correct config to permit it was entered. Turned out in /etc/user_attr.d there was the core-os file that stated root as a role but a leftover SUNWgui-install file that stated root as a user. Removing SUNWgui-install and performing the root as a user role change again fixed this.

    Now I’m no expert in Solaris, although I do work for Oracle, formally Sun in hardware it’s a different department. I loved what I read about ZFS so I wanted it too. For me it’s just a fileserver and dev webserver at home, nothing special. Solaris does make you investigate and think about what your trying to achieve – a challenge I have become to enjoy!

    A well paced tutorial, I will be reading more!

  7. Jon Strabala

    Welcome to bug #1333 on the illumos site https://www.illumos.org/issues/1333 – so it seems like Solaris 11 also exhibits the same behavior ( I just saw it on a fresh install of from sol-11-1111-live-x86.iso )

    “I can’t enable speedstep in poll mode to
    prevent kernel usage being permanently 20-40%
    CPU as it bounces from one speedstep to another”

    Okay you might try setting the following in /etc/system followed by a reconfiguration
    reboot:

    set apix:apic_timer_preferred_mode = 0x

    The real issue is of course in the code specifically “apic_timer.c” but due to Oracle’s non-compliance with CDDL until they abide by the terms I imagine the fix is either viral to Oracle’s code base and/or a major copyright violation.

  8. Gene

    Great article.. Thanks.. But come on Oracle wtf… you change ipadm create-if, delete-if, to create-ip, delete-ip.. But leave show-if, enable-if, and disable-if as they were?

  9. Marc Lobelle

    Thanks for the article. It looks simple.

    However, when trying it on solaris 11/11, the interface already existed in dhcp mode.
    OK; I deleted it as explained but when I tried to set a static address,

    ipadm create-addr -T static -a 130.104.229.230/27 net0/v4

    I got the message

    ipadm: cannot create address: Persistent operation on a temporary object

    notice: net0 was existing. I deleted it with ipadm delete-ip net0, but it was still there in ipadm show-if.

    Has something changed since you wrote the article ?

    What should I do to set a static address in this machine ?

    Thanks

    Marc

    also among the svcs, there is a network/physical:upgrade should it be disabled ? since we enable :default ?

    1. matthew.mattoon Post author

      Hi Marc,

      Perform a reboot (this should clear any in-memory changes to your network config that might be in effect), and then post the following:

      # dladm show-phys
      # dladm show-link
      # ipadm show-if
      # ipadm show-addr

      -matt

      1. matthew.mattoon Post author

        Hi J.Marcos,

        You are correct, networking has changed a bit in the GA of Solaris 11, the above still works on Solaris 11 Express and OpenIndiana. However on the GA version the network/physical:nwam service was depracated, so instead of stopping that and starting network/physical:default we now need to set the ncp mode, the appropriate modes are Automatic (which is NWAM) or DefaultFixed (which is manual configuration).

        I will include an update in the article.

        -matt

  10. Tim

    “But realistically “Solaris 11… Completing 2011″ what marketing department could pass that up?” Too funny and true. They did it on 11/11/11 just to be sure everyone knew this is Solaris 11.

    Great info on the page. Thanks for updating it.