Factor One – Use Case

The use case is the single most important factor, this basically tells you how you plan on using the data.  Is this going to serve files (NFS or CIFS), will you be using it to serve block devices (Fibre Channel or iSCSI), or perhaps you will be using this as a consolidation point of multiple sources of data for use in a backup scenario.  The answer to this question will primarily determine if you are using a ZIL or Cache device or if you are using spares.  This of course will steal slots (where disks could go) so you will want to factor that into your larger capacity planning.

Physical Capacity

Obviously the machine that you buy will have a limit to how many disks you can fit in it.  After that you can plan on adding expansion chassis.  You need to make sure you understand how your zpools will be formatted and the types of disks that you will be using so that you can begin to see how much data you will be able to squeeze onto the spindles.  One easy thing to overlook is the need for drive bays for other purposes (spares, log, cache, system drives, etc).  If you plan on expanding with an additional chassis, will you use up all of your expansion slots with various cards (10Gb, FibreChannel, NICs, SSD, etc).  Also when considering disks you need to keep in mind that capacity is not the only metric to consider.  You also need to keep in mind that your workload will determine how performant the solution must be to meet requirements, for example a backup server will not require the same performance as a storage head which is handing out block devices which have Operating Systems installed on them (read: Virtual Machine environments).

Spare Devices

Depending on your use case you might need to consider allocating some drives as spares to allow you the time to procure and/or replace failed drives.  This will simply protect you from additional failure(s).  If you have the space I like to have 1 spare per 12-15 acting as a spare.  This really straight forward, just make sure that as part of any large system that the company accepts the risks involved with your decision, if they are not willing to accept the risks then provide them with an option which will protect them from that risk.  They will then either pay for the lower risk system or accept the risk of the cheaper system.

Cache Devices

Cache devices (L2ARC) is a really quick and cheap way to increase your read iops on a system wide basis.  Basically as data is read off of ZFS it is read through the cache device, then when subsequent read requests for the data come in then they can be served directly off of the cache, this works in combination with the ARC which is where all of your memory goes in ZFS.  Now a cache device is really necessary if you are using deduplication.  When you enable deduplication all subsequently written files are indexed into the dedup table, before a write can be committed it must be compared to items that are already in the dedup table (to see if anything else is the same – thus if it should actually be written to disk or if a pointer to the previously existing file will exist) initially this will not pose a problem as your dedupe table will be relatively small and it can be stored in the 25% of the ARC that it is relegated to.  However at some point you will run out of space in the ARC and if you do not have a L2ARC then your dedup table will be swapped which of course means that it will be holding up writes to disk, in other words it will be slow.  Since a cache device’s purpose is to speed up reads, then you want to look for a SSD which is slanted towards better performance on the read side.  I like Crucial M4 for this purpose, but please keep in mind they had a serious firmware bug in a previous version so make sure you end up with v0309 on the disk and update to it if you don’t.

Log Devices

Log devices (ZFS Intent Log) is a really cool way to speed up technologies which utilize synchronous writes.  Synchronous writes are used to ensure that data is committed to the disk before additional data is sent, it essentially serializes the connection.  Most commonly you will find synchronous writes in databases, NFS and iSCSI.  So if you are planning on using one of these technologies then you will want to consider using a mirrored ZIL device.  Now I am sure someone out there is saying, “Whoa mirrored?  But that takes up two slots man…” and you would be correct, however keep in mind that if you have a failure of your ZIL, you have actual data committed to it which has not been committed to disk (read: data loss).  As such you really ought to have a mirrored ZIL or none at all, you can also stack multiple mirrors as a ZIL if you had a higher synchronous writes workload.  Since log devices are meant to improve writes (albeit a certain type of write) then you will want to look for a SSD which is slanted towards better write performance.  I tend to like OCZ Vertex 3 for this purpose.

Factor Two – Data Growth

Now we get into actually sizing data.  When you try to project growth you need to make sure that (1) you take use an appropriately sized sample set of data (2) factor in any relevant business or technical factors which could have skewed your sample.  I like to size my storage based off of backup size.  Simply grab the size of your full backups for the data in question, if you are going to use this to serve block devices collecting this data becomes a bit more complex, because now you are talking more of a machine sprawl type of growth which can be harder to account for.  Here is are a few formulas to get you started (make sure you are using the same metric, i.e. B, KB, MB, GB, TB).

Calculate Weekly Growth

(full_week1 - full_week0) = growth_week1
(full_week2 - full_week1) = growth_week2

Calculate Weekly Growth Rate

(growth_week1 / full_week0) = growth_rate_week1

Average Weekly Growth

(growth_week1 + growth_week2 + growth_week3 + growth_week4) / number_of_weeks = average_weekly_growth

Average Weekly Growth Rate

(growth_rate_week1 + growth_rate_week2 + growth_rate_week3 + growth_rate_week4) / number_of_weeks = average_weekly_growth_rate

Annualized Weekly Growth

(average_weekly_growth * 53) = annual_growth

So for example if we start with this example…

Notice that in this example we have a couple of points where I saw it fit to make notes.  One was when our dataset decreased and the other was when it grew sharper than the trend up to that point.  I made these notes because I wanted to point out that if you had a very large deviation then you might want to make some sort of adjustment so as to not skew your results.  In this case both of these happened because of regular course of business so they should be legitimately considered as part of the growth curve.  However if you had some sort of failure in your backups that you were already accounting for that data in another way you would not want to double count it if it came back into the backups in the middle of your sample.

Weekly Growth

week1 (105 - 100) = 5
week2 (120 - 105) = 15
week3 (122 - 120) = 2
week4 (122 - 122) = 0
week5 (118 - 122) = -4
week6 (130 - 118) = 12
week7 (135 - 130) = 5
week8 (136 - 135) = 1

Weekly Growth Rate

week1 (5 / 100) = .05 = 5%
week2 (15 / 105) = .143 = 14.3%
week3 (2 / 120) = .017 = 1.7%
week4 (0 / 122) = 0 = 0%
week5 (-4 / 122) = -.033 = -3.3%
week6 (12 / 118) = .102 = 10.2%
week7 (5 / 130) = .039 = 3.9%
week8 (1 / 135) = .007 = .7%

Average Weekly Growth

(5 + 15 + 2 + 0 + -4 + 12 + 5 + 1) / 8 = 4.5

Average Weekly Growth Rate

(5 + 14.3 + 1.7 + 0 + -3.3 + 10.2 + 3.9 + .7) / 8 = 4.1%

Annualized Growth

(4.5 * 53) = 238.5

Now as you can see from this example the numbers add up quickly, in a year we have easily doubled our dataset, now why is this important?  Basically most businesses perform budgeting at the beginning of the year and as such expenses need to be planned out, and any expansion that you do perform will need to _at least_ last through the year, either way you at least need to know how long you can reasonably expect to be able to use this hardware in its capacity before having to look for upgrades.  Your project will be generally regarded as successful if it meets the technical requirements with minimal involvement from the business (having to ask for more money), especially if you can tell them that this storage will need to be expanded in x number of months.

Factor Three – Data Churn

Churn is freaking scary when it comes to calculating capacity requirements.  Churn is the amount of data change, now some churn is actually growth so we will want to keep that in mind when we are performing our analysis, however churn is the amount of data that changes in a given week. It is amazing the levels of churn that some companies have.  This is especially disconcerting if you plan on utilizing snapshots or send/receive as a form of backup.  Now we still use backups to calculate churn, however instead we will use our mid-week backups instead of our fulls.  Now if you are using incremental backups then you will total all of those during the week to get your dataset size.  If you are using differentials you can calculate this using the final differential in the week.

Now the tricky thing about churn is that growth is churn, but churn is not growth.  So after we calculate our growth and our churn we will subtract our growth from our churn to get our actual churn, otherwise we will be double counting our growth.  I don’t bother doing this until we are talking about the averages and the annualized numbers.  You will notice that these formulas are largely the same as the growth formulas, just remember her that you need to calculate these against your aggregated incrementals or your final differential to get valid numbers.

Calculate Weekly Churn (differentials)

diff_week1 = churn_week1

Calculate Weekly Churn (incrementals)

(inc1_week1 + inc2_week1 + inc3_week1 + inc4_week1 + inc5_week1) = churn_week1

Calculate Weekly Churn Rate

(churn_week1 / full_week0) = churn_rate_week1

Average Weekly Churn

(churn_week1 + churn_week2 + churn_week3 + churn_week4) / number_of_weeks = average_weekly_churn

Average Weekly Churn Rate

(churn_rate_week1 + churn_rate_week2 + churn_rate_week3 + churn_rate_week4) / number_of_weeks = average_weekly_churn_rate

Annualized Weekly Churn

(average_churn_growth * 53) = annual_churn

Expanding on our previous example, if we had churn rates that looked like this…

Weekly Churn Rate

week1 (8 / 100) = .08 = 8%
week2 (6 / 105) = .057 = 5.7%
week3 (5 / 120) = .042 = 4.2%
week4 (2 / 122) = .016 = 1.6%
week5 (2 / 122) = .016 = 1.6%
week6 (15 / 118) = .127 = 12.7%
week7 (6 / 130) = .046 = 4.6%
week8 (1 / 135) = .007 = .7%

Average Weekly Churn

(8 + 6 + 5 + 2 + 2 + 15 + 6 + 1) / 8 = 5.6

Now this is only partly correct, we still need to account for the growth we have already included  by subtracting the average growth from our average churn.

(5.6 - 4.5) = 1.1

Giving us a growth-adjusted weekly average churn of 1.1.

Average Weekly Churn Rate

(8 + 5.7 + 4.2 + 1.6 + 1.6 + 12.7 + 4.6 + .7) / 8 = 4.9%

Again we must remove our growth.

(4.9 - 4.5) = 0.4

Giving us a growth-adjusted weekly average churn of 0.4%

Annualized Churn

(1.1 * 53) = 58.3

Now that we have calculated our storage needs we can start to work out our final configurations based on the goals of our project.  In our example case we have learned that based on our current dataset (136) and our annualized growth (58.3), which means that in 1 year our dataset will increase by 30% which means that if we had a goal of engineering a system which would not need any upgrades (based on current use cases) in the first two years then you would need a minimum size of 252.6.

Now one final consideration when sizing ZFS is pool capacity, ZFS uses copy on write to turn random writes into sequential writes, this is very good for performance, however when a pool exceeds 80% ZFS will not be able to do this as well and as such your writes will become semi-random (since parts of files will have to be written in non-sequential blocks that happen to be free).  So we should also make sure that we have a 20% ceiling on our projections so that we can ensure the same level of performance throughout the systems life.  And with that we have a final number of 303.1.

Now please keep in mind, these formulas will allow you to work through your own sizing exercise, but I am not suggesting that you have a growth rate of 4.5% and a churn rate of 1.1% this was merely an example, you will need to use your own numbers to come up with projections which are applicable to your scenario.  Also you will notice that in my example I did not use a size metric (MB, GB, TB, PB) I did this intentionally so as to not confuse you these formulas will work regardless of your metric, just ensure you use the same metric in all of your calculations.

Happy Sizing!

However regardless of if it is necessary doesn’t mean that you should just do it…  One look at your auth.log will reveal that.

 # tail /var/log/auth.log
Feb  5 21:12:32 mail sshd[3105]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=69-214-200-46.pool.ukrtel.net
Feb  5 21:12:34 mail sshd[3105]: Failed password for invalid user admin from 46.200.214.69 port 1714 ssh2
Feb  5 21:12:35 mail sshd[3107]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=69-214-200-46.pool.ukrtel.net  user=root
Feb  5 21:12:38 mail sshd[3107]: Failed password for root from 46.200.214.69 port 1817 ssh2

The fact of that matter is that once you have SSH open folks will try and hit it and brute force there way in.  Now there are many ways around this.

1) Move SSH to a non-standard port number.
2) Disable Root Logins over SSH and use non-standard usernames.
3) Use fail2ban to proactively disconnect users who are attempting to brute force your server.
4) Use SSH keys to secure your logins, and disable all password authentication.

Now options 1 and 2 are really just garbage.  They don’t actually do anything with regards to security, they simply obfuscate your environment in the hopes that your attacker will give up and go home.  Option 3 is good, and option 4 is a sledge hammer which is crude in its implementation.

Instead I will be implementing a modified version of option 4.  What we will be doing is allowing Public Key authentication from the entire internet while allowing Password authentication from trusted IP space, this can be an entire IP block, or single IPs.  This relies on the match directive in ssh, so please make sure your version of ssh supports this before attempting.

Configure SSH Keys

# ssh-keygen -t rsa

Copy SSH Keys to your Server

# ssh-copy-id root@yourserver.domain

Validate SSH Key Authentication

If you did it properly then you will not be asked for a password.

# ssh root@yourserver.domain

Secure SSH to Only Allow Password Auth from Trusted Networks

Before you mess with this part ensure you have an alternate way of getting into the system in case you make a mistake which keeps you from using SSH.

In the /etc/ssh/sshd_config disable Password Authentication

# cat /etc/ssh/sshd_config
...
PasswordAuthentication no
...

Then add the following to the end of the file, where x.x.x.x and y.y.y.y are your trusted IP addresses, and /32 is used to represent a single IP address.

# cat /etc/ssh/sshd_config
...
Match Address x.x.x.x/24,y.y.y.y/32
Password Authentication yes

Validate your Configuration

Once everything is configured you can restart ssh and test.

# /etc/init.d/ssh restart

The first thing I did was examine the log files, while the service was still in a failed state.

# cat /var/log/vmware/vpx/vpxd.log

I won’t bore you with an entire excerpt, however I will put some of the more helpful pieces of data from the log file here, perhaps it will help someone identify this article if they are having a similar issue but not really gaining any traction.

Unable to get exclusive access to vCenter repository.
Error deleting from VPX_SESSIONL
Alert:false@ /build/mts/release/bora-455964/bora/vpx/vpxd/util/vpxdVdb.cpp:408
Registry Item DB 5 value is ''
Failed to intialize VMware VirtualCenter. Shutting down...

Now this environment is based off of SLES 11 SP1 with an embedded DB2 database.  This might affect other configurations, but I don’t know.  So use at your own risk.

Here is the VMWare KB which “describes” the fix…  But it is not very verbose.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021581

Here is the step-by-step.

Please keep in mind, this article contains two fixes for two similarly logged but different problems.  Fix one should only work if your service crashes and is unable to restart.  Fix two is actually what solved the problem that we were experiencing.

FIX ONE – Which did not work for me.

Gather the Database Connection Information

# cat /etc/vmware-vpx/embedded_db.cfg
EMB_DB_INSTALL_DIR='/opt/db2/current'
EMB_DB_HOME='/opt/db2/home/'
EMB_DB_TYPE='db2'
EMB_DB_SERVER='127.0.0.1'
EMB_DB_PORT='50000'
EMB_DB_INSTANCE='VCDB'
EMB_DB_USER='vc'
EMB_DB_PASSWORD='YOURPASSWORDHERE'

Stop the Service

# /etc/init.d/vmware-vpxd stop
Stopping VMware vSphere Profile-Driven Storage Service...
Stopped VMware vSphere Profile-Driven Storage Service.
Stopping VMware Inventory Service...
Stopped VMware Inventory Service.
Stopping tomcat: success
Stopping vmware-vpxd: success
Shutting down ldap-server..done

Connect to DB2 Database

# /opt/db2/v9.7.2/bin/db2
(c) Copyright IBM Corporation 1993,2007
Command Line Processor for DB2 Client 9.7.2

db2 => connect to vcdb user vc using YOURPASSWORDHERE

Database Connection Information

Database server        = DB2/LINUXX8664 9.7.2
SQL authorization ID   = VC
Local database alias   = VCDB

List Tables to Test Database Connection

db2 => list tables
...
VPX_SESSIONLOCK                 VC              T     2011-11-16-17.28.59.418535
...
287 record(s) selected.

Delete All Rows from VPX_SESSIONLOCK Table

db2 => delete from vpx_sessionlock
DB20000I  The SQL command completed successfully.

Quit and Restart Service

db2 => quit
# /etc/init.d/vmware-vpxd start
Waiting for embedded DB2 database to startup: .success Cleaning session lock table: success Verifying EULA acceptance: success Starting ldap-server..done Starting vmware-vpxd: success Waiting for vpxd to initialize: ....success Starting tomcat: success Executing startup scripts...
Starting VMware Inventory Service...Waiting for VMware Inventory Service........................
VMware Inventory Service started.

Starting VMware vSphere Profile-Driven Storage Service...Waiting for VMware vSphere Profile-Driven Storage Service......
VMware vSphere Profile-Driven Storage Service started.

 FIX TWO – Which did work for me.

Apparently the problem here is that db2 is configured with a transaction log which is too small, which is resulting in the service crashing.  Now according to VMWare’s KB [http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006812] you should expect errors indicating that the transaction log is full, however we did not see these at all.  So we really went out on a limb simply because logically transaction logs made sense.  Also VMWare recommends the sizes that we are going to be configuring the logs to in these steps, so it was relatively low risk.  Also keep in mind we have 3 esxi hosts and maybe 30 guests, so this can happen regardless of your size.

Stop the Service

# service vmware-vpxd stop

Change to db2inst1 and Connect to Database

# su db2inst1
db2inst1@vcenter00:~>db2 connect to vcdb

Retreive Current Database Configuration

db2inst1@vcenter00:~> db2 get database config for vcdb | grep LOG
Catalog cache size (4KB)              (CATALOGCACHE_SZ) = 300
Log buffer size (4KB)                        (LOGBUFSZ) = 256
Log file size (4KB)                         (LOGFILSIZ) = 1024
Number of primary log files                (LOGPRIMARY) = 13
Number of secondary log files               (LOGSECOND) = 4
Changed path to log files                  (NEWLOGPATH) =
Path to log files                                       = /storage/db/db2/home/db2inst1/db2inst1/NODE0000/SQL00001/SQLOGDIR/
Overflow log path                     (OVERFLOWLOGPATH) =
Mirror log path                         (MIRRORLOGPATH) =
Block log on disk full                (BLK_LOG_DSK_FUL) = NO
Block non logged operations            (BLOCKNONLOGGED) = NO
Percent max primary log space by transaction  (MAX_LOG) = 0
Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0
Log retain for recovery enabled             (LOGRETAIN) = OFF
First log archive method                 (LOGARCHMETH1) = OFF
Options for logarchmeth1                  (LOGARCHOPT1) =
Second log archive method                (LOGARCHMETH2) = OFF
Options for logarchmeth2                  (LOGARCHOPT2) =
Log pages during index build            (LOGINDEXBUILD) = OFF

Update Database Configuration

db2inst1@vcenter00:~> db2 update db CFG FOR VCDB USING logprimary 16 logsecond 112 logfilsiz 8192
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W  One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, all
applications must disconnect from this database before the changes become
effective.

Validate Database Configuration

db2inst1@vcenter00:~> db2 get database config for vcdb | grep LOG
Catalog cache size (4KB)              (CATALOGCACHE_SZ) = 300
Log buffer size (4KB)                        (LOGBUFSZ) = 256
Log file size (4KB)                         (LOGFILSIZ) = 8192
Number of primary log files                (LOGPRIMARY) = 16
Number of secondary log files               (LOGSECOND) = 112
Changed path to log files                  (NEWLOGPATH) =
Path to log files                                       = /storage/db/db2/home/db2inst1/db2inst1/NODE0000/SQL00001/SQLOGDIR/
Overflow log path                     (OVERFLOWLOGPATH) =
Mirror log path                         (MIRRORLOGPATH) =
Block log on disk full                (BLK_LOG_DSK_FUL) = NO
Block non logged operations            (BLOCKNONLOGGED) = NO
Percent max primary log space by transaction  (MAX_LOG) = 0
Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0
Log retain for recovery enabled             (LOGRETAIN) = OFF
First log archive method                 (LOGARCHMETH1) = OFF
Options for logarchmeth1                  (LOGARCHOPT1) =
Second log archive method                (LOGARCHMETH2) = OFF
Options for logarchmeth2                  (LOGARCHOPT2) =
Log pages during index build            (LOGINDEXBUILD) = OFF

Exit and Restart Service

db2inst1@vcenter00:~> exit
# service vmware-vpxd start

So there you go.  Hopefully one of these methods will help you resolve this issue in your environment.

# virsh destroy vmname

Which is a HARD power off of your guest.  This is the same as pulling the power cord from the wall, nothing is shutdown, but the noise stops just the same.  This method can result in data corruption if writes are interrupted.

# virsh shutdown vmname

Now in theory this will perform a graceful shutdown, or more importantly it sends a signal to the guest which results in the guest initiating a shutdown.  However this is not guaranteed, if your virtual machine configuration or guest OS is not configured to acknowledge or even receive these signals then nothing will happen.

Enable ACPI on the Virtual Hardware

Now the signal that is sent is an ACPI signal, this means that we need to configure the virtual machine to have acpi on the virtual processor.  This is accomplished by adding <acpi/> inside the features section of the virtual machines XML file.

This will print just the features block out of the XML file, it will not make any changes.

# awk '/<features>/,/<\/features>/' /etc/libvirt/qemu/vmname.xml
<features>
<acpi/>
<apic/>
<pae/>
</features>

Configure the Guest Operating System (Linux)

Linux is pretty simple, just make sure that acpid is installed.

Using APT

# apt-get install acpid

Using YUM

# yum install acpid

Configure the Guest Operating System (Windows)

Post Server 2000 Microsoft decided that by default servers should not respond to any idiot in a Datacenter with at least one finger.  Which is not an unreasonable assumption.  This had the unintended consequence of hamstringing virtualization, since obviously virtual machines don’t have the weakness of a power button which can be “bumped” by a one-fingered idiot.

I have seen different behavior based on the version of the OS you use, as I assume Microsoft has tried to make the behavior a bit more friendly and workable.  Basically as I have noticed.  Windows 2003 has to be logged on fully to have a virsh shutdown be successful.  Windows 2008 has to have the console open and the user has to have pressed CTRL-ALT-DEL, but not necessarily logged in.  Windows 2008 R2 simply needs the console open.

Regardless if you’d like to disable this protection on your VMs there is a registry change which can be made, of course this can also be made via a group policy.

HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\Shutdownwithoutlogon

0=disabled
1=enabled

This of course needs to be done on all of your Windows guests.

I am writing this article from my professional experiences, inside of a software company which provided enterprise support for its products, as well as from the perspective of the Enterprise Customer.  I am intentionally discarding all experiences with support as a home user, and even as a small business owner.  This is specific to Enterprise Support, consumer support plays by different rules.

Value Your Customers Time And Investments

Your customers have invested in your products, once a customer buys a product they are no longer customers, they are partners.  Your partners are extremely intelligent people who chose your product because of that aforementioned intelligence (if this is not true then why do you sell it?).  With regards to their investments your partner’s employees are also an investment, and if you waste their employee’s time, you are robbing your partner of productivity which will NEVER be regained.

This is real money folks.  If your partner has someone making $50/hour on a phone call with support and they spend 1 hour trying to resolve the issue this means that not only is your customer losing the $50 that they are paying them, but there is another invisible $50 which is lost due to the lost productivity.  Your customer is willing to deal with this IF you keep the waste to a minimum.

Don’t Make Support A Profit Center

Support should never be a profit center, once an organization switches support from being a cost center (something that costs the company money) to a profit center (something that makes the company money) the company will start to make decisions which will increase profitability and alienate customers.  This will result in more marketing dollars being spent to neutralize the ill-effects of your support organization.  Don’t misunderstand me, I don’t have a problem with paying for support, and I am not advocating that it be free or even cheap.  Ultimately the support organization should NEARLY pay for itself (unless of course there is some large recall required), what should not happen EVER, is customers should never be pressured to put a failed drive back in a system to see if it would rebuild.  This sort of thing might be common practice, but it shows the customer that we don’t value their data and their business.

Market The Organization Not The Product

Products come and products go…  A good support organization will actually pull in its own Enterprise Customers, I would submit that a portion of your marketing dollars could actually be spent on making critical changes in your support organization.  This approach is invaluable when your organization is small and flexible, and the products down sell themselves.

Once the changes are made and you have a phenomenal support organization then you should spend a little bit of your marketing budget on advertising your support organization, since from an Enterprise Customer perspective it is all about risk mitigation and cost containment.  Highlight this, show the world the changes that you have made.

Build Your Organization For Success

If you are a US-based company, you had better make sure that if you use any overseas support that they speak english really well (this also goes for really deep accents from anywhere).  I don’t have a problem if the support center is in India or Mexico or Singapore or wherever,  but if we cannot communicate clearly then forget it.  It won’t matter how good the support was all I will remember is straining to understand what you are saying.  Remember that when I call you, I am already having a bad day at best, don’t make it worse.  Your employees should always provide attentive customer service, it should be prompt and respectful.  However they should also be technically adept and there should be a clear and logical path of escalations.  If someone needs a script then let them go work for your competitor, your organization will be much better off.

These four components will create a support organization which will actually help sell your product, instead of giving customers a reason to complain about your company.