The High Cost of Testing Software On Production Servers
Overview
Testing software is a great way to make sure you don’t run into issues prior to fully deploying it. From hotfixes, service packs and new software, it is a good idea to test software before you unleash it on your workstations or servers. The problem comes in when you want to run a test on a production server. As you’ll see from this article, there is a high cost of testing software on production servers.
First we’ll cover the potential costs and risk of testing software along with some useful ideas for setting up a proper test environment.
Case Example
Let’s walk through a fictional case to better examine what could go wrong and what could be done to avoid an outage. Let’s say you’ve got an application server that is running your external website. You decide to test a new web checkout application on your production web server. You install the new piece of software but find that it requires a newer version of the Microsoft .NET framework. You install the latest framework and reboot. You find that your website no longer works due to the .NET framework install.
The clock starts ticking. Your external website is down and people are starting to notice. In an attempt to fix the issue, you uninstall the new application and uninstall the latest .NET Framework. You reboot again but find that your website still won’t come back up. You guess that your website must be expecting to use an older version of the .NET Framework. You do some searching and find how to tell IIS to use a specific version of the .NET Framework. After all is said and done, your site was down for a few hours, your boss is pretty upset with you and your customers think a bit less of your company. Not a good impression on you or your company.
Keep reading to find out what went wrong and how to avoid these types of issues in the future.
Cost of Testing Software on Production Servers
Below are some of the biggest costs to testing software on production servers:
Downtime - If your testing takes a turn for the worse, you could bring down an entire system or multiple systems. In the case above, the administrator was lucky the site was only down for a few hours. What if the installed software caused the operating system to fault? Downtime could have been much worse.
Man hours – testing itself can take a long time if done properly, but this can be done on your own schedule. If your testing causes a production outage, it’s ‘all hands aboard’ to get the issue resolved - resulting in a loss of productivity for you and company staff.
Data corruption – installing software on production systems can sometimes lead to hard-to-find issues like data corruption or driver conflicts. The end result is that you may have bad data floating around without your even knowing it.
Performance – without proper testing, performance can suffer. Software like anti-malware utilities are notorious for using precious CPU and memory and without proper tuning can bring a system down to its knees.
Reputation – not only are there hard costs to tests gone wrong, but there are soft costs as well. A bad test result on a production system can damage you and your company’s reputation. In the case described, the external ‘face’ of the company was unavailable for a few hours – certainly not a good way to impress current and potential customers.
Setting Up A Test Environment
In order to properly test, you need to set up a test environment and develop a thorough test plan. If you are short on hardware and time, you can set up a virtual test environment using Hyper-V, VMWare or a free virtual solution called VirtualBox from Sun Microsystems.
If you choose to use VMWare, you can use the free VMWare vCenter Converter to create a virtual clone of a physical server. This saves a lot of time and ensures you have an identical copy of your production machine.
If you choose to use a free solution like Virtualbox, there are some available guides on Brighthub that will walk you through installing Windows Server 2008 and Microsoft SQL 2008 in VirtualBox.
In addition to creating your test environment, you will need to create a test plan. I like to create a spreadsheet with one worksheet for each application – Exchange, SQL, etc. On each worksheet I create a detailed list of functions to test. Go through each feature and sign off with the results – whether your test passed or failed and if it failed any comments related to the result. Make sure any failed tests are properly mitigated before deploying the software to your production environment.
Hopefully by following this advice and setting up a test environment and test plan you’ll save yourself time and cost by properly testing software before installing it into production.
