Successfully Deploying Microsoft Service Packs

Identify Known Issues
Although service packs go through Beta releases and various testing cycles before they are released to the public, nobody can accurately test any software release against the million possible combinations of hardware and software that exist in the real world. Problems and bugs will arise. You should read all of the documentation that is available on a service pack, paying special attention to the release notes which are a list of all known bugs and issues identified with the service pack when it was released. In addition, you should check for all Knowledge Base articles at http://support.microsoft.com in order to identify issues that have surfaced after it was released. (To make your life easier, we compile a collection of support documents and known bugs for every Windows 2000 and Windows XP Service Pack at our Service Pack Resource Center) Specifically, you need to look for the following issues:

• Hardware incompatibilities - These include RAID controllers, NIC cards, peripherals, specific BIOS or motherboard issues, display drivers, printers,  Hardware that is on Microsoft's hardware compatibility list (HCL) for each operating system should have been part of the service pack testing phase and should be less likely (theoretically) to have post release issues than other products.
• Software issues - Because service packs update so many DLL's, registry entries, and other core system components, they also have the capability of causing issues with many applications including Oracle, Novell Clients, graphics programs, antivirus software and more.
• Issues with application service packs - On occasion, Microsoft's operating system service packs have been known to wreck havoc on application service packs for SQL, Exchange, SMS, IIS, or other BackOffice products - and visa versa. In some cases, application service packs may need to be reinstalled after an OS service pack so be sure to check for these issues.
• Issues with Domain Controllers - Changes and updates in group policy, replication, network components and other updates can cause a variety of issues with domain controllers, whether or not they are running Active Directory. 

Evaluate your need
Although it's nice to know that your systems are patched and updated properly, it may not always be necessary to deploy a service pack immediately simply because it exists. Microsoft provides a list of bugs fixed for each service pack, as well as release notes, and other documentation that should be reviewed carefully before deployment. Sure it's easier to install a service pack than the +1000 individual hot fixes that comprise it, but you should still take the time to evaluate what specific issues the service pack will resolve and how those issues impact your environment. If all of the known instabilities and security issues in your environment are covered by hot fixes that are already in place, your need to deploy the service pack shouldn't be that immediate. In fact, many of the large enterprise environments I've worked in are typically 1 or 2 service packs behind the current release. This practice may cause some issues if you ever call Microsoft's technical support, as their blanket answer to many support requests seems to "install the latest service pack" before they'll even begin to discuss the problem. 

Clearly, piling on hundreds of hotfixes is not an viable alternative to a service pack. Hotfixes don't go through the same testing process that applies to service packs. In some cases various hotfixes have been known to break each other and cause instabilities of their own. Microsoft occasionally re-releases a hotfix to address these issues, and resolves similar compatibility problems before they release a service pack. Microsoft may also incorporate a bug fix or security feature into a service pack that isn't made available as a hotfix or incremental update that may force your hand to upgrade. For example, Windows XP Service Pack 1 included a fix for a security flaw that was considered so critical by Microsoft, that it wasn't documented in the release notes or acknowledged until Leo Laporte from TechTV made it public. Until that point, Microsoft "strongly urged" users of XP to install SP1 and listed it as a "recommended" update instead of a "required" one. Waiting too long to deploy a service pack also affects your ability to install "post service pack hotfixes" that require the current service pack version.

The bottom line is that a service pack should do more good than harm to your environment. Rolling out a service pack is not a trivial issue in a large Enterprise class environment with thousands of servers and workstations. In smaller LANs, the task isn't as large, but a higher proportion of the servers in the environment are more likely to be mission critical.

Pre-deployment testing
Before rolling out a service pack into a production environment, you should test it on non production or non critical machines first. The goal here is to be proactive and flush out problems specific to your environment before they become a problem. The larger your environment, the more likely it is you'll uncover an issue in your deployment. Your risk may also increase for every piece of hardware that is not on the Hardware Compatibility List. The choice is simple. You can spend time testing on the front end, or repairing failed installations afterward. In our experience, the time commitment is about the same. Of course, testing occurs on your schedule, doesn't impact business productivity, and won't result in lost data. We also recognize that not every environment has a proper test lab that includes a cross section of the actual hardware in your environment. But if you have these resources you should use them. 

• Phase 1 - Start with an upgrade to a clean installation on a variety of hardware including servers, workstations, laptops, and systems that use special configurations such as multihomed NIC cards and multimonitor. Monitor the actual drive space being used by the service pack across different installations and consider running a few baseline performance tests as well. (Memory usage, boot time, disk performance, and CPU usage) You should test this for systems running with and without a previous service pack installed.
• Phase 2 - Test clean installations of application servers including those running Exchange, SQL, Oracle, IIS, and Terminal Services. Microsoft's Service Packs also have a way of interfering with common third party tools such as Novell Client, VPN and dial up clients, antivirus software, and other applications. If your business depends on it, test it! 
• Phase 3 - Try upgrading a replica of an existing production server by backing up the data and reinstalling it on identical hardware in the test lab. Norton Ghost and other drive imaging software are generally faster than Windows Backup, and can help speed up this process.
• Phase 4 - Test a rudimentary back out plan in the event a service pack installation fails. This includes testing the service pack uninstall feature as well as your backup procedures.
• Phase 5 - Upgrade a few non critical production systems first. Microsoft has a practice called "eating your own dog food" that I've seen adopted by many IT departments. Before rolling out a service pack or update to the production environment, the entire IT staff is updated first. This helps identify issues early and keeps potential problems "in house" where they won't damage your department's reputation.
• Phase 6 - Consider running System Stress for Windows NT 4.0 and Windows 2000 for up to two weeks on a number of non-critical systems to test stability. (A System Stress CD is included with Microsoft Developer Network (MSDN) CD subscriptions).

Caution:  Watch out for Service Packs that "step on" or break each other. Sometimes application service packs break each other (SQL and SMS), operating system service packs break application service packs (Windows 2000 and Exchange or SQL), or the combination of both service packs cause issues that are not present when they're used individually. You may also need to reinstall application service packs after installing an Operating system service pack. Be sure to check BOTH of the service pack's documentation as well as Microsoft's Knowledge Base for known conflicts

Deployment Planning
Deploying a service pack in an Enterprise environment is not a trivial task. Microsoft's operating system service packs range in size from 87 - 130Mb, and can take 30 minutes or more to install on each workstation or server - if all goes well. Good planning and a careful phased deployment can make the process less painful and keep surprises to a minimum.  

• Decide on a deployment method - You can deploy service packs a number of ways, including manual installations using a CD or network share, automated installations using SMS, or Group Policy and MSI files. Many companies use a combination of process, upgrading their workstations using an automated process and upgrading their servers manually. 

• Develop and test your back out plan -  Hopefully, you've already done some rudimentary testing on your backup procedures, but you also need to develop and test a back out plan for every hardware platform in your environment. Make sure you actually test a fully functioning replica of the mission critical servers you've identified earlier. If your e-commerce server won't boot and you have to rebuild it from scratch, could your recover your customer data as well as unprocessed orders? Find out, your career may depend upon it.

• Setup a Pilot program - After testing, you should run a practice deployment on a few non-critical systems. Consider upgrading lab servers, training PC's, and workstation belonging to IT staff. The goal is to flush out problems that may be unique to your environment before deploying the service pack to the general user community. You should test manual installations and any automated installation scripts

• Identify critical systems - Most companies have more than a few systems which contain data or perform functions deemed to be "mission critical" These include e-mail, web servers, data processing, e-commerce, remote access servers, etc., that cannot afford to have any down time.  You'll need to identify a contact for each server in the event something goes wrong (Exchange administrators, DBA's, developers, etc,) and work out a schedule that accommodates the business.   

• Keep all affected parties informed - As you plan your deployment, keep the users, management, help desk, and any other parties that could be affected by your deployment informed. Don't forget the network engineers if you're deploying a service pack across a LAN using an automated script or SMS. The amount of data being transferred in a large deployment can overwhelm network resources if you don't have the infrastructure to support it.

• Start with the workstations - By workstations, we mean desktop systems, not laptops. If you're environment is configured correctly, all of your company's data should be stored on the servers where it's backed up on a regular basis and not on the workstations themselves. While this risk minimizes data loss, most companies have a greater variety of hardware platforms for desktops than servers, and you are more likely to uncover compatibility issues that generate help desk calls.

• Upgrading Laptops - Laptops are a bit trickier than desktop systems as their hardware is usually proprietary and potentially very finicky  Testing should work out most of the bugs, but laptops can still be unpredictable. To make matters worse, laptop users usually store data locally, remote users aren't easy to support and troubleshoot over the phone, and automated deployments are almost impossible via a dial up line.  Be sure to implement a backup solution for all of your laptop users before upgrading them.

• Server deployments - According to Microsoft's recommendations, all of your servers should be on the same service pack level whenever possible. This is especially critical of domain controllers and servers that interact with them like Exchange Server. Your deployment should start with the least complex and least critical servers and work up from there in an order similar to the following:

  1. File and print servers

  2. Non-critical application servers

  3. Internal web servers

  4. DNS/DHCP/WINS servers (test carefully)

  5. External web servers

  6. Critical application servers

  7. Mail servers

  8. Domain controllers

Performing the deployment
Preparation and communication are the keys to a successful deployment. A smooth rollout builds user and management confidence in the IT department, and makes your job easier in the long run. 

• Keeping everyone informed - When you're deploying a service pack, you need to do more than simply send an e-mail. You'll need to coordinate the rollout with the business units in your company to make sure outages won't interrupt operations. Managers and users need to know how much downtime to expect, and what date and time you'll be upgrading their machines. The help desk needs to be informed and staffed properly to handle support requests. Make sure to track all service pack related calls so you can quickly identify trends or recurrent problems early in your deployment. You may consider dedicating one Level 3 person as a single point of contact for support requests and configuring a special support line to handle calls.

• Inventory machines to establish eligibility - Disk space may be an issue on some workstations and servers that can cause your service pack installations to fail or proceed slowly. Low system memory can also slow an installation, or create problems with automated installations. You can use SMS to inventory your environment before the deployment, or consider adding a component to your automated installation script that checks for free disk space on C:\ and reports back to a central log file if it fails.

• Upgrading Workstations - If you have more than a few workstations in your environment, you'll want to upgrade them gradually over several days so your help desk won't be swamped. In large enterprise environments, I've started with as little as 25-50 workstations per day working up to 200-300 within a week (using SMS) to ensure smooth rollout. This gradual approach helps you estimate bandwidth, identify problems unique to your environment, and anticipate call volume to the help desk. If your company's PCs have the ability to "wake" on command, the deployments can be performed at night or on a weekend to minimize downtime and bandwidth utilization. Some companies prefer to deploy service packs to workstations on weekends, using a CD-ROM and assigning 10-20 PC's per IT staffer. While this process is slower, you can address installation issues immediately instead of having an irate user calling the help desk the following morning because his PC won't boot.

• Upgrading Servers - Server upgrades should always be done manually whenever possible. You should backup every server immediately before upgrading it, or plan your upgrades to follow the routine backup schedule. In addition, you should update the startup disks and backup the system state immediately before beginning the service pack upgrade. To save time, you can upgrade several servers at once but you need to be prepared for the worst. (You never know what will happen when you reboot a server that hasn't been turned off in months) If I'm upgrading more than a few servers, I like to keep a spare hard drive or 2 hanging around as well as a power supply. (They can be hard to get on a weekend in the middle of the night.) Have all of the resources you need to rebuild a server at your fingertips, including the software that runs on the server and access to the backup media.