LabMice.net - The Windows 2000\XP\.NET Resource Index
Home | About Us | Search |

Last Updated December 10, 2003

Successfully Deploying Microsoft Service Packs

Service Packs are a compilation of bug fixes and updates to the code base of Microsoft's operating systems (Windows NT/2000/XP) or applications (Office, Exchange, SQL). They are suppose to improve stability and enhance security, but Microsoft's service packs have a been known to create some serious problems of their own. Before you jump right in and deploy the next service pack in your environment, take a minute to review our best practices for testing and deploying them.  If you simply need advice on installing a service pack on a standalone system, check out our article on Surviving a Service Pack
HomeService Pack > Installing and Managing Service Packs
 
  
Bug Pack?
Installing a service pack on a single workstation is usually not a big deal. But deploying a 100Mb service pack across a network consisting of hundreds or even thousands of workstations and servers can be huge undertaking teaming with potential pitfalls. Every administrator I know has horror stories of what Service Pack "X" did to their environment, and are a bit cautious (if not paranoid) whenever Microsoft releases a new Service Pack. In fact, some of Microsoft's service packs have been so bad, they've had to re-release them. Microsoft appears to be getting better at improving their coding process and testing service packs before release, but it's simply impossible to test the millions of combinations of hardware and software that exists in the real world. The keys to successfully deploying a service pack with your wits (and career) intact are relatively simple. Don't rush, don't take shortcuts, and heed the advice in this article!. 

Time is on your side - Learn from the mistakes of others
The early bird may get the worm, but it's the second mouse that gets the cheese. When it comes to service packs, it pays to wait at least 30 days (or even longer) to see what issues surface. There are very few advantages of being an early adopter, and the risk almost always outweigh the rewards. With the release of Windows 2000, Microsoft promised to make service packs a compilation of bug fixes only, and not add new features. Microsoft also integrated Windows Update into both server and desktop operating systems to help administrators keep up on hotfixes and other incremental code updates as needed. These two "advances" should greatly reduce the need to immediately deploy a service pack as soon as it's released. By waiting a few weeks (or even months) after a service pack is released, you'll have a better idea of what issues have been uncovered before you take a chance with your production systems. A 3 to 6 month waiting period should be sufficient time to uncover most of the known issues and allow review and testing in your own environment.

Identify Known Issues
Although service packs go through Beta releases and various testing cycles before they are released to the public, nobody can accurately test any software release against the million possible combinations of hardware and software that exist in the real world. Problems and bugs will arise. You should read all of the documentation that is available on a service pack, paying special attention to the release notes which are a list of all known bugs and issues identified with the service pack when it was released. In addition, you should check for all Knowledge Base articles at http://support.microsoft.com in order to identify issues that have surfaced after it was released. (To make your life easier, we compile a collection of support documents and known bugs for every Windows 2000 and Windows XP Service Pack at our Service Pack Resource Center) Specifically, you need to look for the following issues:

  • Hardware incompatibilities - These include RAID controllers, NIC cards, peripherals, specific BIOS or motherboard issues, display drivers, printers,  Hardware that is on Microsoft's hardware compatibility list (HCL) for each operating system should have been part of the service pack testing phase and should be less likely (theoretically) to have post release issues than other products.
  • Software issues - Because service packs update so many DLL's, registry entries, and other core system components, they also have the capability of causing issues with many applications including Oracle, Novell Clients, graphics programs, antivirus software and more.
  • Issues with application service packs - On occasion, Microsoft's operating system service packs have been known to wreck havoc on application service packs for SQL, Exchange, SMS, IIS, or other BackOffice products - and visa versa. In some cases, application service packs may need to be reinstalled after an OS service pack so be sure to check for these issues.
  • Issues with Domain Controllers - Changes and updates in group policy, replication, network components and other updates can cause a variety of issues with domain controllers, whether or not they are running Active Directory. 

Evaluate your need
Although it's nice to know that your systems are patched and updated properly, it may not always be necessary to deploy a service pack immediately simply because it exists. Microsoft provides a list of bugs fixed for each service pack, as well as release notes, and other documentation that should be reviewed carefully before deployment. Sure it's easier to install a service pack than the +1000 individual hot fixes that comprise it, but you should still take the time to evaluate what specific issues the service pack will resolve and how those issues impact your environment. If all of the known instabilities and security issues in your environment are covered by hot fixes that are already in place, your need to deploy the service pack shouldn't be that immediate. In fact, many of the large enterprise environments I've worked in are typically 1 or 2 service packs behind the current release. This practice may cause some issues if you ever call Microsoft's technical support, as their blanket answer to many support requests seems to "install the latest service pack" before they'll even begin to discuss the problem. 

Clearly, piling on hundreds of hotfixes is not an viable alternative to a service pack. Hotfixes don't go through the same testing process that applies to service packs. In some cases various hotfixes have been known to break each other and cause instabilities of their own. Microsoft occasionally re-releases a hotfix to address these issues, and resolves similar compatibility problems before they release a service pack. Microsoft may also incorporate a bug fix or security feature into a service pack that isn't made available as a hotfix or incremental update that may force your hand to upgrade. For example, Windows XP Service Pack 1 included a fix for a security flaw that was considered so critical by Microsoft, that it wasn't documented in the release notes or acknowledged until Leo Laporte from TechTV made it public. Until that point, Microsoft "strongly urged" users of XP to install SP1 and listed it as a "recommended" update instead of a "required" one. Waiting too long to deploy a service pack also affects your ability to install "post service pack hotfixes" that require the current service pack version.

The bottom line is that a service pack should do more good than harm to your environment. Rolling out a service pack is not a trivial issue in a large Enterprise class environment with thousands of servers and workstations. In smaller LANs, the task isn't as large, but a higher proportion of the servers in the environment are more likely to be mission critical.

Pre-deployment testing
Before rolling out a service pack into a production environment, you should test it on non production or non critical machines first. The goal here is to be proactive and flush out problems specific to your environment before they become a problem. The larger your environment, the more likely it is you'll uncover an issue in your deployment. Your risk may also increase for every piece of hardware that is not on the Hardware Compatibility List. The choice is simple. You can spend time testing on the front end, or repairing failed installations afterward. In our experience, the time commitment is about the same. Of course, testing occurs on your schedule, doesn't impact business productivity, and won't result in lost data. We also recognize that not every environment has a proper test lab that includes a cross section of the actual hardware in your
environment. But if you have these resources you should use them. 

  • Phase 1 - Start with an upgrade to a clean installation on a variety of hardware including servers, workstations, laptops, and systems that use special configurations such as multihomed NIC cards and multimonitor. Monitor the actual drive space being used by the service pack across different installations and consider running a few baseline performance tests as well. (Memory usage, boot time, disk performance, and CPU usage) You should test this for systems running with and without a previous service pack installed.
  • Phase 2 - Test clean installations of application servers including those running Exchange, SQL, Oracle, IIS, and Terminal Services. Microsoft's Service Packs also have a way of interfering with common third party tools such as Novell Client, VPN and dial up clients, antivirus software, and other applications. If your business depends on it, test it! 
  • Phase 3 - Try upgrading a replica of an existing production server by backing up the data and reinstalling it on identical hardware in the test lab. Norton Ghost and other drive imaging software are generally faster than Windows Backup, and can help speed up this process.
  • Phase 4 - Test a rudimentary back out plan in the event a service pack installation fails. This includes testing the service pack uninstall feature as well as your backup procedures.
  • Phase 5 - Upgrade a few non critical production systems first. Microsoft has a practice called "eating your own dog food" that I've seen adopted by many IT departments. Before rolling out a service pack or update to the production environment, the entire IT staff is updated first. This helps identify issues early and keeps potential problems "in house" where they won't damage your department's reputation.
  • Phase 6 - Consider running System Stress for Windows NT 4.0 and Windows 2000 for up to two weeks on a number of non-critical systems to test stability. (A System Stress CD is included with Microsoft Developer Network (MSDN) CD subscriptions).
Caution: 

Watch out for Service Packs that "step on" or break each other. Sometimes application service packs break each other (SQL and SMS), operating system service packs break application service packs (Windows 2000 and Exchange or SQL), or the combination of both service packs cause issues that are not present when they're used individually. You may also need to reinstall application service packs after installing an Operating system service pack. Be sure to check BOTH of the service pack's documentation as well as Microsoft's Knowledge Base for known conflicts 

Deployment Planning
Deploying a service pack in an Enterprise environment is not a trivial task. Microsoft's operating system service packs range in size from 87 - 130Mb, and can take 30 minutes or more to install on each workstation or server - if all goes well. Good planning and a careful phased deployment can make the process less painful and keep surprises to a minimum.  

  • Decide on a deployment method - You can deploy service packs a number of ways, including manual installations using a CD or network share, automated installations using SMS, or Group Policy and MSI files. Many companies use a combination of process, upgrading their workstations using an automated process and upgrading their servers manually. 

  • Develop and test your back out plan -  Hopefully, you've already done some rudimentary testing on your backup procedures, but you also need to develop and test a back out plan for every hardware platform in your environment. Make sure you actually test a fully functioning replica of the mission critical servers you've identified earlier. If your e-commerce server won't boot and you have to rebuild it from scratch, could your recover your customer data as well as unprocessed orders? Find out, your career may depend upon it.

  • Setup a Pilot program - After testing, you should run a practice deployment on a few non-critical systems. Consider upgrading lab servers, training PC's, and workstation belonging to IT staff. The goal is to flush out problems that may be unique to your environment before deploying the service pack to the general user community. You should test manual installations and any automated installation scripts

  • Identify critical systems - Most companies have more than a few systems which contain data or perform functions deemed to be "mission critical" These include e-mail, web servers, data processing, e-commerce, remote access servers, etc., that cannot afford to have any down time.  You'll need to identify a contact for each server in the event something goes wrong (Exchange administrators, DBA's, developers, etc,) and work out a schedule that accommodates the business.   

  • Keep all affected parties informed - As you plan your deployment, keep the users, management, help desk, and any other parties that could be affected by your deployment informed. Don't forget the network engineers if you're deploying a service pack across a LAN using an automated script or SMS. The amount of data being transferred in a large deployment can overwhelm network resources if you don't have the infrastructure to support it.

  • Start with the workstations - By workstations, we mean desktop systems, not laptops. If you're environment is configured correctly, all of your company's data should be stored on the servers where it's backed up on a regular basis and not on the workstations themselves. While this risk minimizes data loss, most companies have a greater variety of hardware platforms for desktops than servers, and you are more likely to uncover compatibility issues that generate help desk calls.

  • Upgrading Laptops - Laptops are a bit trickier than desktop systems as their hardware is usually proprietary and potentially very finicky  Testing should work out most of the bugs, but laptops can still be unpredictable. To make matters worse, laptop users usually store data locally, remote users aren't easy to support and troubleshoot over the phone, and automated deployments are almost impossible via a dial up line.  Be sure to implement a backup solution for all of your laptop users before upgrading them.

  • Server deployments - According to Microsoft's recommendations, all of your servers should be on the same service pack level whenever possible. This is especially critical of domain controllers and servers that interact with them like Exchange Server. Your deployment should start with the least complex and least critical servers and work up from there in an order similar to the following:

    1. File and print servers

    2. Non-critical application servers

    3. Internal web servers

    4. DNS/DHCP/WINS servers (test carefully)

    5. External web servers

    6. Critical application servers

    7. Mail servers

    8. Domain controllers

Performing the deployment
Preparation and communication are the keys to a successful deployment. A smooth rollout builds user and management confidence in the IT department, and makes your job easier in the long run. 

  • Keeping everyone informed - When you're deploying a service pack, you need to do more than simply send an e-mail. You'll need to coordinate the rollout with the business units in your company to make sure outages won't interrupt operations. Managers and users need to know how much downtime to expect, and what date and time you'll be upgrading their machines. The help desk needs to be informed and staffed properly to handle support requests. Make sure to track all service pack related calls so you can quickly identify trends or recurrent problems early in your deployment. You may consider dedicating one Level 3 person as a single point of contact for support requests and configuring a special support line to handle calls.

  • Inventory machines to establish eligibility - Disk space may be an issue on some workstations and servers that can cause your service pack installations to fail or proceed slowly. Low system memory can also slow an installation, or create problems with automated installations. You can use SMS to inventory your environment before the deployment, or consider adding a component to your automated installation script that checks for free disk space on C:\ and reports back to a central log file if it fails.

  • Upgrading Workstations - If you have more than a few workstations in your environment, you'll want to upgrade them gradually over several days so your help desk won't be swamped. In large enterprise environments, I've started with as little as 25-50 workstations per day working up to 200-300 within a week (using SMS) to ensure smooth rollout. This gradual approach helps you estimate bandwidth, identify problems unique to your environment, and anticipate call volume to the help desk. If your company's PCs have the ability to "wake" on command, the deployments can be performed at night or on a weekend to minimize downtime and bandwidth utilization. Some companies prefer to deploy service packs to workstations on weekends, using a CD-ROM and assigning 10-20 PC's per IT staffer. While this process is slower, you can address installation issues immediately instead of having an irate user calling the help desk the following morning because his PC won't boot.

  • Upgrading Servers - Server upgrades should always be done manually whenever possible. You should backup every server immediately before upgrading it, or plan your upgrades to follow the routine backup schedule. In addition, you should update the startup disks and backup the system state immediately before beginning the service pack upgrade. To save time, you can upgrade several servers at once but you need to be prepared for the worst. (You never know what will happen when you reboot a server that hasn't been turned off in months) If I'm upgrading more than a few servers, I like to keep a spare hard drive or 2 hanging around as well as a power supply. (They can be hard to get on a weekend in the middle of the night.) Have all of the resources you need to rebuild a server at your fingertips, including the software that runs on the server and access to the backup media.

Service Pack Installation Tips

Verify the Installation Source 
If at all possible, download the entire network install of the service pack or use the CD that comes with a TechNet subscription instead of using Windows Update on every machine. If you make copies of the service pack CD, or create CD's based on the download, verify the integrity of the disk against the original before you start using it for your deployment. If your using a network installation share, be sure that the source server and network infrastructure can handle the bandwidth requirements based on the estimated number of simultaneous installs. (You may be surprised to find that sometimes CD's work faster than network deployments - depending on your network infrastructure and system CD speed.)
Shut down unnecessary services 
Microsoft recommends that you disable ALL third party services, especially any service that accesses the file system. This includes antivirus scanners, quota managers, and scheduling programs. In addition, you should set all Microsoft application services to manual. This includes SQL Services, Exchange Services, and Spooler on dedicated print servers. 
Perform a clean reboot before installing the service pack  
If you haven't rebooted your server or workstation in a while, you'll want to restart it before installing a service pack. This will close any persistent processes or memory leaks, and allows you to check for startup errors or other issues. It will also rebuild the systems "last known good configuration" in the event your service pack install fails. 
Create an uninstall directory
All operating system service packs since Windows 2000 Service Pack 1 automatically prompt you to create an uninstall directory. You should let the installation create this directory and leave it intact even after the service pack is up and running. If you install programs or drivers that may not be compatible with the Service Pack, you may need to uninstall the Service Pack.
Update the ERD 
If your installation was successful, you'll need to update your Emergency Repair Disk for each server to make sure you can recover the system if it becomes unstable. 

 

  

Send us your feedback!
If you have any questions, comments, or suggestions that would help us improve this page, please drop us a line and let us know!


Dell Business Weekly Promo

Original publication date: October 22, 2002

This site and its contents are Copyright 1999-2003 by LabMice.net. Microsoft, NT, BackOffice, MCSE, and Windows are registered trademarks of Microsoft Corporation. Microsoft Corporation in no way endorses or is affiliated with LabMice.net. The products referenced in this site are provided by parties other than LabMice.net. LabMice.net makes no representations regarding either the products or any information about the products. Any questions, complaints, or claims regarding the products must be directed to the appropriate manufacturer or vendor.