|
Bug
Pack?
Installing a
service pack on a single workstation is usually not a
big deal. But deploying a 100Mb service pack across a
network consisting of hundreds or even thousands of
workstations and servers can be huge undertaking teaming
with potential pitfalls. Every administrator I know
has horror stories of what Service Pack "X"
did to their environment, and are a bit cautious (if not
paranoid) whenever Microsoft releases a new Service
Pack. In fact, some of Microsoft's service packs have
been so bad, they've had to re-release them. Microsoft
appears to be getting better at improving their
coding process and testing service packs before release,
but it's simply impossible to test the millions of
combinations of hardware and software that exists in the
real world. The keys to successfully deploying a service pack
with your wits (and career) intact are
relatively simple. Don't rush, don't take shortcuts, and
heed the advice in this article!.
Time is on your
side - Learn from the mistakes of others
The early bird may get the worm, but it's the second
mouse that gets the cheese. When it comes to service
packs, it pays to wait at least 30 days (or even longer)
to see what issues surface. There are very few advantages of
being an early adopter, and the risk almost always
outweigh the rewards. With the release of Windows 2000, Microsoft promised to
make service packs a compilation of bug fixes only, and
not add new features. Microsoft also integrated Windows
Update into both server and desktop operating systems to
help administrators keep up on hotfixes and other incremental
code updates as needed. These two "advances"
should greatly reduce the need to immediately deploy a
service pack as soon as it's released. By waiting a few
weeks (or even months) after a service pack is released,
you'll have a better idea of what issues have been
uncovered before you take a chance with your production
systems. A 3 to 6 month waiting period should be
sufficient time to uncover most of the known issues and
allow review and testing in your own environment.
|
|
Identify Known
Issues
Although service packs go through Beta releases
and various testing cycles before they are released to
the public, nobody can accurately test any software
release against the million possible combinations of
hardware and software that exist in the real world.
Problems and bugs will arise. You should read all of
the documentation that is available on a service
pack, paying special attention to the release notes
which are a list of all known bugs and issues identified
with the service pack when it was released. In addition,
you should check for all Knowledge Base articles at http://support.microsoft.com
in order to identify issues
that have surfaced after it was released. (To make your
life easier, we compile a collection of support documents
and known bugs for every Windows 2000 and Windows XP
Service Pack at our Service
Pack Resource Center) Specifically, you need to look
for the following issues:
- Hardware
incompatibilities - These include RAID
controllers, NIC cards, peripherals, specific BIOS
or motherboard issues, display drivers,
printers, Hardware that is on Microsoft's
hardware compatibility list (HCL) for each
operating system should have been part of the
service pack testing phase and should be less
likely (theoretically) to have post release issues
than other products.
- Software issues
- Because service packs update so many DLL's,
registry entries, and other core system
components, they also have the capability of
causing issues with many applications including
Oracle, Novell Clients, graphics programs,
antivirus software and more.
- Issues with
application service packs - On occasion,
Microsoft's operating system service packs have
been known to wreck havoc on application service
packs for SQL, Exchange, SMS, IIS, or other
BackOffice products - and visa versa. In some
cases, application service packs may need to be
reinstalled after an OS service pack so be sure to
check for these issues.
- Issues with
Domain Controllers
- Changes and updates in group policy,
replication, network components and other updates
can cause a variety of issues with domain
controllers, whether or not they are running
Active Directory.
Evaluate your need
Although it's nice to know that your systems are patched
and updated properly, it may not always be necessary to
deploy a service pack immediately simply because it
exists. Microsoft provides a list of bugs fixed for each
service pack, as well as release notes, and other
documentation that should be reviewed carefully before
deployment. Sure it's easier to install a service pack
than the +1000
individual hot fixes that comprise it, but you
should still take the time to evaluate what specific
issues the service pack will resolve and how those
issues impact your environment. If all of the known
instabilities and security issues in your environment
are covered by hot fixes that are already in place, your need to deploy the
service pack shouldn't be that immediate. In fact, many
of the large enterprise environments I've worked in are
typically 1 or 2 service packs behind the current
release. This practice may cause some issues if you ever
call Microsoft's technical support, as their
blanket answer to many support requests seems to
"install the latest service pack" before
they'll even begin to discuss the problem.
Clearly, piling on
hundreds of hotfixes is not an viable alternative to a
service pack. Hotfixes don't go through the same testing process that applies to
service packs. In some cases various hotfixes have been
known to break each other and cause instabilities of
their own. Microsoft occasionally re-releases a hotfix
to address these issues, and resolves similar
compatibility problems before they release a service
pack. Microsoft may also incorporate a bug fix or
security feature into a service pack that isn't made
available as a hotfix or incremental update that may
force your hand to upgrade. For example, Windows XP
Service Pack 1 included a fix for a security flaw that
was considered so critical by Microsoft, that it wasn't
documented in the release notes or acknowledged until
Leo Laporte from TechTV made it public. Until that
point, Microsoft "strongly urged" users of XP
to install SP1 and listed it as a
"recommended" update instead of a
"required" one. Waiting too long to deploy a
service pack also affects your ability to install
"post service pack hotfixes" that require the
current service pack version.
The bottom line is that a
service pack should do more good than harm to your
environment. Rolling out a service pack is not a trivial
issue in a large Enterprise class environment with
thousands of servers and workstations. In smaller LANs,
the task isn't as large, but a higher proportion of the
servers in the environment are more likely to be mission
critical.
Pre-deployment testing
Before rolling out a service pack into a production
environment, you should test it on non production or non
critical machines first. The goal here is to be
proactive and flush out problems specific to your
environment before they become a problem. The larger your
environment, the more likely it is you'll uncover an
issue in your deployment. Your risk may also increase for every
piece of hardware that is not on the Hardware
Compatibility List. The choice is simple. You can spend time testing on
the front end, or repairing failed installations
afterward. In our experience, the time commitment is about the same. Of
course, testing occurs on your schedule, doesn't impact business productivity,
and won't result in lost data. We also recognize that not every environment has a
proper test lab that includes a cross section of the actual
hardware in your environment.
But if you have these resources you should use
them.
- Phase 1 -
Start with an upgrade to a clean installation on a
variety of hardware including servers,
workstations, laptops, and systems that use
special configurations such as multihomed NIC
cards and multimonitor. Monitor the actual drive
space being used by the service pack across
different installations and consider running a few
baseline performance tests as well. (Memory usage,
boot time, disk performance, and CPU usage) You
should test this for systems running with and
without a previous service pack installed.
- Phase 2 -
Test clean installations of application servers
including those running Exchange, SQL, Oracle, IIS, and
Terminal Services. Microsoft's Service Packs also
have a way of interfering with common third party
tools such as Novell Client, VPN and dial up
clients, antivirus software, and other
applications. If your business depends on it, test
it!
- Phase 3 - Try
upgrading a replica of an existing production
server by backing up the data and reinstalling it
on identical hardware in the test lab. Norton
Ghost and other drive imaging software are
generally faster than Windows Backup, and can help
speed up this process.
- Phase 4 -
Test a rudimentary back out plan in the event a service
pack installation fails. This includes testing the
service pack uninstall feature as well as your
backup procedures.
- Phase 5 -
Upgrade a few non critical production systems
first. Microsoft has a practice called
"eating your own dog food" that I've
seen adopted by many IT departments. Before
rolling out a service pack or update to the
production environment, the entire IT staff is
updated first. This helps identify issues early
and keeps potential problems "in house"
where they won't damage your department's
reputation.
- Phase 6 -
Consider running System Stress for Windows NT 4.0
and Windows 2000 for up to two weeks on a
number of non-critical systems to test stability.
(A System Stress CD is included with Microsoft
Developer Network (MSDN) CD subscriptions).
 |
Caution: |
|
|
Watch
out for Service Packs that "step
on" or break each other. Sometimes
application service packs break each other
(SQL and SMS), operating system service
packs break application service packs
(Windows 2000 and Exchange or SQL), or the
combination of both service packs cause
issues that are not present when they're
used individually. You may also need to
reinstall application service packs after
installing an Operating system service
pack. Be sure to check BOTH of the service
pack's documentation as well as
Microsoft's Knowledge Base for known
conflicts |
|
|
Deployment Planning
Deploying a service pack in an Enterprise
environment is not a trivial task. Microsoft's
operating system service packs range in size from 87 -
130Mb, and can take 30 minutes or more to install on
each workstation or server - if all goes well. Good
planning and a careful phased deployment can make the process
less painful and keep surprises to a minimum.
-
Decide on a
deployment method - You can deploy service
packs a number of ways, including manual
installations using a CD or network share, automated installations using
SMS, or Group Policy and MSI
files. Many companies use a combination of
process, upgrading their workstations using an
automated process and upgrading their servers
manually.
-
Develop and test
your back out plan - Hopefully, you've
already done some rudimentary testing on your backup
procedures, but you also need to develop and test
a back out plan for every
hardware platform in your environment. Make sure
you actually test a fully functioning replica of the mission critical servers
you've identified earlier. If your e-commerce
server won't boot and you have to rebuild it from
scratch, could your recover your customer data as
well as unprocessed orders? Find out, your career
may depend upon it.
-
Setup
a Pilot program - After testing, you should
run a practice deployment on a few non-critical
systems. Consider upgrading lab servers, training
PC's, and workstation belonging to IT staff. The goal is to flush out problems
that may be unique to your environment before deploying
the service pack to the general user community.
You should test manual installations and any
automated installation scripts
-
Identify critical
systems - Most companies have more than a few
systems which contain data or perform functions
deemed to be "mission critical" These
include e-mail, web servers, data processing,
e-commerce, remote access servers, etc., that
cannot afford to have any down time. You'll
need to identify a contact for each server in the
event something goes wrong (Exchange
administrators, DBA's, developers, etc,) and work
out a schedule that accommodates the
business.
-
Keep all affected
parties informed - As you plan your
deployment, keep the users, management, help desk,
and any other parties that could be affected by
your deployment informed. Don't forget the network
engineers if you're deploying a service pack across
a LAN using an automated script or SMS. The
amount of data being transferred in a large
deployment can overwhelm network resources if you
don't have the infrastructure to support it.
-
Start with the
workstations - By workstations, we mean
desktop systems, not laptops. If you're
environment is configured correctly, all of your
company's data should be stored on the servers
where it's backed up on a regular basis and not on
the workstations themselves. While this risk
minimizes data loss, most companies have a greater
variety of hardware platforms for desktops than
servers, and you are more likely to uncover
compatibility issues that generate help desk
calls.
-
Upgrading
Laptops - Laptops are a bit trickier than
desktop systems as their hardware is usually
proprietary and potentially very finicky
Testing should work out most of the bugs, but
laptops can still be unpredictable. To make
matters worse, laptop users usually store data locally,
remote users aren't easy to support and
troubleshoot over the phone, and automated
deployments are almost impossible via a dial up
line. Be sure to implement a backup solution
for all of your laptop users before upgrading
them.
-
Server deployments
- According to Microsoft's recommendations, all of
your servers should be on the same service pack
level whenever possible. This is especially
critical of domain controllers and servers that
interact with them like Exchange Server. Your
deployment should start with the least complex and
least critical servers and work up from there in
an order similar to the following:
-
File and print
servers
-
Non-critical
application servers
-
Internal web
servers
-
DNS/DHCP/WINS
servers (test carefully)
-
External web
servers
-
Critical
application servers
-
Mail servers
-
Domain
controllers
Performing
the deployment
Preparation and communication are the keys to a
successful deployment.
A smooth rollout builds user and management confidence in
the IT department, and makes your job easier in the long
run.
-
Keeping
everyone informed - When you're deploying a
service pack, you need to do more than simply send
an e-mail. You'll need to coordinate the rollout
with the business units in your company to make sure
outages won't interrupt operations. Managers and
users need to know how much downtime to expect,
and what date and time you'll be upgrading their machines.
The help desk needs to be informed and staffed
properly to handle support requests. Make sure to
track all service pack related calls so you can
quickly identify trends or recurrent problems early
in your deployment. You may consider dedicating
one Level 3 person as a single point
of contact for support requests and configuring a
special support line to handle calls.
-
Inventory
machines to establish eligibility - Disk space
may be an issue on some workstations and servers
that can cause your service pack installations to
fail or proceed slowly. Low system memory can also
slow an installation, or create problems with
automated installations. You can use SMS to
inventory your environment before the deployment,
or consider adding a component to your automated
installation script that checks for free disk
space on C:\ and reports back to a central log
file if it fails.
-
Upgrading
Workstations - If you have more than a few
workstations in your environment, you'll want to
upgrade them gradually over several days so your
help desk won't be swamped. In large enterprise environments,
I've started with as little as 25-50 workstations
per day working up to 200-300 within a week (using
SMS) to ensure smooth rollout. This gradual approach
helps you estimate bandwidth, identify problems
unique to your environment, and anticipate call
volume to the help desk. If your company's PCs
have the ability to "wake" on command, the deployments
can be performed at night or on a weekend to
minimize downtime and bandwidth utilization. Some
companies prefer to deploy service packs
to workstations on weekends, using a CD-ROM and
assigning 10-20 PC's per IT staffer. While this
process is slower, you can address installation
issues immediately instead of having an irate user
calling the help desk the following morning
because his PC won't boot.
-
Upgrading
Servers - Server upgrades should always be
done manually whenever possible. You should backup
every server immediately before upgrading it, or
plan your upgrades to follow the routine backup
schedule. In addition, you should update the
startup disks and backup the system state immediately
before
beginning the service pack upgrade. To
save time, you
can upgrade several servers at once but you need
to be prepared for the worst. (You never know what
will happen when you reboot a server that hasn't
been turned off in months) If I'm upgrading more
than a few servers, I like to keep a spare hard
drive or 2 hanging around as well as a power
supply. (They can be hard to get on a weekend in
the middle of the night.) Have all of the
resources you need to rebuild a server at your
fingertips, including the software that runs on
the server and access to the backup media.
Service
Pack Installation
Tips
|
Verify
the Installation Source
If at all possible,
download the entire network install of the service
pack or use the CD that comes with a TechNet
subscription instead of using Windows Update on
every machine. If you make copies of the service
pack CD, or create CD's based on the download,
verify the integrity of the disk against the
original before you start using it for your
deployment. If your using a network installation
share, be sure that the source server and network
infrastructure can handle the bandwidth
requirements based on the estimated number of simultaneous
installs. (You may be surprised to find that
sometimes CD's work faster than network
deployments - depending on your network infrastructure
and system CD speed.) |
Shut down
unnecessary services
Microsoft
recommends that you disable ALL third party
services, especially any service that accesses the
file system. This includes antivirus scanners, quota
managers, and scheduling programs. In
addition, you should set all Microsoft application services to
manual. This includes SQL Services, Exchange
Services, and Spooler on dedicated print servers. |
Perform
a clean reboot before installing the service pack
If you haven't rebooted your server or
workstation in a while, you'll want to restart it
before installing a service pack. This will close
any persistent processes or memory leaks, and
allows you to check for startup errors or other
issues. It will also rebuild the systems "last
known good configuration" in the event your
service pack install fails. |
Create
an uninstall directory
All operating system
service packs since Windows 2000 Service Pack 1
automatically prompt you to create an uninstall
directory. You should let the installation create
this directory and leave it intact even after the
service pack is up and running. If you install
programs or drivers that may not be compatible with
the Service Pack, you may need to uninstall the
Service Pack. |
Update
the ERD
If your installation was successful,
you'll need to update your Emergency Repair Disk
for each server to make sure you can recover the
system if it becomes unstable. |
|
|
|