Microsoft Exchange Server 2003 and

Microsoft Exchange 2000 Server

Memory Dump File
Concepts and Processes

Microsoft Product Support Services white paper

Written by Steve Justice

Published June 2004
Revised June 2005

Abstract

This white paper describes memory dump files, the tools that are used to collect them, and
the techniques that support engineers use to troubleshoot the processes that Exchange
relies on.
The information contained in this document represents the current view of Microsoft Corporation on the issues
discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it
should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the
accuracy of any information presented after the date of publication.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR
IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under
copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or
transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for
any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights
covering subject matter in this document. Except as expressly provided in any written license agreement from
Microsoft, the furnishing of this document does not give you any license to these patents, trademarks,
copyrights, or other intellectual property.

 2004 Microsoft Corporation. All rights reserved.

Microsoft, Windows, and Windows NT are either registered trademarks or trademarks of Microsoft Corporation
in the United States and/or other countries.
The names of actual companies and products mentioned herein may be the trademarks of their respective
owners.
CONTENTS

INTRODUCTION........................................................................................................... 1

MEMORY DUMP FILE CONCEPTS................................................................................2

Definition of a memory dump file 2
Information that a memory dump file captures 2
Differences between full memory dump files and mini memory dump files 3
Why it takes so long to create a memory dump file 3
Tools that are used to capture memory dump files 4

MEMORY DUMP FILE PROCESSES..............................................................................6

Process stops responding 6
Performance problems 8
Crashes 9
Memory leaks 11
Reducing the time to upload memory dump file information 12

CONCLUSION............................................................................................................ 13

FOR MORE INFORMATION.........................................................................................14

APPENDIX A.............................................................................................................. 15

APPENDIX B.............................................................................................................. 17

APPENDIX C.............................................................................................................. 19

APPENDIX D.............................................................................................................. 20
INTRODUCTION
This white paper provides general guidelines for troubleshooting certain
problematic scenarios. Do not assume that the steps that this white paper
provides are the only methods that you have to use to troubleshoot any problem.
Some circumstances may require more than one approach to determine the root
cause of a problem.

Additionally, memory dump files are not always the best way for a support
engineer to determine the root cause of a problem. Sometimes a memory dump
file helps direct the troubleshooting, but does not point out the exact root cause
of all problems. In those cases, additional troubleshooting steps must be used.
Such additional steps may include using other tools or diagnosis techniques.

If a Product Support Services (PSS) support engineer is helping you solve a

problem, the steps that you take may differ from the steps in this white paper.
Troubleshooting the problems that are listed in this white paper is a complex
process. Because of this, the type of information that is necessary to diagnose a
problem is largely left to the support engineer’s discretion. The complexity of
code-level troubleshooting requires many techniques to gather pertinent
information.

The information in this white paper is not intended to supersede the judgment of
a PSS support engineer who is diagnosing a problem.

The information in this white paper applies to:

 Exchange 2000 Server

 Microsoft Exchange 2000 Enterprise Server

 Exchange Server 2003

 Microsoft Exchange Server 2003 Enterprise Edition

Product Support Services White Paper 1

MEMORY DUMP FILE CONCEPTS
The concepts in this white paper refer only to Exchange-related processes that
run in User mode. If you have to diagnose a Kernel mode process, the steps and
methods are different depending on the nature of the problem.

Definition of a memory dump file

A memory dump file is a copy of the contents of the virtual memory for a
particular process. A debugger can create a memory dump file. The debugger
writes the contents of virtual memory to a file on the hard disk. Later, another
debugger can be used to open that memory dump file and treat the file as if it is
a "frozen" instance of a live process that is being debugged. A support engineer
can use this snapshot of virtual memory to examine things, such as the heap
memory and the call stacks for every thread that was running for that process.

Information that a memory dump file captures

Memory dump files capture the contents of the virtual address space at one
moment in time for a process that is running. That virtual address space contains
almost everything that a process was using at that time to conduct its operations.
Some of the most frequently used information in memory dump files is:

 Heap memory. This information includes objects or variables that are created
by a process which is using statements in C++ such as "free," or "malloc" in
C. Heap memory is generally used to determine what data was loaded at the
time that the memory dump file was collected.

 Call stacks for all threads. These call stacks contain all the function calls that
were being processed at the time that the memory dump file was collected,
together with the variables that were on the stack. Call stacks are used to
determine what was happening at the time that the memory dump file was
collected.

 Thread environment blocks. These blocks contain information about a running

thread. Support engineers can use this information to determine what the
state of the thread was and what the thread ID is.

 Assembly code. The assembly code that each process is executing is loaded in
virtual memory. A memory dump file can capture that assembly code, and a
support engineer can use that code to help debug the problem so that the
support engineer can determine how variables or processor registers reached
their current state.

 Module information. Most processes use multiple files when they are
executing. For example, many DLL files are frequently used to load supporting
libraries for a process. A process might contain files that were created by
Microsoft or by other companies. A memory dump file captures information

Product Support Services White Paper 2

 about those loaded modules so that their version information and assembly
can be examined later.

Differences between full memory dump files and mini memory dump
files
A memory dump file can collect a variety of information. Typically, a support
engineer must have all the contents of virtual memory to troubleshoot a problem.
In other cases, you might want to capture less information to focus on a specific
problem. The debugger is flexible. This flexibility lets you limit the information
that a memory dump file captures by collecting either full memory dump files or
mini memory dump files:

 Full memory dump files. These files contain the contents of virtual memory
for a process. These files are the most useful when you are troubleshooting
unknown issues. A support engineer can use these files to look anywhere in
memory to locate any object, pull up the variable that was loaded on any call
stack, and disassemble code to help diagnose the problem. The disadvantage
of full memory dump files is that they are large. It also may take additional
time to collect these files, and the process that is being recorded must be
frozen while the dump file is created.

 Mini memory dump files. A mini dump file is more configurable than a full
dump file and can range from only several megabytes (MB) up to the size of a
full dump file. The size differs because of the amount of virtual memory that
the debugger is writing to disk. Although you can gather mini memory dump
files quickly and they are small, they also have a disadvantage. Mini dump
files may contain much less information than full dump files. The information
that a mini dump file gathers may be virtually useless to a support engineer if
the area of memory that the support engineer has to troubleshoot was not
captured. For example, if the heap memory is not written to the memory
dump file, a support engineer cannot examine the contents of a message that
was being processed at the time that the problem occurred. Useful
information, such as the subject line and the recipient list, would be lost.

Why it takes so long to create a memory dump file

When a debugger has to copy the contents of virtual memory to disk, all threads
for that process must be frozen at the same time. All threads can remain frozen
until the memory dump file is finished, and then all threads are unfrozen at the
same time. Because of this, a process such as the Store.exe process may cause
clients to stop responding (hang) or report that the Exchange computer is
unavailable. The time that is necessary to write a memory dump file is directly
related to how much information must be written to disk. If a process is using 1.5
gigabytes (GB) of virtual memory, several minutes may pass before the memory
dump file can be created. This behavior occurs because approximately 1.5 GB of

Product Support Services White Paper 3

data is written to the memory dump file. During that time, all threads can remain
frozen to prevent unexpected behavior in the process.

Tools that are used to capture memory dump files

Many tools can capture virtual memory. Each of these tools uses some form of a
debugger to create memory dump files.

The tools that PSS support engineers use most frequently are:

 Dr. Watson. This tool is a "just in time" debugger. Dr. Watson creates
User.dmp files for a process whenever a second chance, unhandled exception
is encountered. In most circumstances, a second chance, unhandled
exception occurs because of an access violation, a stack overflow, or some
condition that an application was not written to handle.

 ADPlus. This is a script that uses the CDB debugger to capture virtual
memory. ADPlus can be configured to create memory dump files when
hardware fails or a process stops responding. ADPlus has additional flexibility;
it can capture multiple processes with one command-line instruction. ADPlus
has two modes of operation: Hang mode and Crash mode:

o Hang mode noninvasively attaches to a process and creates a memory

dump file that is based on command-line parameters. After the dump
file is created, ADPlus detaches the debugger, and the process
continues to run. ADPlus gives you a snapshot of what the process was
doing at a point in time.

A typical ADPlus command line to capture a Hang mode memory dump

file is:

adplus –hang –pn [process name]

o Crash mode invasively attaches to the process and waits for an

exception to occur. By default, ADPlus creates a mini memory dump
file for every first chance exception that is encountered. Whenever a
second chance exception occurs, a full memory dump file is collected
and the process is terminated. If ADPlus is in Crash mode, and the
user decides to detach ADPlus by pressing CTRL+C, a full memory
dump file is created, and then the process is terminated.

A typical ADPlus command line to capture a Crash mode memory

dump file is:

adplus –crash –pn [process name]

 Userdump.exe. This is a command-line tool that can capture the virtual

memory for one process when a process stops responding or an unhandled
exception is encountered. Userdump.exe can create full memory dump files.

Product Support Services White Paper 4

 Microsoft Windows® Debugger. If a support engineer is attached to a process
with the Windows Debugger, the support engineer can create either full dump
files or mini dump files while debugging. This method gives the support
engineer the greatest amount of flexibility to create memory dump files.

Note This white paper describes how to use the Adplus utility to configure
the debugger to collect memory dump files. You can also use the Windows
Debugger, but the syntax that you must use is more complex and beyond the
scope of this white paper. Therefore, the equivalent Windows Debugger
commands are not included for the examples in this white paper.

Product Support Services White Paper 5

MEMORY DUMP FILE PROCESSES
Different approaches must be taken to use memory dump files to diagnose
problems. Support engineers try to minimize the impact that gathering memory
dump files has on a customer, but still try to collect all the information that they
must have to troubleshoot a problem. To minimize the impact of information
gathering, tools have been designed to help customize the way that memory
dump files are gathered. This section provides some general guidelines about
how to approach certain situations to bring problems to a resolution as quickly as
possible. The information in this section is not intended to supersede the
judgment of a PSS support engineer who is diagnosing a problem.

Note All the examples in this section assume that the Debugging Tools for
Windows are installed on the Exchange server. To install the Debugging Tools for
Windows, visit the following Microsoft Web site:

http://www.microsoft.com/whdc/devtools/debugging/default.mspx
It is recommended that you install the Debugging Tools for Windows to the
c:\debuggers folder if you can. Installing to this folder makes executing
command line instructions easier.

Process stops responding

When a process stops responding (hangs), one of the many threads that are
running has entered a condition that prevents that thread from completing an
operation. Other threads may also be affected by the problematic condition that
one thread has entered. For example, if a thread is in an infinite loop, that thread
consumes 100 percent of the processor's utilization until the process is
terminated. If the only data that is collected is one memory dump file of an
infinite loop, it is difficult to determine which thread was spinning. You can use
System Monitor together with a memory dump file to help isolate the problem,
but you may still want multiple dump files. You may want multiple dump files
because the thread may have been in several loops at the time. With one dump
file, the support engineer can determine which loops were being executed, but
with multiple dump files, the support engineer is more likely to be able to isolate
the infinite loop. This is because the support engineer can view multiple
snapshots of the call stack for a thread to see how the stack changed over the
course of several minutes.

When a process stops responding, it is important for the support engineer to be

able to see what the threads are doing when the process stops responding.
Memory dump files only take a snapshot of all the threads. If you capture
multiple snapshots of the same threads, a support engineer can start to identify
patterns of behavior to help isolate the problem. In this scenario, there is no
good method to do this other than capturing memory dump files. (See Appendix
B for an example of a debugging process that has stopped responding.)

Product Support Services White Paper 6

There are several methods that can be used to collect data from a process that
has stopped responding. The method that is used is determined by the behavior
of the server that the process is running on. Factors such as current CPU activity,
memory usage, error messages, event logs, and disk performance determine the
approach that is taken in each case. However, if a process stops responding, a
general approach can be to collect a full memory dump of the process, so that
the state of the threads for that process can be inspected. Optionally, a support
engineer can use mini memory dump files the same way that those files are used
in performance-related problems, to catch thread state changes. The following
example uses ADPlus to capture memory dump files when a process is not
responding:

1. Open a Command Prompt window.

2. Change to the folder where the Debugging Tools for Windows are
installed. (By default, this folder is C:\Program Files\Debugging Tools for
Windows.)

3. Type the following command-line instruction:

adplus –hang –MiniOnSecond –pn [process name]

4. Adplus informs you that a memory dump file is being created. Click OK to
continue.

5. A memory dump file is created in a subfolder of the location where the

Debugging Tools for Windows were installed. The subfolder name is
similar to "Hang_Mode__Date_[MM-DD-YYYY]__Time_[HH-MM-SS]PM."

The memory dump file has a name that is similar to "PID-2748__[process

name]__full_[YYYY-MM-DD_HH-MM-SS]-416_0ABC.dmp."

6. Wait for approximately three minutes, and then repeat step 3. This
creates another mini memory dump file in a new subfolder of the location
where the Debugging Tools for Windows were installed. You will have two
subfolders, and each subfolder will contain a mini memory dump file.

7. Wait for approximately three minutes longer, and then type the following
command-line instruction:

adplus –hang –pn [process name]

This creates another subfolder, but this time a full memory dump file is
created for the process.

8. Collect all three of the subfolders that were created, and then use the
instructions in Appendix D to send the subfolders to PSS.

Note PSS may also request additional information, such as system monitor logs
and event logs, to troubleshoot the problem.

Product Support Services White Paper 7

Performance problems
Contention for a resource can cause multiple threads to back up behind each
other. If a thread is waiting to write to a file on disk, the thread frequently locks
a critical section to do that. For example, if a thread must write to a log file, that
thread acquires a lock for the log file, and then writes to the log file. If there are
other threads that have to write to the log file, those threads must wait until the
file is unlocked, so that they can lock the file and write to the file. If for some
reason the disk cannot handle those write requests in a timely manner, the disk
becomes a bottleneck. At that point, multiple threads start to back up behind
each other while they wait to gain access to the log file. Therefore, the process
stops responding or becomes slow to respond.

In these situations, support engineers typically must have multiple dump files, so
that they can examine the state of the threads at different points in time. Getting
a snapshot of what is occurring over the course of several minutes helps reveal
whether a process has stopped responding, or whether the threads are just
processing slowly. The difference is significant because it determines the direction
that the troubleshooting takes moving forward.

Collecting memory dump files for performance problems can take longer than
usual if a lack of system resources causes a process to run more slowly than
usual. Also, because collecting a memory dump file itself requires system
resources, the server needs time to recover from the memory dump file creation
process. Therefore, the performance of the server immediately after you create a
memory dump file, especially a full memory dump file, should not be considered
typical performance behavior. Typical performance behavior returns after the
process catches up from creating the dump file.

The following example uses ADPlus to capture memory dump files when a
process is experiencing a performance problem:

1. Open a Command Prompt window.

2. Change to the folder where the Debugging Tools for Windows are
installed. (By default, this folder is C:\Program Files\Debugging Tools for
Windows.)

3. Type the following command-line instruction:

adplus –hang –MiniOnSecond –pn [process name]

4. Adplus informs you that a memory dump file is being created. Click OK to
continue.

5. A memory dump file is created in a subfolder of the location where the

Debugging Tools for Windows were installed. The subfolder name is
similar to "Hang_Mode__Date_[MM-DD-YYYY]__Time_[HH-MM-SS]PM."

The memory dump file has a name that is similar to "PID-2748__[process

name]__full_[YYYY-MM-DD_HH-MM-SS]-416_0ABC.dmp."

Product Support Services White Paper 8

 6. Wait for approximately three minutes, and then repeat step 3. This
creates another mini memory dump file in a new subfolder of the location
where the Debugging Tools for Windows were installed. You will have two
subfolders, and each of them will contain a mini memory dump file.

7. Wait for approximately three minutes longer, and then type the following
command-line instruction:

adplus –hang –pn [process name]

8. This creates another subfolder, but this time a full memory dump file is
created for the process.

9. Collect all three of the subfolders that were created, and then use the
instructions in Appendix D to send the subfolders to PSS.

Note PSS may also request additional information, such as system monitor logs
and event logs, to troubleshoot the problem.

Crashes
A crash occurs when a condition arises that the exception handlers for the
process and the operating system cannot handle. Access violations, heap
corruption, stack corruption, or a stack overflow may lead to a crash. By design,
a crash creates one dump file at the time of the crash, and it is possible that the
dump file can be used to isolate the root cause. Sometimes you may have to
capture another memory dump file of the crash. For example, a support engineer
may require more or different memory dump files from the time of the crash if
the memory dump file was corrupted and is unreadable, or if other factors also
lead to the crash.

Dr.Watson will capture a crash if Dr.Watson is configured to be the system's "just

in time" debugger. To make sure that Dr.Watson is configured to catch process
crashes, click Start, click Run, type the following command in the Open box,
and then click OK:

drwtsn32

The Dr.Watson interface is displayed. Examine the Dr.Watson interface to make

sure that the following settings are configured:

 Number of Instructions: 10

 Number of Errors to Save: 10

 Crash Dump Type: Full

 Dump All Thread Contexts is selected

 Append To Existing Log File is selected

 Create Crash Dump File is selected

Product Support Services White Paper 9

To make sure that Dr.Watson is the default "just in time" debugger, click Start,
click Run, type the following command in the Open box, and then click OK:

drwtsn32 –i

You receive a notification that Dr.Watson has been registered as the default
application debugger. Click OK to clear that notification.

You can also configure the debugger to catch crashes by using ADplus or the
Windows Debugger. Those options give you more control of what type of memory
dump file is created and what the name of the memory dump file will be.

The advantage of using Dr.Watson to catch crashes is that the debugger does not
have to be invasively attached to a process to capture a crash. The
disadvantages are that the same file name is always used (User.dmp), and the
information in the User.dmp file will be overwritten if another crash occurs. Also,
Dr.Watson does not capture first-chance exceptions.
To overcome the limitations of Dr.Watson, you can use ADPlus, the Windows
Debugger, or CDB to make collecting memory dump files from processes that
have crashed easier. The following example uses ADplus to configure the
debugger to catch a crash in a process:

1. Open a Command Prompt window.

2. Change to the folder where the Debugging Tools for Windows are
installed. (By default, this folder is C:\Program Files\Debugging Tools for
Windows.)

3. Type the following command-line instruction:

adplus –crash –pn [process name]

4. Adplus informs you that the debugger (CDB.exe) is attached and

monitoring for crashes for the specified process.

5. A full memory dump file is created if any one of the following situations
occur:

 The process encounters an exception that cannot be handled, and

the process crashes

 The process is stopped by any means. This includes stopping a

service or shutting a process down

 A user presses CTRL+C in the CDB Command Prompt window that

ADPlus opened. If a user presses CTRL+C in that window, a
memory dump file is created and the process is terminated

6. The memory dump file is created in a subfolder of the location where the
Debugging Tools for Windows were installed. The subfolder name is
similar to "Crash_Mode__Date_[MM-DD-YYYY]__Time_[HH-MM-SS]PM."

Product Support Services White Paper 10

 The memory dump file has a name that is similar to "PID-2748__[process
name]__full_[YYYY-MM-DD_HH-MM-SS]-416_0ABC.dmp."

Memory leaks
A memory leak is created by code that creates objects in virtual memory but
does not remove those objects when the code finishes using them. For example,
if a process is opening messages in the store process to scan those messages for
viruses, and that process continually opens messages for scanning but never
closes them, virtual memory for the store process increases and is never
released. A support engineer can use a full memory dump file of a process that is
leaking memory to examine the contents of the heap and determine what types
of objects are being leaked and which user logon owns them. In this example,
one dump file may be sufficient to capture the leak, but if the leak is small or
slow to develop, a support engineer may require multiple dump files to examine
the contents of virtual memory. Other tools, such as LeakDiag, UMDH, VADump,
and System Monitor, are typically employed to diagnose a leak.

Memory leaks can be captured using either an invasive attachment or a

noninvasive attachment of the debugger. You can use a noninvasive attachment
in this case because the thread state of a process is not as important as the
contents of virtual memory when you capture memory leaks. The following
example uses the process for collecting a memory dump file of a memory leak:

1. Open a Command Prompt window.

2. Change to the folder where the Debugging Tools for Windows are
installed. (By default, this folder is C:\Program Files\Debugging Tools for
Windows.)

3. Type the following command-line instruction:

adplus –hang –pn [process name]

4. Adplus informs you that a memory dump file is being created. Click OK to
continue.

5. A memory dump file is created in a subfolder of the location where the

Debugging Tools for Windows were installed. The subfolder name is
similar to "Hang_Mode__Date_[MM-DD-YYYY]__Time_[HH-MM-SS]PM."

The memory dump file has a name that is similar to "PID-2748__[process

name]__full_[YYYY-MM-DD_HH-MM-SS]-416_0ABC.dmp."

6. Collect the subfolder that was created, and then use the instructions in
Appendix D to send the subfolder to PSS.

Note PSS may also request additional information, such as system monitor logs
and event logs, to troubleshoot the problem.

Product Support Services White Paper 11

Reducing the time to upload memory dump file information
Full memory dumps files can exceed 1 GB. Therefore, the time that is required to
upload those files to PSS can be significant. To speed up the file transfer process,
you can extract a subset of information from the full memory dump file, so that
you can quickly send the information to PSS. (An example of the size difference
would be a full memory dump file that is 1.1 GB; this file can generate a mini
memory dump file that is 2.2 MB.) The PSS support engineer can then start
analyzing the subset of information while the full memory dump file is being
uploaded.

You can use the Windows Debugger to create a new mini memory dump file that
is based on the full memory dump file.

To create the new file, follow these steps:

1. Start WinDbg. WinDbg starts the Windows Debugger.

2. On the File menu, click Open Crash Dump.

3. Open the full memory dump file that captured the process at the time of
the failure.

4. Ignore any error messages regarding symbols that are displayed. To do

this, click OK to close the messages.

5. At the command line in the Windows Debugger, type the following

command:

.dump /mi c:\newdump.dmp

This creates a new mini memory dump file that is named

C:\Newdump.dmp.

6. Wait until you see the debugger report "Dump successfully written."

7. On the File menu, click Exit. If you are prompted about whether you
want to save workspace information, click No.

Make sure that you also upload the full memory dump file to PSS. PSS will
probably also need the full memory dump file to complete a detailed investigation
of the problem.

Product Support Services White Paper 12

CONCLUSION
Memory dump files capture the contents of the virtual address space at one
moment in time for a process that is running. That captured virtual address space
contains almost everything that a process was using at that time to conduct its
operations, such as stacks, heap memory, and thread information.

There are two general types of memory dump files: full memory dump files and
mini memory dump files. Full memory dump files contain all the contents of
virtual memory for a process. Mini memory dump files may contain much less
information than full memory dump files, but mini memory dump files are faster
to collect and smaller than full memory dumps.

Different approaches must be taken to use memory dump files to diagnose

problems. If you are diagnosing a process that crashed, one full memory dump
file may be sufficient to troubleshoot the problem. If a process stops responding
(hangs), you typically have to gather multiple memory dump files several
minutes apart to diagnose the problem.

In any case, support engineers strive to minimize the impact that gathering
memory dump files has on a customer while still trying to collect all the
information that they must have to troubleshoot a problem.

Product Support Services White Paper 13

FOR MORE INFORMATION
For the latest information about Exchange, visit the following Microsoft Web sites:

 http://support.microsoft.com/default.aspx?pr=exch2003

 http://support.microsoft.com/default.aspx?pr=exch2k

 http://www.microsoft.com/exchange/

For additional information about how to collect memory dump files and scenarios
where memory dump files can be useful on Exchange Server 2003 computers and
Exchange 2000 Server computers, click the following article numbers to view the
articles in the Microsoft Knowledge Base:

241215 How to use the Userdump.exe tool to create a dump file

823150 How to gather data to troubleshoot Exchange Server 2003 virtual

memory
325044 HOW TO: Troubleshoot virtual memory fragmentation in Exchange
2003 and Exchange 2000

286350 Use ADPlus to troubleshoot "hangs" and "crashes"

Product Support Services White Paper 14

APPENDIX A
This appendix includes a table to use for reference when you start to gather data
for certain conditions. Note that the information that this table contains is not
intended to supersede the judgment of a PSS support engineer who is diagnosing
a problem.

The main body of this white paper contains information about collecting mini
memory dump files. This information is also included in the following table. Some
of those points are restated in this appendix for clarity.

The table includes preferred options and secondary options for both a process
that stops responding (hangs) and slow performance. The preferred options
involve gathering several full memory dump files during the problematic
condition. This option is preferred because support engineers can use all the
contents of virtual memory for a running process to investigate the problem. This
helps the support engineer examine the objects that were being used at the time
that the problem occurred.

The secondary options involve gathering one full memory dump file, and then
gathering at least two more mini memory dump files during the problematic
condition. The disadvantage of the secondary option is that the mini memory
dump files will not contain the objects that were being used during the
problematic condition. The support engineer might be able to use the data in the
full memory dump file to locate objects that are referenced in the mini memory
dump file, but that may not be possible.

Another significant difference between the preferred option and the secondary
option is the time that it takes to complete them. Full memory dump files take
several minutes longer to create than mini memory dump files. Therefore, the
time that it takes to complete the preferred option is longer than the time that it
takes to complete the secondary option by at least several minutes. The time
difference depends on the amount of virtual memory that a process is using at
the time that the memory dump file is collected.

Product Support Services White Paper 15

 Exchange Server 2003 and Exchange 2000 Server
memory dump file processes
Condition Required data Optional data
Crash  One full memory dump file at the time that the x number of mini
(A process process terminated dump files created
suddenly o You can also use the .dump /mi leading up to the
terminates) [filename] command in the Windows failure
Debugger to create a mini memory file
from the full memory dump file. (Only available in
 Application event logs ADPlus attached
 System event logs in Crash mode)
Hang One of the following:
(A process  Preferred option. Two mini memory dump
stops files at least 2 minutes apart, but no more than
responding but 15 minutes apart
still appears to  Secondary Option. Two full memory dump
be running) files at least 2 minutes apart, but no more than
15 minutes apart

 One full memory dump file

 System Monitor logs
 Application event logs
 System event logs
Slow One of the following:
Performance  Preferred option. Two mini memory dump
(A process is files at least 2 minutes apart, but no more than
running but 15 minutes apart
users are  Secondary option. Two full memory dump
experiencing files at least 2 minutes apart, but no more than
delays) 15 minutes apart

 One full memory dump file

 System Monitor logs (Specific counters will be
determined at the time that the problem is
occurring)
 Application event logs
 System event logs
Memory Leak  One full memory dump file during the high memory Possibly
(Memory is use condition LeakDiag or
being used and  System Monitor logs UMDH tracking
never released)

Product Support Services White Paper 16

APPENDIX B
This appendix includes a mock memory dump file to illustrate what a support
engineer may have to do to debug and troubleshoot a process that has stopped
responding.

From a memory dump file, you can verify that multiple threads are backed up
behind one thread. In the following example, thread 102 is blocking the others.

CritSec at 1bfe1544. Owned by thread 113.

Waiting Threads: 57 216 232 240 266 277 288 289 328 332
CritSec at 1bc2e5c4. Owned by thread 102.
Waiting Threads: 113

After you have this information, you must determine what thread 102 doing. To
determine what thread 102 is doing, view that call stack.

ChildEBP RetAddr
NTDLL!ZwReadFile+0xb
KERNEL32!ReadFile+0x125
MSExchange!OpIFSRead+0x84
MSExchange!OpRead+0x413
MSExchange!IStreamRTF::Read+0x1b
inetmail!CStreamLockBytes::Read+0x43
inetmail!CInetStream::_HrGetNext+0x33
inetmail!CMessageTree::_HrBindOffset+0x71
inetmail!CMessageTree::Load+0x5e
MSExchange!CConverter::Initialize+0x229
MSExchange!CIMailConvert::HrOpenProperty+0x88
MSExchange!MSG::OpFurnishProp+0x539
MSExchange!MSG::OpGetBasicProp+0x16f
exlib!RPC_INTERFACE::DispatchTo+0x5e
exlib!OSF_SCALL::Dispatch+0xac
exlib!LOADABLE_TRANSPORT::ProcessIO+0x14a
exlib!ProcessIOWrapper+0x9
exlib!BasOpachedThread+0x4f
exlib!ThreadStart+0x18
KERNEL32!ThreadStart+0x52

From this call stack, you can verify that this thread was trying to read a file from
disk. This information is recorded at the top of the stack, where the store called
IFSRead, which called into ReadFile.

This is useful information, but you cannot be sure that the problem is only a disk
performance problem. Next, you must determine what type of data was being
read from the file at the time. In this case, this stack is processing a rule. For
example, you know that you are doing work in the following folder.

16e34658 "/NON_IPM_SUBTREE/Deferred Action"

16e34698 ""

Product Support Services White Paper 17

The only way to determine whether that thread has stopped responding is to
obtain another dump file of all the threads, and then see whether thread 102 is
still stuck in ReadFile several minutes later, and also several minutes after that.
If thread 102 remains in ReadFile for that long, there might be a problem with
the disk or SAN.

At this point, the information indicates why the server is performing slowly or has
stopped responding, but you do not know for sure. You know that at one moment
this thread was in ReadFile. That is not sufficient information to confirm that you
have a disk performance problem.

When you obtain the three snapshots of memory, if they show that your threads
are no longer backed up but the problem persists, you can examine all the
threads to determine what was happening that caused the process to stop
responding. You can use System Monitor logs and event logs to help gain a better
understanding of why the threads are in this state, and why the process stopped
responding.

Typically, PSS asks for multiple dump files because PSS support engineers have
to know what the threads are doing at the time of failure during multiple
moments in time.

Product Support Services White Paper 18

APPENDIX C
This appendix describes how to use the Product Support Services File Transfer
Application (PSSFTA) to receive files from Microsoft Support:

1. You will receive an e-mail message from your support engineer at Microsoft.
The e-mail message will contain a Web site address, or Uniform Resource
Locator (URL).

2. Your support engineer will contact you again, either by e-mail message or
telephone, to give you a password:

 You must keep these e-mail messages confidential at all times, to

help avoid unauthorized access to your files.

 Files will only remain at the Web site address for 24 hours. If you
do not retrieve your file in 24 hours, it will be deleted. If this occurs,
please contact your support engineer and ask to the have the files sent
again.

3. Start your Web browser, and visit the Web site address that your support
engineer provided in the e-mail message.

4. Click Receive Files from Microsoft.

5. Type the password that your support engineer provided, and then click GO.
You are given access to the location that contains the files that you want.

6. Right-click the file name, and then click Save Target As.

7. In the Save As dialog box, select the destination path on the computer where
you want to save the file, and then click Save. Click Close to close the dialog
box. The file is copied to that destination.

8. If you have multiple files in this location, repeat steps 6 and 7 to make sure
that all the files are copied.

9. Although Microsoft takes precautions to help make sure that the files we send
are clean of any viral infection, always use your antivirus vendor's software to
scan the files to help make sure that no infection exists. In the unlikely event
that your antivirus software detects an infection, please contact the Microsoft
support engineer who you are working with.

10. When you have received your files, exit the Web browser to close your
session.

Important If you click any .txt, .htm, .html, or .jpg files, the file may
automatically be displayed in the Web browser. For files that have these
extensions, right-click the link, and then click Save Target As to save the file. If
the file is large, the system may hang, depending on the capacity of the hard
disk on your computer.

Product Support Services White Paper 19

APPENDIX D
This appendix describes how to use the Product Support Services File Transfer
Application (PSSFTA) to send files to Microsoft Support:

1. You will receive an e-mail message from your support engineer at Microsoft.
The e-mail message will contain a Web site address, or Uniform Resource
Locator (URL).

2. Your support engineer will contact you again, either by e-mail message or
telephone, to give you a password.

 Keep these e-mail messages confidential at all times, to help avoid

unauthorized access to your files.

 Files will only remain at this location for 24 hours. If for some
reason your support engineer does not download your file in 24 hours, it
will be deleted. If this happens, you may be asked to send the file again.
3. Start your Web browser, and then visit the Web site address that your
support engineer provided in the e-mail message.

4. Click Send Files to Microsoft.

5. Type the password that your support engineer provided.

6. Click Browse, locate the file that you want to send, and then click Upload. If
you need to send an additional file, click Send, and then repeat the process.

7. When you finish sending your files, quit your Web browser to close your
session.

Important Do not upload a file that is larger than 1.93 GB. If you try to upload
a file that is larger than 1.93 GB, the upload will fail. If you have to transfer a file
that is larger than 1.93 GB, please contact your support engineer.

Product Support Services White Paper 20

FREQUENTLY ASKED QUESTIONS

Question: What are the file size limitations of the PSSFTA?

The PSSFTA will accept files that are up to 1.93 gigabytes (GB) for
sending files to Microsoft and receiving files from Microsoft using PC
hardware. Macintosh users are limited to 300 megabytes (MB) for sending
files, and 1.93 GB for receiving files from Microsoft. If you have to transfer
files that are larger than that, please contact your support engineer. Your
support engineer can advise you of the appropriate choices that are
available.

Question: What is the hard disk drive space requirement for receiving files?
To receive files from Microsoft, available hard disk drive space must be
double the size of the file that is being received.

Question: What type of encryption do you use to help secure my data?

All authentication and transfers with the PSSFTA application use industry
standard Secure Sockets Layer (SSL). To learn more about Secure
Sockets Layer, visit the following Microsoft Web site and examine the
"Understanding SSL" section:

http://www.microsoft.com/technet/treeview/default.asp?
url=/technet/prodtechnol/iis/maintain/featusability/c06iis.asp

Question: I cannot access my folders any longer and the application says my
password is invalid. What happened?

All folders are removed every 15 days unless your support engineer
receives an extension for that folder to continue resolving your support
issue. The folders that the PSSFTA uses are designed to be used for an
individual support issue. If you have to transfer files that are related to
other issues to and from your support engineer, your support engineer will
create additional folders for you.

Question: Who has access to my file?

Only the support engineer troubleshooting your issue, the support

engineer's direct manager, and any additional support personnel who are
required to help resolve your support issue will have access to your file.
Any person with whom you share the location and password e-mail
messages that your support engineer sent you will also have access to this
data file. Do not share these e-mail messages at any time with any entity
who you do not want to have unrestricted access to the files in those
folders.

Question: What should I do if I believe my location and password information

have been compromised?

Product Support Services White Paper 21

 Immediately stop use of the compromised location and contact your
support engineer.

Question: What operating systems and Web browsers has the PSSFTA
application been tested with?

This application has been tested with the following operating system and
Web browser combinations:
o Macintosh with Microsoft Internet Explorer 5.1.x for Macintosh
o Macintosh with Microsoft Internet Explorer 5.2.x for Macintosh
o Microsoft Windows 2000 with Microsoft Internet Explorer 5.0
o Windows 2000 with Microsoft Internet Explorer 5.5
o Windows 2000 with Microsoft Internet Explorer 6
o Microsoft Windows 98 with Microsoft Internet Explorer 5.0
o Windows 98 with Microsoft Internet Explorer 5.5
o Windows 98 with Microsoft Internet Explorer 6
o Windows 98 with Netscape 4
o Microsoft Windows Millennium Edition (Me) with Microsoft Internet
Explorer 5.5
o Microsoft Windows NT 4.0 with Microsoft Internet Explorer 5.0
o Microsoft Windows XP Professional with Microsoft Internet Explorer
6
Question: What do I do when the file upload page is expired?
If this happens, please start again from the beginning; visit the Web page
that the Microsoft support professional provided, type the provided password,
click Send Files to Microsoft, and then click Go.

Product Support Services White Paper 22