Keys to optimizing your backup environment: Veritas NetBackup
Jeff Harbert
Storage Consultant GlassHouse Technologies, Inc. jharbert@glasshouse.com
Introduction
Audience Profile Storage Management Interdependence Case Study Backup Architecture and Performance
Server Tape Network Clients
Audience Response
Are you a NetBackup Administrator?
Yes No 1. 0-6 months 2. 6-12 months 3. 12-24 months 4. More than 2 years
How long have you been using the product?
Case Study Biotechnology Company
Technical Issues
Backup architecture unable to scale with increasing storage capacities and server growth Network architecture unable to support throughputs required for backup Lack of NBU / OS tuning: Implementation did not include optimization Client-side issues impacting backup success (OS, antivirus, network, applications)
Business Problems
Limited budget and staff Reliability and performance problems for both hardware and software Regulatory / data retention requirements Disconnect between application design and storage management Shrinking backup windows Lack of data management policies
NetBackup Architecture
Tape/SAN/Disk Technology
SD
SD SD
HEWLETT PACKARD
NetBackup Server
5411 Enterprise Switch 2 3 P OWER MGMT
Clients
10 00 BASE X AMBER AC TIVI TY = GREEN LINK O K = FL ASHING GREEN = DISABLED
10/ 100 BASE TX 4 5 6 7 8 9 1 0 11 1 1 2 2 3 3 R 3 3
10
11
3R
Network
Media Server
Case Study
Technical Issues
Backup server underpowered for size of environment Lacking in CPUs, RAM, and Network throughput No media servers to balance load
Solution
Upgraded hardware to HP rp7410 Master: 4GBs of RAM, 4 CPUs, Gb Ethernet cards Partitioned server into master and media server Upgraded hosts with more than 200 GBs into SAN media servers
NetBackup not properly tuned for optimal performance
Modified NetBackup server-side variables to maximize performance
NetBackup Server Performance
Application Performance Tuning
Backup Schedule Balancing Number of Backup Jobs Multi-streaming and multiplexing Specific Tuning Variables NET_BUFFER_SZ DISABLE_RESOURCE_BUSY CLEAN_IN_BACKGROUND Pre-Process Interval
NetBackup Performance Tuning Guide
UNIX: http://seer.support.veritas.com/docs/240733.htm Windows: http://seer.support.veritas.com/docs/248373.htm
Average Media Server Volume
1,200 1,000 800 600 400 200 0 Sunday Monday Tuesday Wednesday Server 4 Server 3 Server 2 Server 1 Friday Saturday Server 1 Server 2 Server 3 Server 4
Data Transferred (GB)
NetBackup Architecture
Tape/SAN/Disk Technology
SD SD SD
Thursday
HEWLETT PACKARD
NetBackup Server
54 Enterprise Switch 11 2 3 PO WER MGMT
Clients
10 00 BASE X AMBER AC TIVI TY = GREEN LINK O K = FL ASHING GREEN = DISABLED
10/ 100 BASE TX 4 5 6 7 8 9 1 0 11 1 1 2 2 3 3 R 3 3
10
11
3R
Network
Media Server
Case Study
Technical Issues
Veritas Shared Storage Option deployed with STK L700 and 20 LTO-1 drives Frequent drive problems caused multiple backup failures Overall Throughput was less than expected
Solution
Modified SSO to limit drive sharing to smaller number of servers Updated persistent bindings to minimize impact of server reboots and SCSI-locks Tape drives were shoeshining due to a lack of data
Increased the number of multiple streams written to each drive Reduced the number of drives available for backup Modified SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS variables on media servers
Tape Tuning Strategies
Storage Unit Fragment Size
Rule-of-Thumb = 2GB Reduces shoe-shining effect Too much can negatively impact restore speed. Requires additional hardware, but enables tapes to be sent offsite quicker Best Practice: Utilize hardware compression unless network-constrained.
Multiplexing Data
In-Line Duplication
Software vs. Hardware Compression
Disk-Based Backups with NetBackup v5.0
Enhanced functionality of v5.0 makes disk-based backups a viable option. Disk as a Primary Storage Device
Enterprise-class solutions are limited due to tremendous storage requirements.
Disk-Based Backups with NetBackup v5.0 (II)
Disk as a Staging Device
v5.0 includes Staging functionality. Data is automatically moved based on preestablished policies. Leverages speed of disk for backup, with tape for long-term storage
Disk as Hybrid-Solution
Direct some backups to disk, others to tape. Eliminates additional step of staging, but provides advantages of both disk and tape
Storage Unit Distribution
Amount of Data Transferred for a 7-Day Period (15.4 TB)
STK9940A531, 0, 0% STK9940A317, 0.3TB, 2% STK9940A258, 0.4TB, 2% STK9940B317, 1.1TB, 7%
STK9940B258, 5.9TB, 39% STK9940B531, 3.6TB, 23%
STK9940B532, 4TB, 27%
NetBackup Architecture
Tape/SAN/Disk Technology
SD SD SD
HEWLETT PACKARD
NetBackup Server
54 Enterprise Switch 11 2 3 PO WER MGMT
Clients
10 00 BASE X AMBER AC TIVI TY = GREEN LINK O K = FL ASHING GREEN = DISABLED
10/ 100 BASE TX 4 5 6 7 8 9 1 0 11 1 1 2 2 3 3 R 3 3
10
11
3R
Network
Media Server
Case Study
Technical Issues
Separate Backup Network was created to reduce impact on corporate network
Clients did not always use backup network for data transfer
Solution
Modified Required Interface option to bind client backups to appropriate network interface Hard-code network settings to 100 Mb, Full-Duplex
Intermittent backup failures following host reboots
NICs and Ports set to autonegotiate
Log files indicate that Media servers were waiting for full buffers during network backups
Slowed backup performance
Tuned NET_BUFFER_SZ Variable on both media servers and clients
Network Design
Dedicated Private VLAN architectures
Logical network segregation & physical network segregation Use jumbo frames and gigabit VLANs. Improves streaming efficiencies to tape Etherchannel configurations Load balancing for network, switches, server HBAs
Multiple NICs for client network load balancing
Network Design, II
NetBackup Tuning Variable
NET_BUFFER_SZ: Determines size of network communications buffer used to receive data from network If set too small, media servers cant empty buffers fast enough. If set too large, media servers have to wait for data. Client settings should match media servers.
Sample Charts Available From GlassHouse
Buffer Information Summary Data Buffer Analysis per Media Server
50,000
Number of Times
45,000 40,000 35,000 30,000 25,000 20,000 15,000 10,000 5,000
0
Waited for full buffer Waited for empty buffer
Server 1
Server 2
Waited for empty buffer Waited for full buffer
Server 3 Server 4
Media Servers
NetBackup Architecture
Tape/SAN/Disk Technology
SD SD SD
HEWLETT PACKARD
NetBackup Server
54 Enterprise Switch 11 2 3 PO WER MGMT
Clients
10 00 BASE X AMBER AC TIVI TY = GREEN LINK O K = FL ASHING GREEN = DISABLED
10/ 100 BASE TX 4 5 6 7 8 9 1 0 11 1 1 2 2 3 3 R 3 3
10
11
3R
Network
Media Server
Case Study
Technical Issues
Oracle Agent was implemented for Hot Oracle Database Backups
Backups of some servers ran outof-window
Solution
Implemented Block-Level Incremental backups of Oracle
30% Faster than standard RMAN backups
Network Appliance Filers backed up via NFS mounts
Full backups took 5 days
Implemented NDMP backups to dedicated tape drives
Testing indicated that 3rd-Party Remote backups were significantly slower
Backups of Netware clients ran out-of-window
Tuned Netware variables per Veritas-recommended settings
Reduced backup times by 33%
Client Considerations
Include/Exclude list size
Maintenance is difficult, but necessary. Exclude Open Files, Temp files, etc.
Open File Processing Block-Level / Flash Backups Client options settings
Pre-Process Interval (set on master server) Increasing default interval
Number and size of files Frequency of data change
Client Backup Performance
Backup Jobs over 1 GB in Volume
16.0 14.0
Throughput (MB/S)
12.0 10.0 8.0 6.0 4.0 2.0 0.0
Client 10
Client 12
Client 13
Client 14
Client 15
Client 16
Client 2
Client 4
Client 7
Client 17
To receive charts similar to those seen in this presentation specific to your environment go to:
www.glasshouse.com/backup
For a cheat sheet see the highlighter in your conference bag.
Client 11
Client 18
Client 1
Client 3
Client 5
Client 6
Client 8
Client 9
Questions? Thank you.
See me at Ask the Expert Tuesday 5:00-6:00 PM
