Agenda

 Overview of IBM Spectrum Protect for Virtual Environments (TSM4VE)

 Product Components Overview
 Tivoli Storage Manager Server Backend Structure
 Backup and restore types
 Backup Transport Options
 DataMover (aka vStorage Backup Server)
 Options File (dsm.opt)
 Client Schedules
 Log Files, Stats and Reporting
 Application Aware-Backups
 Issues and Common Error Codes
 Extras
 Questions
Who am I ?

Steven Soave
Twitter: @StevenSoave

Current Position: Wintel & VMware Technical Specialist

Worked in IT for around 14 years (VMware products around 8 years)

VMware Certification: VCP6-NV

Disclaimer

 The information in this presentation are from my own experiences and do not reflect the
views or configures of the product at my employer.
Poll
 Who has used/is using IBM Spectrum Protect for Virtual Environments?
Overview of IBM Spectrum Protect for Virtual Environments
(TSM4VE)
 What is it?
 Image level virtual machine backup product leveraging VMware snapshots/change block tracking (CBT)

 Who are some of its direct competitors?

 Veeam
 CommVault
 Veritas NetBackup

 Supports VMware (vCenter / vCloud Director) and Hyper-V - (Windows and Linux VMs)

 Working with this product since 2014 when it was called “Tivoli Storage Manager for Virtual Environments 6.4
(TSM4VE)” to present where it is now called “IBM Spectrum Protect for Virtual Environments 8.1” (name
changed circa v7.1.3)

 Licensing model I’ve been exposed to: ‘data volume licensing’ – which means you can use all the IBM Spectrum
Protect suite of products. Hence in essence corporations that have this licensing won’t let you use anything else as
they will see it as paying double (through experience) 

(cont.)
 Pre using an imaged based backup solution like TSM4VE all my experiences were backing
up at the file level.

 With TSM4VE if all the stars are in aligned its been known to achieve 100% backup success
rates….. For a day or two ;-)
More on this later as “It Depends” on a number of components all working optimally
Product Components Overview

Physical Components Overview

 Configuration View  DataMover Services View
Some Firewall Stuff (FYI)
Tivoli Storage Manager Server Backend Structure
 Policy Domain (Web Console Links to the Default
Management Class)
 When using the web console you can only manage
the default management class, schedules,
backups and restores etc.
 All nodes (backup endpoints i.e. TSM agents on
servers or DataMovers) belong to the Policy
Domain
 If you want to use an alternative Management
Class it needs to by specified in opt file and use
CLI to perform backups and restores
 Backup Retentions - 35 days
 Storage Pools - Disk based, not tape (you can use
tape but I highly recommend you don’t)
 Storage Pools are linked to the management class
Backup and restore types
 Periodic Full
 Full backup
 Useful for long term backups

 Incremental forever
 An incremental forever backup strategy minimizes backup windows while providing faster
recovery of your data.
 Performs an automatic Full if one doesn’t exist
 Reduces the amount of data that goes across the network.
 Reduces data growth because all incremental backups contain only the blocks that changed
since the previous backup.
 Reduces the length of the backup window.

https://www.ibm.com/support/knowledgecenter/en/SS8TDQ_7.1.8/ve.user/t_ve_vsph_backupguitsm.html
 Full VM Restore
 Restore a full or incremental VM backup. The entire VM is restored to the state
it existed in when originally backed up.
 Note: When restoring a VM the original vNIC MAC addresses are retained, as long as
the original VM has been deleted or the vNIC removed from it so the system doesn’t
detect a MAC conflict.

 Full VM Instant Restore

 The restored VM becomes available for instant use, either for validating the backed up
VM or for restoring the VM to permanent.
 Basically gives you instant access while the full restore is happening in the background.

 Full VM Instant Access

 The restored VM becomes available for instant use for validating the backed up VM as
at the restore point specified.
 Any changes while using this restore will be cleaned up when the restore point is
unmounted, no changes are written back to the backup.
 VM Restore GUI
 Limitation: No search option (need to find where the VM currently is using vCenter)
 CLI: If restoring using CLI commands then you just specify the VM name
 File Restore Methods
 The are three methods to do file level restores. All three use the “Recovery Agent” and
iSCSI
 Method One: Using the “Mount Proxy” via the Web Console
 Method Two: Using the Recovery Agent directly
 Individually select and mount volumes required
 No indicator as you what the OS logical volume label is
 Is mounted as a logical disk/disks on the DataMover OS to browse the backed up files
 Method Three: Using the File Restore UI web console (End User Self Service)
Backup Transport Options
dsm.opt = vmvstortransport option to default (san:hotadd:nbdssl:nbd)

 san
 You want to offload backup traffic from the LAN, and instead, move it over a SAN.
 Required the DataMover to be a physical server

 hotadd
 You are using a data mover node that is installed on a virtual machine to back up other
virtual machines.
 Mounts target VMs vmdk using SCSI on the DataMover.
 Maximum of 15 SCSI disks per controller and a maximum of four controllers = 60 SCSI
disks.
 ndb
 You are backing up virtual machines over an Ethernet LAN; you either do not have, or
do not want to use, a SAN to offload backup traffic from the LAN.
 Uses port TCP 902 Network File Copy (NFC) from the ESXi host management interface
to the DataMover. Note: if using any bandwidth restricting technology (BladeSystems)
then ensure you have taken this into account as it could elongate the backups.

 nbdssl
 You are backing up virtual machines over an Ethernet LAN and you want to use SSL to
encrypt the data..
 Note that encrypting data can degrade backup performance
DataMover (aka vStorage Backup Server)
 Proxy Sizing
https://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/f731037e-c0cf-436e-88b5-862b9a6597c3/page/41a14c27-1ef1-4205-bf9e-
d2880970d877/attachment/969be223-c832-44bf-9a54-1df64497b514/media/TSM-VE-Proxy-Sizing_26may.pdf

 General rule of thumb is to use a data mover for every 100 TB of vSphere data
 10 GbE (or HotAdd / SAN equivalent) available on all data paths in the environment,
specifically the path from the datastore to the data mover and then to the IBM Spectrum
Protect server
 5% average daily change rate
 8 hour backup window

 Data mover hardware specification

The following hardware specification is suggested for each data mover:
16 cores (2.8 GHz) if using client-side deduplication
8 cores (2.8 GHz) if using server-side deduplication
8 GB RAM
10 GbE network
 DataMover Nodes are service instances configured via the TSM client agent on a VM or physical
server:
 TSM Client Acceptor Service (dsmcad.exe) = Handles comms with TSM server and listens for connection from Remote Client Agent
to start a backup or restore. Also listens for prompted schedule changes on behalf of the Client Scheduler.
 TSM Client Scheduler Service (dsmcsvc.exe) = Handles the Client Schedules created on the TSM server and running them
 TSM Remote Client Agent Service (dsmagent.exe) = Used to run Run Once and Scheduled jobs when prompted by the Acceptor
service

 Multiple DataMover instances can run on a single server with varied dsm.opt instruction sets which can
by linked to certain client schedules.

 Not a good idea to run multiple DataMover servers that target the same client schedule at the
same time in order to decrease backup time.

 DataMovers can target vCenter Clusters, Hosts, VMs, Datastores or Folders

‘dsm.opt’ file options
VMBACKUPTYPE FULLVM
VMCHOST <vCenter server name>
VMCUSER <vCenter user account>

SCHEDMODE PROMPTED

ERRORLOGRETENTION 35 D
SCHEDLOGRETENTION 35 D

COMPRESSION YES
COMPRESSALWAYS YES
DEDUPLICATION YES
*DEDUPCACHESIZE 2048
*DEDUPCACHEPATH "C:\Program Files\Tivoli\TSM\baclient\TSMDEDUPDB\STD"
ENABLEDEDUPCACHE NO

VMENABLETEMPLATEBACKUPS YES
VMPROCESSVMWITHPRDM YES
VMPROCESSVMWITHINDEPENDENT YES
VMSKIPMAXVIRTUALDISKS YES
VMMAXVIRTUALDISKS 8

VMLIMITPERDatastore 2
VMLIMITPERHost 8
VMMAXParallel 12

VMVSTORTRANSPORT "NBD:HOTADD"

TESTFLAG VMBACKUP_UPDATE_UUID

VMVERIFYIFAction FORCEfull

INCLUDE.VMSNAPSHOTATTEMPTS * 1 1
 VMCUSER Permissions in vCenter (Create a Role)
http://www-01.ibm.com/support/docview.wss?uid=swg27047438
Client Schedules
 Unsupported characters in VM or datacenter name
Data Protection for VMware does not support backing up VMs or datacenters that contain any of the following
characters in their name (ignore the brackets)
(“) Double quotation mark (‘) Single quotation mark (:) Colon (;) Semicolon (*) Asterisk (?) Question mark (,) Comma (<) Less than sign (>) Greater than sign
(/) Forward slash (\) Backward slash (|) Vertical bar

 Schedules are associated with a DataMover instance (service)

 No other schedule can be run using a DataMover instance that currently has a scheduled job running. Hence you will
see the need for multiple DataMover instances if breaking out longer running backups etc.

 DataMovers can have multiple schedules assigned to it. (Note: Care must be taken as the later schedule may not
run if current schedule run time supersedes the latter's start time (see Schedule Duration)

 Schedules have Duration Windows. If schedule is due to run but another is still running then the latter will still run as
long as its start time is within:
- Start time + Duration Window

 Two types of schedule:

- Run Now
- Scheduled
Log Files, Stats and Reporting
 DataMover Log Files are you friend and I use them daily for troubleshooting:
 Located here by default: C:\Program Fiels\Tivoli\TSM\baclient

 Most used logs (http://www-01.ibm.com/support/docview.wss?uid=swg21408654):

 dsierror.log = Used to show tsm server errors
 ANS0266I The dsmEndTxn vote is ABORT, so check the reason field.
 ANS1311E Server out of data storage space
 ANS1301E This operation cannot continue due to an error on the TSM server

 dsmerror.log = Used to log errors by each of the client applications: dsmc, dsmadmc, dsmcad, dsmj, Web Client
 ANS0361I DIAG: VmVerifyAllBitmapCtlAreInJobGroup(): Ctl file from bitmap.dat is missing in job folder! MegaBlock number:11299 Job
number:550005
 ANS9919E Failed to find the expected control files for TESTVM
 ANS9920W Forcing a full vm backup for TESTVM
 ANS9386W Changed block tracking is not supported for virtual machine TESTVM2'. A FULL backup of the disk will be performed and will include both
used and unused areas of the disk.

 dsmsched.log = Used to write client schedule information, and scheduler errors for applications: dsmc scheduler, dsmcad
 2017-10-18 05:27:16 Processing snapshot
disk: [DATASTORE_NAME] TESTVM/TESTVM_1.vmdk (Hard Disk 2)
Capacity: 42,949,672,960
Data to Send: 3,023,110,144
Transport: (nbd)[sending]

 dsmwebcl.log - Used to write Web Client messages and Client GUI messages for applications: dsmcad, dsmj, Web Client.
 Typical Schedule Run Stats for a DataMover targeting one Production vCenter cluster
2017-10-11 09:30:51 Accessing as node: DATACENTER NODE
2017-10-11 09:30:51 Aggregate statistics for Backup VM command.
2017-10-11 09:30:51 Total number of objects inspected: 368
2017-10-11 09:30:51 Total number of objects backed up: 368
2017-10-11 09:30:51 Total number of objects updated: 0
2017-10-11 09:30:51 Total number of objects rebound: 0
2017-10-11 09:30:51 Total number of objects deleted: 0
2017-10-11 09:30:51 Total number of objects expired: 0
2017-10-11 09:30:51 Total number of objects failed: 0
2017-10-11 09:30:51 Total number of objects skipped: 0
2017-10-11 09:30:51 Total number of objects encrypted: 0
2017-10-11 09:30:51 Total objects deduplicated: 364
2017-10-11 09:30:51 Total number of objects grew: 0
2017-10-11 09:30:51 Total number of retries: 0
2017-10-11 09:30:51 Total number of bytes inspected: 75.80 TB
2017-10-11 09:30:51 Total number of bytes processed: 1.92 TB
2017-10-11 09:30:51 Total bytes before deduplication: 1.92 TB
2017-10-11 09:30:51 Total bytes after deduplication: 972.11 GB
2017-10-11 09:30:51 Total number of bytes transferred: 486.98 GB
2017-10-11 09:30:51 Data transfer time: 140,756.01 sec
2017-10-11 09:30:51 Network data transfer rate: 14,634.47 KB/sec
2017-10-11 09:30:51 Aggregate data transfer rate: 64,138.02 KB/sec
2017-10-11 09:30:51 Objects compressed by: 50%
2017-10-11 09:30:51 Deduplication reduction: 50.52%
2017-10-11 09:30:51 Total data reduction ratio: 99.38%
2017-10-11 09:30:51 Elapsed processing time: 08:55:16
2017-10-11 09:30:52 Backup VM command complete

 Typically backup (incremental) approx. 1500 VMs over a 10 hour backup window, multiple TSM4VE environments
 Reporting: Predatar (SilverString)

 TSM Manager
 IBM Spectrum Protect for Virtual Environment Web UI
Application-Aware Backups

 Supports
 Microsoft Exchange Server
 Microsoft SQL Servers
 Microsoft Active Directory Domain Controllers

 More info: https://www.ibm.com/support/knowledgecenter/en/SS8TDQ_7.1.2/com.ibm.itsm.ve.doc/c_ve_plan_dpapps.html

Issues and Common Error Codes
 Consolidating VM backups
After a VM backup, the VM might contain pre-existing snapshots even though no snapshots are present in the
Snapshot Manager. For example, the VM hard disk points to snapshot VMDK files (for example *-000001.vmdk)
instead of regular VMDK files.

 Rare: When removing snapshot after backup VM is powered down

When logged with VMware they say its Storage I/O / timeout related where vCenter will power off the VM by
design?

 Be aware of snapshot stun times, I’ve known this to affect SQL servers mostly. I believe this is better in vSphere 6 >

 Centrally (Help Desk) managed file restores is a major pain to achieve. Lots of support calls with IBM support

 RC -50 - TCP/IP disconnect issues between DataMover and TSM server instance
 RC 6584 - Kind of a false positive. Quiesce snapshot failed but crash consistent successful (net result VM backed
up)
 RC 4390 - Target specific VM in schedule and VM no longer exists in vCenter inventory
 RC 11 - Normally Storage Pool is full
 RC 4379 - Normally issues around VMTools
Extras
 New version of this product called “IBM Spectrum Protect Plus”
https://www.ibm.com/us-en/marketplace/ibm-spectrum-protect-plus
https://www.ibm.com/blogs/systems/vm-protection-and-availability-made-easy/
 Includes RESTful API
 New file restore (search and recovery)
 Rumoured Q1 2018

 IBM Spectrum Protect for Virtual Environments as of v8.1 can now do:
https://www.ibm.com/support/knowledgecenter/en/SSERB6_8.1.0/ve.user/r_techchg_ve.html
 Enhancements for backup and restore performance for VMs:
Optimized backup operations for multiple virtual disks by using parallel sessions, one session for each virtual disk
Optimized backup operations for a single virtual disk by using parallel sessions, multiple sessions for each virtual disk
 VVols Support

 Veeam Availability Suite – Not-For-resale (NFR) License

https://www.veeam.com/blog/how-to-get-free-veeam-nfr-key.html
 If you have a Home Lab and need a backup solution check this out!
Thank you

Questions