ZDLRA – Deep Dive

Daniele Massimi – Principal Consultant

BE-IMS

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
 Agenda

1. ZDLRA – Functionality under the hood

2. Installation and Configuration from bottom up

3. Have a look inside – (just particularities)

4. From protected Databases to backups

5. Backup and Recovery Capabilities

6. Recovery Appliance Views and Utilities

7. Conclusion

2 21.06.2017 ZDLRA - in Action

 Who am I
Daniele Massimi, Trivadis AG (CH, Bern)
Principal Consultant daniele.massimi@trivadis.com

Seit 2012 bei Trivadis IMS (Infrastructure Service Management)

Oracle DBA seit 2000 (Exadata > 6 Jahre)

Infrastruktur-Architekturen, Operations & Administration, Upgrades and Migration,

High Availability, Backup & Recovery, Virtualizierung
Engineered Systems (Exadata, ODA, ZDLRA, PCA)

Trainer (Oracle Architectur and Internal, 12c New Features, Exadata)

Oracle Certifications
Oracle Certified Master (11g)
Oracle Certified Professional (8i – 12c)
Oracle Certified Expert (RAC)
Oracle Implementation Specialist (Exadata, OVM)
3 21.06.2017 ZDLRA - in Action
 Our company.

Trivadis is a market leader in IT consulting, system integration, solution engineering

and the provision of IT services focusing on and
technologies
in Switzerland, Germany, Austria and Denmark. We offer our services in the following
strategic business fields:

OPERATION

Trivadis Services takes over the interactive operation of your IT systems.

4 21.06.2017 © Trivadis – The Company

 ZDLRA
Functionality under the Hood

5 21.06.2017 ZDLRA - in Action

 Common problems with Oracle backups

RPO depends on archive backup frequency (many minutes or hours)

Backup window and nightly batches compete for resources
High network bandwidth usage for full backups
Low-speed backups and restores
Backup and restore success ratio penalized by shared infrastructure
Critical databases require Data Guard for transaction safety
– Additional storage space, license and computation required
Expiration of backup set on ML: need to crosscheck and delete expired
Lack of backup validation
…

6 21.06.2017 ZDLRA - in Action

 Recovery Appliance Architecture

Source: Oracle

7 21.06.2017 ZDLRA - in Action

 Minimal Backup Overhead

Traditional Backup Recovery Appliance

– Read/Write for Disk/Tape Backup – Offloading operations
– Deduplication – Delta Push
– Compression – Delta Store
– Validation
– Deletion
– Merging
All on database host
 Degrade performance
Source: Oracle

8 21.06.2017
 Delta Push
Delta Push is a highly optimized form of source-
Backuped Database side optimization
– Through RMAN block change tracking
– No reading of unchanged data
Two operations on the protected Database
– Incremental "forever" backup
– Real time redo transport
One-time full backup as prerequirement
Afterwards Incremental forever backup
Validate incoming Backups against corrupted
Changed Data physical blocks
Source: Oracle Compress the Backups using special block-level
Algorithm

 Less data, less I/O, less network traffic

9 21.06.2017 ZDLRA - in Action

 Eliminate data loss – Real Time redo transport

Real Time redo transport

– Data Guard like but asynchronous
– Changes out of Logbuffer transfered
– Validate and writes to a stage area
Redolog switch on database Source: Oracle

– RA converts the staged redo data to a

compressed archived redolog backup  Reduces RPO to (near) zero
– Writes this archlog backup to catalog
– Ready for future restores
Explicit Archivelog backup no more necessary

10 21.06.2017 ZDLRA - in Action

 Delta Store – the “brains” of the Recovery Appliance

Backup
Day N Virtual Full
Totality of all Database Backup in a
Storage Location Day 1 Virtual Full
validates the incoming blocks Day 0 Day 1 Day N
compresses, indexes and stores Full Incr Incr
deduplicate, less storage usage
High number of virtual full backups
Higher recover window
Source: Oracle
Delta Store

11 21.06.2017 ZDLRA - in Action

 Delta Store – efficient restore/recover
Day ‘N’ Full Backup
Restore
Creates a Virtual Full Database
Backups
No need of transfer and apply
incremental backups during restore
operations Day 0 Day 1 Day N
Roll forward with restored redologs Full Incr Incr
Uses Exadata performance and
scalability

Delta Store Source: Oracle

12 21.06.2017 ZDLRA - in Action

 Delta Pools

Chunks inside the Delta Store

Backups will be indexed and stored inside the Delta Pools
Set of datafiles blocks from which RA constucts Virtual Full Backups Source: Oracle

Each backuped datafile will have his own delta Pool

Automated delta pool space management
– Protection Policy will be inherited to database retention policy
– Delete automatically expired or obseleted Backups
– Periodically reorganising for performance improving
– Delta pool optimization when old blocks are deleted and new incremental Backup
arriving

13 21.06.2017 ZDLRA - in Action

 Delta Push – how it works (1/2)
Clients address the HTTP Servlet on ZDLRA
RMAN ships incremental Backup to Staging Area
Data will be validated and Metadata will be stored on RAC Database
Data Blocks of Backupsets will be indexed and stored on Delta Store

Source: Oracle
14 21.06.2017 ZDLRA - in Action
 Delta Push – how it works (2/2)
Optionally Real Time Redo Transport could be activated
ZDLRA will be using Data Guard Technology (RFS on RA)
Validated Redo Blocks will be stored on Delta Store
Archive Logs will be generated whenever a Log Switch happen on Prod DB
Optionally you can replicate or copy to a remote ZDLRA or Tape

15 21.06.2017 ZDLRA - in Action Source: Oracle

 Restore – how it works

Clients address the HTTP Servlet on ZDLRA

ZDLRA re-assemble and validate "virtual Backupsets"
Data Blocks will be shipped back to Client

Source: Oracle

16 21.06.2017 ZDLRA - in Action

 Policy-Based Database Protection as a Service
Gold Policy – Customer Critical
Standardized
Disk: 45 days protection policies
Tape: 90 days

– Recovery window
Silver Policy – Internal Critical – Retention
Disk: 30 days – Replication
Tape: 45 days

Bronze Policy - Test/Dev

Disk: 15 days
Tape: 30 days

17 21.06.2017 ZDLRA Workshop - Introduction

 Protection Policies Attributes
Storage Location  Recovery Appliance storage location for storing backups
Recovery Window Disk  Tape recovery window goal for the protected database
Recovery Window Tape  Tape recovery window goal for the protected database
Max Retention Policy  maximum length of time that the Recovery Appliance retains
backups for databases that use this retention policy
Guaranteed Copy  guaranteed copy setting, which determines whether backups
protected by this policy must be copied to tape or replicated before being considered
for deletion
Polling Policy  optional backup polling policy that determines whether Recovery
Appliance polls a storage location for backups or deletion
Unprotected Window maximum acceptable difference between the current time
and the latest time that the database can be restored

18 21.06.2017 ZDLRA Workshop - Protection Policies

 Recovery Window – Index Efficiency (1/2)
Only new version of Data Blocks will be backed up (inc level 1)
Recovery windows controls deletion policy of the blocks
Main scope is to maintain the restorability within the recovery windows
Old and no more needed blocks will be deleted
If enough space is available the Recovery Windows can be overachieved

19 21.06.2017 ZDLRA - in Action Source: Oracle

 Recovery Window – Index Efficiency (2/2)
Once the old blocks was deleted, we will have a defragmentation within the
storage
This situation will provide random I/O instead of sequential I/O  Performance
degradation
Intermittently the RA will automatically reorganize the blocks
Implicitly all touched blocks will be phisical and logical checked on his integrity

Source: Oracle
20 21.06.2017 ZDLRA - in Action
 Storage Locations
Main Storage Location are the Container within the ASM Diskgroup
It is possible to have more than one Storage Location in ASM

Backup Polling Locations are Storage Location outside from ZDLRA (e.g. NFS Mount)
• Backups are placed directly to this location without interacting with ZDLRA
• With Polling Policies you can define how often and where
is the polling directory
• Once all requirement meets  e.g. backup related to a
protected database a copy to the RA will be created
• After copying process the protected database will be delete
the backup Automatically from polling directory Source: Oracle

21 21.06.2017 ZDLRA - in Action

 Installation and Configuration
from bottom up

22 21.06.2017 ZDLRA - in Action

 Installation Steps

Create the ZDLRA configuration using

the latest OEDA

Start the Re-Imaging and Setup from the first

Computing Node (like an Exadata Setup)

[root@zdldbadm01 linux-x64]# ./install.sh -cf ./WorkDir/zdl-zdl.xml –r 1-16

23 21.06.2017 ZDLRA - in Action

 Installation Steps
Recovery Appliance specific Installation Steps
[root@zdldbadm01 install]# ./ra_install.pl --help
ra_install.pl - Recovery Appliance Installer
You must be logged in as root to run this command.
All steps should be run successfully for install.
Usage: ./ra_install.pl --help | --step=STEP_NUMBER {--install|--uninstall} [--
stdout]
Options:
--help: displays this help message
--step=number: Which step to run
--stdout: Print all messages to stdout rather then the log file
--install: Installs the step [Default]
--uninstall: Uninstalls the step
Step Numbers:
Step 1 - Validation and Configuration Prep
Step 2 - OS Setup
Step 3 - Oracle User Setup
Step 4 - DBFS Setup
Step 5 - Tape Backup configuration
Step 6 - ZDLRA DB Backup Setup
Step 7 - Enable ZDLRA Services

24 21.06.2017 ZDLRA - in Action

 Configuration Steps

Deployment of the Agents on the compute Nodes

Recovery Appliance Discovery
Installation of the Backup Module on any Clients
[oracle@odax30 zdlra]$ java -jar ra_install.jar -dbUser ra_orcl02 -dbPass
manager -host zdlingest-scan -port 1521 -serviceName zdlra -walletDir
/u01/app/oracle/admin/orcl02/xdb_wallet -libDir $ORACLE_HOME/lib

Recovery Appliance Install Tool, build 2015-11-03

Recovery Appliance credentials are valid.
Recovery Appliance wallet created in directory
/u01/app/oracle/admin/orcl02/xdb_wallet.
Recovery Appliance initialization file
/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/raorcl02.ora created.
Downloading Recovery Appliance Software Library from file ra_linux64.zip.
Downloaded 27189305 bytes in 5 seconds. Transfer rate was 5437861 bytes/second.
Download complete.
[oracle@odax30 zdlra]$

25 21.06.2017 ZDLRA - in Action

 Have a look inside
(just particularities)

26 21.06.2017 ZDLRA - in Action

 On the computing Nodes
Metadata Database
– RAC Database (also used for Recovery Catalog (VPC))
– Non Container Database
– DBFS Mount-Points for Backup and Replication purpose
– DBFS added as Cluster Resources
oracle@zdldbadm01:~/ [+ASM1] df -h | grep dbfs
dbfs-@repdbfs.local:/
957M 120K 956M 1% /dbfs_repdbfs
dbfs-@obdbfs.local:/ 300G 23G 278G 8% /dbfs_obdbfs

– HTTP Servlet inside the database will be started for communication between the
clients and the ZDLRA (dbms_ra.startup_recovery_appliance)
SQL> exec dbms_ra.startup_recovery_appliance;

PL/SQL procedure successfully completed.

27 21.06.2017 ZDLRA - in Action

 On the computing Nodes
– Three new FS for internally purposes are provided
[oracle@zdldbadm01 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbSys1
30G 15G 14G 54% /
tmpfs 253G 53M 253G 1% /dev/shm
/dev/sda1 504M 72M 407M 15% /boot
/dev/mapper/VGExaDb-LVDbOra1
99G 59G 36G 63% /u01
/dev/mapper/VGExaDb-LVRADump
296G 191M 281G 1% /radump  Dump destination for internal Stuff
/dev/mapper/VGExaDb-LVRABackup
99G 351M 94G 1% /rabackup
/dev/mapper/VGExaDb-LVOBIndex
296G 191M 281G 1% /obindex
dbfs-@repdbfs.local:/
957M 120K 956M 1% /dbfs_repdbfs
dbfs-@obdbfs.local:/ 300G 4.6M 300G 1% /dbfs_obdbfs

28 21.06.2017 ZDLRA - in Action

 On the computing Nodes
Backup of Metadata Database
– Done using Export of RASYS Schema every 2 Hours
– Backup Location on DBFS FS: /dbfs_obdbfs/OSB/ra_export
– Scheduling via "anacron"
[root@zdldbadm01 ~]# cat /etc/anacrontab
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
#RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command

1 65 cron.daily nice run-parts /etc/cron.daily
7 70 cron.weekly nice run-parts /etc/cron.weekly
30 75 cron.monthly nice run-parts /etc/cron.monthly

[root@zdldbadm01 cron.hourly]# ls -l /etc/cron.hourly/

total 4
-rwx------ 1 root root 409 Nov 10 2015 0anacron
lrwxrwxrwx 1 root root 46 Aug 25 16:20 ra_export.pl -> /opt/oracle.RecoveryAppliance/bin/ra_export.pl

29 21.06.2017 ZDLRA - in Action

 On the computing Nodes
ASM Configuration
– Two Diskgroups  DELTA (for Backups), CATALOG (Metadatas)
– New special sub-Directory CONTAINER for storing Container Files which
contains Delta Pools and Delta Stores
– Each Container File as a size of 2 TB
ASMCMD [+delta/zdlra] > ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTAINER/
Y DATAFILE/
Y TEMPFILE/

ASMCMD [+delta/zdlra/CONTAINER] > ls -s

Block_Size Blocks Bytes Space Name
512 4294959104 2199019061248 4398059094016 con.365.920823681
512 4294959104 2199019061248 4398059094016 con.366.920823687
512 4294959104 2199019061248 4398059094016 con.367.920823699
...

30 21.06.2017 ZDLRA - in Action

 On the cell Nodes

Fortunatelly nothing special ☺

Same appearance as in Exadata Storage Cell
During the Installation the "Write Back" Option will be enabled automatically for the
Flashcache
Little difference on the Grid Disks compared with the Exadata, only 10 Grid Disks
instead of 12 for the CATALOG ASM Disk Group

31 21.06.2017 ZDLRA - in Action

 From protected Databases
to backups

32 21.06.2017 ZDLRA - in Action

 Protected Databases

Once the Environment meets all the requirements we can add the databases to the
ZDLRA

The requirement are:

• Network connectivity
• Backup Module is installed for every Oracle Home
• Definition of Protection Policies are created (Retention Time)

 Block Change Tracking not mandatory, but recommended !!!

Two Methods are possible:

• Enterprise Manager
• Manually  DBMS_RA PL/SQL Package

33 21.06.2017 ZDLRA - in Action

 Prerequisite for Adding Protected Database
(EM and manually)
Create an VPD User on the Metadata Database
SQL> create user ra_orcl03 identified by <pwd>;

Grant the Create Session privilege

SQL> grant create session to ra_orcl03;

34 21.06.2017 ZDLRA - in Action

 Adding Protected Database with EM
Add Protected Database on Recovery Appliance Page
– Choose the Protection Policy

35 21.06.2017 ZDLRA - in Action

 Adding Protected Database with EM

Configuring Backup Settings

– Register database
– Add RMAN configuration (configure...)
– Enabling Redo Transport
(add parameter to spfile and restart DB)

36 21.06.2017 ZDLRA - in Action

 Scheduling Protected Database with EM

Protected Database is ready for backing up

– We need a Schedule

37 21.06.2017 ZDLRA - in Action

 Scheduling Protected Database with EM

Configure the repeating interval for

the Incremental-Forever Backups
Configure the Archive Log Backups and
deletion rule

38 21.06.2017 ZDLRA - in Action

 Scheduling Protected Database with EM
Also useful as Script Generator  here the Script is not modifiable !!!

39 21.06.2017 ZDLRA - in Action

 Scheduling Protected Database with EM
Output of an Backup Schedule on EM13c

40 21.06.2017 ZDLRA - in Action

 Adding Protected Database manually
Add Database for the protected database
begin
dbms_ra.add_db (
db_unique_name => 'orcl03',
protection_policy_name => 'bronze',
reserved_space => '250G');
end;

Grant access to Virtual Private Catalog

begin
dbms_ra.grant_db_access (
db_unique_name => 'orcl03',
username => 'ra_orcl03');
end;

Configure the RMAN Channel for ZDLRA

CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet
credential_alias=zdlingest-scan:1521/zdlra:dedicated')";

41 21.06.2017 ZDLRA - in Action

 Adding Protected Database manually
Enable Real-Time Redo Transport
ALTER SYSTEM SET REDO_TRANSPORT_USER=ra_orcl03 SCOPE=BOTH;

Restart the Database

[oracle@odax30 ~]$ srvctl stop database -db orcl03

[oracle@odax30 ~]$ srvctl start database -db orcl03

42 21.06.2017 ZDLRA - in Action

 Execute Backup manually
Backups works like in traditional way
Only "forever" incremental level 1 backups should be schedule
Level 0 Backup are of course always possible
[oracle@odax30 ~]$ rman target / catalog ra_orcl02/manager@zdlingest-scan:1521/zdlra
...
connected to target database: ORCL02 (DBID=3592015831)
connected to recovery catalog database

RMAN> backup incremental level 1 database;

It works also with old fashion syntax  soft link from libra.so to libobk.so is
needed !!!
run {
allocate channel c1 type sbt_tape;
backup incremental level 1 database plus archivelog delete input;
}
...
ORA-27211: Failed to load Media Management Library
Scheduling is also free selectable, EM is recommended !
43 21.06.2017 ZDLRA - in Action
 Copy to Tape

44 21.06.2017 ZDLRA - in Action

 Purpose of Copy to Tape

Tape Infrastructure provides a second or third copy of your backups

Tape Infrastructure could be located on different Data Center  DR capability !

Tape backups are "cheaper" for long-term Storage

High efficiency is given in combination with ZDLRA

Oracle Secure Backup just pre-installed on ZDLRA

45 21.06.2017 ZDLRA - in Action

 Setting up Copy to Tape
Configuring Media Management Library
• Library Name
• Number of Drives
• Number of Restore Drives
• Media Manager Location (libobk.so)

46 21.06.2017 ZDLRA - in Action

 Setting up Copy to Tape Template

Template helps to define specific attributes

• Backup types
• Database or Protection Policies
• Recovery Windows will be inherited
• Number of copies
• Runtime Window
• Schedule

47 21.06.2017 ZDLRA - in Action

 Executing Copy to Tape

With the scheduling will be iniatiate the copy to tape process

backupset of defined database or protection policy will be copied to tape
Long Term Backup should be done with copy_backup procedure

Restores of backupset from tape will be retrieved transparently !!!

48 21.06.2017 ZDLRA - in Action

 Recovery Capabilities

49 21.06.2017 ZDLRA - in Action

 General statement about Recoveries of protected
Databases
Recoveries of protected databases can be done with EM or manually
With EM several Recovery-Scenarios are possible
All Recovery scenarios with EM are full automated
(e.g. Creation of auxiliary Databases, etc.)
EM generates RMAN Scripts for each Scenario
with the possibility of adaption
Possibility of selection between full and point-in-time
recoveries
Duplicate Database (for standby) need some preparation
(e.g. FS-Structure (admin tree))

50 21.06.2017 ZDLRA - in Action

 Protected Database Recovery with EM

From database Page under Availability  Backup & Recovery  Perform Recovery

51 21.06.2017 ZDLRA - in Action

 Protected Database Recovery with EM

Choose the needed Recovery Scenario

... You can choose an other restore

location if needed...

52 21.06.2017 ZDLRA - in Action

 Protected Database Recovery with EM

A schedule will be created

... And you can adapt the script if needed and submit the job...

53 21.06.2017 ZDLRA - in Action

 Protected Database Recovery with EM
The Job can be monitored by clicking the "View Job" button...

54 21.06.2017 ZDLRA - in Action

 Protected Database Recovery manually

Restore can be done also manually

– of course db must be mounted before...

– RMAN Client should be configured otherwise parameter are needed (e.g. ENV=...)
run {
restore database;
recover database;
alter database open;
}

Or a point-in-time Recovery
run {
set until time '08.09.2016 08:45:00';
restore database;
recover database;
alter database open resetlogs;
}

55 21.06.2017 ZDLRA - in Action

 Real Time Redo Transport – what happens really ?
We tested what happens really when the protected database crash…
▪ We did a backup of a protected database with enabled real time redo transport
RMAN> backup incremental level 1 database plus archivelog delete input;

▪ then we shutdown the database inconsistently (abort)… before we checked the actual SCN
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
28933241
SQL> shutdown abort

▪ the ZDLRA figure out the crash of the protected database and catalog the redo stream from the
staging area as archivelog with the latest SCN he know… that will be latest consistent SCN for
restore and recover the database
RMAN> list backup of archivelog all;
.
.
1 26 28933197 18-APR-17 28933202 18-APR-17
1 27 28933202 18-APR-17 28933232 18-APR-17
1 28 28933232 18-APR-17 28933246 18-APR-17
56 21.06.2017 ZDLRA - in Action
 Recovery Appliance Views and
Utilities

57 21.06.2017 ZDLRA - in Action

 Some RA Views

It is also possible to read the Recovery Appliance Information over Views from Recover
Appliace Metadata Database
View Name Description

RA_DATABASE Information about Protected Databases

RA_DB_ACCESS Describes User Accounts that have access to which protected database

RA_PROTECTION_POLICY Protection Policies defined on the Recovery Appliance

RA_DATABASE_STORAGE_USAGE Storage usage for each protected database

RA_RESTORE_RANGE Describe Restore Range for each protected database

RA_INCIDENT_LOG Describe Recovery Appliance Incidents

RA_REPLICATION_SERVER Replication Server Information (if used)

58 21.06.2017 ZDLRA - in Action

 rastat.pl Utility
rastat Utility help to evaluate the performance of Recovery Appliance
– NETBACKUP/NETRESTORE
• Measure the network performance
– ASMREAD/ASMWRITE
• Measure the ASM Disk I/O Performance
– CONTAINERREAD/CONTAINERWRITE/CONTAINERALLOC
• Measure the Container Disk I/O Performance
[oracle@zdldbadm01 ~]$ perl /opt/oracle.RecoveryAppliance/client/rastat.pl --test=CONTAINERWRITE --
filesize=2048M --rasys=rasys/manager@zdlingest-scan:1521/zdlra --diskgroup=/:DELTA
RECOVERY APPLIANCE WRITE IO TEST TO CONTAINER FILES

Disk Group: /:DELTA

2048 MB, .75s write IO time, .51s CPU time, 2724.80 MB/s, 68.51% CPU usage

PL/SQL procedure successfully completed.

59 21.06.2017 ZDLRA - in Action

 dbms_ra Package

dbms_ra provides administration function from inside the RA Database

Ca. 50 Procedure for different types of administration
– Start/Stop of Recovery Appliance
– Add/Delete Protected Databases
– Grants Access to specific User to enable to backup and restore
– Add/Update Protection Policies
– Manage Tape Configuration (if available)
– Add Repliaction Server (if available)

But, the utilization of Enterprise Manager is recommended !!!

60 21.06.2017 ZDLRA - in Action

 Conclusion

61 21.06.2017 ZDLRA - in Action

 Conclusion

ZDLRA worked in our Tests as

expected
Very good implementation of well
known Backup and Recovery
Technology on Engineered System
RPO is (near) zero with Real Time
Redo Transport
Even RTO ist much better then
existing systems
Very simple Management with EM
 ZDLRA will find it’s
 Risk reduced, cost reduced,
qualitiy improved customer !
62 21.06.2017 ZDLRA - in Action
Questions and Answers...
Daniele Massimi – Principal Consultant
Tel. +41 58 459 50 92
daniele.massimi@trivadis.com

63 21.06.2017 ZDLRA - in Action