AWS Certified SysOps

Administrator - Associate
By Cloud Mentor Pro

© Copyright by Cloud Mentor Pro | Confidential 1

 Section 6
• AWS Lambda
• Amazon EC2 Storage & Data Management

© Copyright by Cloud Mentor Pro | Confidential 2

AWS Lambda
It’s a serverless world

© Copyright by Cloud Mentor Pro | Confidential 3

 Why AWS Lambda
• Virtual Servers in the Cloud
• Limited by RAM and CPU
• Continuously running
Amazon EC2
• Scaling means intervention to add / remove servers

• Virtual functions – no servers to manage!

• Limited by time - short executions
• Run on-demand
Amazon Lambda
• Scaling is automated!
© Copyright by Cloud Mentor Pro | Confidential 4
 Benefits of AWS Lambda
• Easy Pricing:
• Pay per request and compute time
• Free tier of 1,000,000 AWS Lambda requests and 400,000 GBs of compute time

• Integrated with the whole AWS suite of services

• Integrated with many programming languages
• Easy monitoring through AWS CloudWatch
• Easy to get more resources per functions (up to 10GB of RAM!)
• Increasing RAM will also improve CPU and network!

© Copyright by Cloud Mentor Pro | Confidential 5

 AWS Lambda language support
• Node.js (JavaScript)
• Python
• Java
• C# (.NET Core) / Powershell
• Ruby
• Custom Runtime API (community supported, example Rust or Golang)

• Lambda Container Image

• The container image must implement the Lambda Runtime API
• ECS / Fargate is preferred for running arbitrary Docker images
© Copyright by Cloud Mentor Pro | Confidential 6
 AWS Lambda Integrations Main ones

© Copyright by Cloud Mentor Pro | Confidential 7

 Example: Serverless Thumbnail creation

© Copyright by Cloud Mentor Pro | Confidential 8

 Example: Serverless CRON Job

Trigger
Every 1 hour

CloudWatch Events AWS Lambda Function

EventBridge Perform a task

© Copyright by Cloud Mentor Pro | Confidential 9

 AWS Lambda Pricing: example
• You can find overall pricing information here:
https://aws.amazon.com/lambda/pricing/
• Pay per calls :
• First 1,000,000 requests are free
• $0.20 per 1 million requests thereafter ($0.0000002 per request)
• Pay per duration: (in increment of 1 ms )
• 400,000 GB -seconds of compute time per month for FREE
• == 400,000 seconds if function is 1GB RAM
• == 3,200,000 seconds if function is 128 MB RAM
• After that $1.00 for 600,000 GB-seconds
• It is usually very cheap to run AWS Lambda so it’s very popular
© Copyright by Cloud Mentor Pro | Confidential 10
 CloudWatch Events / EventBridge

© Copyright by Cloud Mentor Pro | Confidential 11

 S3 Events Notifications
• S3:ObjectCreated, S3:ObjectRemoved,
S3:ObjectRestore, S3:Replication…
• Object name filtering possible (*.jpg)
• Use case: generate thumbnails of images uploaded
to S3

• S3 event notifications typically deliver events in

seconds but can sometimes take a minute or
longer
• If two writes are made to a single non- versioned
object at the same time, it is possible that only a
single event notification will be sent
• If you want to ensure that an event notification is
sent for every successful write, you can enable
versioning on your bucket.
© Copyright by Cloud Mentor Pro | Confidential 12
 Simple S3 Event Pattern – Metadata Sync

© Copyright by Cloud Mentor Pro | Confidential 13

 Lambda Execution Role (IAM Role)
• Grants the Lambda function permissions to AWS services / resources
• Sample managed policies for Lambda:
• AWSLambdaBasicExecutionRole – Upload logs to CloudWatch.
• AWSLambdaKinesisExecutionRole – Read from Kinesis
• AWSLambdaDynamoDBExecutionRole – Read from DynamoDB Streams
• AWSLambdaSQSQueueExecutionRole – Read from SQS
• AWSLambdaVPCAccessExecutionRole – Deploy Lambda function in VPC
• AWSXRayDaemonWriteAccess – Upload trace data to X-Ray.

• When you use an event source mapping to invoke your function, Lambda
uses the execution role to read event data.
• Best practice: create one Lambda Execution Role per function
© Copyright by Cloud Mentor Pro | Confidential 14
 Lambda Resource Based Policies
• Use resource-based policies to give other accounts and AWS services
permission to use your Lambda resources
• Similar to S3 bucket policies for S3 bucket
• An IAM principal can access Lambda:
• if the IAM policy attached to the principal authorizes it (e.g. user access)
• OR if the resource-based policy authorizes (e.g. service access)

• When an AWS service like Amazon S3 calls your Lambda function, the
resource-based policy gives it access.

© Copyright by Cloud Mentor Pro | Confidential 15

 Lambda Logging & Monitoring
• CloudWatch Logs:
• AWS Lambda execution logs are stored in AWS CloudWatch Logs
• Make sure your AWS Lambda function has an execution role with an IAM policy
that authorizes writes to CloudWatch Logs
• CloudWatch Metrics:
• AWS Lambda metrics are displayed in AWS CloudWatch Metrics
• Invocations, Durations, Concurrent Executions
• Error count, Success Rates, Throttles
• Async Delivery Failures
• Iterator Age (Kinesis & DynamoDB Streams)

© Copyright by Cloud Mentor Pro | Confidential 16

 Lambda Tracing with X-Ray
• Enable in Lambda configuration (Active Tracing)
• Runs the X-Ray daemon for you
• Use AWS X-Ray SDK in Code
• Ensure Lambda Function has a correct IAM Execution Role
• The managed policy is called AWSXRayDaemonWriteAccess
• Environment variables to communicate with X-Ray
• _X_AMZN_TRACE_ID: contains the tracing header
• AWS_XRAY_CONTEXT_MISSING: by default, LOG_ERROR
• AWS_XRAY_DAEMON_ADDRESS: the X-Ray Daemon IP_ADDRESS:PORT

© Copyright by Cloud Mentor Pro | Confidential 17

 Lambda Function Configuration
• RAM:
• From 128MB to 10GB in 1MB increments
• The more RAM you add, the more vCPU credits you get
• At 1,792 MB, a function has the equivalent of one full vCPU
• After 1,792 MB, you get more than one CPU, and need to use multi-threading in
your code to benefit from it (up to 6 vCPU)
• If your application is CPU-bound (computation heavy), increase RAM

• Timeout: default 3 seconds, maximum is 900 seconds (15 minutes)

© Copyright by Cloud Mentor Pro | Confidential 18

 Lambda Execution Context
• The execution context is a temporary runtime environment that
initializes any external dependencies of your lambda code
• Great for database connections, HTTP clients, SDK clients…
• The execution context is maintained for some time in anticipation of
another Lambda function invocation
• The next function invocation can “re-use” the context to execution time
and save time in initializing connections objects
• The execution context includes the /tmp directory

© Copyright by Cloud Mentor Pro | Confidential 19

 Initialize outside the handler

© Copyright by Cloud Mentor Pro | Confidential 20

 Lambda Functions /tmp space
• If your Lambda function needs to download a big file to work…
• If your Lambda function needs disk space to perform operations…
• You can use the /tmp directory
• Max size is 10GB
• The directory content remains when the execution context is frozen,
providing transient cache that can be used for multiple invocations
(helpful to checkpoint your work)
• For permanent persistence of object (non temporary), use S3
• To encrypt content on /tmp, you must generate KMS Data Keys

© Copyright by Cloud Mentor Pro | Confidential 21

 Lambda Concurrency and Throttling
• Concurrency limit: up to 1000 concurrent executions

• Can set a “reserved concurrency” at the function level (=limit)

• Each invocation over the concurrency limit will trigger a “Throttle”
• Throttle behavior:
• If synchronous invocation => return ThrottleError - 429
• If asynchronous invocation => retry automatically and then go to DLQ
• If you need a higher limit, open a support ticket
© Copyright by Cloud Mentor Pro | Confidential 22
 Lambda Concurrency Issue
• If you don’t reserve (=limit) concurrency, the following can happen:

© Copyright by Cloud Mentor Pro | Confidential 23

 Concurrency and Asynchronous Invocations
• If the function doesn't have enough
concurrency available to process all events,
additional requests are throttled.
• For throttling errors (429) and system
errors (500-series), Lambda returns the
event to the queue and attempts to run
the function again for up to 6 hours.
• The retry interval increases exponentially
from 1 second after the first attempt to a
maximum of 5 minutes.

© Copyright by Cloud Mentor Pro | Confidential 24

 Cold Starts & Provisioned Concurrency
• Cold Start:
• New instance => code is loaded and code outside the handler run (init)
• If the init is large (code, dependencies, SDK…) this process can take some time.
• First request served by new instances has higher latency than the rest
• Provisioned Concurrency:
• Concurrency is allocated before the function is invoked (in advance)
• So the cold start never happens and all invocations have low latency
• Application Auto Scaling can manage concurrency (schedule or target utilization)
• Note:
• Note: cold starts in VPC have been dramatically reduced in Oct & Nov 2019
• https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/

© Copyright by Cloud Mentor Pro | Confidential 25

 Reserved and Provisioned Concurrency

https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html
© Copyright by Cloud Mentor Pro | Confidential 26
 Lambda Monitoring – CloudWatch Metrics
• Invocations – number of times your function is invoked (success/failure)
• Duration – amount of time your function spends processing an event
• Errors – number of invocations that result in a function error
• Throttles – number of invocation requests that are throttled (no concurrency
available)
• DeadLetterErrors – number of times Lambda failed to send an event to a DLQ
(async invocations)
• IteratorAge – time between when a Stream receives a record and when the Event
Source Mapping sends the event to the function (for Event Source Mapping that
reads from Stream)
• ConcurrentExecutions – number of function instances that are processing events

© Copyright by Cloud Mentor Pro | Confidential 27

 Example Lambda Metrics Dashboard

© Copyright by Cloud Mentor Pro | Confidential 28

 Lambda Monitoring – CloudWatch Alarms
• Example 1 – No Lambda Invocations in the last hour using Invocations
CloudWatch Metric

• Example 2 – When error > 0 using Errors CloudWatch Metric

• Example 3 – When throttles > 0 using Throttles CloudWatch Metric

© Copyright by Cloud Mentor Pro | Confidential 29

 Lambda Monitoring – CloudWatch Logs

© Copyright by Cloud Mentor Pro | Confidential 30

 Lambda Monitoring – CloudWatch Logs Insights
• Allows you to search through all your Lambda functions logs

© Copyright by Cloud Mentor Pro | Confidential 31

 Lambda Monitoring – CloudWatch Logs Insights

© Copyright by Cloud Mentor Pro | Confidential 32

 Lambda Monitoring – Lambda Insights
• Collects, aggregates, and summarizes:
• System-level Metrics – CPU time, memory, disk,
network
• Diagnostic Information – cold starts, Lambda
worker shutdowns
• Helps you isolate issues with your Lambda
functions and resolve them quickly
• Uses a CloudWatch Lambda Extension
(provided as a Lambda layer)

© Copyright by Cloud Mentor Pro | Confidential 33

Amazon EC2 Storage & Data
Management
EBS, Instance Store & EFS

© Copyright by Cloud Mentor Pro | Confidential 34

 What’s an EBS Volume?
• An EBS (Elastic Block Store) Volume is a network drive you can attach to
your instances while they run
• It allows your instances to persist data, even after their termination
• They can only be mounted to one instance at a time (at the CCP level)
• They are bound to a specific availability zone

• Analogy: Think of them as a “network USB stick”

• Free tier: 30 GB of free EBS storage of type General Purpose (SSD) or
Magnetic per month

© Copyright by Cloud Mentor Pro | Confidential 35

 EBS Volume
• It’s a network drive (i.e. not a physical drive)
• It uses the network to communicate the instance, which means there might be a
bit of latency
• It can be detached from an EC2 instance and attached to another one quickly

• It’s locked to an Availability Zone (AZ)

• An EBS Volume in us-east-1a cannot be attached to us-east-1b
• To move a volume across, you first need to snapshot it

• Have a provisioned capacity (size in GBs, and IOPS)

• You get billed for all the provisioned capacity
• You can increase the capacity of the drive over time
© Copyright by Cloud Mentor Pro | Confidential 36
 EBS Volume - Example

© Copyright by Cloud Mentor Pro | Confidential 37

 EC2 Instance Store
• EBS volumes are network drives with good but “limited” performance
• If you need a high-performance hardware disk, use EC2 Instance Store

• Better I/O performance

• EC2 Instance Store lose their storage if they’re stopped (ephemeral)
• Good for buffer / cache / scratch data / temporary content
• Risk of data loss if hardware fails
• Backups and Replication are your responsibility

© Copyright by Cloud Mentor Pro | Confidential 38

 Local EC2 Instance Store

© Copyright by Cloud Mentor Pro | Confidential 39

 EBS Volume Types
• EBS Volumes come in 6 types
• gp2 / gp3 (SSD): General purpose SSD volume that balances price and performance for a wide
variety of workloads
• io1 / io2 Block Express (SSD): Highest-performance SSD volume for mission-critical low-latency or
high-throughput workloads
• st1 (HDD): Low cost HDD volume designed for frequently accessed, throughput- intensive
workloads
• sc1 (HDD): Lowest cost HDD volume designed for less frequently accessed workloads

• EBS Volumes are characterized in Size | Throughput | IOPS (I/O Ops Per Sec)
• When in doubt always consult the AWS documentation – it’s good!
• Only gp2/gp3 and io1/io2 Block Express can be used as boot volumes

© Copyright by Cloud Mentor Pro | Confidential 40

 EBS Volume Types Use cases
General Purpose SSD
• Cost effective storage, low-latency
• System boot volumes, Virtual desktops, Development and test
environments
• 1 GiB - 16 TiB
• gp3:
• Baseline of 3,000 IOPS and throughput of 125 MiB/s
• Can increase IOPS up to 16,000 and throughput up to 1000 MiB/s independently
• gp2:
• Small gp2 volumes can burst IOPS to 3,000
• Size of the volume and IOPS are linked, max IOPS is 16,000
• 3 IOPS per GB, means at 5,334 GB we are at the max IOPS
© Copyright by Cloud Mentor Pro | Confidential 41
 EBS Volume Types Use cases
Provisioned IOPS (PIOPS) SSD
• Critical business applications with sustained IOPS performance
• Or applications that need more than 16,000 IOPS
• Great for databases workloads (sensitive to storage perf and consistency)
• io1 (4 GiB - 16 TiB):
• Max PIOPS: 64,000 for Nitro EC2 instances & 32,000 for other
• Can increase PIOPS independently from storage size
• io2 Block Express (4 GiB – 64 TiB):
• Sub-millisecond latency
• Max PIOPS: 256,000 with an IOPS:GiB ratio of 1,000:1
• Supports EBS Multi-attach
© Copyright by Cloud Mentor Pro | Confidential 42
 EBS Volume Types Use cases
Hard Disk Drives (HDD)
• Cannot be a boot volume
• 125 GiB to 16 TiB
• Throughput Optimized HDD (st1)
• Big Data, Data Warehouses, Log Processing
• Max throughput 500 MiB/s – max IOPS 500
• Cold HDD (sc1):
• For data that is infrequently accessed
• Scenarios where lowest cost is important
• Max throughput 250 MiB/s – max IOPS 250

© Copyright by Cloud Mentor Pro | Confidential 43

 EBS –Volume Types Summary

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#solid-state-drives

© Copyright by Cloud Mentor Pro | Confidential 44

 EBS Multi-Attach – io1/io2 family
• Attach the same EBS volume to multiple EC2
instances in the same AZ
• Each instance has full read & write permissions to
the high-performance volume
• Use case:
• Achieve higher application availability in clustered Linux
applications (ex: Teradata)
• Applications must manage concurrent write operations
• Up to 16 EC2 Instances at a time
• Must use a file system that’s cluster-aware (not
XFS, EXT4, etc…)
© Copyright by Cloud Mentor Pro | Confidential 45
 EBS Volume Resizing
• You can only increase the EBS volumes:
• Size (any volume type)
• IOPS (only in IO1)
• After resizing an EBS volume, you need to
repartition your drive
• After increasing the size, it’s possible for the volume
to be in a long time in the “optimisation” phase. The
volume is still usable
• You can’t decrease the size of your EBS volume
(create another smaller volume then migrate data)

© Copyright by Cloud Mentor Pro | Confidential 46

 EBS Snapshots
• Make a backup (snapshot) of your EBS volume at a point in time
• Not necessary to detach volume to do snapshot, but recommended
• Can copy snapshots across AZ or Region

© Copyright by Cloud Mentor Pro | Confidential 47

 Amazon Data Lifecycle Manager
• Automate the creation, retention, and deletion
of EBS snapshots and EBS-backed AMIs
• Schedule backups, cross-account snapshot
copies, delete outdated backups, …
• Uses resource tags to identify the resources
(EC2 instances, EBS volumes)
• Can’t be used to manage snapshots/AMIs
created outside DLM
• Can’t be used to manage instance-store backed
AMIs
© Copyright by Cloud Mentor Pro | Confidential 48
 EBS Snapshots – Fast Snapshot Restore (FSR)
• EBS Snapshots stored in S3
• By default, there’s a latency of I/O operations the
first time each block is accessed (block must be
pulled from S3)
• Solution: force the initialization of the entire
volume (using the dd or fio command), or you can
enable FSR
• FSR helps you to create a volume from a snapshot
that is fully initialized at creation (no I/O latency)
• Enabled for a snapshot in a particular AZ (billed
per minute – very expensive $$$)
• Can be enabled on snapshots created by Data
Lifecycle Manager
© Copyright by Cloud Mentor Pro | Confidential 49
 EBS Snapshots Features
• EBS Snapshot Archive
• Move a Snapshot to an ”archive tier” that is
75% cheaper
• Takes within 24 to 72 hours for restoring
the archive
• Recycle Bin for EBS Snapshots
• Setup rules to retain deleted snapshots so
you can recover them after an accidental
deletion
• Specify retention (from 1 day to 1 year)

© Copyright by Cloud Mentor Pro | Confidential 50

 EBS Migration
• EBS Volumes are only locked to a specific AZ
• To migrate it to a different AZ (or region):
• Snapshot the volume
• (optional) Copy the volume to a different region
• Create a volume from the snapshot in the AZ of your choice

• Let’s practice!

© Copyright by Cloud Mentor Pro | Confidential 51

 EBS Encryption
• When you create an encrypted EBS volume, you get the following:
• Data at rest is encrypted inside the volume
• All the data in flight moving between the instance and the volume is encrypted
• All snapshots are encrypted
• All volumes created from the snapshot
• Encryption and decryption are handled transparently (you have nothing
to do)
• Encryption has a minimal impact on latency
• EBS Encryption leverages keys from KMS (AES-256)
• Copying an unencrypted snapshot allows encryption
• Snapshots of encrypted volumes are encrypted
© Copyright by Cloud Mentor Pro | Confidential 52
 Encryption: encrypt an unencrypted EBS volume
• Create an EBS snapshot of the volume
• Encrypt the EBS snapshot ( using copy )
• Create new ebs volume from the snapshot ( the volume will also be
encrypted )
• Now you can attach the encrypted volume to the original instance

© Copyright by Cloud Mentor Pro | Confidential 53

 Amazon EFS – Elastic File System
• Managed NFS (network file system) that can be mounted on many EC2
• EFS works with EC2 instances in multi-AZ
• Highly available, scalable, expensive (3x gp2), pay per use

© Copyright by Cloud Mentor Pro | Confidential 54

 Amazon EFS – Elastic File System
• Use cases: content management, web serving, data sharing, Wordpress
• Uses NFSv4.1 protocol
• Uses security group to control access to EFS
• Compatible with Linux based AMI (not Windows)
• Encryption at rest using KMS

• POSIX file system (~Linux) that has a standard file API

• File system scales automatically, pay-per-use, no capacity planning!

© Copyright by Cloud Mentor Pro | Confidential 55

 EFS – Performance & Storage Classes
• EFS Scale
• 1000s of concurrent NFS clients, 10 GB+ /s throughput
• Grow to Petabyte-scale network file system, automatically
• Performance Mode (set at EFS creation time)
• General Purpose (default) – latency-sensitive use cases (web server, CMS, etc…)
• Max I/O – higher latency, throughput, highly parallel (big data, media processing)
• Throughput Mode
• Bursting – 1 TB = 50MiB/s + burst of up to 100MiB/s
• Provisioned – set your throughput regardless of storage size, ex: 1 GiB/s for 1 TB storage
• Elastic – automatically scales throughput up or down based on your workloads
• Up to 3GiB/s for reads and 1GiB/s for writes
• Used for unpredictable workloads

© Copyright by Cloud Mentor Pro | Confidential 56

 EFS – Storage Classes
• Storage Tiers (lifecycle management feature – move
file after N days)
• Standard: for frequently accessed files
• Infrequent access (EFS-IA): cost to retrieve files, lower price
to store.
• Archive: rarely accessed data (few times each year), 50%
cheaper
• Implement lifecycle policies to move files between storage
tiers
• Availability and durability
• Standard: Multi-AZ, great for prod
• One Zone: One AZ, great for dev, backup enabled by
default, compatible with IA (EFS One Zone-IA)
• Over 90% in cost savings
© Copyright by Cloud Mentor Pro | Confidential 57
 EBS vs EFS – Elastic Block Storage
• EBS volumes…
• one instance (except multi-attach io1/io2)
• are locked at the Availability Zone (AZ) level
• gp2: IO increases if the disk size increases
• gp3 & io1: can increase IO independently
• To migrate an EBS volume across AZ
• Take a snapshot
• Restore the snapshot to another AZ
• EBS backups use IO and you shouldn’t run them
while your application is handling a lot of traffic
• Root EBS Volumes of instances get
terminated by default if the EC2 instance
gets terminated. (you can disable that)
© Copyright by Cloud Mentor Pro | Confidential 58
 EBS vs EFS – Elastic File System
• Mounting 100s of instances across AZ
• EFS share website files (WordPress)
• Only for Linux Instances (POSIX)

• EFS has a higher price point than EBS

• Can leverage Storage Tiers for cost savings

• Remember: EFS vs EBS vs Instance Store

© Copyright by Cloud Mentor Pro | Confidential 59

 EFS – Access Points
• Easily manage applications access to NFS
environments
• Enforce a POSIX user and group to use
when accessing the file system
• Restrict access to a directory within the
file system and optionally specify a
different root directory
• Can restrict access from NFS clients using
IAM policies

© Copyright by Cloud Mentor Pro | Confidential 60

 EFS - Operations
• Operations that can be done in place:
• Lifecycle Policy (enable IA or change IA settings)
• Throughput Mode and Provisioned Throughput
Numbers
• EFS Access Points
• Operations that require a migration using
DataSync (replicates all file attributes and
metadata)
• Migration to encrypted EFS
• Performance Mode (e.g. Max IO)

© Copyright by Cloud Mentor Pro | Confidential 61

 EFS – CloudWatch Metrics
• PercentIOLimit
• How close the file system reaching the I/O limit
(General Purpose)
• If at 100%, move to Max I/O (migration)
• BurstCreditBalance
• The number of burst credits the file system can
use to achieve higher throughput levels
• StorageBytes
• File system’s size in bytes (15 minutes interval)
• Dimensions: Standard, IA, Total (Standard + IA)

© Copyright by Cloud Mentor Pro | Confidential 62

 Exercise
• API Serverless with AWS
SAM and Github Action

© Copyright by Cloud Mentor Pro | Confidential 63