Dynamic, shared, and highly available storage for OpenShift

applications

1
 OVERVIEW RED HAT CONFIDENTIAL

What is OpenShift Container Storage ?

Highly scalable, production-grade persistent storage
● For stateful applications running in Red Hat® OpenShift

● Optimized for Red Hat OpenShift Infrastructure services

● Developed, released and deployed in synch with Red Hat OpenShift

● Supported via a single contract with Red Hat OpenShift

● Complete persistent storage fabric across hybrid cloud for OCP

2
 OVERVIEW RED HAT CONFIDENTIAL

Why do you need Persistent Storage?

OCP Infrastructure OCP Application Local/Ephemeral Storage

Registry
Service 1
Metrics
Prometheus
Logging
Service 2

OpenShift Container Storage Focus

RWX/RWO backed by File, Block, S3
3
 OVERVIEW RED HAT CONFIDENTIAL

Consistent storage management, and operations

RED HAT OPENSHIFT CONTAINER STORAGE

VIRTUAL HYBRID LEGACY
BARE METAL CONTAINERS
MACHINES CLOUD STORAGE

ANY CLOUD. ANY APP. NO LOCK IN

4
Future Proof against cloud or infrastructure lock-in
 OVERVIEW RED HAT CONFIDENTIAL

Complete Storage for Container Platform

RWO - Block RWX - File Object S3

RED HAT OPENSHIFT CONTAINER STORAGE

BARE VIRTUAL
VIRTUAL CONTAINE HYBRID
HYBRID LEGACY
LEGACY
BARE METAL CONTAINERS
METAL MACHINES
MACHINES RS CLOUD
CLOUD STORAGE
STORAGE

Provides Storage for All Apps and infrastructure Services

5 in their native interfaces
 FOCUS AREAS RED HAT CONFIDENTIAL

Focus Areas

EASE OF USE HYBRID CLOUD STORAGE

DAY 1 & DAY 2 & DATA SERVICES ENHANCEMENTS

6
 PRESENT & FUTURE RED HAT CONFIDENTIAL

OCS Operator based on Operator SDK

with Operator Lifecycle Manager (OLM)

Object Bucket Claim

7
 RED HAT CONFIDENTIAL

OCP 4 with OCS 4 - Technology Stack

Easy & Automated Highly Resilient & Multi-Cloud & Hybrid

Management with Scalable Storage Object Storage
Operators System

8
 TERMINOLOGY

9
Terminology
● CRD: Custom Resource Definition; Schema Extension to Kubernetes API
● CR: Custom Resource; One record/instance/object, conforming to a CRD
● OPERATOR: Daemon that watches for changes to resources
● STORAGE CLASS: “class” of storage service
● PVC: Persistent Volume Claim, attach persistent storage to a pod
● POD: a group of one or more containers managed by Kubernetes
Storage access modes
● RWO - ReadWriteOnce: volume can be mounted as read-write by a single node

● ROX - ReadOnlyMany: volume can be mounted read-only by many nodes

● RWX - ReadWriteMany: volume can be mounted as read-write by many nodes

 OPERATOR PATTERN

12
Operator pattern
● Codifies domain expertise to deploy and manage an application
○ Automates actions a human would normally do
● Apply user’s desired state
○ Observe - discover current actual state of cluster
○ Analyze - determine differences from desired state
○ Act - perform operations to drive actual towards desired
What’s an Operator?
An Operator is an entity that runs just like any of your applications in your
OpenShift cluster. So the Operator manages the life-cycle of a storage cluster.

The Operator primarily role is to honor a “desired state”, which may be

“maintain a Storage Cluster healthy”.

→ Rook does that for Ceph.

→ OCS is a meta-operator that bootstraps Rook and Nooba operators, as well as their respective cluster
CRs (CephCluster, Object, File, Block).
 Ceph

15
 Ceph RED HAT CONFIDENTIAL

Ceph - Highly Resilient & Scalable

Distributed Storage
● High availability and resiliency
● Data protection
● Consistent storage platform across hybrid cloud
● Block, file & object storage service
● Scale up/down
● Monitoring and alerts

16
 RED HAT CONFIDENTIAL

Architectural components
APP HOST/VM CLIENT

RGW RBD CEPHFS

A web services gateway A reliable, A distributed file
for object storage, fully-distributed block system with POSIX
compatible with S3 and device with cloud semantics and
Swift platform integration scale-out metadata
management
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby, PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors
17
 RED HAT CONFIDENTIAL

RADOS components

Monitors:
▪ Maintain cluster membership and state

M ▪ Provide consensus for distributed decision-making

▪ Small, odd number (typically 3 or 5)
▪ Do not serve data

Object Storage Daemon:

▪ 10s to 10000s in a cluster
▪ One per disk (or one per SSD, RAID group…)
▪ Serve stored objects to clients
18 ▪ Intelligently peer for replication & recovery
 RED HAT CONFIDENTIAL

RADOS components

Managers:

M ▪ Tightly coupled with Monitor

▪ Manage cluster “logistic” PG and maps
▪ Provide additional monitoring and interfaces to external
monitoring and management systems
▪ Pluggable python interface to develop modules
▪ Some modules:
■ Balancer
■ PG auto-scaler
■ Dashboard
■ RESTful
19 ■ Prometheus
■ Rook
 RED HAT CONFIDENTIAL

RADOS components

Metadata Server:
▪ Manages metadata for a POSIX-compliant shared
filesystem
▪ Directory hierarchy
▪ File metadata (owner, timestamps, mode, etc.)
▪ Stores metadata in RADOS
▪ Does not serve file data to clients
▪ Only required for shared filesystem
▪ Multiple MDS are supported
 RED HAT CONFIDENTIAL

RADOS components

Rados Gateway:
▪ REST-based object storage proxy
▪ Uses RADOS to store objects
▪ API supports buckets, accounts
▪ Usage accounting for billing
▪ Compatible with S3 and Swift applications
▪ Multi-site replication

21
 RED HAT CONFIDENTIAL

RADOS Cluster

APPLICATION

M M

M M

M
22 RADOS CLUSTER
 OBJECT PLACEMENT WITH CRUSH
Controlled Replication Under Scalable Hashing

23
CRUSH: Data is organized into pools

10 11 10 01
10 01 01 11
OBJECTS POOL
A 01 01 01 10 OSD 0 OSD 1

01 10 11 10
POOL 01 01 10 01
OBJECTS
B 10 01 01 01 OSD 2 OSD 3

OBJECTS POOL 10 01 10 11
C 01 11 10 10

OSD 4 OSD 5
01 10 01 01

OBJECTS POOL 11 10 01 10
D 10 10 01 01

01 01 10 01 OSD 6 OSD 7

POOLS CLUSTER
(CONTAINING PGs)
CRUSH: dynamic placement

▪ Pseudo-random placement algorithm

▪ Fast calculation, no lookup
▪ Repeatable, deterministic
▪ Statistically uniform distribution
▪ Stable mapping
▪ Limited data migration on change
▪ Rule-based configuration
▪ Infrastructure topology aware (zone/datacenter etc…)
▪ Adjustable replication
▪ Weighting
 ROOK AND CSI

26
Ceph-CSI
Ceph CSI plugin implements an interface between CSI enabled Container
Orchestrator (CO) and Ceph cluster.

It allows dynamically provisioning Ceph volumes and attaching them to workloads.

“That’s THE way to provide persistent storage to containers.”

Ceph-CSI driver
● Responds to client requests for storage from a storage class
○ Pods request storage with PVCs
● Provisions RBD or CephFS volumes
● Attaches and mounts the volumes to the pods
Storage Class: RBD
Supported access mode:

● RWO: ReadWriteOnce: the volume can be mounted as read-write by a single

node
○ Traditional use-case
● RWX - ReadWriteMany: the volume can be mounted as read-write by many
nodes

○ Virtual machine block live migration

Storage Class: CephFS
Supported access mode:

● RWO: ReadWriteOnce: the volume can be mounted as read-write by a single

node
● RWX - ReadWriteMany: the volume can be mounted as read-write by many
nodes

○ Application reading and writing from different sources

What happens if a node fails?
Pods will be rescheduled onto different nodes and storage will be re-attached.

Since Ceph block and filesystem are distributed it’s a matter of:

● Re-mapping the block onto a different machine

● Re-mounting the filesystem onto a different machine
 ROOK ARCHITECTURE

32
Architectural Layers
● Rook:
○ The operator owns the management of Ceph
● Ceph-CSI:
○ CSI driver dynamically provisions and connects client pods to
the storage
● Ceph:
○ Data layer: Storage Provider
○ Block/File/Object storage
Rook Components: Pods
Application Storage: Provisioning
Application Storage: Data Path
 ENVIRONMENT OVERVIEW

37
What’s running?
How do daemons run? 1/2
Rook does **not** rely on a “ceph.conf” configuration file, instead everything
is CLI based. So don’t be afraid if you see an empty ceph.conf or nothing at
all.

This means all the Ceph commands have all their configuration passed via a
CLI flag.
How do daemons run? 2/2
E.g for a monitor container:
How do I run Ceph commands?
As a direct consequence of not using a “ceph.conf” configuration file, if you exec
into any container, you won’t be able to easily run Ceph commands.

Instead you should be using the “toolbox” or the Operator container, which will
allow you to run any Ceph commands.
Deployment description: Monitor
initContainers:

1. chown-container-data-dir → chowns Ceph log data directory on the host

2. init-mon-fs → runs monitor “ceph-mon --mkfs” to initialize monitor data

Containers:

● mon → runs the “ceph-mon” process on foreground

Deployment description: Manager|MDS|RGW
initContainers:

1. chown-container-data-dir → chowns Ceph log data directory on the host

Containers:

● mgr|rgw|mds → runs the process on foreground

Deployment description: OSD prepare
initContainers:

1. copy-bins → copy “rook” and “tini” binaries from a container to a given directory. Basically
copies binaries from the Operator image into the Ceph image. Later during the provision
container, the “rook” CLI will be called to perform several actions (preparing the disk).

Containers:

● provision → runs the “ceph-volume lvm prepare” command to prepare the disk

One OSD prepare job per PVC.

Deployment description: OSD activate
initContainers:

1. config-init → generates “ceph.conf” configuration file

2. copy-bins → copy “rook” and “tini” binaries from a container to a given directory
3. chown-container-data-dir → chowns Ceph log data directory on the host

Containers:

● osd
○ runs “ceph-volume lvm activate” command
○ runs the “ceph-osd” process on foreground

Each OSD has its own pod.

Pods are running on PVC
All daemons requiring data are running on PVC using AWS GP2 Storage Class:

● Monitor data store

● OSD data (user data)
 CEPH TROUBLESHOOTING

47
Get ready to troubleshoot
● “Toolbox” to the rescue!
○ Just adapt the namespace and the container image of this YAML
https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/toolbox.yaml
● If you don’t want to run the toolbox, you can exec into the Operator pod and run:

export CEPH_ARGS=”-c /var/lib/rook/openshift-storage/openshift-storage.config”

Then you can run any Ceph commands.

Get pod details
Get daemon logs
Enable more logging
● Enable logging on file for a given daemon
○ If a daemon keeps crashing and you want to collect a log file instead of reading stdout, run the
following command from the toolbox
○ ceph config set mon.a log_to_file true
○ SSH on the host and look into /var/lib/rook/<ns>/log/
● Enable debug logs:
○ Ceph config set mon.a debug_ms 10

This change will persist across restart of the daemon, so the config change is
permanent.
Get daemon local configuration
I want to verify the value of osd_memory_target_cgroup_limit_ratio:

The command works locally by talking to the daemon socket, so it does not
connect to the monitors at all.

This command only works when exec’ed into the specific container.
Debug failing daemon 1/2
Scenario:

● OSD pod keeps failing to activate the OSD 0

● The pod is in a crash loop
● You need to get into this container but you have prevent it from failing
● We have to patch the entrypoint command and replace it
● Instead of running the daemon CLI line we will sleep
Debug failing daemon 2/2

Now you can exec into this pod and will get the right environment to start
debugging.

Bonus point: if you are debugging an OSD, don’t forget to run:

ceph-volume lvm activate --no-systemd --bluestore $ROOK_OSD_ID

 Hybrid and Data Services
Multi Cloud Object Gateway

55
 MULTI-CLOUD OBJECT GATEWAY RED HAT CONFIDENTIAL

Start lean

A single lightweight pod for basic development

and tests

Scale locally

Scale with local volumes or Red Hat Ceph

Storage

Workload portability

Easily mirror data to other cluster or native cloud

storage

56
 MULTI CLOUD OBJECT GATEWAY RED HAT CONFIDENTIAL

DEPLOY AND MANAGE DATA SERVICES

App Multi-Cloud Buckets

App S3 API Hybrid Buckets

App
Multi-site Buckets

57
EFFICIENCY AND SECURITY BY DEFAULT RED HAT CONFIDENTIAL

S3 Write Fragment Dedupe Compress Encrypt Store

London DC

New York DC

Paris DC
 CONFIDENTIAL Designator

BUCKET CLAIMS

59
 OBJECT BUCKET CLAIM CONFIDENTIAL Designator

Bucket Create new bucket

App Bucket
Claim Create new account

Read
Write

60
 BUCKET CLAIM CONFIDENTIAL Designator

OBJECT BUCKET CLAIM

61

OpenShift Container Storage 4 Sales Enablement

 BUCKET CLAIM CONFIDENTIAL Designator

OBJECT BUCKET CLAIM

● BUCKET_HOST - The endpoint to use in the application

● BUCKET_PORT - The port available for this application
● BUCKET_NAME - Requested or generated bucket name
● AWS_ACCESS_KEY_ID - The access key which is part of the credentials
● AWS_SECRET_ACCESS_KEY - The secret access key which is part of the
credentials

62
 CONFIDENTIAL Designator

MCG TROUBLESHOOTING

63
 NOOBAA STATUS CONFIDENTIAL Designator

#------------------#
INFO[0000] CLI version: 2.0.8 #- Mgmt Addresses -#
INFO[0000] noobaa-image: noobaa/noobaa-core:5.2.10 #------------------#
INFO[0000] operator-image: noobaa/noobaa-operator:2.0.8 ExternalDNS : [https://noobaa-mgmt-openshift-storage.apps.cluster-ocs-e12b.ocs-e12b.example.opentlc.com
INFO[0000] Namespace: openshift-storage https://a1839c3200b7511eab3bf12891326d01-456461733.us-east-1.elb.amazonaws.com:443]
INFO[0000] ExternalIP : []
INFO[0000] CRD Status: NodePorts : [https://10.0.140.19:32561]
INFO[0001] ✅ Exists: CustomResourceDefinition "noobaas.noobaa.io" InternalDNS : [https://noobaa-mgmt.openshift-storage.svc:443]
INFO[0002] ✅ Exists: CustomResourceDefinition "backingstores.noobaa.io" InternalIP : [https://172.30.46.100:443]
INFO[0002] ✅ Exists: CustomResourceDefinition "bucketclasses.noobaa.io" PodPorts : [https://10.131.2.15:8443]
INFO[0002] ✅ Exists: CustomResourceDefinition "objectbucketclaims.objectbucket.io" #--------------------#
INFO[0002] ✅ Exists: CustomResourceDefinition "objectbuckets.objectbucket.io" #- Mgmt Credentials -#
INFO[0002] #--------------------#
INFO[0002] Operator Status: email : admin@noobaa.io
INFO[0002] ✅ Exists: Namespace "openshift-storage" password : a5F/I4qh56qlUpFb5NJVFw==
INFO[0002] ✅ Exists: ServiceAccount "noobaa" #----------------#
INFO[0003] ✅ Exists: Role "ocs-operator.v0.0.1-hnwlz" #- S3 Addresses -#
INFO[0003] ✅ Exists: RoleBinding "ocs-operator.v0.0.1-hnwlz-noobaa-m2272" #----------------#
INFO[0003] ✅ Exists: ClusterRole "ocs-operator.v0.0.1-vmkpp" ExternalDNS : [https://s3-openshift-storage.apps.cluster-ocs-e12b.ocs-e12b.example.opentlc.com
INFO[0003] ✅ Exists: ClusterRoleBinding "ocs-operator.v0.0.1-vmkpp-noobaa-j28q2" https://a183dc3760b7511eab3bf12891326d01-841639993.us-east-1.elb.amazonaws.com:443]
INFO[0003] ✅ Exists: Deployment "noobaa-operator" ExternalIP : []
INFO[0003] NodePorts : [https://10.0.140.19:30052]
INFO[0003] System Status: InternalDNS : [https://s3.openshift-storage.svc:443]
INFO[0004] ✅ Exists: NooBaa "noobaa" InternalIP : [https://172.30.147.48:443]
INFO[0004] ✅ Exists: StatefulSet "noobaa-core" PodPorts : [https://10.131.2.15:6443]
INFO[0004] ✅ Exists: Service "noobaa-mgmt" #------------------#
INFO[0004] ✅ Exists: Service "s3" #- S3 Credentials -#
INFO[0004] ✅ Exists: Secret "noobaa-server" #------------------#
INFO[0004] ✅ Exists: Secret "noobaa-operator" AWS_ACCESS_KEY_ID : kOevajmwLAMe2o7TVMCc
INFO[0005] ✅ Exists: Secret "noobaa-admin" AWS_SECRET_ACCESS_KEY : eWevwwF+0TwiC2LdG/a9Lyh8bX+LyvRHrSL/fnwc
INFO[0005] ✅ Exists: StorageClass "openshift-storage.noobaa.io" #------------------#
INFO[0005] ✅ Exists: BucketClass "noobaa-default-bucket-class" #- Backing Stores -#
INFO[0005] ✅ (Optional) Exists: BackingStore "noobaa-default-backing-store" #------------------#
INFO[0005] ✅ (Optional) Exists: CredentialsRequest "noobaa-cloud-creds" NAME TYPE TARGET-BUCKET PHASE AGE
INFO[0005] ✅ (Optional) Exists: PrometheusRule "noobaa-prometheus-rules" noobaa-default-backing-store aws-s3 noobaa-backing-store-a1ebfdd3-880a-40f6-b659-be4b588cd1c4 Ready 2h9m10s
INFO[0006] ✅ (Optional) Exists: ServiceMonitor "noobaa-service-monitor" #------------------#
INFO[0006] ✅ (Optional) Exists: Route "noobaa-mgmt" #- Bucket Classes -#
INFO[0006] ✅ (Optional) Exists: Route "s3" #------------------#
INFO[0006] ✅ Exists: PersistentVolumeClaim "db-noobaa-core-0" NAME PLACEMENT PHASE AGE
INFO[0006] ✅ System Phase is "Ready" noobaa-default-bucket-class {Tiers:[{Placement: BackingStores:[noobaa-default-backing-store]}]} Ready 2h9m10s
INFO[0006] ✅ Exists: "noobaa-admin" #-----------------#
#- Bucket Claims -#
#-----------------#
NAMESPACE NAME BUCKET-NAME STORAGE-CLASS BUCKET-CLASS PHASE
64
openshift-storage obc-test obc-test-noobaa-74f10ace-e5fc-4210-a106-84ac02f2caaa openshift-storage.noobaa.io
Bound
 TROUBLESHOOTING CONFIDENTIAL Designator

● Symptom: No Object Service

Additional info: oc get sts noobaa-core
oc get pod |grep noobaa
Possible reasons:
○ If noobaa-core doesn’t exist, problem in
OCS-Operator. Check its logs

65
 TROUBLESHOOTING CONFIDENTIAL Designator

Symptom: Pods exist, but don’t get into Running state

Additional info:
● oc logs noobaa-operator-xxx
● oc describe pod/sts/deployment
● oc logs -c core noobaa-core-0
Possible reasons:
● Lack of cpu/memory resources in OCS nodes
● Problem with the PV (check with oc get pvc)
● Failure to create the default backing store with
AWS or RGW
66
 TROUBLESHOOTING CONFIDENTIAL Designator

● OCS dashboard issues

○ https://<management
url>/metric - check the raw data
provided by MCG
● NooBaa CLI - noobaa status

● NooBaa UI - check resource/buckets. Check

connections status, bucket status, etc.

67
Nothing helped? Must gather it!
 Platform

68
 RED HAT CONFIDENTIAL

Supported Platform

● AWS
● VMware vSphere

Refer KCS : https://access.redhat.com/articles/4731161

69
 Workloads

70
 OCS WORKLOADS RED HAT CONFIDENTIAL

Persistent Volume Object Service

Block Shared File System

● Primary for DB and Transactional ● POSIX-compliant shared file system ● Media, AI/ML training data,

workloads ● Interface for legacy workloads Archiving, Backup, Health Records

● Low latency ● CI/CD Pipelines ● Great Bandwidth performance

● Messaging ● AI/ML Data Aggregation ● Object API (S3/Blob)

Provided by Rook-Ceph Provided by Rook-Ceph Provided by Multi-Cloud Object Gateway

71
 OCS Must-Gather

72
 RED HAT CONFIDENTIAL

Lifecycle of must-gather

73
 RED HAT CONFIDENTIAL

Behind the scenes

● Once the must-gather is invoked, the oc client creates a temporary namespace

where the must-gather-pod is deployed.
● It also deploys a service account for the must-gather along with the cluster-role
binding.
● The dump is initially generated under /must-gather directory of the must-gather
pod and then gets copied to local host from where it has been invoked.
● Once the transfer process between the must-gather pod and and local system is
completed, the oc client deletes the temporary namespace along with the
cluster-role binding and service account.

74
 RED HAT CONFIDENTIAL

Directory Structure of ocs-must-gather

dump directory

75
 RED HAT CONFIDENTIAL

Directory structure of pod logs

76
Where to look for a resource ?(Core)
Resource Location

Pods ● All pod spec of a namespace can be found under

A kubernetes pod is a group of ceph/namespaces/<namespace-name>/core/pods.yaml. (or
containers that are deployed together on output of oc get pods -o yaml -n <namespace-name>).
the same host.
● Pod logs can be found under
ceph/<namespace-name>/pods/<pod-name>.

ConfigMaps ● All configmap spec of a namespace can be found under

ConfigMaps allow you to decouple ceph/namespaces/<namespace-name>/core/configmaps.yam
configuration artifacts from image content l. (or output of oc get configmaps -o yaml -n
to keep containerized applications <namespace-name>).
portable

Events ● All events spec of a namespace can be found under

Kubernetes events are objects that ceph/namespaces/<namespace-name>/core/events.yaml.
provide insight into what is happening (or output of oc get events -o yaml -n
inside a cluster.
<namespace-name>).

PersistentVolumeClaims ● All pvc spec of a namespace can be found under

A PVC is a request for storage by a user ceph/namespaces/<namespace-name>/core/persistentvolu
meclaims.yaml. (or output of oc get
persistentvolumeclaims -o yaml -n <namespace-name>).
Where to look for a resource ?(Core)
Resource Location

ReplicationControllers ● All replicationcontrollers spec of a namespace can be found

A Replication Controller ensures that a under
specified number pod replicas are running ceph/namespaces/<namespace-name>/core/replicationcont
at a time. rollers.yaml. (or output of oc get
replicationcontrollers -o yaml -n <namespace-name>).

Secrets ● All secrets spec of a namespace can be found under

Kubernetes secret objects let you store ceph/namespaces/<namespace-name>/core/secrets.yaml.
and manage sensitive info such password, (or output of oc get secrets -o yaml -n
OAuth tokens, etc.
<namespace-name>).

Services ● All services of a namespace can be found under

A service routes traffics across a set of ceph/namespaces/<namespace-name>/core/services.yaml.
pods. (or output of oc get services -o yaml -n
<namespace-name>).
Where to look for a resource ?(apps)
Resource Location

Daemonsets ● All daemonsets spec of a namespace can be found under

DaemonSets are used to ensure that some ceph/namespaces/<namespace-name>/apps/daemonsets.yaml.
or all k8s run a copy of a pod. (or output of oc get daemonsets -o yaml -n
<namespace-name>).

Deployments ● All deployments spec of a namespace can be found under

Deployments ensures that only a certain ceph/namespaces/<namespace-name>/apps/deployments.yaml.
number of pods are down while they are (or output of oc get deployments -o yaml -n
being updated.
<namespace-name>).

Replicasets ● All replicasets spec of a namespace can be found under

Replicaset ensures the number of replica of ceph/namespaces/<namespace-name>/apps/replicasets.yaml.
pod that should be running. (or output of oc get replicasets -o yaml -n
<namespace-name>).

Statefulsets ● All statefulsets spec of a namespace can be found under

Statefulset is the workload API object used ceph/namespaces/<namespace-name>/apps/statefulsets.yaml.
to manage stateful applications. (or output of oc get statefulsets -o yaml -n
<namespace-name>).
Where to look for a resource ?(apps.openshift.io)

Resource Location

Deploymentconfigs ● All deploymentconfigs spec of a namespace can be found

under
ceph/namespaces/<namespace-name>/apps.openshift.io/depl
oymentconfigs.yaml. (or output of oc get deploymentconfigs
-o yaml -n <namespace-name>).
Where to look for a resource ? (autoscaling)

Resource Location

Horizontalpodautoscalers ● All horizontalpodautoscalers spec of a namespace can be

Horizontalpodautoscaler automatically found under
scales the number of pods in a replication ceph/namespaces/<namespace-name>/autoscaling/horizontal
controller, deployment, replicaset or podautoscalers.yaml. (or output of oc get
statefulset based on observed CPU
horizontalpodautoscalers -o yaml -n <namespace-name>).
utilization
Where to look for a resource ? (batch)

Resource Location

Cronjobs ● All cronjobs spec of a namespace can be found under

Cron jobs are useful for creating periodic ceph/namespaces/<namespace-name>/batch/cronjobs.yaml. (or
and recurring tasks, like running backups or output of oc get cronjobs -o yaml -n <namespace-name>).
sending emails

Jobs ● All jobs spec of a namespace can be found under

A job in kubernetes is a supervisor for pods ceph/namespaces/<namespace-name>/batch/jobs.yaml. (or
carrying out batch processes output of oc get jobs -o yaml -n <namespace-name>).
Where to look for a resource ? (ceph.rook.io)

Resource Location

Cephblockpools ● All cephblockpools spec of a namespace can be found under

Cephblockpool is the CRD for creating and ceph/namespaces/<namespace-name>/ceph.rook.io/<cephblockpool
customizing storage pools in the rook ceph -name>.yaml. (or output of oc get cephblockpools <name> -o yaml -n
cluster <namespace-name>).
https://rook.io/docs/rook/v1.1/ceph-pool-crd.html

Cephclusters ● All cephclusters spec of a namespace can be found under

Cephcluster is the CRD for creating and ceph/namespaces/<namespace-name>/ceph.rook.io/<cephcluster-n
customizing the ceph storage cluster ame>.yaml. (or output of oc get cephclusters <name> -o yaml -n
https://rook.io/docs/rook/v1.1/ceph-cluster-crd.ht <namespace-name>).
ml

Cephfilesystems ● All cephfilesystems spec of a namespace can be found under

Cephfilesystem is the CRD for creating and ceph/namespaces/<namespace-name>/ceph.rook.io/<cephfilesyste
customizing the filesystem in the rook ceph m-name>.yaml. (or output of oc get cephfilesystems <name> -o yaml
cluster -n <namespace-name>).
https://rook.io/docs/rook/v1.1/ceph-filesystem-crd
.html

Cephobjectstores ● All cephobjectstores spec of a namespace can be found under

Cephobjectstore is the CRD for creating and ceph/namespaces/<namespace-name>/ceph.rook.io/<cephobjectsto
customizing the object store in the rook ceph re-name>.yaml. (or output of oc get cephobjectstores <name> -o
cluster yaml -n <namespace-name>).
https://rook.io/docs/rook/v1.1/ceph-object-st
ore-crd.html
Where to look for a resource ? (ceph.rook.io)
Resource Location

Cephobjectstoreusers ● All cephobjectstoreuser spec of a namespace can be found

Cephobjectstoreuser is the CRD for under
creating and customizing the object store ceph/namespaces/<namespace-name>/ceph.rook.io/<cephobj
user in the rook ceph cluster ectstoreuser-name>.yaml. (or output of oc get
https://rook.io/docs/rook/v1.1/ceph-object-st cephobjectstoreusers <name> -o yaml -n
ore-user-crd.html
<namespace-name>).
Where to look for a resource ? (operators.coreos.com)

Resource Location

Clusterserviceversions ● All clusterserviceversions spec of a namespace can be found

A clusterserviceversion(CSV) is a yaml under
manifest created from Operator metadata ceph/namespaces/<namespace-name>/operators.coreos.com/
that assists the operator lifecycle clusterserviceversions.yaml. (or output of oc get
manager(OLM) in running the operator in a
clusterserviceversions -o yaml -n <namespace-name>).
cluster
https://docs.openshift.com/container-platfor
m/4.1/applications/operator_sdk/osdk-gener
ating-csvs.html
Where to look for a resource ? (noobaa.io)

Resource Location

Backingstores ● All backingstores spec of a namespace can be found under

noobaa/namespaces/<namespace-name>/noobaa.io/<backings
tores-name>.yaml. (or output of oc get backingstores
<name> -o yaml -n <namespace-name>).

Bucketclasses ● All bucketclasses spec of a namespace can be found under

noobaa/namespaces/<namespace-name>/noobaa.io/<bucketcl
asses-name>.yaml. (or output of oc get backingstores
<name> -o yaml -n <namespace-name>).

Noobaas ● All noobaas spec of a namespace can be found under

noobaa/namespaces/<namespace-name>/noobaa.io/<noobaa-n
ame>.yaml. (or output of oc get backingstores <name> -o
yaml -n <namespace-name>).
Where to look for a resource ? (route.openshift.io)

Resource Location

Routes ● All routes spec of a namespace can be found under

An openshift route is a way to expose a ceph/namespaces/<namespace-name>/route.openshift.io/ro
service by giving it an externally-reachable utes.yaml. (or output of oc get routes -o yaml -n
hostname like www.example.com <namespace-name>).
Where to look for a resource ? (miscellaneous)

Resource Location

Ceph command outputs ● All ceph command outputs of a ocs cluster can be found
The output of commonly used ceph under
commands for debugging ceph/namespaces/<namespace-name>/must_gather_commands.

Osd prepare volume logs ● All osd prepare volume logs outputs of a ocs cluster can be
The osd prepare volume logs which gets found under
stored in the nodes ceph/namespaces/<namespace-name>/osd_prepare_volume_lo
gs.
● Osd prepare volume logs resides under the nodes where the
pvc was prepared. So in must-gather dump it will be under
the node-name directory.
 Deployment

89
 EASE OF USE RED HAT CONFIDENTIAL

Operator Driven Install from OLM

OCS
Operator

90
 EASE OF USE RED HAT CONFIDENTIAL

Simple Install

91
 RED HAT CONFIDENTIAL

Integrated Dashboard

92
 RED HAT CONFIDENTIAL

Persistent Storage Dashboard

93
 RED HAT CONFIDENTIAL

Object Service Dashboard

94
 RED HAT CONFIDENTIAL

Day 2 Operations

95
 Thank you linkedin.com/company/red-hat

youtube.com/user/RedHatVideos
Red Hat is the world’s leading provider of

enterprise open source software solutions.

facebook.com/redhatinc
Award-winning support, training, and consulting

services make twitter.com/RedHat

Red Hat a trusted adviser to the Fortune 500.

96