Loading...

Follow Red Hat Storage on Feedspot

Continue with Google
Continue with Facebook
or

Valid
By Marko Karg and Annette Clewett

At the end of the first post in this blog series, we deployed a MySQL instance using StatefulSets (STS). Today, we want to use that same deployment method and connect a WordPress instance to the MySQL database and see what happens if the database fails. Remember, we’re using a MySQL pod created from a StatefulSet.

OpenShift on AWS test environment

All the posts in this series use an OCP-on-AWS setup that includes 8 EC2 instances deployed as 1 master node, 1 infra node, and 6 worker nodes that also run OCS gluster and heketi pods. The 6 worker nodes are basically the storage provider (OpenShift Container Storage [OCS]) and persistent storage consumers (MySQL). As the following figure shows, the ocs worker nodes are of instance type m5.2xlarge with 8 vCPUs, 32 GB Mem, and 3x100GB gp2 volumes attached to each node for OCP and 1 1TB gp2 volume for OCS storage cluster. The AWS region us-west-2 has availability zones (AZs) us-west-2a, us-west-2b, us-west-2c, and the 6 worker nodes are spread across the 3 AZs, two nodes in each AZ. This mean the OCS storage cluster is “stretched” across these 3 AZs.

MySQL setup

We’ve created a headless MySQL service already, using an OCS-based persistent volume claim (PVC):

oc get services
NAME        TYPE CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
mysql       ClusterIP             <none>     3306/TCP    16h

The STS we need is also created as described in the first post of this series. Once the STS has been created and the container has been started, we have a running MySQL instance running:

oc get pods
NAME                READY   STATUS    RESTARTS    AGE
mysql-ocs-0         1/1     Running   0           21h

With a service and a database up, we can move forward to get our application deployed.

Although there are templates available that would set up WordPress with a preconfigured database in one shot, we want to take the long way and start from scratch to illustrates the required steps.

WordPress setup

Let’s create a new php application to run WordPress:

# oc new-app php~https://github.com/wordpress/wordpress

After a few seconds, we have the required pods in our project:

# oc get pods
NAME                READY     STATUS      RESTARTS   AGE
mysql-ocs-0         1/1       Running     0          21h
wordpress-1-build   0/1       Completed   0          22h
wordpress-1-q5jts   1/1       Running     0          22h

To make our WordPress instance available to the world, we need to expose it:

# oc expose service wordpress

Now two services are available, one for MySQL and one for WordPress:

# oc get service
NAME                                                     TYPE       
CLUSTER-IP      EXTERNAL-IP  PORT(S)             AGE
glusterfs-dynamic-31a07eb1-3a72-11e9-96fc-02e7350e98d2   ClusterIP  
172.30.165.245  <none>       1/TCP               1m
mysql-ocs                                                ClusterIP  
172.30.210.183  <none>       3306/TCP            1m
wordpress                                                ClusterIP  
172.30.1.139    <none>       8080/TCP,8443/TCP   28s

Let’s connect to the web interface of WordPress now. To do so, we take the HOST / PORT portion from the following command:

oc get route wordpress
NAME        HOST/PORT                                         PATH 
SERVICES    PORT       TERMINATION     WILDCARD
wordpress   wordpress-marko.apps.ocpocs311.ocpgluster.com             
wordpress   8080-tcp None              None

In our case, that’s wordpress-marko.apps.ocpocs311.ocpgluster.com. Take that string and put it into a browser to go to the WordPress interface. The WordPress web interface will guide us through setting up the database connection now.

We have most of the above values pre-defined in our MySQL STS, so the database name is “wordpress”, the username is “admin”, and the password is “secret”.

cat mysql-sts.yaml
….omitted….
   spec:
      terminationGracePeriodSecods: 10
      containers:
      - name: mysql-ocs
        image: mysql:5.7
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        - name: MYSQL_DATABASE
          value: wordpress
        - name: MYSQL_USER
          value: admin
        - name: MYSQL_PASSWORD
          value: secret
….omitted….

Our database host can be found by running this command:

oc get services
NAME        TYPE CLUSTER-IP       EXTERNAL-IP     PORT(S)
AGE
NAME                                                     TYPE
CLUSTER-IP     EXTERNAL-IP      PORT(S)             AGE
glusterfs-dynamic-31a07eb1-3a72-11e9-96fc-02e7350e98d2   ClusterIP
172.30.165.245 <none>           1/TCP               1m
mysql-ocs                                                ClusterIP 
               <none>           3306/TCP            1m
wordpress                                                ClusterIP
172.30.1.139   <none>           8080/TCP,8443/TCP   28s 

So we will use mysql-ocs for the database host.

If information is entered is correct, WordPress will guide us through the rest of the installation process:

We now need to enter some information for the web front end:

The installation takes some time and finally presents this screen:

So now our deployment is done. Next, we’ll log into WordPress and create some test content:

Failure scenario

So now that we have everything in place, we want to see what happens when our MySQL container fails. To simulate that, we’ve set up a client that checks the website using the “curl” command.

Because we’re only interested in the HTTP response over a longer time, we run it in a loop, trimming the output to what we’re interested in:

while true; do date; curl -I 
http://wordpress-marko.apps.ocpocs311.ocpgluster.com/2019/02/26/lorem-ipsum/ 
2>&1 | grep HTTP; sleep 1; done

As a first test we simply kill the MySQL container, watching the preceding loop closely:

# oc get pods
NAME                READY STATUS     RESTARTS  AGE
mysql-ocs-0         1/1   Running    0         21h
wordpress-1-build   0/1   Completed  0         22h
wordpress-1-q5jts   1/1   Running    0         22h

Delete the MySQL pod:

oc delete pod mysql-ocs-0

Here’s the output from the preceding curl loop:

Mi 27. Feb 09:40:44 UTC 2019
HTTP/1.1 200 OK
Mi 27. Feb 09:40:45 UTC 2019
HTTP/1.1 500 Internal Server Error
…
Mi 27. Feb 09:40:59 UTC 2019
HTTP/1.1 500 Internal Server Error
Mi 27. Feb 09:41:03 UTC 2019
HTTP/1.1 200 OK

So the pod failure effectively caused our WordPress instance to be unavailable for 18 seconds (09:40:45 to 09:41;)3), give or take a few seconds for the curl command. We’ve run a larger number of the same test and ended up with an average value of 12 seconds. This is the time that the WordPress application is unavailable due to the MySQL pod being deleted. Once the pod is re-created and the OCS storage is mounted in the pod, then WordPress is available again.

This test only deletes the pod that is running the database. What we cannot be sure of so far is that the MySQL pod actually moves from one node to another. To have that happen, we have to cordon the node on which the pod currently runs and then delete the pod. Cordoning the node means that it will take no new containers and, as a consequence, the new incarnation of our database pod will have to be started on a different node.

As a first step, we need to find the node on which mysql-ocs is currently running:

# ocs get pod mysql-ocs-0 -o wide

ocs get pod mysql-ocs-0 -o wide

NAME          READY   STATUS    RESTARTS      AGE      IP            NODE           
NOMINATED NODE
mysql-ocs-0   1/1     Running   0             5m       10.129.2.44   
ip-172-16-27-161.us-west-2.compute.internal   <none>

Now we cordon the node ip-172-16-27-161.us-west-2.compute.internal and then delete the mysql-ocs-0 pod:

# oc adm cordon ip-172-16-27-161.us-west-2.compute.internal
node/ip-172-16-27-161.us-west-2.compute.internal cordoned
# oc delete pod mysql-ocs-0
pod "mysql-ocs-0" deleted

Again, we’ve run a series of the same test and ended up with an average value of 12 seconds. Therefore, it does not matter if the pod must be relocated to another node or is re-created on the same node again.

Conclusion

The goal behind this post was to show how an application like WordPress in OCP can be connected to a database, as well as how fast a pod can fail-over to another node when it is using Red Hat OCS as a storage platform. An average time of 12 seconds is what we can reproduce persistently for one MySQL pod. The exact time is, of course, specific to every setup, as it depends on a lot of different factors, but the reproducibility and the deterministic time is something that’s common to OpenShift deployments.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
By Sagy Volkov and Annette Clewett

Greetings from Red Hat’s storage architect team! With this post, we’re kicking off a series in which we’ll demonstrate a step-by-step deployment of a stateful application on OpenShift Container Platform (OCP) using OpenShift Container Storage (OCS). This series, based on the 3.11 version of both OCP and OCS, will not cover how to install OCP or OCS.

We’ll start with creating one MySQL pod (using OCP StatefulSets and OCS), and then add the application that uses the MySQL database on persistent storage. As we progress in this series, we’ll show more advanced topics, such as OCP multi-tenant scenarios, MySQL performance on OCS, failover scenarios, and more.

OpenShift on AWS test environment

All the posts in this series use an OCP-on-AWS setup that includes 8 EC2 instances deployed as 1 master node, 1 infra node, and 6 worker nodes that also run OCS gluster and heketi pods. The 6 worker nodes are basically the storage provider (OCS) and persistent storage consumers (MySQL). As shown in the following, the OCS worker nodes are of instance type m5.2xlarge with 8 vCPUs, 32 GB Mem, and 3x100GB gp2 volumes attached to each node for OCP and a single 1TB gp2 volume for OCS storage cluster. The AWS region us-west-2 has Availability Zones (AZs) us-west-2a, us-west-2b, us-west-2c, and the 6 worker nodes are spread across the 3 AZs, two nodes in each AZ. This means the OCS storage cluster is “stretched” across these 3 AZs.

MySQL deployment with StatefulSets

This post revolves around deploying a MySQL pod using OCS and StatefulSets (STS), so let’s get started.

Stateful applications need persistent volume(s) (PVs) to support failover scenarios in which, when a pod (or pods) move(s) to a different worker node, the data it/they use(s) must be persistent after the pod(s) move(s).

STS were introduced in Kubernetes 1.9 and have a few advantages over “simple” deployments:

  1. Pod creation can be ordered when creating (and reversed ordered when scaling down). This is especially important in master/slave scenarios and/or distributed databases.
  2. Pods can have an easy naming convention and retain the name when migrating from one node to another after a failover.
  3. The persistent volume claims (PVCs) are not deleted when the STS is deleted to keep the data intact for future usage.

The first step in creating a PVC is making sure we have a storage class we can use to dynamically create the volume in OCP:

oc get sc
NAME                PROVISIONER      AGE
glusterfs-storage   kubernetes.io/glusterfs   9d
gp2 (default)       kubernetes.io/aws-ebs     21d
gp2-xfs             kubernetes.io/aws-ebs     18d

As you can see, we have 3 storage classes in our OCP cluster. For the MySQL deployment, we will be using the glusterfs-storage class, which is created with the installation of OCS when deploying OCP using Ansible playbooks and specific OCS inventory file options. This means that every time a claim is made for storage it will be the glusterfs-storage class that will provide it because it is configured into our STS definition file. If you want to see the content of any of the storageclass (SC) resources, run “oc get sc <storageclass_name> -o yaml”.

Because we are going to use STS, one of the requirements is to create a headless service for our MySQL application. We’re going to use the following yaml file:

cat headless-service-mysql.yaml
apiVersion: v1
kind: Service
metadata:
    name: mysql-ocs
    labels:
       app: mysql-ocs
spec:
    ports:
    - port: 3306
       name: mysql-ocs
    clusterIP: None
    selector:
       app: mysql-ocs

And then create the service.

oc create -f headless-service-mysql.yaml
service/mysql-ocs created
$oc get svc
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP    PORT(S)    AGE
mysql-ocs   ClusterIP   None         <none>         3306/TCP   6s

Now that we have a storageclass and a headless service, let’s look at our STS yaml. This is a simple example, and as we progress in this series, we’ll update and add to this file.

Note: It is neither secure nor recommended to have plain-text password sets in yaml files. Instead, use secrets. For our example, to make things simple, we’ll use plain text.

cat mysql-sts.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: mysql-ocs
spec:
  selector:
         matchLabels:
      app: mysql-ocs
  serviceName: "mysql-ocs"
  podManagementPolicy: Parallel
  replicas: 1
  template:
    metadata:
      labels:
        app: mysql-ocs
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mysql-ocs
        image: mysql:5.7
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        - name: MYSQL_DATABASE
          value: wordpress
        - name: MYSQL_USER
          value: admin
        - name: MYSQL_PASSWORD
          value: secret
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-ocs-data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: mysql-ocs-data
    spec:
      storageClassName: glusterfs-storage
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 8Gi

Most of the container definitions are similar to that of a “DeploymentConfig” type. We’re using the headless service “mysql-ocs” that we previously created and specified MySQL 5.7 as the image to be used. The interesting part is at the bottom of the preceding file; the “volumeClaimTemplates” definition is how we create a persistent volume (PV), then claim it (PVC) and attach it to the newly created MySQL pod. As you can also see, we’re using the storage class we have from the OCP/OCS installation (glusterfs-storage), and we request a volume size of 8 GB to be created and use in a “ReadWriteOnce” mode.

To create our STS, we run the following command:

oc create -f mysql-sts.yaml
statefulset.apps/mysql-ocs created
Deployment validation

Let’s check that the pod is running. Please note that, depending on the hardware used, the MySQL container image download speed, size of volume requested, and availability of existing PVCs, this action can take between from a few seconds to around a minute.

oc get pods
NAME          READY     STATUS      RESTARTS     AGE
mysql-ocs-0   1/1       Running     0            31s

Let’s look at the PVC we created with this STS.

oc get pvc
NAME                         STATUS     VOLUME
CAPACITY ACCESS MODES     STORAGECLASS          AGE
mysql-ocs-data-mysql-ocs-0   Bound 
pvc-cb25b2c0-3a12-11e9-96fc-02e7350e98d2     8Gi       RWO 
glusterfs-storage 1m

And the PV that is associated with the PVC:

oc get pv
NAME                                       CAPACITY ACCESS MODES  RECLAIM 
POLICY   STATUS    CLAIM .                         STORAGECLASS 
REASON    AGE
pvc-cb25b2c0-3a12-11e9-96fc-02e7350e98d2   8Gi      RWO .         Delete 
Bound     sagy/mysql-ocs-data-mysql-ocs-0   glusterfs-storage          3m

If you want to see the connection/relationship between Kubernetes, gluster, heketi, and our persistent storage volume, we can run a few commands to show it. We know the PV name from our “oc get pvc” we ran previously, so we’ll use “oc describe” and search for Path.

oc describe pv pvc-d1fc687c-3a14-11e9-96fc-02e7350e98d2|grep Path
    Path:           vol_82f64c461e4796213160f30519f318f8

In our case, the volume name is vol_82f64c461e4796213160f30519f318f8, and this is the same volume name in gluster., If you log in to the container inside the MySQL pod, we can see the same volume and the directory it is mounted to.

oc rsh mysql-ocs-0
$ df -h|grep vol_82f64c461e4796213160f30519f318f8
172.16.26.120:vol_82f64c461e4796213160f30519f318f8  8.0G 325M 7.7G 4% 
/var/lib/mysql

We can see that the volume is mounted on /var/lib/mysql (what we specified in our STS yaml file) and size is 8.0G.

If we want to check heketi for more info, we must first make sure that heketi-client package is installed on the server you’re trying to run it from. The following file must be sourced to export the environment before using heketi-client commands.

cat heketi-export-app-storage
export HEKETI_POD=$(oc get pods -l glusterfs=heketi-storage-pod -n 
app-storage -o jsonpath='{.items[0].metadata.name}')
export HEKETI_CLI_SERVER=http://$(oc get route/heketi-storage -n app-storage 
-o jsonpath='{.spec.host}')
export HEKETI_CLI_USER=admin
export HEKETI_CLI_KEY=$(oc get pod/$HEKETI_POD -n app-storage -o 
jsonpath='{.spec.containers[0].env[?(@.name=="HEKETI_ADMIN_KEY")].value}')
export HEKETI_ADMIN_KEY_SECRET=$(echo -n ${HEKETI_CLI_KEY} | base64)

The  heketi volume name is the gluster volume name without the “vol_”, which can be found using the following command:

oc describe pv pvc-d1fc687c-3a14-11e9-96fc-02e7350e98d2|grep Path|awk '{print 
$2}'|awk -F 'vol_' '{print $2}'
82f64c461e4796213160f30519f318f8

And now, after we made sure heketi-cli is installed and sourced the environment variables, the heketi-cli command can be used to get more information about this gluster volume.

heketi-cli volume info 82f64c461e4796213160f30519f318f8
Name: vol_82f64c461e4796213160f30519f318f8
Size: 8
Volume Id: 82f64c461e4796213160f30519f318f8
Cluster Id: f05418936dc63638041af2831914c37d
Mount: 172.16.26.120:vol_82f64c461e4796213160f30519f318f8
Mount Options: 
backup-volfile-servers=172.16.53.212,172.16.39.190,172.16.56.45,172.16.27.161
,172.16.44.7
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3
Snapshot Factor: 1.00
Deleting StatefulSet and persistent storage

So far, we’ve seen how to create a MySQL pod using STS and OCS storage, but what happens when we want to delete a pod or the storage? First, let’s look at our PVC.

oc get pvc
NAME                         STATUS    VOLUME                        
CAPACITY ACCESS MODES     STORAGECLASS        AGE
mysql-ocs-data-mysql-ocs-0   Bound 
pvc-d1fc687c-3a14-11e9-96fc-02e7350e98d2   8Gi        RWO 
glusterfs-storage 20h

Now let’s delete our STS for MySQL.

$ oc delete -f mysql-sts.yaml
statefulset.apps "mysql-ocs" deleted

And let’s check the PVC again after MySQL STS is deleted.

oc get pvc
NAME                         STATUS    VOLUME                        
CAPACITY ACCESS MODES     STORAGECLASS        AGE
mysql-ocs-data-mysql-ocs-0   Bound 
pvc-d1fc687c-3a14-11e9-96fc-02e7350e98d2    8Gi       RWO 
glusterfs-storage 20h

As you can see. the PVC remains with the data intact and will be used again if we will redeploy the same STS.

If you want to delete the PVC, run the following command:

$ oc delete pvc mysql-ocs-data-mysql-ocs-0
persistentvolumeclaim "mysql-ocs-data-mysql-ocs-0" deleted

And you monitor the PV and watch how it gets deleted, as well (PV is first released).

oc get pv
NAME                                       CAPACITY ACCESS MODES RECLAIM 
POLICY STATUS     CLAIM                              STORAGECLASS   
REASON  AGE
pvc-d1fc687c-3a14-11e9-96fc-02e7350e98d2   8Gi      RWO          Delete 
Released   sagy/mysql-ocs-data-mysql-ocs-0    glusterfs-storage 
20h

And if we query again, the PV will be gone and deleted.

oc get pvc
No resources found.
Conclusion

In this post, we’ve shown the first step toward running on OCP an application that needs persistent data. We used the glusterfs-storage storageclass that is provided by OCS to create a PVC and attached the volume to a MySQL pod. We automated the process using an STS. We also explained the relationship between OCS, heketi, the PV, PVC, and the MySQL pod.

In our next post we’ll show how to connect a WordPress pod to our database pod.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
A SAAS solution to monitor your Ceph storage infrastructure By Ilan Rabinovitch (Datadog) and Federico Lucifredi (Red Hat) Monitoring a distributed system

Red Hat Ceph Storage is a highly scalable, fault-tolerant platform for object, block, and file storage that delivers excellent data resiliency (we default to keeping three copies of a customer’s data at all times), with service availability capable of enduring the loss of a single drive, of a cluster node, or even of an entire rack of storage without users experiencing any interruption. Like its resiliency, Ceph’s ability to scale is another outcome of its distributed architecture.

Distributed systems’ architectures break with the common assumptions made by most traditional monitoring tools in defining the health of an individual device or service. In a somewhat obvious example, Nagios’ Red/Green host (or drive) health status tracking becomes inadequate, as the loss of a drive either generates unnecessary alerts or fails to highlight enough what is likely to be a more urgent condition, like the loss of a MON container. The system can withstand the loss of multiple drives or storage nodes without needing immediate action from an operator, as long as free storage capacity remains available on other nodes. More urgent events, like the loss of a MON bringing the cluster from HA+1 to HA status, would, however, be mixed in with all the other “red” false alarms and lost in all the noise.

Distributed systems need monitoring tools that are aware of their distributed nature to ensure that pagers go off only for alerts truly critical in nature. Most hardware failure reports naturally found in a large enough system are managed weekly or monthly as part of recurring maintenance activity. An under-marketed advantage of distributed systems is that the swapping of failed drives or the replacement of PSUs becomes a scheduled activity, not an emergency one.

Red Hat Ceph Storage’s built-in monitoring tools are designed to help you keep tabs on your clusters’ health, performance, and resource usage, while avoiding distributed-systems awareness shortcomings. Red Hat has a long history of customer choice, and for that reason we include with Red Hat Ceph Storage documentation information on how to use external monitoring tools. Nagios may be sub-optimal for the details, but it is still the closest thing we have to a standard in the open source community’s fragmented monitoring space.

Datadog is much more interesting.

Monitoring is a service

Unlike traditional monitoring solutions, Datadog’s software-as-a-service (SaaS) platform was built specifically for dynamic distributed systems like Red Hat Ceph Storage. Datadog automatically aggregates data from ephemeral infrastructure components, so you can maintain constant visibility even as your infrastructure scales up or down. And because Datadog is fully hosted, it self-updates with features so you only get alerted when it matters most. Field-tested machine learning algorithms distinguish between normal and abnormal trends, and you can configure alerts to trigger only on truly urgent occurrences (e.g., the loss of a Ceph monitor rather than the loss of a single storage node). To enable data-driven collaboration and troubleshooting, Datadog automatically retains your monitoring data for more than a year and makes that data easily accessible from one central platform.

Datadog provides an out-of-the-box integration with Red Hat Ceph Storage to help you get more real-time visibility into the health and performance of your clusters with near-zero setup delay. Datadog also delivers template integrations with more than 250 other technologies, including services that are commonly used with Ceph storage, like OpenStack and Amazon S3, so you can get more comprehensive insights into every layer of your stack in one place.

We will explore a few ways in which you can use Datadog to monitor Red Hat Ceph Storage in full context with the rest of your stack. Then we’ll explain how to set up Datadog to start getting clearer insights into your Ceph deployment in three easy steps.

Key Ceph metrics at a glance

Datadog automatically collects metrics from your Red Hat Ceph Storage clusters and makes it easy to explore, visualize, and alert on this data at the cluster, pool, and node levels. The integration includes a template Ceph dashboard that displays an overview of health and performance data from your monitor and storage nodes. You can use the template variables at the top of the dashboard to filter metrics by individual clusters, pools, and Object Storage Daemons (OSDs) to get more granular insights.

Red Hat Ceph Storage includes robust features for high availability and performance and, by monitoring its built-in health checks, you can help you ensure everything is running smoothly.

Datadog automatically queries Ceph for the status of these health checks, along with other key information about your nodes, including:

  • Object Storage Daemon (OSD) statusQuickly find out if an OSD is down, so you can try restarting the node or troubleshooting potential issues (e.g., networking, disk capacity).

  • Monitor status: If you’re running more than one monitor in your cluster (as recommended for high availability), Ceph requires a quorum of monitor nodes to reach a consensus about the latest version of the cluster map. Monitor nodes fall out of the quorum when they become unavailable, or when they fall behind and cannot access the latest version of the map. If your cluster cannot maintain a quorum, clients will be unable to read or write data from the cluster. With Datadog, you can track the number of available monitor nodes in your cluster, as well as the real-time quorum status (the number of monitors in the quorum). You can also set up an alert that notifies you when the number of monitors in the quorum decreases, so you can have enough time to troubleshoot the issue or deploy more nodes if needed.
  • Storage capacity: Datadog’s Red Hat Ceph Storage integration reports OSD storage capacity metrics so you can take action before any OSD runs out of disk space (at which point Ceph will stop writing data to the OSD to guard against data loss). You can set up an alert to detect when any OSD reaches a “NEARFULL” state (85 percent capacity, by default), which gives you enough time to add more OSDs, as recommended in the documentation. You can also use Datadog’s forecasting algorithms to get notified a certain amount of time before any OSD, pool, or cluster is predicted to run out of disk space.

Increased visibility across Ceph metrics

Datadog’s integration also reports other metrics from Ceph, including the rate of I/O operations and commit latency. See the full list of metrics collected as part of this integration in our documentation.

Although it’s important to monitor these metrics, they provide only part of the picture. In the next section, we’ll explore a few of the other ways you can use Datadog to monitor Red Hat Ceph Storage alongside all the other services in your environment.

Monitoring Ceph in context

Your infrastructure depends on Ceph for storage, but it also relies on a range of other systems, services, and applications. To help you monitor Red Hat Ceph Storage in context with other components of your stack, Datadog also integrates with more than 250 technologies, including Amazon S3 and OpenStack Nova.

Monitoring OpenStack + Ceph

If you’ve deployed Ceph on OpenStack, Datadog can help you get clearer insights into your infrastructure across multiple dimensions. Datadog’s OpenStack integration includes a default dashboard that provides a high-level overview of metrics from the hypervisors, Nova servers, tenants, and other components of your OpenStack Compute cluster.

To learn more about integrating Datadog with OpenStack, consult the documentation.

Monitoring Amazon S3 + Ceph

If you’re using Amazon S3 alongside Red Hat Ceph Storage, it’s important to track your S3 activity in real time. Datadog’s AWS S3 integration automatically collects metrics related to request throughput, HTTP errors, and latency. Upon setting up the AWS S3 integration, you’ll see all these key metrics displayed in an out-of-the-box dashboard.

To learn more about integrating Datadog with AWS S3, consult the documentation.

More visibility with APM and logs

Datadog’s distributed tracing and APM can help you monitor the performance of applications and services that use Ceph. Datadog APM is fully integrated with the rest of Datadog, so you can easily navigate from inspecting a distributed request trace to viewing system-level metrics from the specific host that executed that unit of work.

You can also use log processing and analytics to collect and monitor Ceph logs in the same place as your metrics and distributed request traces. Simply follow the configuration steps described here.

Setup guide

It only takes a few minutes to set up Datadog’s Ceph integration. The open source Datadog Agent collects data (including system-level metrics like CPU and memory usage) from all your nodes, as well as the services running on those nodes, so that you can view, correlate, and alert on this data in real time.

Installing the Agent on a node usually only takes a single command—see the instructions for your platform here. You can also deploy the Agent across your entire Ceph cluster with configuration management tools like Chef and Ansible, if desired.

Configure metric collection

To configure the Datadog Agent to collect Ceph metrics, you’ll need to create a configuration file for the integration on your Ceph nodes. The Agent comes with an example config that you can use as a template. Navigate to the “ceph.d” directory within your Agent’s configuration directory, and locate the example configuration file: **conf.yaml.example**.

Copy the example to a new **conf.yaml** file, and edit the new file to include the correct path to your Ceph executable. The Agent check expects the executable to be located at “/usr/bin/ceph”, but you can specify a different path if needed:

init_config:

instances:
  - ceph_cmd: /path/to/your/ceph # default is /usr/bin/ceph
    use_sudo: true               # only if the ceph binary needs sudo on your nodes

As the preceding example shows, you can also enable “sudo” access if it’s required to execute “ceph” commands on your nodes. If you enable the “use_sudo” option, you must also add the Datadog Agent to your sudoers file, as described in the documentation. For example:

dd-agent ALL=(ALL) NOPASSWD:/usr/bin/ceph
Restart the Agent

Save and exit the configuration file. Restart the Agent using the command for your platform (as specified here) to pick up the Agent configuration change. Then run the Agent status command to ensure that the Agent can successfully connect to Ceph and retrieve data from your cluster. When the integration is working properly, you should see a “ceph” section in the terminal output, similar to the following snippet:

  Running Checks
  ==============
    ceph (unversioned)
    ------------------
      Total Runs: 124
      Metric Samples: 27, Total: 3348
      Events: 0, Total: 0
      Service Checks: 19, Total: 2356
      Average Execution Time : 2025ms

In the Datadog platform, navigate to the Ceph integration tile of your Datadog account and click the “Install Integration” button.

Now that the Datadog Agent is collecting metrics from Ceph, you should start to see data flowing into the built-in Ceph dashboard in your Datadog account.

After you deploy Datadog, your Ceph data is available for visualization, alerting, and correlation with monitoring data from the rest of your infrastructure and applications. The template Ceph dashboard provides a high-level overview of your cluster at a glance, but you can easily customize it to highlight the information that matters most. And dashboards are just the tip of the iceberg—You can use features..

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Red Hat Ceph Storage 3.2 is now available! The big news with this release is full support for the BlueStore Ceph backend, offering significantly increased performance for both object and block applications.

First available as a Technology Preview in Red Hat Ceph Storage 3.1, Red Hat has conducted extensive performance tuning and testing work to verify that BlueStore is now ready for use in production environments. With the 3.2 release, Red Hat Ceph Storage has attributes that make it suitable for a wide range of use cases and workloads, including:

  • Data analytics: As a data lake, Red Hat Ceph Storage uses object storage to deliver massive scalability and high availability to support demanding multitenant analytics workloads. Disparate analytics clusters can be consolidated to reduce cost of ownership, lower administrative burden, and increase service levels. BlueStore helps improve performance, while support for erasure coding helps reduce overall storage costs for data protection over simple replication.
  • Hybrid cloud applications: Red Hat Ceph Storage is ideal for on-premise storage clouds. Because Red Hat Ceph Storage supports the Amazon Web Services (AWS) Simple Storage Service (S3) interface, applications can access their storage with the same API, whether in public or private clouds.
  • OpenStack applications. Red Hat Ceph Storage is very popular for OpenStack applications. Red Hat Ceph Storage 3.2 can offer improved performance for OpenStack deployments, including Red Hat OpenStack Platform. Erasure coding for RADOS Block Device (RBD) is available as a Technology Preview in this release.
  • Backup target. A growing list of software vendors have certified their backup applications with Red Hat Ceph Storage as a backup storage target:
    • Veritas NetBackup for Symantec OpenStorage (OST) cloud backup – versions 7.7 and 8.0  
    • Rubrik Cloud Data Management (CDM) – versions 3.2 and later  
    • NetApp AltaVault – versions 4.3.2 and 4.4  
    • Trilio, TrilioVault – versions 3.0
    • Veeam Backup & Replication – version 9.x
BlueStore performance

BlueStore is all about performance. For hard disk drive (HDD) based clusters, BlueStore architecturally removes the double-write penalty incurred by the traditional FileStore backend. Additionally, BlueStore provides significant performance enhancements in configurations that use all solid-state drives (SSDs) or Non Volatile Memory Express (NVM Express, or NVMe) drives.

The architectural shift to a BlueStore backend has already shown performance improvements on community Ceph distributions. Testing by Micron in 2018 demonstrated up to 2x increases in performance with the BlueStore over the traditional FileStore backend.

Micron conducted BlueStore vs. FileStore object testing and reported significant performance improvements in terms of both improved throughput and reduced latency.

4MB objects

100% writes

  • 88% increase in throughput
  • 47% decrease in average latency
70%/30% reads/writes
  • 64% increase in throughput
  • 40% decrease in average latency

Micron also conducted BlueStore vs. FileStore block testing and reported higher IOPS and lower latency.

4K random blocks

100% writes

  • 18% higher I/O operations (IOPS)
  • 5% lower average latency
  • Up to 70%+ reduced 99.999% latency
70%/30% reads/writes
  • 14% higher IOPS
  • 80%+ lower read tail latency
  • 70%+ lower write tail latency
Upgrades and new installs

Importantly, both the BlueStore and FileStore backends coexist in Red Hat Ceph Storage 3.2. Existing Red Hat Ceph Storage 2.5 and 3.1 clusters retain the FileStore backend when upgrading to version 3.2. Newly created Red Hat Ceph Storage clusters default to the BlueStore backend. Those wishing to upgrade existing clusters to the BlueStore backend should contact Red Hat Support.

For more information on how Red Hat Ceph can tackle your toughest data storage challenges, please visit our Ceph product page.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Red Hat Ceph Storage 3.2 is now available! The big news with this release is full support for the BlueStore Ceph backend, offering significantly increased performance for both object and block applications.

First available as a Technology Preview in Red Hat Ceph Storage 3.1, Red Hat has conducted extensive performance tuning and testing work to verify that BlueStore is now ready for use in production environments. With the 3.2 release, Red Hat Ceph Storage has attributes that make it suitable for a wide range of use cases and workloads, including:

  • Data analytics: As a data lake, Red Hat Ceph Storage uses object storage to deliver massive scalability and high availability to support demanding multitenant analytics workloads. Disparate analytics clusters can be consolidated to reduce cost of ownership, lower administrative burden, and increase service levels. BlueStore helps improve performance, while support for erasure coding helps reduce overall storage costs for data protection over simple replication.
  • Hybrid cloud applications: Red Hat Ceph Storage is ideal for on-premise storage clouds. Because Red Hat Ceph Storage supports the Amazon Web Services (AWS) Simple Storage Service (S3) interface, applications can access their storage with the same API, whether in public or private clouds.
  • OpenStack applications. Red Hat Ceph Storage is very popular for OpenStack applications. Red Hat Ceph Storage 3.2 can offer improved performance for OpenStack deployments, including Red Hat OpenStack Platform. Erasure coding for RADOS Block Device (RBD) is available as a Technology Preview in this release.
  • Backup target. A growing list of software vendors have certified their backup applications with Red Hat Ceph Storage as a backup storage target:
    • Veritas NetBackup for Symantec OpenStorage (OST) cloud backup – versions 7.7 and 8.0  
    • Rubrik Cloud Data Management (CDM) – versions 3.2 and later  
    • NetApp AltaVault – versions 4.3.2 and 4.4  
    • Trilio, TrilioVault – versions 3.0
    • Veeam Backup & Replication – version 9.x
BlueStore performance

BlueStore is all about performance. For hard disk drive (HDD) based clusters, BlueStore architecturally removes the double-write penalty incurred by the traditional FileStore backend. Additionally, BlueStore provides significant performance enhancements in configurations that use all solid-state drives (SSDs) or Non Volatile Memory Express (NVM Express, or NVMe) drives.

The architectural shift to a BlueStore backend has already shown performance improvements on community Ceph distributions. Testing by Micron in 2018 demonstrated up to 2x increases in performance with the BlueStore over the traditional FileStore backend.

Micron conducted BlueStore vs. FileStore object testing and reported significant performance improvements in terms of both improved throughput and reduced latency.

4MB objects

100% writes

  • 88% increase in throughput
  • 47% decrease in average latency
70%/30% reads/writes
  • 64% increase in throughput
  • 40% decrease in average latency

Micron also conducted BlueStore vs. FileStore block testing and reported higher IOPS and lower latency.

4K random blocks

100% writes

  • 18% higher I/O operations (IOPS)
  • 5% lower average latency
  • Up to 70%+ reduced 99.999% latency
70%/30% reads/writes
  • 14% higher IOPS
  • 80%+ lower read tail latency
  • 70%+ lower write tail latency
Upgrades and new installs

Importantly, both the BlueStore and FileStore backends coexist in Red Hat Ceph Storage 3.2. Existing Red Hat Ceph Storage 2.5 and 3.1 clusters retain the FileStore backend when upgrading to version 3.2. Newly created Red Hat Ceph Storage clusters default to the BlueStore backend. Those wishing to upgrade existing clusters to the BlueStore backend should contact Red Hat Support.

For more information on how Red Hat Ceph can tackle your toughest data storage challenges, please visit our Ceph product page.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
By Annette Clewett and Luis Rico

The snapshot capability in Kubernetes is in tech preview at present and, as such, backup/recovery solution providers have not yet developed an end-to-end Kubernetes volume backup solution. Fortunately, GlusterFS, an underlying technology behind Red Hat OpenShift Container Storage (RHOCS), does have a mature snapshot capability. When combined with enterprise-grade backup and recovery software, a robust solution can be provided.

This blog post details how backup and restore can be done when using RHOCS via GlusterFS. As of the Red Hat OpenShift Container Platform (OCP) 3.11 release, there are a limited number of storage technologies (EBS, Google Cloud E pDisk, and hostPath) that support creating and restoring application data snapshots via Kubernetes snapshots. This Kubernetes snapshot feature is in tech preview, and the implementation is expected to change in concert with upcoming Container Storage Interface (CSI) changes. CSI, a universal storage interface (effectively an API) between container orchestrators and storage providers, is ultimately where backup and restore for OCS will be integrated in the future using volume snapshots capability.

Traditionally, backup and restore operations involve two different layers. One is the application layer. For example, databases like PostgreSQL have their own procedures to do an application consistent backup. The other is the storage layer. Most storage platforms provide a way for backup software like Commvault or Veritas NetBackup to integrate, obtain storage level snapshots, and perform backups and restores accordingly. An application layer backup is driven by application developers and is application specific. This study will focus on traditional storage layer backup and restore using Commvault Complete Backup and Recovery Software for this purpose. Other backup software tools can be used in a similar manner if they supply the same capabilities as used with Commvault.

RHOCS can be deployed in either converged mode or independent mode, and both are supported by the process described in this article. Converged mode, formerly known as Container Native Storage (CNS), means that Red Hat Gluster Storage is deployed in containers and uses the OCP host storage and networking. Independent mode, formerly known as Container Ready Storage (CRS), is deployed as a stand-alone Red Hat Gluster Storage cluster that provides persistent storage to OCP containers. Both modes of RHOCS deployment use heketi in a container on OCP for provisioning and managing GlusterFS volumes.

Storage-level backup and restore for RHOCS

If a backup is performed at the Persistent Volume (PV) level, then it will not capture the OCP Persistent Volume Claim (PVC) information. OCP PVC to PV mapping is required to identify which backups belong to which application. This leaves a gap: Which GlusterFS PV goes with which OCP PVC?

In a traditional environment, this is solved by naming physical volumes such that the administrator has a way of identifying which volumes belong to which application. This naming method can now be used in OCP (as of OCP 3.9) by using custom volume naming in the StorageClass resource. Before OCP 3.9, the names of the dynamically provisioned GlusterFS volumes were auto-generated with random vol_UUID naming. Now, by adding a custom volume name prefix in the StorageClass, the GlusterFS volume name will include the OCP namespace or project as well as the PVC name, thereby making it possible to map the volume to a particular workload.

OCS custom volume naming

Custom volume naming requires a change to the StorageClass definition. Any new RHOCS persistent volumes claimed using this StorageClass will be created with a custom volume name. The custom volume name will have prefix, project or namespace, PVC name and UUID (<myPrefix>_<namespace>_<claimname>_UUID).

The following glusterfs-storage StorageClass has custom volume naming enabled by adding the volumenameprefix parameter.

# oc get sc glusterfs-storage -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: glusterfs-storage
parameters:
  resturl: http://heketi-storage-app-storage.apps.ocpgluster.com
  restuser: admin
  secretName: heketi-storage-admin-secret
  secretNamespace: app-storage
  volumenameprefix: gf ❶
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete

❶ Custom volume name support: <volumenameprefixstring>_<namespace>_<claimname>_UUID

As an example, using this StorageClass for a namespace of mysql1 and PVC name of mysql the volume name would be gf_mysql1_mysql_043e08fc-f728-11e8-8cfd-028a65460540 (the UUID portion of the name will be unique for each volume name).

Note: If custom volume naming cannot be used, then it is important to collect information about all workloads using PVCs, their OCP PV associated, and the GlusterFS volume name (contained in Path variable in description of OCP PV).

RHOCS backup process

The goal of this blog post is to provide a generic method to back up and restore OCS persistent volumes used with OCP workloads. The example scripts and .ini files have no specific dependency on a particular backup and restore product. As such, they can be used with a product such as Commvault, where the scripts can be embedded in the backup configuration. Or, they can be used standalone, assuming that basic backup/recovery of the mounted gluster snapshots will be done via standard RHEL commands.

Note: The methods described here apply only for gluster-file volumes and currently will not work for gluster-block volumes.

For this approach, a “bastion host” is needed for executing the scripts, mounting GlusterFS snapshot volumes, and providing a place to install the agent if using backup and restore software. The bastion host should be a standalone RHEL7 machine separate from the OCP nodes and storage nodes in your deployment.

Requirements for the bastion host

The bastion host must have network connectivity to both the backup and restore server (if used), as well as the OCP nodes with the gluster pods (RHOCS converged mode) or the storage nodes (RHOCS independent mode). The following must be installed or downloaded to the bastion host:

  • backup and restore agent, if used
  • heketi-client package
  • glusterfs-fuse client package
  • atomic-openshift-clients package
  • rhocs-backup scripts and .ini files
RHOCS backup scripts

The github repository rhocs-backup contains unsupported example code that can be used with backup and restore software products. The two scripts, rhocs-pre-backup.sh and rhocs-post-backup.sh, have been tested with Commvault Complete Backup and Recovery Software. The rhocs-pre-backup.sh script will do the following:

  • Find all gluster-file volumes using heketi-client
  • Create a gluster snapshot for each volume
  • Mount the the snapshot volumes on the bastion host that has the backup agent installed
  • Protect the heketi configuration database by creating a json file for the database in the backup directory where all gluster snapshots are going to be mounted

Once the mounted snapshot volumes have been backed up, the rhocs-post-backup.sh script will do the following:

  • Unmount the snapshot volumes
  • Delete the gluster snapshot volumes

The two .ini files, independent_vars.ini and converged_vars.ini, are used to specify parameters specific to your RHOCS mode of deployment. Following are example parameters for converged_var.ini.

## Environment variables for RHOCS Backup:
## Deployment mode for RHOCS cluster: converged (CNS) or independent (CRS)
export RHOCSMODE="converged"

## Authentication variables for accessing OpenShift cluster or
## Gluster nodes depending on deployment mode
export OCADDRESS="https://master.refarch311.ocpgluster.com:443"
export OCUSER="openshift"
export OCPASS="redhat"
export OCPROJECT="app-storage" ## OpenShift project where gluster cluster lives

## Any of the Gluster servers from RHOCS converged cluster
## used for mounting gluster snapshots
export GLUSTERSERVER=172.16.31.173

## Directory for temporary files to put the list of
## Gluster volumes /snaps to backup
export VOLDIR=/root
export SNAPDIR=/root

## Destination directory for mounting snapshots of Gluster volumes:
export PARENTDIR=/mnt/source

## Heketi Route and Credentials
export USERHEKETI=admin ## User with admin permissions to dialog with Heketi
export SECRETHEKETI="xzAqO62qTPlacNjk3oIX53n2+Z0Z6R1Gfr0wC+z+sGk=" ## Heketi user key
export HEKETI_CLI_SERVER=http://heketi-storage-app-storage.apps.refarch311.ocpgluster.com ##
Route where Heketi pod is listening

## Provides Logging of this script in the dir specified below:
export LOGDIR="/root"

The pre-backup script, when executed, uses the heketi-client for the list of current gluster-file volumes. Because of this, for the script to work properly the heketi container must be online and reachable from bastion host. Additionally, for the scripts to work properly, all GlusterFS nodes or peers of RHOCS cluster must be online, as GlusterFS snapshot operation requires all bricks of a GlusterFS volume be available.

Manual execution of pre- and post-backup scripts

This section assumes that the bastion host has been created and has the necessary packages, scripts, and .ini files are installed on this machine. Currently, the pre- and post-backup scripts run as the root user. Because of this, backing up volumes for RHOCS independent mode will require that the bastion host can SSH as the root user with passwordless access to one of the GlusterFS storage nodes. This access should be verified before attempting to run the following scripts.

The scripts can be manually executed in the following manner for RHOCS converged mode:

sudo ./rhocs-pre-backup.sh /<path_to_file>/converged_vars.ini

Followed by this script to unmount the snapshot volumes and to remove the snapshot volumes from the RHOCS Heketi database and GlusterFS converged cluster:

sudo ./rhocs-post-backup.sh /<path_to_file>/converged_vars.ini

A variation of these scripts for RHOCS independent mode can be manually executed in the following manner:

sudo ./rhocs-pre-backup.sh /<path_to_file>/independent_vars.ini

Followed by this script to unmount the snapshot volumes and to remove the snapshot volumes from the RHOCS Heketi database and GlusterFS independent cluster:

sudo ./rhocs-post-backup.sh /<path_to_file>/independent_vars.ini

For each execution of the pre- or post-backup script a log file will be generated and placed in the directory specified in the .ini file (default is /root).

Note: Pre-backup scripts can be modified as needed for specific scenarios to achieve application-level consistency, like quiescing a database before taking a backup. Also, if special features are used with RHOCS, like SSL encryption or geo-replication, scripts will have to be customized and adjusted to be compatible with those features.

Commvault backup process

Note that this blog post does not cover the tasks to install and configure Commvault to back up and restore data. In addition to having the Commvault Console and Agent in working order, this section also assumes that the bastion host has been created and has the necessary packages, scripts, and .ini files installed.

The use of these scripts to back up OCP PVs is compatible with any backup frequency or retention configured in the Commvault backup policy. But, as we are mounting gluster snapshots in newly created folders with date and time information, the backup application will always consider contents as new, so even if backup policy is incremental, it will effectively do a full backup. Also, the backup content will consist of dozens or even hundreds of very small filesystems (1-10 GB), that could run faster under a “always do full backup” strategy.

Detailed process for backup using Commvault

Once the scripts and .ini files are on the bastion host and a Commvault Agent is installed, a backup can be done using the Commvault Commcell Console or the Commvault Admin Console. The following views show how to do the backup using the Commcell Console and validate the backup using the Admin Console.

A Subclient must be created before a backup can be done and a unique name must be specified.

When creating a Subclient, you must input where on the bastion host you want the backup to be done from on the bastion host with Commvault Agent (e.g., /mnt/source).

Choose what schedule you want the backup done on (or Do Not Schedule; start backup manually using Console instead).

And last or the Subclient configuration, add the path to the pre- and post-backup scripts, as well as the path to the appropriate .ini file, converged_vars.ini or independent_vars.ini. Once this is done and the subclient has been saved, you are ready to take a backup of the gluster-file snapshot volumes.

The backup can then be done by selecting the desired subclient and issuing an immediate backup or letting the selected schedule do the backups when configured (e.g., daily).

For an immediate backup, you can choose full or incremental. As already stated, a full backup will be done every time, because the pre-backup script always creates a new directory to mount the gluster-file snapshot volume.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
Red Hat Storage by Colleen Corrice - 5M ago

Our top 3 storage-for-containers things to look forward to at KubeCon By Steve Bohac, OpenShift Storage Product Marketing

Season greetings!

As always, much going on with Red Hat OpenShift Container Storage!

Of course, Kubernetes 1.13 was released this week, Container Journal recently published an article I authored, and KubeCon Seattle is coming up next week… By the way, did you see the latest Forrester Wave Enterprise Container Platform Software Suites where Red Hat OpenShift was named a Leader? Good stuff!

Red Hat OpenShift Container Storage helps organizations standardize storage across multiple environments and easily integrates with Red Hat OpenShift to deliver a persistent storage layer for containerized applications that require long-term, stateful storage. Enterprises can benefit from a simple, integrated solution including the container platform, registry, application development environment, and storage—all in one, supported by a single vendor.

December is always a busy month with industry conferences (not to mention holiday planning!), so as I finalized my own KubeCon plans, I wanted to pause and take a quick breath and outline my top 3 things I’m looking forward to at KubeCon Seattle 2018 next week:

  1. Assorted Kubernetes announcements (whatever they are!). Yes, who knows what kind of interesting things will be announced next week… but they’ll likely be exciting! The Kubernetes ecosystem has gotten so large now, there is always a plethora of interesting products and technologies announced at KubeCon. It’s always interesting to see how these new announcements dictate where things are going with Kubernetes and cloud native technologies in general. (By the way, for a great overview of the “third era” of Kubernetes, check out PodCTL #54 with our own Brian Gracely and Tyler Britten.)
  2. For the first time ever, there will be a Cloud Native Storage Day as one of the co-located events at KubeCon. Like the other co-located events, it takes place next Monday before the KubeCon show officially kicks off. The day’s agenda includes customers and industry leaders like Red Hat (I’ll be there with a few colleagues presenting) discussing current implementations and future directions of container storage. This should be very educational and interactive for everyone! And…. the sessions will be recorded (look back here for a post-KubeCon blog after the show for links to the recordings!).
  3. Catching up on the status of the Rook project. What is Rook? Rook is a persistent storage orchestrator that is designed to run as a native Kubernetes service. Consider it the glue between storage and the container—the thing that makes automation work. This is an interesting development around storage for containers, and I’m looking forward to meeting up with colleagues and “fellow travelers” to understand more.

Anyway, it should be a good one at KubeCon next week (did I mention it is sold out!?). In between sessions, make sure to visit us in Booth D1 in the Expo Hall for product demonstrations, to speak with Red Hat OpenShift Container Storage experts and other community leaders about upstream projects, and to snag some of our giveaways (while supplies last!).

We hope to see you there! If we don’t catch you in person, we’ll be tweeting (and re-tweeting) all week! If you don’t already, make sure to follow us on Twitter at @RedHatStorage.

Not attending KubeCon? No sweat! You can still learn more and get hands on with a more intimate understanding of how Red Hat OpenShift and OpenShift Container Storage work together with a test drive.

Still want to learn more? Check out the Red Hat OpenShift Container Storage datasheet.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
By Annette Clewett and Luis Rico

The snapshot capability in Kubernetes is in tech preview at present and, as such, backup/recovery solution providers have not yet developed an end-to-end Kubernetes volume backup solution. Fortunately, GlusterFS, an underlying technology behind Red Hat OpenShift Container Storage (RHOCS), does have a mature snapshot capability. When combined with enterprise-grade backup and recovery software, a robust solution can be provided.

This blog post details how backup and restore can be done when using RHOCS via GlusterFS. As of the Red Hat OpenShift Container Platform (OCP) 3.11 release, there are a limited number of storage technologies (EBS, Google Cloud E pDisk, and hostPath) that support creating and restoring application data snapshots via Kubernetes snapshots. This Kubernetes snapshot feature is in tech preview, and the implementation is expected to change in concert with upcoming Container Storage Interface (CSI) changes. CSI, a universal storage interface (effectively an API) between container orchestrators and storage providers, is ultimately where backup and restore for OCS will be integrated in the future using volume snapshots capability.

Traditionally, backup and restore operations involve two different layers. One is the application layer. For example, databases like PostgreSQL have their own procedures to do an application consistent backup. The other is the storage layer. Most storage platforms provide a way for backup software like Commvault or Veritas NetBackup to integrate, obtain storage level snapshots, and perform backups and restores accordingly. An application layer backup is driven by application developers and is application specific. This study will focus on traditional storage layer backup and restore using Commvault Complete Backup and Recovery Software for this purpose. Other backup software tools can be used in a similar manner if they supply the same capabilities as used with Commvault.

RHOCS can be deployed in either converged mode or independent mode, and both are supported by the process described in this article. Converged mode, formerly known as Container Native Storage (CNS), means that Red Hat Gluster Storage is deployed in containers and uses the OCP host storage and networking. Independent mode, formerly known as Container Ready Storage (CRS), is deployed as a stand-alone Red Hat Gluster Storage cluster that provides persistent storage to OCP containers. Both modes of RHOCS deployment use heketi in a container on OCP for provisioning and managing GlusterFS volumes.

Storage-level backup and restore for RHOCS

If a backup is performed at the Persistent Volume (PV) level, then it will not capture the OCP Persistent Volume Claim (PVC) information. OCP PVC to PV mapping is required to identify which backups belong to which application. This leaves a gap: Which GlusterFS PV goes with which OCP PVC?

In a traditional environment, this is solved by naming physical volumes such that the administrator has a way of identifying which volumes belong to which application. This naming method can now be used in OCP (as of OCP 3.9) by using custom volume naming in the StorageClass resource. Before OCP 3.9, the names of the dynamically provisioned GlusterFS volumes were auto-generated with random vol_UUID naming. Now, by adding a custom volume name prefix in the StorageClass, the GlusterFS volume name will include the OCP namespace or project as well as the PVC name, thereby making it possible to map the volume to a particular workload.

OCS custom volume naming

Custom volume naming requires a change to the StorageClass definition. Any new RHOCS persistent volumes claimed using this StorageClass will be created with a custom volume name. The custom volume name will have prefix, project or namespace, PVC name and UUID (<myPrefix>_<namespace>_<claimname>_UUID).

The following glusterfs-storage StorageClass has custom volume naming enabled by adding the volumenameprefix parameter.

# oc get sc glusterfs-storage -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: glusterfs-storage
parameters:
  resturl: http://heketi-storage-app-storage.apps.ocpgluster.com
  restuser: admin
  secretName: heketi-storage-admin-secret
  secretNamespace: app-storage
  volumenameprefix: gf ❶
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete

❶ Custom volume name support: <volumenameprefixstring>_<namespace>_<claimname>_UUID

As an example, using this StorageClass for a namespace of mysql1 and PVC name of mysql the volume name would be gf_mysql1_mysql_043e08fc-f728-11e8-8cfd-028a65460540 (the UUID portion of the name will be unique for each volume name).

Note: If custom volume naming cannot be used, then it is important to collect information about all workloads using PVCs, their OCP PV associated, and the GlusterFS volume name (contained in Path variable in description of OCP PV).

RHOCS backup process

The goal of this blog post is to provide a generic method to back up and restore OCS persistent volumes used with OCP workloads. The example scripts and .ini files have no specific dependency on a particular backup and restore product. As such, they can be used with a product such as Commvault, where the scripts can be embedded in the backup configuration. Or, they can be used standalone, assuming that basic backup/recovery of the mounted gluster snapshots will be done via standard RHEL commands.

Note: The methods described here apply only for gluster-file volumes and currently will not work for gluster-block volumes.

For this approach, a “bastion host” is needed for executing the scripts, mounting GlusterFS snapshot volumes, and providing a place to install the agent if using backup and restore software. The bastion host should be a standalone RHEL7 machine separate from the OCP nodes and storage nodes in your deployment.

Requirements for the bastion host

The bastion host must have network connectivity to both the backup and restore server (if used), as well as the OCP nodes with the gluster pods (RHOCS converged mode) or the storage nodes (RHOCS independent mode). The following must be installed or downloaded to the bastion host:

  • backup and restore agent, if used
  • heketi-client package
  • glusterfs-fuse client package
  • atomic-openshift-clients package
  • rhocs-backup scripts and .ini files
RHOCS backup scripts

The github repository rhocs-backup contains unsupported example code that can be used with backup and restore software products. The two scripts, rhocs-pre-backup.sh and rhocs-post-backup.sh, have been tested with Commvault Complete Backup and Recovery Software. The rhocs-pre-backup.sh script will do the following:

  • Find all gluster-file volumes using heketi-client
  • Create a gluster snapshot for each volume
  • Mount the the snapshot volumes on the bastion host that has the backup agent installed
  • Protect the heketi configuration database by creating a json file for the database in the backup directory where all gluster snapshots are going to be mounted

Once the mounted snapshot volumes have been backed up, the rhocs-post-backup.sh script will do the following:

  • Unmount the snapshot volumes
  • Delete the gluster snapshot volumes

The two .ini files, independent_vars.ini and converged_vars.ini, are used to specify parameters specific to your RHOCS mode of deployment. Following are example parameters for converged_var.ini.

## Environment variables for RHOCS Backup:
## Deployment mode for RHOCS cluster: converged (CNS) or independent (CRS)
export RHOCSMODE="converged"

## Authentication variables for accessing OpenShift cluster or
## Gluster nodes depending on deployment mode
export OCADDRESS="https://master.refarch311.ocpgluster.com:443"
export OCUSER="openshift"
export OCPASS="redhat"
export OCPROJECT="app-storage" ## OpenShift project where gluster cluster lives

## Any of the Gluster servers from RHOCS converged cluster
## used for mounting gluster snapshots
export GLUSTERSERVER=172.16.31.173

## Directory for temporary files to put the list of
## Gluster volumes /snaps to backup
export VOLDIR=/root
export SNAPDIR=/root

## Destination directory for mounting snapshots of Gluster volumes:
export PARENTDIR=/mnt/source

## Heketi Route and Credentials
export USERHEKETI=admin ## User with admin permissions to dialog with Heketi
export SECRETHEKETI="xzAqO62qTPlacNjk3oIX53n2+Z0Z6R1Gfr0wC+z+sGk=" ## Heketi user key
export HEKETI_CLI_SERVER=http://heketi-storage-app-storage.apps.refarch311.ocpgluster.com ##
Route where Heketi pod is listening

## Provides Logging of this script in the dir specified below:
export LOGDIR="/root"

The pre-backup script, when executed, uses the heketi-client for the list of current gluster-file volumes. Because of this, for the script to work properly the heketi container must be online and reachable from bastion host. Additionally, for the scripts to work properly, all GlusterFS nodes or peers of RHOCS cluster must be online, as GlusterFS snapshot operation requires all bricks of a GlusterFS volume be available.

Manual execution of pre- and post-backup scripts

This section assumes that the bastion host has been created and has the necessary packages, scripts, and .ini files are installed on this machine. Currently, the pre- and post-backup scripts run as the root user. Because of this, backing up volumes for RHOCS independent mode will require that the bastion host can SSH as the root user with passwordless access to one of the GlusterFS storage nodes. This access should be verified before attempting to run the following scripts.

The scripts can be manually executed in the following manner for RHOCS converged mode:

sudo ./rhocs-pre-backup.sh /<path_to_file>/converged_vars.ini

Followed by this script to unmount the snapshot volumes and to remove the snapshot volumes from the RHOCS Heketi database and GlusterFS converged cluster:

sudo ./rhocs-post-backup.sh /<path_to_file>/converged_vars.ini

A variation of these scripts for RHOCS independent mode can be manually executed in the following manner:

sudo ./rhocs-pre-backup.sh /<path_to_file>/independent_vars.ini

Followed by this script to unmount the snapshot volumes and to remove the snapshot volumes from the RHOCS Heketi database and GlusterFS independent cluster:

sudo ./rhocs-post-backup.sh /<path_to_file>/independent_vars.ini

For each execution of the pre- or post-backup script a log file will be generated and placed in the directory specified in the .ini file (default is /root).

Note: Pre-backup scripts can be modified as needed for specific scenarios to achieve application-level consistency, like quiescing a database before taking a backup. Also, if special features are used with RHOCS, like SSL encryption or geo-replication, scripts will have to be customized and adjusted to be compatible with those features.

Commvault backup process

Note that this blog post does not cover the tasks to install and configure Commvault to back up and restore data. In addition to having the Commvault Console and Agent in working order, this section also assumes that the bastion host has been created and has the necessary packages, scripts, and .ini files installed.

The use of these scripts to back up OCP PVs is compatible with any backup frequency or retention configured in the Commvault backup policy. But, as we are mounting gluster snapshots in newly created folders with date and time information, the backup application will always consider contents as new, so even if backup policy is incremental, it will effectively do a full backup. Also, the backup content will consist of dozens or even hundreds of very small filesystems (1-10 GB), that could run faster under a “always do full backup” strategy.

Detailed process for backup using Commvault

Once the scripts and .ini files are on the bastion host and a Commvault Agent is installed, a backup can be done using the Commvault Commcell Console or the Commvault Admin Console. The following views show how to do the backup using the Commcell Console and validate the backup using the Admin Console.

A Subclient must be created before a backup can be done and a unique name must be specified.

When creating a Subclient, you must input where on the bastion host you want the backup to be done from on the bastion host with Commvault Agent (e.g., /mnt/source).

Choose what schedule you want the backup done on (or Do Not Schedule; start backup manually using Console instead).

And last or the Subclient configuration, add the path to the pre- and post-backup scripts, as well as the path to the appropriate .ini file, converged_vars.ini or independent_vars.ini. Once this is done and the subclient has been saved, you are ready to take a backup of the gluster-file snapshot volumes.

The backup can then be done by selecting the desired subclient and issuing an immediate backup or letting the selected schedule do the backups when configured (e.g., daily).

For an immediate backup, you can choose full or incremental. As already stated, a full backup will be done every time, because the pre-backup script always creates a new directory to mount the gluster-file snapshot volume.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
Red Hat Storage by Colleen Corrice - 6M ago

Our top 3 storage-for-containers things to look forward to at KubeCon By Steve Bohac, OpenShift Storage Product Marketing

Season greetings!

As always, much going on with Red Hat OpenShift Container Storage!

Of course, Kubernetes 1.13 was released this week, Container Journal recently published an article I authored, and KubeCon Seattle is coming up next week… By the way, did you see the latest Forrester Wave Enterprise Container Platform Software Suites where Red Hat OpenShift was named a Leader? Good stuff!

Red Hat OpenShift Container Storage helps organizations standardize storage across multiple environments and easily integrates with Red Hat OpenShift to deliver a persistent storage layer for containerized applications that require long-term, stateful storage. Enterprises can benefit from a simple, integrated solution including the container platform, registry, application development environment, and storage—all in one, supported by a single vendor.

December is always a busy month with industry conferences (not to mention holiday planning!), so as I finalized my own KubeCon plans, I wanted to pause and take a quick breath and outline my top 3 things I’m looking forward to at KubeCon Seattle 2018 next week:

  1. Assorted Kubernetes announcements (whatever they are!). Yes, who knows what kind of interesting things will be announced next week… but they’ll likely be exciting! The Kubernetes ecosystem has gotten so large now, there is always a plethora of interesting products and technologies announced at KubeCon. It’s always interesting to see how these new announcements dictate where things are going with Kubernetes and cloud native technologies in general. (By the way, for a great overview of the “third era” of Kubernetes, check out PodCTL #54 with our own Brian Gracely and Tyler Britten.)
  2. For the first time ever, there will be a Cloud Native Storage Day as one of the co-located events at KubeCon. Like the other co-located events, it takes place next Monday before the KubeCon show officially kicks off. The day’s agenda includes customers and industry leaders like Red Hat (I’ll be there with a few colleagues presenting) discussing current implementations and future directions of container storage. This should be very educational and interactive for everyone! And…. the sessions will be recorded (look back here for a post-KubeCon blog after the show for links to the recordings!).
  3. Catching up on the status of the Rook project. What is Rook? Rook is a persistent storage orchestrator that is designed to run as a native Kubernetes service. Consider it the glue between storage and the container—the thing that makes automation work. This is an interesting development around storage for containers, and I’m looking forward to meeting up with colleagues and “fellow travelers” to understand more.

Anyway, it should be a good one at KubeCon next week (did I mention it is sold out!?). In between sessions, make sure to visit us in Booth D1 in the Expo Hall for product demonstrations, to speak with Red Hat OpenShift Container Storage experts and other community leaders about upstream projects, and to snag some of our giveaways (while supplies last!).

We hope to see you there! If we don’t catch you in person, we’ll be tweeting (and re-tweeting) all week! If you don’t already, make sure to follow us on Twitter at @RedHatStorage.

Not attending KubeCon? No sweat! You can still learn more and get hands on with a more intimate understanding of how Red Hat OpenShift and OpenShift Container Storage work together with a test drive.

Still want to learn more? Check out the Red Hat OpenShift Container Storage datasheet.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Customers can realize more value and greater simplicity with cost-effective, open source, integrated compute and storage delivered in a compact footprint By Daniel Gilfix, Red Hat Cloud Storage and Hyperconverged Infrastructure

Hyperconverged Infrastructure (HCI) emerged as an infrastructure category about a decade ago aimed at a few specific use cases and has been dominated by proprietary software vendors offering appliances built on their hardware, or rigid configurations delivered with OEM hardware partners.

What’s new?

Today we announced the next iteration of our enterprise-grade, open source approach in this space—Red Hat Hyperconverged Infrastructure for Virtualization 1.5, which benefits from the combined strength of Red Hat Enterprise Linux, Red Hat Virtualization, Red Hat Gluster Storage, and Red Hat Ansible Automation.

Where’s the beef?

Red Hat Hyperconverged Infrastructure for Virtualization (RHHI-V) is an optimized, hyperconverged infrastructure (HCI) that has helped organizations across industries like energy, retail, banking, telco, and the public sector make the most of business-critical applications that must be deployed with limited space, budget, and IT staff, including departmental and lines of business ops, remote sites, and development and test environments. Integration with Red Hat Ansible Automation helps reduce manual errors normally associated with downtime while enabling a more streamlined and speedy deployment. Simplified administration via a single user interface means you can consolidate your infrastructure and adopt a software-defined datacenter more efficiently. Such adoption includes using RHHI-V in lieu of a more expensive VMware “lock-in” environment or transitioning from it under professional guidance with the Red Hat infrastructure migration solution.

What’s inside?

Red Hat Hyperconverged Infrastructure for Virtualization 1.5 now features advanced data reduction capabilities for even greater efficiencies as well as a series of validated server configurations for optimized workloads to reduce or eliminate the guesswork out of infrastructure deployment. Details follow:

  • Data reduction via deduplication and compression. Made possible through embedded Virtual Data Optimizer (VDO) code in Red Hat Enterprise Linux, you can now efficiently eliminate duplicate instances of repeating data and compress the reduced data set. This results in improved storage utilization and enables more affordable high-performance storage options.
  • Virtual graphics processing unit (vGPU). With the vGPU capability, you can assign GPU slices to VMs to accelerate 3D graphics and to offload computationally heavy jobs, including applications in computational science, workloads in oil and gas and manufacturing, as well as emerging AI and machine learning applications processing.
  • Open Virtual Network support. Support for software-defined networking via Open Virtual Network (OVN) helps improve scalability while enabling live migration of virtual networking components in a hyperconverged Linux environment.
  • Deep Ansible integration. Red Hat Ansible Automation enables true “ops value” at deploy and runtime, thereby paving the way toward your broader automation goals. We also deliver Ansible playbooks to enable remote replication and recovery of RHHI-V environments.
  • Validated hardware configurations. To help ensure RHHI-V users deploy sound infrastructure configurations, Red Hat has tested a number of use cases with our hardware partners and documents configuration guidelines for optimized workloads. These configurations, along with our new RHHI-V sizing tool, can help you anticipate platform requirements based on their usage patterns, taking the guesswork out of deploying a software-defined HCI platform, and reducing time to value. You can choose among industry standard hardware and enjoy more predictable performance for their desired deployment patterns.
Who benefits?

While RHHI-V was initially targeted at remote office/branch office deployment, we’ve experienced steadily increasing demand to support more mission-critical applications, such as remote tactical operations for public sector, field analysis and oil rig operations in the energy sector, and managing data from a myriad of sensors in factories across both process and discrete manufacturing. Now integrated even more broadly across the Red Hat software stack, RHHI-V is a powerful, general purpose platform for anyone seeking to jumpstart edge computing or modernize their existing data center to accommodate new workloads with greater degrees of efficiency. 

How can you learn more?

For more information on Red Hat Hyperconverged Infrastructure for Virtualization, check out this article by Storage Switzerland. Feel free to also attend our upcoming webinar on December 11. You can always simply access us on the web.

Read Full Article

Read for later

Articles marked as Favorite are saved for later viewing.
close
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Separate tags by commas
To access this feature, please upgrade your account.
Start your free month
Free Preview