Skip to content

Commit

Permalink
docs: clarify docs for wording or obsolete info
Browse files Browse the repository at this point in the history
Update the getting started guide, prereqs, architecture,
and other basic docs with more concise wording and removed
some obsolete information. Some links are removed so they
don't distract the user from reading the critical information
that we are conveying in the rook docs.

Signed-off-by: travisn <[email protected]>
  • Loading branch information
travisn committed Apr 27, 2023
1 parent 844712a commit ecdeb7a
Show file tree
Hide file tree
Showing 8 changed files with 99 additions and 86 deletions.
42 changes: 21 additions & 21 deletions Documentation/Getting-Started/Prerequisites/prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,22 @@ and Rook is granted the required privileges (see below for more information).

## Minimum Version

Kubernetes **v1.21** or higher is supported for the Ceph operator.
Kubernetes **v1.21** or higher is supported.

## CPU Architecture

Architectures supported are `amd64 / x86_64` and `arm64`.

## Ceph Prerequisites

In order to configure the Ceph storage cluster, at least one of these local storage types is required:
To configure the Ceph storage cluster, at least one of these local storage types is required:

* Raw devices (no partitions or formatted filesystems)
* Raw partitions (no formatted filesystem)
* LVM Logical Volumes (no formatted filesystem)
* Persistent Volumes available from a storage class in `block` mode

You can confirm whether your partitions or devices are formatted with filesystems with the following command:
Confirm whether the partitions or devices are formatted with filesystems with the following command:

```console
$ lsblk -f
Expand All @@ -34,7 +34,7 @@ vda
vdb
```

If the `FSTYPE` field is not empty, there is a filesystem on top of the corresponding device. In this example, you can use `vdb` for Ceph and can't use `vda` or its partitions.
If the `FSTYPE` field is not empty, there is a filesystem on top of the corresponding device. In this example, `vdb` is available to Rook, while `vda` and its partitions have a filesystem and are not available.

## Admission Controller

Expand All @@ -43,23 +43,23 @@ Enabling the Rook admission controller is recommended to provide an additional l
To deploy the Rook admission controllers, install the cert manager before Rook is installed:

```console
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.11.1/cert-manager.yaml
```

## LVM package

Ceph OSDs have a dependency on LVM in the following scenarios:

* OSDs are created on raw devices or partitions
* If encryption is enabled (`encryptedDevice: "true"` in the cluster CR)
* A `metadata` device is specified

LVM is not required for OSDs in these scenarios:

* Creating OSDs on PVCs using the `storageClassDeviceSets`
* OSDs are created on raw devices or partitions
* OSDs are created on PVCs using the `storageClassDeviceSets`

If LVM is required for your scenario, LVM needs to be available on the hosts where OSDs will be running.
Some Linux distributions do not ship with the `lvm2` package. This package is required on all storage nodes in your k8s cluster to run Ceph OSDs.
If LVM is required, LVM needs to be available on the hosts where OSDs will be running.
Some Linux distributions do not ship with the `lvm2` package. This package is required on all storage nodes in the k8s cluster to run Ceph OSDs.
Without this package even though Rook will be able to successfully create the Ceph OSDs, when a node is rebooted the OSD pods
running on the restarted node will **fail to start**. Please install LVM using your Linux distribution's package manager. For example:

Expand Down Expand Up @@ -93,23 +93,23 @@ Ceph requires a Linux kernel built with the RBD module. Many Linux distributions
have this module, but not all.
For example, the GKE Container-Optimised OS (COS) does not have RBD.
You can test your Kubernetes nodes by running `modprobe rbd`.
If it says 'not found', you may have to rebuild your kernel and include at least
the `rbd` module, install a newer kernel, or choose a different Linux distribution.
Test your Kubernetes nodes by running `modprobe rbd`.
If the rbd module is 'not found', rebuild the kernel to include the `rbd` module,
install a newer kernel, or choose a different Linux distribution.

Rook's default RBD configuration specifies only the `layering` feature, for
broad compatibility with older kernels. If your Kubernetes nodes run a 5.4
or later kernel you may wish to enable additional feature flags. The `fast-diff`
and `object-map` features are especially useful.
or later kernel, additional feature flags can be enabled in the
storage class. The `fast-diff` and `object-map` features are especially useful.

```yaml
imageFeatures: layering,fast-diff,object-map,deep-flatten,exclusive-lock
```

### CephFS

If you will be creating volumes from a Ceph shared file system (CephFS), the recommended minimum kernel version is **4.17**.
If you have a kernel version less than 4.17, the requested PVC sizes will not be enforced. Storage quotas will only be
If creating RWX volumes from a Ceph shared file system (CephFS), the recommended minimum kernel version is **4.17**.
If the kernel version is less than 4.17, the requested PVC sizes will not be enforced. Storage quotas will only be
enforced on newer kernels.

## Distro Notes
Expand All @@ -118,21 +118,21 @@ Specific configurations for some distributions.

### NixOS

When you use NixOS, the kernel modules will be found in the non-standard path `/run/current-system/kernel-modules/lib/modules/`,
For NixOS, the kernel modules will be found in the non-standard path `/run/current-system/kernel-modules/lib/modules/`,
and they'll be symlinked inside the also non-standard path `/nix`.

For Rook Ceph containers to be able to load the required modules, they need read access to those locations.
Rook containers require read access to those locations to be able to load the required modules.
They have to be bind-mounted as volumes in the CephFS and RBD plugin pods.

If you install Rook with Helm, uncomment these example settings in `values.yaml`:
If installing Rook with Helm, uncomment these example settings in `values.yaml`:

* `csi.csiCephFSPluginVolume`
* `csi.csiCephFSPluginVolumeMount`
* `csi.csiRBDPluginVolume`
* `csi.csiRBDPluginVolumeMount`

If you deploy without Helm, add those same values to the corresponding environment variables in the operator pod,
or the corresponding keys in its `ConfigMap`:
If deploying without Helm, add those same values to the settings in the `rook-ceph-operator-config`
ConfigMap found in operator.yaml:

* `CSI_CEPHFS_PLUGIN_VOLUME`
* `CSI_CEPHFS_PLUGIN_VOLUME_MOUNT`
Expand Down
32 changes: 16 additions & 16 deletions Documentation/Getting-Started/example-configurations.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Example Configurations
---

Configuration for Rook and Ceph can be configured in multiple ways to provide block devices, shared filesystem volumes or object storage in a kubernetes namespace. We have provided several examples to simplify storage setup, but remember there are many tunables and you will need to decide what settings work for your use case and environment.
Configuration for Rook and Ceph can be configured in multiple ways to provide block devices, shared filesystem volumes or object storage in a kubernetes namespace. While several examples are provided to simplify storage setup, settings are available to optimize various production environments.

See the **[example yaml files](https://github.com/rook/rook/blob/master/deploy/examples)** folder for all the rook/ceph setup example spec files.

Expand All @@ -16,7 +16,7 @@ The [crds.yaml](https://github.com/rook/rook/blob/master/deploy/examples/crds.ya
kubectl create -f crds.yaml -f common.yaml
```

The examples all assume the operator and all Ceph daemons will be started in the same namespace. If you want to deploy the operator in a separate namespace, see the comments throughout `common.yaml`.
The examples all assume the operator and all Ceph daemons will be started in the same namespace. If deploying the operator in a separate namespace, see the comments throughout `common.yaml`.

## Operator

Expand All @@ -31,13 +31,13 @@ Settings for the operator are configured through environment variables on the op

## Cluster CRD

Now that your operator is running, let's create your Ceph storage cluster. This CR contains the most critical settings
Now that the operator is running, create the Ceph storage cluster with the CephCluster CR. This CR contains the most critical settings
that will influence how the operator configures the storage. It is important to understand the various ways to configure
the cluster. These examples represent a very small set of the different ways to configure the storage.
the cluster. These examples represent several different ways to configure the storage.

* [`cluster.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/cluster.yaml): This file contains common settings for a production storage cluster. Requires at least three worker nodes.
* [`cluster.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/cluster.yaml): Common settings for a production storage cluster. Requires at least three worker nodes.
* [`cluster-test.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/cluster-test.yaml): Settings for a test cluster where redundancy is not configured. Requires only a single node.
* [`cluster-on-pvc.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/cluster-on-pvc.yaml): This file contains common settings for backing the Ceph Mons and OSDs by PVs. Useful when running in cloud environments or where local PVs have been created for Ceph to consume.
* [`cluster-on-pvc.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/cluster-on-pvc.yaml): Common settings for backing the Ceph Mons and OSDs by PVs. Useful when running in cloud environments or where local PVs have been created for Ceph to consume.
* [`cluster-external.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/cluster-external.yaml): Connect to an [external Ceph cluster](../CRDs/Cluster/ceph-cluster-crd.md#external-cluster) with minimal access to monitor the health of the cluster and connect to the storage.
* [`cluster-external-management.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/cluster-external-management.yaml): Connect to an [external Ceph cluster](../CRDs/Cluster/ceph-cluster-crd.md#external-cluster) with the admin key of the external cluster to enable
remote creation of pools and configure services such as an [Object Store](../Storage-Configuration/Object-Storage-RGW/object-storage.md) or a [Shared Filesystem](../Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md).
Expand All @@ -47,27 +47,27 @@ See the [Cluster CRD](../CRDs/Cluster/ceph-cluster-crd.md) topic for more detail

## Setting up consumable storage

Now we are ready to setup [block](https://ceph.com/ceph-storage/block-storage/), [shared filesystem](https://ceph.com/ceph-storage/file-system/) or [object storage](https://ceph.com/ceph-storage/object-storage/) in the Rook Ceph cluster. These kinds of storage are respectively referred to as CephBlockPool, CephFilesystem and CephObjectStore in the spec files.
Now we are ready to setup Block, Shared Filesystem or Object storage in the Rook cluster. These storage types are respectively created with the CephBlockPool, CephFilesystem and CephObjectStore CRs.

### Block Devices

Ceph can provide raw block device volumes to pods. Each example below sets up a storage class which can then be used to provision a block device in kubernetes pods. The storage class is defined with [a pool](http://docs.ceph.com/docs/master/rados/operations/pools/) which defines the level of data redundancy in Ceph:
Ceph provides raw block device volumes to pods. Each example below sets up a storage class which can then be used to provision a block device in application pods. The storage class is defined with a Ceph pool which defines the level of data redundancy in Ceph:

* [`storageclass.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass.yaml): This example illustrates replication of 3 for production scenarios and requires at least three worker nodes. Your data is replicated on three different kubernetes worker nodes and intermittent or long-lasting single node failures will not result in data unavailability or loss.
* [`storageclass-ec.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml): Configures erasure coding for data durability rather than replication. [Ceph's erasure coding](http://docs.ceph.com/docs/master/rados/operations/erasure-code/) is more efficient than replication so you can get high reliability without the 3x replication cost of the preceding example (but at the cost of higher computational encoding and decoding costs on the worker nodes). Erasure coding requires at least three worker nodes. See the [Erasure coding](../CRDs/Block-Storage/ceph-block-pool-crd.md#erasure-coded) documentation for more details.
* [`storageclass-test.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-test.yaml): Replication of 1 for test scenarios and it requires only a single node. Do not use this for applications that store valuable data or have high-availability storage requirements, since a single node failure can result in data loss.
* [`storageclass.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass.yaml): This example illustrates replication of 3 for production scenarios and requires at least three worker nodes. Data is replicated on three different kubernetes worker nodes. Intermittent or long-lasting single node failures will not result in data unavailability or loss.
* [`storageclass-ec.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml): Configures erasure coding for data durability rather than replication. Ceph's erasure coding is more efficient than replication so you can get high reliability without the 3x replication cost of the preceding example (but at the cost of higher computational encoding and decoding costs on the worker nodes). Erasure coding requires at least three worker nodes. See the [Erasure coding](../CRDs/Block-Storage/ceph-block-pool-crd.md#erasure-coded) documentation.
* [`storageclass-test.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-test.yaml): Replication of 1 for test scenarios. Requires only a single node. Do not use this for production applications. A single node failure can result in full data loss.

The storage classes are found in different sub-directories depending on the driver:
The block storage classes are found in the examples directory:

* `csi/rbd`: The CSI driver for block devices. This is the preferred driver going forward.
* `csi/rbd`: the CSI driver examples for block devices

See the [Ceph Pool CRD](../CRDs/Block-Storage/ceph-block-pool-crd.md) topic for more details on the settings.
See the [CephBlockPool CRD](../CRDs/Block-Storage/ceph-block-pool-crd.md) topic for more block storage settings.

### Shared Filesystem

Ceph filesystem (CephFS) allows the user to 'mount' a shared posix-compliant folder into one or more hosts (pods in the container world). This storage is similar to NFS shared storage or CIFS shared folders, as explained [here](https://ceph.com/ceph-storage/file-system/).
Ceph filesystem (CephFS) allows the user to mount a shared posix-compliant folder into one or more application pods. This storage is similar to NFS shared storage or CIFS shared folders, as explained [here](https://ceph.com/ceph-storage/file-system/).

File storage contains multiple pools that can be configured for different scenarios:
Shared Filesystem storage contains configurable pools for different scenarios:

* [`filesystem.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/filesystem.yaml): Replication of 3 for production scenarios. Requires at least three worker nodes.
* [`filesystem-ec.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/filesystem-ec.yaml): Erasure coding for production scenarios. Requires at least three worker nodes.
Expand Down
Loading

0 comments on commit ecdeb7a

Please sign in to comment.