Ceph Storage (Rook)

Overview

The Ceph Storage plugin deploys a Rook-managed Ceph cluster on your Kubernetes nodes and turns their spare raw disks into block, shared file and object storage. Once installed, create a DiskPool to contribute disks, then a BlockStorage or FileStorage to provision a StorageClass that any PersistentVolumeClaim can reference, or an ObjectStorage to provision S3-compatible buckets through ObjectBucketClaims.

Installed versions

• Rook v1.21.1
• Ceph v19.2.7
• Ceph CSI v3.18.2

Key Features

• Automatic disk discovery: nodes are scanned for raw block devices, which appear in the console as Disks
• Declarative disk pools: a DiskPool selects which discovered disks to contribute to the shared cluster
• Block, file and object storage classes: a BlockStorage produces a ready-to-use ReadWriteOnce StorageClass; a FileStorage produces a ReadWriteMany StorageClass backed by a shared filesystem with active and standby metadata servers; an ObjectStorage produces an S3-compatible object store (Ceph RGW) whose StorageClass provisions buckets for ObjectBucketClaims
• Tunable replication: set 1, 2 or 3 replicas per storage object, or leave it unset to derive the count from the number of nodes contributing disks

Use Cases

• Persistent block volumes for stateful workloads (databases, message queues)
• Shared volumes many pods can mount at once (shared caches, content repositories)
• S3-compatible buckets for in-cluster workloads (artifacts, backups, media)
• ReadWriteOnce and ReadWriteMany volumes on bare-metal or VM clusters
• Cost-effective in-cluster storage using spare raw disks

Requirements

• Raw, unpartitioned disks on the cluster nodes. Disks that already hold a filesystem or partition table are not offered.
• Cluster-admin-equivalent permissions. The Rook operator runs privileged, host-networked device-discovery DaemonSets, installs the Ceph CSI driver, and manages its own cluster-wide RBAC, so this plugin requests wildcard permissions and runs as cluster-admin on the cluster you install it into.

Limitations

• The S3 endpoint is reachable in-cluster only; exposing it externally (and standalone S3 users) is not offered yet.
• All disk pools feed one shared Ceph cluster, and data is placed across every disk in it. Additional BlockStorage/FileStorage/ObjectStorage objects give you additional StorageClasses over the same disks.