Ceph isn’t just another distributed file system; it’s a unified, distributed storage system designed to be incredibly resilient and scale to exabytes.
Let’s see it in action. Imagine we have three ceph-node machines and we want to set up a basic Ceph cluster. First, we’ll need to install the cephadm package on each of them.
# On each ceph-node:
sudo apt update
sudo apt install cephadm -y
Now, we bootstrap the cluster from one of the nodes. This initializes the first monitor and manager daemons.
# On one ceph-node (e.g., ceph-node-1):
sudo cephadm bootstrap --mon-ip 192.168.1.101
This command will output a ceph.conf file and a ceph.client.admin.keyring. We need to copy these to our local machine to interact with the cluster.
# On ceph-node-1, then copy to your admin machine:
sudo cephadm shell -- ceph config generate-minimal-conf > ceph.conf
sudo cephadm shell -- ceph auth get client.admin > ceph.client.admin.keyring
On your admin machine, you’ll need ceph-common installed to use ceph commands.
# On your admin machine:
sudo apt install ceph-common -y
Now, let’s add our other nodes. We’ll need to copy the ceph.conf and ceph.client.admin.keyring to each node, and then use cephadm to add them to the cluster.
# On your admin machine, for each other node (e.g., ceph-node-2, ceph-node-3):
sudo scp ceph.conf ceph.client.admin.keyring user@ceph-node-2:/etc/ceph/
sudo scp ceph.conf ceph.client.admin.keyring user@ceph-node-3:/etc/ceph/
# Then, on each node (e.g., ceph-node-2, ceph-node-3):
sudo cephadm add-host ceph-node-2 192.168.1.102
sudo cephadm add-host ceph-node-3 192.168.1.103
The core problem Ceph solves is providing a single, unified storage pool from potentially many disparate devices, managed as a single entity. It achieves this through a distributed metadata server (MDS) for file systems, object storage daemons (OSDs) that manage actual storage devices, and monitors (MONs) that maintain the cluster map. All communication and state management are distributed, meaning there’s no single point of failure.
The cluster map is the heart of Ceph’s distributed intelligence. It contains information about the location of all OSDs, MONs, and other daemons, as well as the placement groups (PGs) that map data to OSDs. When a client needs to access data, it queries a monitor for the relevant PG’s location and then directly interacts with the OSDs holding that data.
To create a distributed storage pool, we need to add OSDs. Let’s assume each node has an available disk, say /dev/sdb.
# On your admin machine, targeting ceph-node-1:
sudo cephadm shell -- ceph orch apply osd --data /dev/sdb --host ceph-node-1
# Repeat for ceph-node-2 and ceph-node-3
sudo cephadm shell -- ceph orch apply osd --data /dev/sdb --host ceph-node-2
sudo cephadm shell -- ceph orch apply osd --data /dev/sdb --host ceph-node-3
Once the OSDs are up and running, Ceph will automatically start creating and populating placement groups. These PGs are how Ceph shards and replicates data across your OSDs. You can check their status:
# On your admin machine:
sudo cephadm shell -- ceph pg stat
The most surprising true thing about Ceph’s distributed nature is how it handles data placement. Instead of a central index or a complex distributed hash table, Ceph uses a CRUSH (Controlled Replication Under Scalable Hashing) algorithm. This algorithm deterministically calculates where data should reside based on the cluster’s topology and desired redundancy, eliminating the need for a lookup service for data placement itself. This is crucial for scalability and performance.
To make this storage available as an object store, we’d create a pool.
# On your admin machine:
sudo cephadm shell -- ceph osd pool create my_replicated_pool 64 64 replicated
Here, my_replicated_pool is the name, 64 is the number of placement groups (PGs) for this pool, and replicated specifies the data distribution method. Ceph automatically creates the necessary PGs and ensures data is replicated according to the cluster’s default or pool-specific settings.
The next concept you’ll typically encounter is configuring CephFS, Ceph’s native distributed file system, which relies on the Ceph object store and its own metadata servers.