CephFS driver¶
The CephFS driver enables manila to export shared filesystems backed by Ceph’s File System (CephFS) using either the Ceph network protocol or NFS protocol. Guests require a native Ceph client or an NFS client in order to mount the filesystem.
When guests access CephFS using the native Ceph protocol, access is controlled via Ceph’s cephx authentication system. If a user requests share access for an ID, Ceph creates a corresponding Ceph auth ID and a secret key if they do not already exist, and authorizes the ID to access the share. The client can then mount the share using the ID and the secret key. To learn more about configuring Ceph clients to access the shares created using this driver, please see the Ceph documentation
And when guests access CephFS through NFS, an NFS-Ganesha server (or CephFS NFS service) mediates access to CephFS. The driver enables access control by managing the NFS-Ganesha server’s exports.
Supported Operations¶
The following operations are supported with CephFS backend:
Create, delete, update and list share
Allow/deny access to share
Only
cephx
access type is supported for CephFS native protocol.Only
ip
access type is supported for NFS protocol.read-only
andread-write
access levels are supported.
Extend/shrink share
Create, delete, update and list snapshot
Create, delete, update and list share groups
Delete and list share group snapshots
Important
Share group snapshot creation is no longer supported in mainline CephFS. This feature has been removed from manila W release.
Prerequisites¶
Important
A manila share backed by CephFS is only as good as the underlying filesystem. Take care when configuring your Ceph cluster, and consult the latest guidance on the use of CephFS in the Ceph documentation.
Ceph testing matrix¶
As Ceph and Manila continue to grow, it is essential to test and support combinations of releases supported by both projects. However, there is little community bandwidth to cover all of them. For simplicity sake, we are focused on testing (and therefore supporting) the current Ceph active releases. Check out the list of Ceph active releases here.
Below is the current state of testing for Ceph releases with this project. Adjacent components such as devstack-plugin-ceph and tripleo are added to the table below. Contributors to those projects can determine what versions of ceph are tested and supported with manila by those components; however, their state is presented here for ease of access.
OpenStack release |
Manila |
devstack-plugin-ceph |
---|---|---|
Wallaby |
Pacific |
Pacific |
Xena |
Pacific |
Quincy |
Yoga |
Quincy |
Quincy |
Zed |
Quincy |
Quincy |
2023.1 (“Antelope”) |
Quincy |
Quincy |
2023.2 (“Bobcat”) |
Quincy |
Reef |
2024.1 (“Caracal”) |
Reef |
Reef |
2024.2 (“Dalmation”) |
Reef |
Reef |
Additionally, it is expected that the version of the Ceph client available to manila is aligned with the Ceph server version. Mixing server and client versions is strongly unadvised.
In case of using the NFS Ganesha driver, it’s also a good practice to use the versions that align with the Ceph version of choice.
Common Prerequisites¶
A Ceph cluster with a filesystem configured (See Create ceph filesystem on how to create a filesystem.)
python3-rados
andpython3-ceph-argparse
packages installed in the servers running the manila-share service.Network connectivity between your Ceph cluster’s public network and the servers running the manila-share service.
Authorizing the driver to communicate with Ceph¶
Capabilities required for the Ceph manila identity have changed from the Wallaby release. The Ceph manila identity configured no longer needs any MDS capability. The MON and OSD capabilities can be reduced as well. However new MGR capabilities are now required. If not accorded, the driver cannot communicate to the Ceph Cluster.
Important
The driver in the Wallaby (or later) release requires a Ceph identity with a different set of Ceph capabilities when compared to the driver in a pre-Wallaby release.
When upgrading to Wallaby you’ll also have to update the capabilities of the Ceph identity used by the driver (refer to Ceph user capabilities docs) E.g. a native driver that already uses client.manila Ceph identity, issue command ceph auth caps client.manila mon ‘allow r’ mgr ‘allow rw’
If you are deploying the CephFS driver with Native CephFS or using an NFS service deployed with ceph management commands, the auth ID should be set as follows:
ceph auth get-or-create client.manila -o manila.keyring \
mgr 'allow rw' \
mon 'allow r'
If you’re deploying the CephFS NFS driver with a standalone NFS-Ganesha service, we use a specific pool to store exports (configurable with the config option “ganesha_rados_store_pool_name”). The client.manila ceph user requires permission to access this pool. So, the auth ID should be set as follows:
ceph auth get-or-create client.manila -o manila.keyring \
osd 'allow rw pool=<ganesha_rados_store_pool_name>" \
mgr 'allow rw' \
mon 'allow r'
manila.keyring
, along with your ceph.conf
file, will then need to be
placed on the server running the manila-share service.
Important
To communicate with the Ceph backend, a CephFS driver instance (represented as a backend driver section in manila.conf) requires its own Ceph auth ID that is not used by other CephFS driver instances running in the same controller node.
In the server running the manila-share service, you can place the
ceph.conf
and manila.keyring
files in the /etc/ceph directory. Set the
same owner for the manila-share process and the manila.keyring
file. Add the following section to the ceph.conf
file.
[client.manila]
client mount uid = 0
client mount gid = 0
log file = /opt/stack/logs/ceph-client.manila.log
admin socket = /opt/stack/status/stack/ceph-$name.$pid.asok
keyring = /etc/ceph/manila.keyring
It is advisable to modify the Ceph client’s admin socket file and log file locations so that they are co-located with manila services’s pid files and log files respectively.
Enabling snapshot support in Ceph backend¶
From Ceph Nautilus, all new filesystems created on Ceph have snapshots enabled by default. If you’ve upgraded your ceph cluster and want to enable snapshots on a pre-existing filesystem, you can do so:
ceph fs set {fs_name} allow_new_snaps true
Configuring CephFS backend in manila.conf¶
Space considerations¶
The CephFS driver reports total and free capacity available across the Ceph
cluster to manila to allow provisioning. All CephFS shares are thinly
provisioned, i.e., empty shares do not consume any significant space
on the cluster. The CephFS driver does not allow controlling oversubscription
via manila. So, as long as there is free space, provisioning will continue,
and eventually this may cause your Ceph cluster to be over provisioned and
you may run out of space if shares are being filled to capacity. It is advised
that you use Ceph’s monitoring tools to monitor space usage and add more
storage when required in order to honor space requirements for provisioned
manila shares. You may use the driver configuration option
reserved_share_percentage
to prevent manila from filling up your Ceph
cluster, and allow existing shares to grow.
Known restrictions¶
A CephFS driver instance, represented as a backend driver section in manila.conf, requires a Ceph auth ID unique to the backend Ceph Filesystem. Using a non-unique Ceph auth ID will result in the driver unintentionally evicting other CephFS clients using the same Ceph auth ID to connect to the backend.
Snapshots are read-only. A user can read a snapshot’s contents from the
.snap/{manila-snapshot-id}_{unknown-id}
folder within the mounted share.
Security¶
Each share’s data is mapped to a distinct Ceph RADOS namespace. A guest is restricted to access only that particular RADOS namespace. https://docs.ceph.com/en/latest/cephfs/file-layouts/
Configuration Reference¶
Configuration option = Default value |
Description |
---|---|
[DEFAULT] |
|
|
(String) The name of the ceph auth identity to use. |
|
(String) The name of the cluster in use, if it is not the default (‘ceph’). |
|
(String) Fully qualified path to the ceph.conf file. |
|
(String) The type of protocol helper to use. Default is CEPHFS. |
|
(Boolean) Whether the NFS-Ganesha server is remote to the driver. |
|
(String) The IP address of the NFS-Ganesha server. |
|
(String) The type of protocol helper to use. Default is CEPHFS. |
|
(String) The username to authenticate as in the remote NFS-Ganesha server host. |
|
(String) The path of the driver host’s private SSH key file. |
|
(String) The password to authenticate as the user in the remote Ganesha server host. This is not required if ‘cephfs_ganesha_path_to_private_key’ is configured. |
|
(String) List of IPs to export shares. If not supplied, then the value of ‘cephfs_ganesha_server_ip’ will be used to construct share export locations. |
|
(String) The read/write/execute permissions mode for CephFS volumes, snapshots, and snapshot groups expressed in Octal as with linux ‘chmod’ or ‘umask’ commands. |
|
(String) The name of the filesystem to use, if there are multiple filesystems in the cluster.” |
|
(String) Provide a unique string value to make the driver ensure all of the shares it has created during startup. Ensuring would re-export shares and this action isn’t always required, unless something has been administratively modified on CephFS. |
|
(String) The ID of the NFS cluster to use. |