Select Page

Ceph: Increasing placement groups in production

by | Oct 25, 2017 | Ceph, Storage und Backup

Ceph LogoIf you are running a Ceph cluster for several years you will come to the point where the number of placement groups (PGs) no longer fits to the size of your cluster. Maybe you increased the number of OSDs several times, maybe you misjudged the growth of a pool years ago.
Too few PGs per OSD can lead to drawbacks regarding performance, recovery time and uneven data distribution. The latter can be examined with the command ceph osd df. It shows the usage, weight, variance and number of PGs for each OSD as well the min/max variance and the standard deviation of your OSD usage.
Increasing the number of the PGs can lead to a more even data distribution,
but there are several warnings in blogs or mailings lists, especially for production environments. Doubling the PGs (e.g. from 1024 to 2048) may bring down your cluster for some minutes, because creating, activating and peering the new PGs may strongly influence your client’s traffic.
Regarding this warnings, we decided to increase the PGs in slices of 128. Between each slice we waited until all PGs peered successfully, which needed only a few seconds and did not influence the client traffic at all.
Increasing the number of PGs is done with two simple commands:

$ ceph osd set <pool> pg_num <int>
$ ceph osd set <pool> pgp_num <int>

Increasing pg_num creates new PGs, but the data rebalancing and backfilling will only start after increasing pgp_num (the number of placement groups for placement), too. pg_num and pgp_num should always have the same value. Increasing your PGs usually comes with a huge amount of backfilling which should not be a problem for a well configured cluster.

Achim Ledermüller
Achim Ledermüller
Lead Senior Systems Engineer

Der Exil Regensburger kam 2012 zu NETWAYS, nachdem er dort sein Wirtschaftsinformatik Studium beendet hatte. In der Managed Services Abteilung ist unter anderem für die Automatisierung des RZ-Betriebs und der Evaluierung und Einführung neuer Technologien zuständig.
More posts on the topicCeph | Storage und Backup

Ceph OSDs mit BlueStore erstellen

BlueStore ist das neue Speicher-Backend für Ceph ab Luminous. v12.2.x. Es wird standardmäßig verwendet, wenn neue OSDs durch ceph-disk, ceph-deploy oder ceph-volume erzeugt werden. Es bietet einen großen Vorteil in Bezug auf Leistung, Robustheit und Funktionalität...

NWS OpenStack – automatisierte Snapshots

Unser Ziel war und ist es, eine Plattform zu schaffen, die sehr hohe Flexibilität, Sicherheit und Komfort bietet. Unsere Kunden sollen in eigenen, isolierten Projekten ihrer Kreativität freien Lauf lassen und nahezu uneingeschränkt sein, ohne sich um essenzielle Dinge...

Ceph Mimic | Using loop devices as OSD

For quite some time we have been using ceph-deploy to deploy OSD in folders during the Ceph trainings held by Netways. This worked perfectly with jewel, but newer versions don't allow this behaviour anymore. There are several reasons for this, however, as we have a...


Sep 21

Icinga 2 Fundamentals Training | Nürnberg

September 21 @ 09:00 - September 24 @ 17:00
NETWAYS Headquarter | Nürnberg
Sep 21

Ansible Advanced Training | Online

September 21 @ 09:00 - September 22 @ 17:00
Sep 28

GitLab Fundamentals Training | Online

September 28 @ 09:00 - September 29 @ 17:00
Sep 28

Kubernetes Quick Start | Nürnberg

September 28 @ 09:00 - 17:00
NETWAYS Headquarter | Nürnberg
Oct 05

Linux Basics Training | Nürnberg

October 5 @ 09:00 - October 7 @ 17:00
NETWAYS Headquarter | Nürnberg