NETWAYS Blog

For quite some time we have been using ceph-deploy to deploy OSD in folders during the Ceph trainings held by Netways. This worked perfectly with jewel, but newer versions don’t allow this behaviour anymore.
There are several reasons for this, however, as we have a quite regulated setup for our training notebooks, we had to come up with some kind of workaround. This approach is, while working fine in our training setup, not recommended for production!
The following steps apply to a current CentOS 7 system.
As stated before, we will deploy an OSD on a block device. Though you could use a separate partition for this, we will use a loop device. For this, the first step is to create a file:
For this, create an OSD directory
 
[bash]$ mkdir -p /home/training/my-cluster/osd-$HOSTNAME
$ cd /home/training/my-cluster/osd-$HOSTNAME/[/bash]
in this folder, create a file for later use
[bash]
$ fallocate -l 30G 30GB.img[/bash]
test it
[bash]
# losetup -l -P /dev/loop1 "/home/training/my-cluster/osd-$HOSTNAME/30GB.img"
# wipefs -a /dev/loop1
# lsblk
[/bash]
This should then display your new loopdevice.
As loop devices are not reboot safe, you need to go some steps further. If you like to use rc.local for this, you’re free to do so.
We’re going to create a service, which will essentially execute the prior mentioned losetup command. For this, we need a script with the command and a .service file, which will execute the script:
[bash]
rebloop.sh
#!/bin/bash
sudo losetup -l -P /dev/loop1 "/home/training/my-cluster/osd-$HOSTNAME/30GB.img"
[/bash]
and the service file:
[bash]
rebloop.service
[Unit]
Description=Reattach loop device after reboot
[Service]
Type=simple
ExecStart=/bin/bash /usr/bin/rebloop.sh
[Install]
WantedBy=multi-user.target
[/bash]
These files have to be executable and be copied to the correct folders. Afterwards, the service must be enabled and can be started.
[bash]
# chmod +x rebloop.*
# cp rebloop.sh /usr/bin/rebloop.sh
# cp rebloop.service /etc/systemd/system
# systemctl enable rebloop.service
# systemctl start rebloop.service
[/bash]
Ceph, however, will still not want to create an OSD on this device, instead give you following error message:
[bash]
–> RuntimeError: Cannot use device (/dev/mapper/<name>). A vg/lv path or an existing device is needed [/bash]
You have to make changes to /usr/lib/python2.7/site-packages/ceph_volume/util/disk.py on the OSD host:
in line 201, add „or TYPE==’loop'“:
[python]# use lsblk first, fall back to using stat
TYPE = lsblk(dev).get(‚TYPE‘)
if TYPE:
return TYPE == ‚disk‘ or TYPE == ‚loop'[/python]
and in line 286, change the „skip_loop“ switch from „True“ to „False“:
[python] def get_block_devs(sys_block_path="/sys/block", skip_loop=False): [/python]
For testing purposes, simply reboot your system and verify if the loop device gets reattached correctly.
If yes, you can deploy an OSD. We’re using ceph-deploy here:
[bash]$ ceph-deploy osd create –data /dev/loop1 $HOSTNAME[/bash]
When the command was successfully executed on your hosts, you can create your first pool
[bash]# ceph osd pool create rbd 100 100 replicated[/bash]
examine ceph status
[bash]# ceph status[/bash]
tag the pool with an application
[bash]# ceph osd pool application enable rbd rbd[/bash]
As you can see, there are quite some changes to be made, and each of it is failure-prone.
Best practice is to simply use a real block device and for production you really should stick to this.
If, however, there are certain needs to be fulfilled, ceph can be convinced to comply.
 
Sources:
http://tracker.ceph.com/issues/36603
https://tracker.ceph.com/issues/23337
https://github.com/NETWAYS/ceph-training
 
 
 

Tim Albert

Senior Systems Engineer

---

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert. Seit Anfang 2016 ist er bei uns tätig. Zuerst im Managed Services Team, dort kümmerte Tim sich um Infrastrukturthemen und den internen Support, um dann 2019 - zusammen mit Marius - Gründungsmitglied der ITSM Abteilung zu werden. In seiner Freizeit engagiert sich Tim in der Freiwilligen Feuerwehr – als Maschinist und Atemschutzgeräteträger -, spielt im Laientheater Bauernschwänke und ist auch handwerklich ein absolutes Allroundtalent. Angefangen von Mauern hochziehen bis hin zur KNX-Verkabelung ist er jederzeit...

Lies mehr von Tim und triff unser Team