Ceph环境搭建 （Debian发行版为例）

Posted on

Ceph环境搭建 （Debian发行版为例）

ref:http://docs.ceph.com/docs/master/start

注：一个ceph存储集群需要至少一个Ceph Monitor和至少两个Ceph OSD Daemons

安装前的了解

安装版本

L版本安装：echo deb http://download.ceph.com/debian-luminous/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

普通用户免密码sudo设置
1
2
3
4
su root
chmod u+w /etc/sudoers
vim /etc/sudoers
添加 lihongbo01 ALL=(ALL) NOPASSWD: ALL这一行，替换`lihongbo01`为你的用户名即可；
ssh默认端口号问题 （不是22)

解决方案：编辑~/.ssh/config文件，内容如下：

1
2
3
4
5
6
7
8
9
10
Port 32200
Host cld-unknown212867
    Hostname cld-unknown212867
    User lihongbo01
Host cld-unknown212868
    Hostname cld-unknown212868
    User lihongbo01
Host cld-unknown212869
    Hostname cld-unknown212869
    User lihongbo01

安装步骤 <在此之前请配置好免密登录和节点时间同步，参考官网Pre-flight>

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
echo deb http://download.ceph.com/debian-{ceph-stable-release}/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list 【版本如：luminous】
sudo apt-get update && sudo apt-get install ceph-deploy

1.注意：debian安装mimic版本时，用ceph-deploy安装不能安装到最新的，只能安装到Candidate: 10.2.11-2的，可以通过apt-cache policy ceph-common查看；而lumious可以安装到12.2.10。

2.软件不知道怎么安装可以用： apt-file search xxx

接着执行：

mkdir my-cluster
cd my-cluster
ceph-deploy new cld-unknown212867
vim ceph.conf
添加：osd pool default size = 2 ## 放到ceph.conf中修改副本数为2
ceph-deploy install --release luminous cld-unknown212867 cld-unknown212868 cld-unknown212869
ceph-deploy mon create-initial
ceph-deploy admin cld-unknown212867 cld-unknown212868 cld-unknown212869
ceph-deploy mgr create cld-unknown212867 *Required only for luminous+ builds, i.e >= 12.x builds*
1. 以loop来做虚拟盘<考虑到当前节点没有额外盘> —- 失败

接着我们来添加OSDs，我们首先虚拟一个块设备（用dd命令）：

dd if=/dev/zero of=1G.img bs=1M count=1000  # OSD节点上虚拟一个1G的块设备
ceph-deploy osd create --data /home/lihongbo01/dev/1G.img cld-unknown212868 【MON节点上执行的】

经过多次尝试，用loop来做设备失败。 【可能还没有完全支持】

2. 【添加盘后添加OSD成功】 —- 成功
ceph-deploy osd create --data /dev/vdd cld-unknown212868 
ceph-deploy osd create --data /dev/vdd cld-unknown212869

创建OSD时最终会调用ceph-volume，这时--data已经不能是目录了，而须一个disk，可参见问题部分查看disk.py代码来理解。

查看成功
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
lihongbo01@cld-unknown212867:~/my-cluster$ sudo ceph health
HEALTH_OK
lihongbo01@cld-unknown212867:~/my-cluster$ sudo ceph -s
  cluster:
    id:     67170b2c-c545-4efb-80cb-810256529f34
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum cld-unknown212867
    mgr: cld-unknown212867(active)
    osd: 2 osds: 2 up, 2 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0B
    usage:   2.00GiB used, 98.0GiB / 100GiB avail
    pgs:

一定要加sudo查看

中途安装失败删除命令：
1
2
3
4
ceph-deploy purge {ceph-node} [{ceph-node}]
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys
rm ceph.*

注意要加上–release luminous才会安装该版本的，不然就安装默认的了，当前ceph-deploy是2.0.1版本，Debian上默认安装Ceph是10.2.11版本的。


以下是以loop虚拟设备来做OSD时遇到的问题即解决方法：

问题

添加OSD不成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
[node4][DEBUG ]  stderr: 12: (main()+0x2c58) [0x7f8e8dc18c18]
[node4][DEBUG ]  stderr: 13: (__libc_start_main()+0xf5) [0x7f8e808c03d5]
[node4][DEBUG ]  stderr: 14: (()+0x384ab0) [0x7f8e8dcf0ab0]
[node4][DEBUG ]  stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
[node4][DEBUG ] --> Was unable to complete a new OSD, will rollback changes
[node4][DEBUG ] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.3 --yes-i-really-mean-it
[node4][DEBUG ]  stderr: purged osd.3
[node4][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdc --block.wal /dev/sdb2 --block.db /dev/sdb1
[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs

修复：
node节点卸载挂载分区
[root@node4 ~]# umount /var/lib/ceph/osd/ceph-3
查看lvs

[root@node4 ~]# lvs
  LV                                             VG                                        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-block-ab4df89f-8682-4233-b683-e05c161089a4 ceph-183df03c-83fb-441c-a82e-d7b560e1a611 -wi-a----- 50.00g                                                    
  root                                           cl                                        -wi-ao---- 35.12g                                                    
  swap                                           cl                                        -wi-ao----  3.88g        

删除lv，vg，pv
[root@node4 ~]# lvremove osd-block-ab4df89f-8682-4233-b683-e05c161089a4 ceph-183df03c-83fb-441c-a82e-d7b560e1a611
  Volume group "osd-block-ab4df89f-8682-4233-b683-e05c161089a4" not found
  Cannot process volume group osd-block-ab4df89f-8682-4233-b683-e05c161089a4
Do you really want to remove active logical volume ceph-183df03c-83fb-441c-a82e-d7b560e1a611/osd-block-ab4df89f-8682-4233-b683-e05c161089a4? [y/n]: y
  Logical volume "osd-block-ab4df89f-8682-4233-b683-e05c161089a4" successfully removed

[root@node4 ~]# vgremove  ceph-183df03c-83fb-441c-a82e-d7b560e1a611
  Volume group "ceph-183df03c-83fb-441c-a82e-d7b560e1a611" successfully removed

[root@node4 ~]# pvremove /dev/sdc
  Labels on physical volume "/dev/sdc" successfully wiped.

删除/var/lib/ceph/osd/ 所有
[root@node4 ~]# rm /var/log/ceph/ceph-* -rf

删除 ceph 日志
[root@node4 ~]# rm -rf /var/lib/ceph/*

monitor1 上重置node4 的sdc分区
[root@monitor1 ceph-cluster]# ceph-deploy disk zap node4 /dev/sdc

monitor1 上重新创建osd节点
[root@monitor1 ceph-cluster]# ceph-deploy osd create node4 --bluestore --data /dev/sdc --block-db /dev/sdb1 --block-wal /dev/sdb2

【实践失败】

loop设备做块设备

需要修改两处：

1
2
vim /usr/lib/python2.7/dist-packages/ceph_volume/util/disk.py +332  修改为：return TYPE == 'disk' or 'loop'
vim /usr/lib/python2.7/dist-packages/ceph_volume/util/disk.py +417  True ==> False

ref:https://www.netways.de/blog/2018/11/14/ceph-mimic-using-loop-devices-as-osd/

清理操作

sudo lvs (vgscan)
sudo vgremove ceph-9eee83f3-8b17-4b3f-b53e-62bf4977733b -y
lsblk


华丽的分割线

Ceph对象网关部署 (可部署多个)

使用命令在 Ceph 对象网关节点上新建一个 Ceph对象网关实例：

1
ceph-deploy rgw create cld-unknown212868

需要开放7480端口：iptables -I INPUT 1 -p tcp -m tcp –dport 7480 -j ACCEPT

测试及效果：

1
2
3
4
5
6
7
8
9
10
11
curl http://cld-unknown212868:7480

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Owner>
    <ID>anonymous</ID>
    <DisplayName></DisplayName>
  </Owner>
  <Buckets>
  </Buckets>
</ListAllMyBucketsResult>

即表示安装成功。

创建S3用户
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
lihongbo01@cld-unknown212867:~/my-cluster$ sudo radosgw-admin user create --uid="lhb" --display-name="First User"
{
    "user_id": "lhb",
    "display_name": "First User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "lhb",
            "access_key": "KSOF3PIEDDNI7KP4I1HC",
            "secret_key": "f6N9wvBWbfviV0R5TaGUHYCXZKWCTfmMPDIQf4hF"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

目前只能使用path-style来访问，hosted-style需要另外配置。

修改了配置文件（如网关端口号）
ceph-deploy --overwrite-conf config push <gateway-node> [<other-nodes>] # 将该配置文件推送到你的 Ceph 对象网关节点
sudo service radosgw restart id=rgw.<short-hostname>  # 重启Ceph对象网关
提供泛域名解析能力

Ceph对象存储网关支持两种访问模式：path-style和host-style；默认的支持path-style，即http://cld-unknown212868/bucket；要想以类似S3的host-style进行访问，即用http://bucket.cld-unknown212868/obj来进行访问，需要配置泛域名解析，此处用到dnsmasq工具。

sudo apt-get install dnsmasq -y
vim /etc/resolv.conf    【改为：nameserver 127.0.0.1】
vim /etc/resolv.dnsmasq.conf    【将原来resolv.conf的内容写在这个文件里面】

接着编辑dnsmqsq配置文件，在/etc/dnsmasq.conf中加入一下内容：

1
2
3
4
5
address=/.cld-unknown212868/10.202.42.147
address=/.cld-unknown212868.i.nease.net/10.202.42.147
resolv-file=/etc/resolv.dnsmasq.conf
conf-dir=/etc/dnsmasq.d
conf-dir=/etc/dnsmasq.d/,*.conf

此配置的含义是将xxx.cld-unknown212868的路径都转到10.202.42.147这个地址；

然后重启dnsmasq服务：/etc/init.d/dnsmasq restart

效果
1
2
3
4
5
6
7
8
9
10
lihongbo01@cld-unknown212868:~/workspace/ceph_workspace$ ping bucket.cld-unknown212868
PING bucket.cld-unknown212868 (10.202.42.147) 56(84) bytes of data.
64 bytes from cld-unknown212868.i.nease.net (10.202.42.147): icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from cld-unknown212868.i.nease.net (10.202.42.147): icmp_seq=2 ttl=64 time=0.064 ms
64 bytes from cld-unknown212868.i.nease.net (10.202.42.147): icmp_seq=3 ttl=64 time=0.065 ms
64 bytes from cld-unknown212868.i.nease.net (10.202.42.147): icmp_seq=4 ttl=64 time=0.043 ms
^C
--- bucket.cld-unknown212868 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3075ms
rtt min/avg/max/mdev = 0.043/0.054/0.065/0.011 ms