基础:
磁盘镜像技术DRBD
DRBD(Distributed Replicated Block Device)是由内核模块和相关脚本构成，用以构建高可用性的集群。其实现方式是通过网络来镜像整个设备。它允许用户在远程机器上建立一个本地块设备的实时镜像。也可以把它看成一个网络RAID1.
工作原理：
DRBD负责接收数据，把数据写到本地磁盘，然后发送给另一个主机。另一个主机再将数据存到自己的磁盘中。目前，DRBD每次只允许对一个节点进行读写访问，这对于通常的故障切换高可用性集群已经够了。
DRBD协议：
A：数据一旦写入磁盘并发送到网络中就认为完成了写入操作
B：收到接受确认就认为完成了写入操作
C：收到写入确认就完成了写入操作
目前运用最多的就是C协议
DRBD的三个进程：
drbd0——worker：主进程
drbd0——asender：primary上drbd0的数据发送进程
drbd0——receiver；secondary上drbd0的数据接受进程
配置DRBD前需要注意几点：
mount drbd设备以前必须把设备切换到primary状态
两个节点中，同一时刻只能有一台处于primary状态，另一台处于secondary状态，处于secondary状态的服务器上不能加载drbd设备。主备服务器同步的两个分区大小最好相同，这样不至于浪费磁盘空间，因为drbd镜像相当于网络raid1.
测试环境:
– 两个节点 (node1:192.168.1.110 node2:192.168.1.111 hosts做绑定 )
- 两块硬盘 (node1与node2节点各自添加相同大小的一块硬盘)
– 保持两个节点时间同步
– Selinux 关闭 防火墙7788端口打开(或者关闭防火墙)
开始安装:
安装第三方源
rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpmyum makecacheyum install *gcc* make* #源码包安装是的依赖
安装DRBD(yum):
[root@node1 ~]# yum -y install drbd83-utils kmod-drbd83[root@node2 ~]# yum -y install drbd83-utils kmod-drbd83node1与node2执行"yum install kernel-devel kernel kernel-headers flex " ,否则会报错:FATAL: Module drbd not found.安装后reboot
手动加入drbd模块:
# depmod
说明：在编译并准备好一个Linux内核加载模块后，modprobe前必须先执行命令depmod，此命令会生成新的modules.dep。
/sbin/modprobe drbd
lsmod |grep drbd
安装DRBD(源码):
[root@node1 ~]#wget http://oss.linbit.com/drbd/8.3/drbd-8.3.16.tar.gz && tar -zxvf drbd-8.3.16.tar.gz #drbd共有两部分组成：内核模块和用户空间的管理工具。[root@node1 ~]#cd drbd-8.3.16[root@node1 drbd-8.3.16]#./configure --prefix=/usr/local/drbd --with-km #"--with-km"启用drbd内核模块(这里可以使用./configure --help查询)[root@node1 drbd-8.3.16]#make KDIR=/usr/src/kernels/2.6.32-358.el6.x86_64/ #KDIR指的是自己的内核路径[root@node1 drbd-8.3.16]#make install[root@node1 drbd-8.3.16]#mkdir -p /usr/local/drbd/var/run/drbd #创建drbd运行所需目录[root@node1 drbd-8.3.16]#cp /usr/local/drbd/etc/rc.d/init.d/drbd /etc/rc.d/init.d/ #添加启动脚本[root@node1 drbd-8.3.16]#chkconfig --add drbd && chkconfig drbd on #设置开机启动安装drbd模块:[root@node1 drbd-8.3.16]#cd drbd[root@node1 drbd-8.3.16]#make clean[root@node1 drbd-8.3.16]#make KDIR=/usr/src/kernels/2.6.32-358.el6.x86_64/[root@node1 drbd-8.3.16]#cp drbd.ko /lib/modules/`uname -r`/kernel/lib/ [root@node1 drbd-8.3.16]# depmod[root@node1 drbd-8.3.16]#/sbin/modprobe drbd[root@node1 drbd-8.3.16]#lsmod |grep drbd
DRBD分区:
在两个节点为drbd单独分区
[root@node1 ~]# fdisk -cu /dev/sdb[root@node2 ~]# fdisk -cu /dev/sdb
如下:
[root@node1 yum.repos.d]# fdisk -cu /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabelBuilding a new DOS disklabel with disk identifier 0x2a0f1472.Changes will remain in memory only, until you decide to write them.After that, of course, the previous content won't be recoverable.Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)Command (m for help): pDisk /dev/sdb: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylinders, total 4194304 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x2a0f1472Device Boot Start End Blocks Id SystemCommand (m for help): nCommand actione extendedp primary partition (1-4)pPartition number (1-4): 1First sector (2048-4194303, default 2048):Using default value 2048Last sector, +sectors or +size{K,M,G} (2048-4194303, default 4194303):Using default value 4194303Command (m for help): wThe partition table has been altered! mkfs.ext4 /dev/sdb1
创建分布式的复制的块设备文件:
[root@node1 ~]# vi /etc/drbd.d/clusterdb.resresource clusterdb { startup { wfc-timeout 30; outdated-wfc-timeout 20; degr-wfc-timeout 30; }net { cram-hmac-alg sha1; shared-secret sync_disk; }syncer { rate 10M; al-extents 257; on-no-data-accessible io-error; } on node1 { device /dev/drbd0; disk /dev/sdb1; address 192.168.1.110:7788; flexible-meta-disk internal; } on node2 { device /dev/drbd0; disk /dev/sdb1; address 192.168.1.111:7788; meta-disk internal; } } 确认在两个节点做了hosts解析
/etc/hosts 192.168.1.110 node1 node1.example.com 192.168.1.111 node2 node2.example.com
时间同步(crond加入定时执行)
crontab -e5 * * * * root ntpdate time-nw.nist.gov:wqcrontab -l5 * * * * root ntpdate time-nw.nist.govservice crond restart
拷贝DRBD配置文件至node2节点:
[root@node1 ~]# scp /etc/drbd.d/clusterdb.res node2:/etc/drbd.d/clusterdb.res
初始化两个节点的DRBD元数据:
[root@node1 ~]# drbdadm create-md clusterdb[root@node2 ~]# drbdadm create-md clusterdbYou want me to create a v08 style flexible-size internal meta data block.There appears to be a v08 flexible-size internal meta data blockalready in place on /dev/sdb1 at byte offset 2146430976Do you really want to overwrite the existing v08 meta-data?[need to type 'yes' to confirm] yesWriting meta data...initializing activity logNOT initialized bitmapNew drbd meta data block successfully created.
报错:
md_offset 2146758656al_offset 2146725888bm_offset 2146660352Found ext3 filesystem 2096448 kB data area apparently used 2096348 kB left usable by current configurationDevice size would be truncated, whichwould corrupt data and result in'access beyond end of device' errors.You need to either * use external meta data (recommended) * shrink that filesystem first * zero out the device (destroy the filesystem)Operation refused.Command 'drbdmeta 0 v08 /dev/sdb1 internal create-md' terminated with exit code 40drbdadm create-md clusterdb: exited with code 40
执行:dd if=/dev/zero of=/dev/sdb1 bs=1M count=1 然后重新初始化可解决报错.
在两个节点开启DRBD进程:
[root@node1 ~]# service drbd start[root@node2 ~]# service drbd start
运行drbdadm设置node1为主节点命令:
[root@node1 ~]# drbdadm -- --overwrite-data-of-peer primary all注：也可以在要设置为Primary的节点上使用如下命令来设置主节点：drbdsetup primary all或[root@node1 ~]# drbdsetup /dev/drbd0 primary -o而后再次查看状态，可以发现数据同步过程已经开始：
检查磁盘初始同步是否完成,确认你是否在主节点上：
[root@node1 yum.repos.d]# cat /proc/drbd version: 8.3.16 (api:88/proto:86-97) GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build32R6, 2013-09-27 15:59:12 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:78848 nr:0 dw:0 dr:79520 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:2017180 [>....................] sync'ed: 4.0% (2017180/2096028)K finish: 0:02:58 speed: 11,264 (11,264) K/secns:1081628 nr:0 dw:33260 dr:1048752 al:14 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0]
注:"/proc/drbd"中显示了drbd当前的状态.第一行的st表示两台主机的状态,都是”备机”状态.ds是磁盘状态.
在分布式复制的块设备上创建文件系统：
[root@node1 yum.repos.d]# /sbin/mkfs.ext4 /dev/drbd0mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 131072 inodes, 524007 blocks 26200 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=536870912 16 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: doneThis filesystem will be automatically checked every 26 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.
在你的主节点挂载DRBD分区:
mkdir /datamount /dev/drbd0 /data[root@node1 ~]# df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/vg_unixmencentos65-lv_root 19G 3.6G 15G 20% /tmpfs 1.2G 44M 1.2G 4% /dev/shm/dev/sda1 485M 80M 380M 18% /boot/dev/drbd0 2.0G 36M 1.9G 2% /data
创建测试文件:
[root@node1 ~]#dd if=/dev/zero of=/data/tempfile1.tmp bs=104857600 count=2#创建200M的测试文件[root@node1 data]# lltotal 204816drwx------ 2 root root 16384 Jan 25 00:14 lost+found-rw-r--r-- 1 root root 209715200 Jan 25 00:15 tempfile1.tmp
关闭主/设置辅节点:
(从主节点卸载data目录,同时将主节点降为辅助节点)
(设置辅助节点为主节点,挂载data目录)
[root@node1 ~]#umount /data #卸载drbd设备
[root@node1 ~]#drbdadm secondary clusterdb #关闭node1的主节点,设置为辅节点
[root@node1 data]# cat /proc/drbd version: 8.3.16 (api:88/proto:86-97)GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r----- ns:774292 nr:8 dw:476096 dr:300622 al:123 bm:23 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@node2 ~]#drbdadm -- --overwrite-data-of-peer primary all #设置node2为主节点,关闭辅节点
[root@node2 ~]#mount /dev/drbd0 /data #挂载drbd0到data目录
[root@node2 data]# cat /proc/drbd version: 8.3.16 (api:88/proto:86-97)GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:32 nr:774292 dw:774324 dr:2030 al:2 bm:23 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
查看data目录:
[root@node2 data]# lltotal 204816drwx------ 2 root root 16384 Jan 25 02:24 lost+found-rw-r--r-- 1 root root 209715200 Jan 25 02:25 tempfile1.tmp
注意:
修改主节点为辅节点时一定要卸载挂载的目录,否则会报错!
翻译自:
可参考:
DRBD脑裂处理:
DRBD介绍、工作原理及脑裂故障处理:
DRBD脑裂处理:
源码下载地址:http://oss.linbit.com/drbd/