为什么ceph-mon不能放在基于ceph的openstack的虚拟机上

发布者：phpor 于为什么ceph-mon不能放在基于ceph的openstack的虚拟机上留下评论

由于对ceph-mon不甚了解，做了下面一件事情：

  1. openstack存储完全基于ceph存储
  2. ceph存储节点单独部署
  3. 现有ceph-mon节点2个（mon-1，mon-2），想重装mon-1；但是，没有了ceph-1之后，ceph-2似乎认为脑裂了，不提供服务；于是想找个临时mon顶上
  4. 在openstack集群上很轻松地申请了一台机器，很快变安装了一个mon-3，由于安全组（没有放开6789）的原因，虽然mon-3能够找到mon-2，ceph -s也能看到，但是mon-1却因为不能连接到mon-3使得mon-3没有完全加入，调整安全组后，似乎一切变的正常，当mon-1去掉之后，ceph -s 卡住了，mon-3中正在执行的yum也卡住了
  5. 观察发现，mon-2 出于probing状态，mon-3出于reop状态，没有leader了
    1. 因为mon-3 依赖ceph，但是ceph没有了leader，mon-3就不能落地数据
    2. mon-3不能落地数据的话，mon-2、mon-3之间就没法选举出来leader
    3. 如此，事情进入僵局，死锁了
  6. 分析：
    1. 每个mon上都存在一个db，里面放着monmap信息，启动的时候，就根据monmap中的信息加入集群，如果monmap中只有自己，直接启动就可以了，如果有多个mon节点，并且当前不存在leader就得选举
    2. 如果能把monmap中的mon节点修改成只有自己，就能正常启动
  7.  解决办法：（参考: http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-mon/#recovering-a-monitor-s-broken-monmap )
    1. 停掉mon-2，不停掉是操作不了monmap的db的
    2. 导出monmap：
      ceph-mon -i ID-FOO –extract-monmap /tmp/monmap
    3. 查看monmap：
      monmaptool –print -f /tmp/monmap
    4. 删除mon-3
      monmaptool –rm mon-3  -f /tmp/monmap
    5. 注入monmap
      ceph-mon -i ID –inject-monmap /tmp/monmap
    6. 启动mon-2

 

Inject a monmap into the monitor

Usually the safest path. You should grab the monmap from the remaining monitors and inject it into the monitor with the corrupted/lost monmap.

These are the basic steps:

  1. Is there a formed quorum? If so, grab the monmap from the quorum:

  2. No quorum? Grab the monmap directly from another monitor (this assumes the monitor you’re grabbing the monmap from has id ID-FOO and has been stopped):

  3. Stop the monitor you’re going to inject the monmap into.

  4. Inject the monmap:

  5. Start the monitor

Please keep in mind that the ability to inject monmaps is a powerful feature that can cause havoc with your monitors if misused as it will overwrite the latest, existing monmap kept by the monitor.

What if the state is probing?

This means the monitor is still looking for the other monitors. Every time you start a monitor, the monitor will stay in this state for some time while trying to find the rest of the monitors specified in the monmap. The time a monitor will spend in this state can vary. For instance, when on a single-monitor cluster, the monitor will pass through the probing state almost instantaneously, since there are no other monitors around. On a multi-monitor cluster, the monitors will stay in this state until they find enough monitors to form a quorum – this means that if you have 2 out of 3 monitors down, the one remaining monitor will stay in this state indefinitively until you bring one of the other monitors up.

If you have a quorum, however, the monitor should be able to find the remaining monitors pretty fast, as long as they can be reached. If your monitor is stuck probing and you have gone through with all the communication troubleshooting, then there is a fair chance that the monitor is trying to reach the other monitors on a wrong address. mon_status outputs the monmap known to the monitor: check if the other monitor’s locations match reality. If they don’t, jump to Recovering a Monitor’s Broken monmap; if they do, then it may be related to severe clock skews amongst the monitor nodes and you should refer to Clock Skews first, but if that doesn’t solve your problem then it is the time to prepare some logs and reach out to the community (please refer to Preparing your logs on how to best prepare your logs).

留下评论

邮箱地址不会被公开。 必填项已用*标注

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据。