PHPor 的Blog – 第28页

watch + ssh 远程top

通过ssh远程top一次ceph用户的所有进程：

ssh ceph-14 top -u ceph -b -n 1

1	ssh ceph-14 top -u ceph -b -n 1

如果想查看多次（3次）：

ssh ceph-14 top -u ceph -b -n 3

1	ssh ceph-14 top -u ceph -b -n 3

如果想以本机的top效果查看，则可以：

watch ssh ceph-14 top -u ceph -b -n 1

1	watch ssh ceph-14 top -u ceph -b -n 1

其实，根本不用watch ：

ssh -t ceph-14 top -u ceph

1	ssh -t ceph-14 top -u ceph

但是如果想在一个机器上查看2个机器的top呢？

watch "ssh ceph-14 top -u ceph -b -n 1 ; ssh ceph-4 top -u ceph -b -n 1"

1	watch "ssh ceph-14 top -u ceph -b -n 1 ; ssh ceph-4 top -u ceph -b -n 1"

如果如果想在一个机器上查看4个机器的top呢？写4遍ssh好繁琐：

watch 'for h in ceph-4 ceph-14; do echo -e "$h\n"; ssh $h top -u ceph -b -n 1; printf "=%.0s" {1..80}; echo;done '

1	watch 'for h in ceph-4 ceph-14; do echo -e "$h\n"; ssh $h top -u ceph -b -n 1; printf "=%.0s" {1..80}; echo;done '

分割线的打印，参考： https://stackoverflow.com/questions/5349718/how-can-i-repeat-a-character-in-bash

cloud-init 源码阅读

网络初始化：

/usr/lib/python2.7/site-packages/cloudinit/stages.py：

    def apply_network_config(self, bring_up):
        netcfg, src = self._find_networking_config()
        if netcfg is None:
            LOG.info("network config is disabled by %s", src)
            return

        try:
            LOG.debug("applying net config names for %s" % netcfg)
            self.distro.apply_network_config_names(netcfg)
        except Exception as e:
            LOG.warn("Failed to rename devices: %s", e)

        if (self.datasource is not NULL_DATA_SOURCE and
                not self.is_new_instance()):
            LOG.debug("not a new instance. network config is not applied.")
            return

        LOG.info("Applying network configuration from %s bringup=%s: %s",
                 src, bring_up, netcfg)
        try:
            return self.distro.apply_network_config(netcfg, bring_up=bring_up)
        except NotImplementedError:
            LOG.warn("distro '%s' does not implement apply_network_config. "
                     "networking may not be configured properly." %
                     self.distro)
            return

def apply_network_config(self, bring_up):

netcfg, src = self._find_networking_config()

if netcfg is None:

LOG.info("network config is disabled by %s", src)

return

try:

LOG.debug("applying net config names for %s" % netcfg)

self.distro.apply_network_config_names(netcfg)

except Exception as e:

LOG.warn("Failed to rename devices: %s", e)

if (self.datasource is not NULL_DATA_SOURCE and

not self.is_new_instance()):

LOG.debug("not a new instance. network config is not applied.")

return

LOG.info("Applying network configuration from %s bringup=%s: %s",

src, bring_up, netcfg)

try:

return self.distro.apply_network_config(netcfg, bring_up=bring_up)

except NotImplementedError:

LOG.warn("distro '%s' does not implement apply_network_config. "

"networking may not be configured properly." %

self.distro)

return

分析：

每次都查找网络配置
如果找到配置，则 self.distro.apply_network_config_names(netcfg)
1. 对于新机器，则 self.distro.apply_network_config(netcfg, bring_up=bring_up) ，包括
  1. 写 ifcfg-eth0 配置文件
  2. if bring_up ,则，启动该网络设备

/usr/lib/python2.7/site-packages/cloudinit/stages.py：

    def _find_networking_config(self):
        disable_file = os.path.join(
            self.paths.get_cpath('data'), 'upgraded-network')
        if os.path.exists(disable_file):
            return (None, disable_file)

        cmdline_cfg = ('cmdline', cmdline.read_kernel_cmdline_config())
        dscfg = ('ds', None)
        if self.datasource and hasattr(self.datasource, 'network_config'):
            dscfg = ('ds', self.datasource.network_config)
        sys_cfg = ('system_cfg', self.cfg.get('network'))

        for loc, ncfg in (cmdline_cfg, sys_cfg, dscfg):
            if net.is_disabled_cfg(ncfg):
                LOG.debug("network config disabled by %s", loc)
                return (None, loc)
            if ncfg:
                return (ncfg, loc)
        return (net.generate_fallback_config(), "fallback")

def _find_networking_config(self):

disable_file = os.path.join(

self.paths.get_cpath('data'), 'upgraded-network')

if os.path.exists(disable_file):

return (None, disable_file)

cmdline_cfg = ('cmdline', cmdline.read_kernel_cmdline_config())

dscfg = ('ds', None)

if self.datasource and hasattr(self.datasource, 'network_config'):

dscfg = ('ds', self.datasource.network_config)

sys_cfg = ('system_cfg', self.cfg.get('network'))

for loc, ncfg in (cmdline_cfg, sys_cfg, dscfg):

if net.is_disabled_cfg(ncfg):

LOG.debug("network config disabled by %s", loc)

return (None, loc)

if ncfg:

return (ncfg, loc)

return (net.generate_fallback_config(), "fallback")

分析：

先尝试从三个不同的地方获取网络配置（注意，这里是有优先级的）：cmdline、system_cfg、datasource中的network_config；只要其中一个地方明确禁用网络或存在配置则返回；

如果没有找到任何配置，则进入预定义的配置逻辑net/__init__.py: net.generate_fallback_config() ：

def generate_fallback_config():
    """Determine which attached net dev is most likely to have a connection and
       generate network state to run dhcp on that interface"""
    # get list of interfaces that could have connections
    invalid_interfaces = set(['lo'])
    potential_interfaces = set(get_devicelist())
    potential_interfaces = potential_interfaces.difference(invalid_interfaces)
    # sort into interfaces with carrier, interfaces which could have carrier,
    # and ignore interfaces that are definitely disconnected
    connected = []
    possibly_connected = []
    for interface in potential_interfaces:
        if interface.startswith("veth"):
            continue
        if os.path.exists(sys_dev_path(interface, "bridge")):
            # skip any bridges
            continue
        carrier = read_sys_net_int(interface, 'carrier')
        if carrier:
            connected.append(interface)
            continue
        # check if nic is dormant or down, as this may make a nick appear to
        # not have a carrier even though it could acquire one when brought
        # online by dhclient
        dormant = read_sys_net_int(interface, 'dormant')
        if dormant:
            possibly_connected.append(interface)
            continue
        operstate = read_sys_net_safe(interface, 'operstate')
        if operstate in ['dormant', 'down', 'lowerlayerdown', 'unknown']:
            possibly_connected.append(interface)
            continue

    # don't bother with interfaces that might not be connected if there are
    # some that definitely are
    if connected:
        potential_interfaces = connected
    else:
        potential_interfaces = possibly_connected

    # if eth0 exists use it above anything else, otherwise get the interface
    # that we can read 'first' (using the sorted defintion of first).
    names = list(sorted(potential_interfaces))
    if DEFAULT_PRIMARY_INTERFACE in names:
        names.remove(DEFAULT_PRIMARY_INTERFACE)
        names.insert(0, DEFAULT_PRIMARY_INTERFACE)
    target_name = None
    target_mac = None
    for name in names:
        mac = read_sys_net_safe(name, 'address')
        if mac:
            target_name = name
            target_mac = mac
            break
    if target_mac and target_name:
        nconf = {'config': [], 'version': 1}
        nconf['config'].append(
            {'type': 'physical', 'name': target_name,
             'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]})
        return nconf
    else:
        # can't read any interfaces addresses (or there are none); give up
        return None

def generate_fallback_config():

"""Determine which attached net dev is most likely to have a connection and

generate network state to run dhcp on that interface"""

# get list of interfaces that could have connections

invalid_interfaces = set(['lo'])

potential_interfaces = set(get_devicelist())

potential_interfaces = potential_interfaces.difference(invalid_interfaces)

# sort into interfaces with carrier, interfaces which could have carrier,

# and ignore interfaces that are definitely disconnected

connected = []

possibly_connected = []

for interface in potential_interfaces:

if interface.startswith("veth"):

continue

if os.path.exists(sys_dev_path(interface, "bridge")):

# skip any bridges

continue

carrier = read_sys_net_int(interface, 'carrier')

if carrier:

connected.append(interface)

continue

# check if nic is dormant or down, as this may make a nick appear to

# not have a carrier even though it could acquire one when brought

# online by dhclient

dormant = read_sys_net_int(interface, 'dormant')

if dormant:

possibly_connected.append(interface)

continue

operstate = read_sys_net_safe(interface, 'operstate')

if operstate in ['dormant', 'down', 'lowerlayerdown', 'unknown']:

possibly_connected.append(interface)

continue

# don't bother with interfaces that might not be connected if there are

# some that definitely are

if connected:

potential_interfaces = connected

else:

potential_interfaces = possibly_connected

# if eth0 exists use it above anything else, otherwise get the interface

# that we can read 'first' (using the sorted defintion of first).

names = list(sorted(potential_interfaces))

if DEFAULT_PRIMARY_INTERFACE in names:

names.remove(DEFAULT_PRIMARY_INTERFACE)

names.insert(0, DEFAULT_PRIMARY_INTERFACE)

target_name = None

target_mac = None

for name in names:

mac = read_sys_net_safe(name, 'address')

if mac:

target_name = name

target_mac = mac

break

if target_mac and target_name:

nconf = {'config': [], 'version': 1}

nconf['config'].append(

{'type': 'physical', 'name': target_name,

'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]})

return nconf

else:

# can't read any interfaces addresses (or there are none); give up

return None

分析：

lo 排除在外
list目录 SYS_CLASS_NET = “/sys/class/net/” ，可以找到多个网络设备，如：

分析现有的网络设备，看看哪个最适合连网
“””Determine which attached net dev is most likely to have a connection and
generate network state to run dhcp on that interface”””

如果是veth开头的，认为是设备对，不考虑
如果存在 /sys/class/net/{$name}/bridge 则认为是（肯定是）网桥，也不考虑

如果是插着线(carrier)的，则加入可以考虑的接口列表，如何判断是否插着线呢？
参考：

def read_sys_net_int(iface, field):
    val = read_sys_net_safe(iface, field)
    if val is False:
        return None
    try:
        return int(val)
    except TypeError:
        return None

def read_sys_net_int(iface, field):

val = read_sys_net_safe(iface, field)

if val is False:

return None

try:

return int(val)

except TypeError:

return None

简单说就是： cat /sys/class/net/{$name}/carrier 如果结果是整数，则是插着线呢，否则就没插线，如下：

注意：
插着线的意思是，线的两端都是加了电的网络设备，即：数据链路层是UP的;
有些网络设备的该文件是不能cat的，如：

如果cat /sys/class/net/{$name}/dormant 是一个大于0的整型值，也可以考虑
参考文件 /sys/class/net/{$name}/operstate ，这里记录了设备的状态，如果状态是： ‘dormant’, ‘down’, ‘lowerlayerdown’, ‘unknown’，也可以考虑
最后，对可以考虑的列表进行排序；排在前面的优先考虑；不过还有个例外，程序里面定义了一个默认的网络设备 DEFAULT_PRIMARY_INTERFACE，就是 eth0，排序后被特意添加到了列表的最前面；不出意外的话，后续胜出的基本就是eth0了

根据上面得到的列表，查找设备的mac地址，只要有一个设备有mac地址，该设备就胜出了，后面的就没戏了

最后，返回网络配置

    if target_mac and target_name:
        nconf = {'config': [], 'version': 1}
        nconf['config'].append(
            {'type': 'physical', 'name': target_name,
             'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]})
        return nconf

if target_mac and target_name:

nconf = {'config': [], 'version': 1}

nconf['config'].append(

{'type': 'physical', 'name': target_name,

'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]})

return nconf

总结：

分析可知，如果不对网络进行特殊配置的话，cloud-init只能帮我们配置一个网卡；一般来讲，大部分需求已经满足了。

我们如果看 cloud-init （ /var/log/cloud-init.log ）的日志的话，会发现，在多个网卡的时候，虽然其他网卡的信息也被read了，但是最终却没有得到和eth0相同的待遇，现在也就真相大白了

堆栈：

关于cmdline的获取方法：（util.py)

容器的话，cat /proc/1/cmdline
非容器的话， cat /proc/cmdline

结果：

BOOT_IMAGE=/boot/vmlinuz-4.10.10-1.el7.elrepo.x86_64 root=UUID=0356e691-d6fb-4f8b-a905-4230dbe62a32 ro console=tty0 console=ttyS0,115200n8 crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8

1	BOOT_IMAGE=/boot/vmlinuz-4.10.10-1.el7.elrepo.x86_64 root=UUID=0356e691-d6fb-4f8b-a905-4230dbe62a32 ro console=tty0 console=ttyS0,115200n8 crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8

变化的mac地址

一个bridge诞生时，会有一个mac地址；当向bridge上addif veth2时，bridge的mac地址就跟随了veth2的mac地址，难道br0就不能固定一个mac地址吗？

可以的：

默认情况下，bridge总是跟随port中mac地址最小的那个port的mac地址，如果不想让mac地址总是变化，则可以设置bridge的首选mac地址，方法就是显式地给bridge设置mac地址：

ip link set dev br0 address fa:11:11:11:11:11

1	ip link set dev br0 address fa:11:11:11:11:11

参考：

http://blog.csdn.net/fanwenbo/article/details/2131193

查找网线在交换机哪个口的方法

已知：

我可以登录服务器
我可以登录交换机
服务器上有多块网卡，当前登录走网卡em1

任务：

要给em4配置ip地址

步骤：

首先找到em4在交换机的哪个口

在服务器上：

ip link set em4 up

1

ip link set em4 up
在服务器上：

ping -I em4 1.1.1.1

1

ping -I em4 1.1.1.1
在交换机上

dis mac-address | include xxxx #其中xxxx是em4的mac地址

1

dis mac-address | include xxxx #其中xxxx是em4的mac地址
啥？没有？有可能，继续

在交换机上

display interface brief   | include 100M    #看看有没有speed是100M的端口,现在都是千兆交换机，应该不会出现 100M， 如果有的话，就include 10M

1	display interface brief \| include 100M #看看有没有speed是100M的端口,现在都是千兆交换机，应该不会出现 100M，如果有的话，就include 10M

在服务器上

ethtool -s em4 speed 100 duplex half #将网卡设置为100M

1

ethtool -s em4 speed 100 duplex half #将网卡设置为100M
在交换机上

display interface brief | include 100M #应该可以看到有一个端口是100M的了，那么该端口就是em4网线使用的端口

1

display interface brief | include 100M #应该可以看到有一个端口是100M的了，那么该端口就是em4网线使用的端口
把网卡重新设置为自动协商模式：

ethtool -s em4 autoneg on

1

ethtool -s em4 autoneg on

然后配置交换机上该端口
配置服务器IP

参考：

http://blog.csdn.net/ggxiaobai/article/details/53583290

fuser or lsof

如果要卸载一个目录，那么目录下任何一个文件被占用都是不能卸载的，但是通常会不假思索地使用fuser 挂载目录，如挂载目录为 /data :

我们发现，fuser 并没有觉得谁在占用 /data ，然而lsof 却能知道；实际上fuser仅仅查看当前的inode有没有被占用，还是lsof更适合此场景

其实，fuser -m 选项是可以的; 而且还可以fuser -k 直接杀掉相关的进程；

注意fuser 选项-m与-M的区别：

-m 会根据指定的位置计算该位置所在的挂载点，然后列出该挂载点文件系统相关的进程

-M 必须指定一个确切的挂载点，对于目录可以没有’ / ‘；这个有时更加安全，有时候，对于已经卸载了的挂载点，就已经属于别的文件系统了

gops 工作原理

gops用来查看系统中存在的go进程。

注意事项：

gops有可能没发现你的go进程：

一方面可能进程是藏在容器中的，在宿主机上gops就发现不了

另一种情况，就是gops参考的是进程文件的特征，该特征在不同go版本可能不一样，比如，我的gops能发现go 1.11的go进程，就发现不了go 1.14的go进程

如下：

gops能list出来系统中的go进程，如果内置了agent的话，还能添加个星号，带星号的进程可以进行深度的分析（因为有agent呀）

那么，gops是如何找到所有go进程，又是如何识别出来agent的呢？

关键代码是这么写的：

// isGo looks up the runtime.buildVersion symbol
// in the process' binary and determines if the process
// if a Go process or not. If the process is a Go process,
// it reports PID, binary name and full path of the binary.
func isGo(pr ps.Process) (path, version string, agent, ok bool, err error) {
	if pr.Pid() == 0 {
		// ignore system process
		return
	}
	path, err = pr.Path()
	if err != nil {
		return
	}
	var versionInfo goversion.Version
	versionInfo, err = goversion.ReadExe(path)
	if err != nil {
		return
	}
	ok = true
	version = versionInfo.Release
	pidfile, err := internal.PIDFile(pr.Pid())
	if err == nil {
		_, err := os.Stat(pidfile)
		agent = err == nil
	}
	return path, version, agent, ok, nil
}

// isGo looks up the runtime.buildVersion symbol

// in the process' binary and determines if the process

// if a Go process or not. If the process is a Go process,

// it reports PID, binary name and full path of the binary.

func isGo(pr ps.Process) (path, version string, agent, ok bool, err error) {

if pr.Pid() == 0 {

// ignore system process

return

}

path, err = pr.Path()

if err != nil {

return

}

var versionInfo goversion.Version

versionInfo, err = goversion.ReadExe(path)

if err != nil {

return

}

ok = true

version = versionInfo.Release

pidfile, err := internal.PIDFile(pr.Pid())

if err == nil {

_, err := os.Stat(pidfile)

agent = err == nil

}

return path, version, agent, ok, nil

}

拿到所有的系统进程
分析进程文件可以检测是否go进程
在进程的用户目录下查找 .config/gops/{$pid} 文件，如果存在，就认为是有内置了agent了，该文件中存放的是agentlisten的端口号

如下：

display arp vs display mac-address

首先，对交换机的操作很不熟悉；

需求：我想从交换机上查看我的服务器是插在了交换机的哪个端口上，进而做一些设置。

已知：我知道我的服务器的mac地址

理论上来讲，交换机里面维护这一个mac <—-> interface 的对照表，我肯定是可以查出来的。

操作：记得display arp可以看到很多mac地址与interface的对应关系，如下：

注意到没？都是192.168.4.* 的，其实我要查看的是10.88.10.* 的，而且，10.88.10.* 属于vlan 4000；难道是： display arp vlan 4000 ？结果更不对

思考：

该交换机的ip为192.168.4.12，上面只显示192.168.4.* 是否和该交换机有192.168.4.12 有关系？如果该交换机上添加了10.88.10.* 的ip的话，是不是也能显示10.88.10.*的mac地址呢？应该是吧。但是交换机是工作正常的，交换机知道哪个mac的包去哪个端口的，这个不会已是否存在10.88.10.*而改变的。display arp可以显示所有的arp请求得到的mac地址，display arp vlan N 则，可以过滤某个vlan下的arp信息。其实：
arp 不等于 mac ，我想要的命令应该是 display mac-address ：

为什么ceph-mon不能放在基于ceph的openstack的虚拟机上

由于对ceph-mon不甚了解，做了下面一件事情：

openstack存储完全基于ceph存储
ceph存储节点单独部署
现有ceph-mon节点2个（mon-1，mon-2），想重装mon-1；但是，没有了ceph-1之后，ceph-2似乎认为脑裂了，不提供服务；于是想找个临时mon顶上
在openstack集群上很轻松地申请了一台机器，很快变安装了一个mon-3，由于安全组（没有放开6789）的原因，虽然mon-3能够找到mon-2，ceph -s也能看到，但是mon-1却因为不能连接到mon-3使得mon-3没有完全加入，调整安全组后，似乎一切变的正常，当mon-1去掉之后，ceph -s 卡住了，mon-3中正在执行的yum也卡住了
观察发现，mon-2 出于probing状态，mon-3出于reop状态，没有leader了
1. 因为mon-3 依赖ceph，但是ceph没有了leader，mon-3就不能落地数据
2. mon-3不能落地数据的话，mon-2、mon-3之间就没法选举出来leader
3. 如此，事情进入僵局，死锁了
分析：
1. 每个mon上都存在一个db，里面放着monmap信息，启动的时候，就根据monmap中的信息加入集群，如果monmap中只有自己，直接启动就可以了，如果有多个mon节点，并且当前不存在leader就得选举
2. 如果能把monmap中的mon节点修改成只有自己，就能正常启动
解决办法：（参考: http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-mon/#recovering-a-monitor-s-broken-monmap )
1. 停掉mon-2，不停掉是操作不了monmap的db的
2. 导出monmap：
  ceph-mon -i ID-FOO –extract-monmap /tmp/monmap
3. 查看monmap：
  monmaptool –print -f /tmp/monmap
4. 删除mon-3
  monmaptool –rm mon-3 -f /tmp/monmap
5. 注入monmap
  ceph-mon -i ID –inject-monmap /tmp/monmap
6. 启动mon-2

Inject a monmap into the monitor

Usually the safest path. You should grab the monmap from the remaining monitors and inject it into the monitor with the corrupted/lost monmap.

These are the basic steps:

Is there a formed quorum? If so, grab the monmap from the quorum:

$ ceph mon getmap -o /tmp/monmap

1

$ ceph mon getmap -o /tmp/monmap

No quorum? Grab the monmap directly from another monitor (this assumes the monitor you’re grabbing the monmap from has id ID-FOO and has been stopped):

$ ceph-mon -i ID-FOO --extract-monmap /tmp/monmap

1

$ ceph-mon -i ID-FOO --extract-monmap /tmp/monmap

Stop the monitor you’re going to inject the monmap into.

Inject the monmap:

$ ceph-mon -i ID --inject-monmap /tmp/monmap

1

$ ceph-mon -i ID --inject-monmap /tmp/monmap

Start the monitor

Please keep in mind that the ability to inject monmaps is a powerful feature that can cause havoc with your monitors if misused as it will overwrite the latest, existing monmap kept by the monitor.

http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-mon/

What if the state is probing?

This means the monitor is still looking for the other monitors. Every time you start a monitor, the monitor will stay in this state for some time while trying to find the rest of the monitors specified in the monmap. The time a monitor will spend in this state can vary. For instance, when on a single-monitor cluster, the monitor will pass through the probing state almost instantaneously, since there are no other monitors around. On a multi-monitor cluster, the monitors will stay in this state until they find enough monitors to form a quorum – this means that if you have 2 out of 3 monitors down, the one remaining monitor will stay in this state indefinitively until you bring one of the other monitors up.

If you have a quorum, however, the monitor should be able to find the remaining monitors pretty fast, as long as they can be reached. If your monitor is stuck probing and you have gone through with all the communication troubleshooting, then there is a fair chance that the monitor is trying to reach the other monitors on a wrong address. mon_status outputs the monmap known to the monitor: check if the other monitor’s locations match reality. If they don’t, jump to Recovering a Monitor’s Broken monmap; if they do, then it may be related to severe clock skews amongst the monitor nodes and you should refer to Clock Skews first, but if that doesn’t solve your problem then it is the time to prepare some logs and reach out to the community (please refer to Preparing your logs on how to best prepare your logs).

Start request repeated too quickly

参考： http://blog.csdn.net/styshoo/article/details/64091744

openstack 网络问题

同网段的机器ping的时候报没有路由的问题
1. arp -an查看，发现mac地址都没查到
2. 通过宿主机直接看虚拟机，发现虚拟机的网卡、ip都是在的
3. 解决：
  1. 虚拟机桥接的网桥重启过，而虚拟没有重启，相当于“网线断了”
4. 结论：
  1. 不要随便重启网桥，重启之后端口信息就没了，其下属的机器就相当于断了网线，后果很严重
openstack启动的机器我们都可以看到libvirtd的xml配置文件，xml配置文件中有关于网卡机器对应的tap的信息；我们直接virt-manager上通过桥接方式配置的网卡，在xml文件中找不到关于tap的名字的信息，但是对应的网桥下确实会多出来一个类似于vnet的网络设备