PHPor 的Blog

docker & cache

缘起：

dubbo-monitor容器swap使用率超过 20% ，而且在不断上升，为什么呢？

下面是容器dubbo-monitor在drop-caches前后的对比：

CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
dubbo_monitor 6.25% 1.4 GiB / 2 GiB 69.99% 0 B / 0 B 6.306 GB / 54.5 GB 0

[root@CX-DOCKER-BASE-1-10.139.105.202 ~]# echo 3 &gt; /proc/sys/vm/drop_caches

[root@CX-DOCKER-BASE-1-10.139.105.202 ~]# docker stats dubbo_monitor
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
dubbo_monitor 3.87% 595.7 MiB / 2 GiB 29.09% 0 B / 0 B 6.373 GB / 54.5 GB 0

CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

dubbo_monitor 6.25% 1.4 GiB / 2 GiB 69.99% 0 B / 0 B 6.306 GB / 54.5 GB 0

[root@CX-DOCKER-BASE-1-10.139.105.202 ~]# echo 3 > /proc/sys/vm/drop_caches

[root@CX-DOCKER-BASE-1-10.139.105.202 ~]# docker stats dubbo_monitor

CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

dubbo_monitor 3.87% 595.7 MiB / 2 GiB 29.09% 0 B / 0 B 6.373 GB / 54.5 GB 0

drop-cache后，容器的swap使用率没有变化

结论：

docker stats 显示的内存使用是包含cache的
~~该容器的部分cache信息会被swap出内存~~
1. cache和swap之间存在一些不默契；cache认为内存够多，还可以cache更多信息；swap机制为，经内存充裕，只要是不常被访问的内存数据，尽量swap出来；于是乎，就形成了一个恶性循环，导致swap占用越来越大
2. ~~解决办法：定期drop一下cache~~
系统不会傻到把cache内容换出到swap中的，swap确实为进程使用的（虚拟）内存的一部分
一个允许最大使用2g内存的java进程，swap出来500m或1g不足为奇，如果给他的swap大小为2g的话，占用50%的swap都有可能

awk替换redis aof中db的方法

测试数据：

# cat appendonly.6387.aof|head -n 20
*2
$6
SELECT
$1
1
*3
$3
SET
$47
mission_58c7be1b47ddd5.20005951_start_wait_time
$10
1489485343
*3
$3
SET
$47
mission_577239de3d58a2.51913045_start_wait_time
$10
1467105181

# cat appendonly.6387.aof|head -n 20

SELECT

SET

$47

mission_58c7be1b47ddd5.20005951_start_wait_time

$10

1489485343

SET

$47

mission_577239de3d58a2.51913045_start_wait_time

$10

1467105181

测试目标：把 db 1 中的数据导入到 db 10 里面去

awk脚本：

BEGIN{ll="";l="";}
{
if($0=="SELECT\r"){
	ll="SELECT\r";
}else if(ll=="SELECT\r"){
	if (l=="") {
		l=$0;
	}else {
		if($0 == "1\r"){
			l="$2\r";$0="10\r";
		}
		print ll; print l; print $0;
		ll="";l="";
	}
}else {
	print $0;
}
}

BEGIN{ll="";l="";}

{

if($0=="SELECT\r"){

ll="SELECT\r";

}else if(ll=="SELECT\r"){

if (l=="") {

l=$0;

}else {

if($0 == "1\r"){

l="$2\r";$0="10\r";

}

print ll; print l; print $0;

ll="";l="";

}

}else {

print $0;

}

测试：

# cat a.aof | awk -f /tmp/a.awk
*2
$6
SELECT
$2
10
*3
$3
SET
$47
mission_58c7be1b47ddd5.20005951_start_wait_time
$10
1489485343
*3
$3
SET
$47
mission_577239de3d58a2.51913045_start_wait_time
$10
1467105181

# cat a.aof | awk -f /tmp/a.awk

SELECT

SET

$47

mission_58c7be1b47ddd5.20005951_start_wait_time

$10

1489485343

SET

$47

mission_577239de3d58a2.51913045_start_wait_time

$10

1467105181

一般情况下，这样做事没问题的，特殊情况下会不好使（加入数据中包含极似协议的内容）

spring 注解之 @ConfigurationProperties

当一个类添加了如下注解的话，装配类的时候，其setter方法（注意，确切说不是属性）就会参考application.properties 文件总对应的配置的，而且，在IDE中添加这种配置时，IDE也会给出提示

ngx.resp.get_headers()

syntax: headers = ngx.resp.get_headers(max_headers?, raw?)

context: set_by_lua*, rewrite_by_lua*, access_by_lua*, content_by_lua*, header_filter_by_lua*, body_filter_by_lua*, log_by_lua*, balancer_by_lua*

Returns a Lua table holding all the current response headers for the current request.

 <span class="pl-k">local</span> h <span class="pl-k">=</span> ngx.<span class="pl-smi">resp</span>.<span class="pl-c1">get_headers</span>()
 <span class="pl-k">for</span> k, v <span class="pl-k">in</span> <span class="pl-c1">pairs</span>(h) <span class="pl-k">do</span>
     <span class="pl-c1">...</span>
 <span class="pl-k">end</span>

				
					
				1
2
3
4

						 <span class="pl-k">local</span> h <span class="pl-k">=</span> ngx.<span class="pl-smi">resp</span>.<span class="pl-c1">get_headers</span>()
 <span class="pl-k">for</span> k, v <span class="pl-k">in</span> <span class="pl-c1">pairs</span>(h) <span class="pl-k">do</span>
     <span class="pl-c1">...</span>
 <span class="pl-k">end</span>

					

			

This function has the same signature as ngx.req.get_headers except getting response headers instead of request headers.

参考: https://github.com/openresty/lua-nginx-module#ngxrespget_headers

注意：字段名都会转换成小写

expires:Thu, 19 Nov 1981 08:52:00 GMT
content-type:text/html; charset=utf-8
connection:keep-alive
cache-control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
pragma:no-cache
content-encoding:gzip

expires:Thu, 19 Nov 1981 08:52:00 GMT

content-type:text/html; charset=utf-8

connection:keep-alive

cache-control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0

pragma:no-cache

content-encoding:gzip

将pillar信息存入mysql：

https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.mysql.html

其实，还有现成的模块可以将pillar信息存储mongodb、ldap、redis等

https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.pillar_ldap.html

https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.redismod.html

https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.postgres.html

https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.git_pillar.html

持续构建之maven + Jenkins + Nexus

一般来讲，maven deploy时要部署到Nexus，需要在pom.xml 中添加如下类似的东东：

    <distributionManagement>
        <repository>
            <id>releases</id>
            <name>Nexus Release Repository</name>
            <url>http://nexus.phpor.net/nexus/content/repositories/releases/</url>
        </repository>
        <snapshotRepository>
            <id>snapshots</id>
            <name>Nexus Snapshot Repository</name>
            <url>http://nexus.phpor.net/nexus/content/repositories/snapshots/</url>
        </snapshotRepository>
    </distributionManagement>

<id>releases</id>

<name>Nexus Release Repository</name>

<url>http://nexus.phpor.net/nexus/content/repositories/releases/</url>

</repository>

<id>snapshots</id>

<name>Nexus Snapshot Repository</name>

<url>http://nexus.phpor.net/nexus/content/repositories/snapshots/</url>

</snapshotRepository>

</distributionManagement>

如此的话，该pom.xml 就不便于分享，还有人喜欢deploy到自己的Nexus呢；

其实，还可以在mvn deploy 时通过参数的方式来指定，如：

mvn deploy -D altDeploymentRepository=snapshots::default::http://nexus.phpor.net/nexus/content/repositories/snapshots/

1	mvn deploy -D altDeploymentRepository=snapshots::default::http://nexus.phpor.net/nexus/content/repositories/snapshots/

这样的话，不管snapshot还是release都会deploy到相同位置；正确的姿势为分别设置snapshot和release的位置，如下：

mvn deploy -D altSnapshotDeploymentRepository=snapshots::default::http://nexus.phpor.net/nexus/content/repositories/snapshots/
    -D altReleaseDeploymentRepository=releases::default::http://nexus.phpor.net/nexus/content/repositories/releases/

1 2	mvn deploy -D altSnapshotDeploymentRepository=snapshots::default::http://nexus.phpor.net/nexus/content/repositories/snapshots/ -D altReleaseDeploymentRepository=releases::default::http://nexus.phpor.net/nexus/content/repositories/releases/

参考：https://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html

docker swarm 模式下证书过期后daemon无法启动

报错信息：

Error creating cluster component: error while loading TLS Certificate

1	Error creating cluster component: error while loading TLS Certificate

解决办法：把 Docker Root Dir (通过docker info查看）下的swarm目录删掉就好了

xfire 代理设置

xfire 可以通过如下方式设置代理：

client.setProperty(CommonsHttpMessageSender.HTTP_PROXY_HOST, httpProxyHost);
client.setProperty(CommonsHttpMessageSender.HTTP_PROXY_PORT, httpProxyPort);

1 2	client.setProperty(CommonsHttpMessageSender.HTTP_PROXY_HOST, httpProxyHost); client.setProperty(CommonsHttpMessageSender.HTTP_PROXY_PORT, httpProxyPort);

需要注意的是，该代理配置并不作用于下载wsdl