PHPor 的Blog – 第73页

json日志分析工具： jq

上篇文章介绍了jsawk，由于一些不足，这里又发现了一个更好的工具： jq

安装：

yum install -y jq

1	yum install -y jq

用法：

man jq

man jq

体验一下：

项目主页： https://stedolan.github.io/jq/

指南： https://stedolan.github.io/jq/tutorial/

手册： https://stedolan.github.io/jq/manual/

字符串连接

# echo '[{"from":"1","to":"2"},{"from":"1", "to":"2"}]'|jq -r '.[]| .from + ":" + .to'
1:2
1:2

# echo '[{"from":"1","to":"2"},{"from":"1", "to":"2"}]'|jq -r '.[]| .from + ":" + .to'

1:2

（注意：这里连接的是字符串，数字是不能直接当做字符串进行连接的）

类型转换：

# echo '[{"ID":1,"Name":"N1"},{"ID":2,"Name":"N2"}]'|jq '(.[].ID|tostring) + ":" + .[].Name'
"1:N1"
"2:N1"
"1:N2"
"2:N2"

# echo '[{"ID":1,"Name":"N1"},{"ID":2,"Name":"N2"}]'|jq '(.[].ID|tostring) + ":" + .[].Name'

"1:N1"

"2:N1"

"1:N2"

"2:N2"

这里通过圆括号来单独处理一部分，然后通过tostring将输出内容转换为字符串；也可以通过 @text 实现：

# echo '[{"ID":1,"Name":"N1"},{"ID":2,"Name":"N2"}]'|jq '(.[].ID|@text) + ":" + .[].Name'
"1:N1"
"2:N1"
"1:N2"
"2:N2"

# echo '[{"ID":1,"Name":"N1"},{"ID":2,"Name":"N2"}]'|jq '(.[].ID|@text) + ":" + .[].Name'

"1:N1"

"2:N1"

"1:N2"

"2:N2"

join:

# echo '[{"from":"1","to":"2"},{"from":"1", "to":"2"}]'|jq -r '.[]| join(":")' 
1:2
1:2

# echo '[{"from":"1","to":"2"},{"from":"1", "to":"2"}]'|jq -r '.[]| join(":")'

1:2

获取对象的key：

# echo '{"a":"A", "b":"B"}'|jq -r 'to_entries[].key'
a
b

# echo '{"a":"A", "b":"B"}'|jq -r 'to_entries[].key'

等效于

# echo '{"a":"A", "b":"B"}'|jq -r 'to_entries[] |.key'
a
b

# echo '{"a":"A", "b":"B"}'|jq -r 'to_entries[] |.key'

等效于：

# echo '{"a":"A", "b":"B"}'|jq -r 'to_entries | .[] |.key'
a
b

# echo '{"a":"A", "b":"B"}'|jq -r 'to_entries | .[] |.key'

过滤：（通过select 过滤出Name为default的条目，然后只显示名字和ID列）

# openstack security group list -fjson|jq '.[]|select(.Name == "default")|.Name + ":" + .ID'
"default:02dadaae-9d78-45ba-8fe5-496841b3c839"
"default:1bfd33d7-6278-4718-9680-fd6f3328561d"

# openstack security group list -fjson|jq '.[]|select(.Name == "default")|.Name + ":" + .ID'

"default:02dadaae-9d78-45ba-8fe5-496841b3c839"

"default:1bfd33d7-6278-4718-9680-fd6f3328561d"

错误的写法： (下面这个写法把Name和ID排列组合输出了)

openstack security group list -fjson|jq '.[].Name + ":" + .[].ID'

1	openstack security group list -fjson\|jq '.[].Name + ":" + .[].ID'

如：

# echo '[{"ID":"1","Name":"N1"},{"ID":"2","Name":"N2"}]'|jq '.[].ID + ":" + .[].Name'
"1:N1"
"2:N1"
"1:N2"
"2:N2"

# echo '[{"ID":"1","Name":"N1"},{"ID":"2","Name":"N2"}]'|jq '.[].ID + ":" + .[].Name'

"1:N1"

"2:N1"

"1:N2"

"2:N2"

根据path 和 select获取感兴趣的值：

# ceph status -f json|jq '(paths | select( .[-2:] == ["metadata", "ceph_version"])) as $path| getpath($path)'
"ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)"
"ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)"
"ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)"
"ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)"

# ceph status -f json|jq '(paths | select( .[-2:] == ["metadata", "ceph_version"])) as $path| getpath($path)'

"ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)"

参考： https://stackoverflow.com/questions/25780807/can-i-use-a-relative-path-or-a-wildcard-in-jq

关联数组的key需要使用双引号，不能使用单引号：

# echo '{"a":"A", "b":"B"}'|jq -r '.["a"]'
A

1 2	# echo '{"a":"A", "b":"B"}'\|jq -r '.["a"]' A

# echo '{"a":"A", "b":"B"}'|jq -r ".['a']"
jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?) at &lt;top-level&gt;, line 1:
.['a']
jq: 1 compile error

# echo '{"a":"A", "b":"B"}'|jq -r ".['a']"

jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?) at <top-level>, line 1:

.['a']

jq: 1 compile error

修改部分值

# echo '{"a":"A", "b":"B"}'|jq '.a="AA"' -c
{"a":"AA","b":"B"}

1 2	# echo '{"a":"A", "b":"B"}'\|jq '.a="AA"' -c {"a":"AA","b":"B"}

# echo '{"a":"A", "b":"B"}'|jq '.a="AA"|.b="BB"' -c
{"a":"AA","b":"BB"}

1 2	# echo '{"a":"A", "b":"B"}'\|jq '.a="AA"\|.b="BB"' -c {"a":"AA","b":"BB"}

在任何一个级别中查找指定字段：

echo '[[{"a":1}]]' | jq ´..|.a?´

1	echo '[[{"a":1}]]' \| jq ´..\|.a?´

构造新对象：

关于paths的用法: 1.3 版本中还没有paths：

这里的paths放在圆括号内，不影响输出，就好比bash中的圆括号是在单独子进程中执行一样，不影响当前的环境，同时，使用as使得圆括号的输出保存在变量中，不影响后续的输入；这样，使得getpath所在环境中的 dot 依然是原始的输入

参考资料： https://linuxtoy.org/archives/jq.html

Extract keys using JQ

https://github.com/stedolan/jq/issues/885

在线测试： https://jqplay.org/jq?

注意事项：

字符串只能使用双引号，不能使用单引号
if 语句必须有else, 如果其中一个分支实在没啥说的就 select(false) 相当于empty

jsawk

主页： https://github.com/micha/jsawk

用途：一直以来，日志都采用 | 分隔的多个匿名的字段组成的，每个字段代表什么含义是基于位置确定的，应该说是很不用户友好的，对于不太熟悉系统的新手是不太愿意看这样的日志的，所以，如果把日志记录成json格式的，应该说是一个进步；但是，对于经常使用awk分析基于|分隔的日志的我们来讲，awk分析json似乎就太不方便了，那么，jsawk就是我们需要的分析json的awk级别的工具

体验一下：

感受：直接把一个有很多行json的日志文件直接cat给jsawk是不行的，jsawk每次只解析一个json对象（或一个包含很多json对象的数组）；如果要聚合一些信息的话，还需要输出给另一个程序（awk？）进一步处理

要不改造一下jsawk？

stat荟萃

awstats

Advanced Web Statistics

ethstatus

Console-based ethernet statistics monitor

gitstats

Generates statistics based on GIT repository activity

iptstate

A top-like display of IP Tables state table entries

iostat

Report Central Processing Unit (CPU) statistics and input/output statistics for
devices, partitions and network filesystems (NFS)

netstat

netstat – Print network connections, routing tables, interface statistics, masquerade connec-
tions, and multicast membership

netstat-nat

A tool that displays NAT connections

ifstatus

Command line real time interface graphs using ncurses

pipestat

Watches data flowing over an anonymous pipe

vnstat

Console-based network traffic monitor

smemstat

Shared memory usage monitoring tool

sysstat

The sar and iostat system monitoring commands

statsd

A simple, lightweight network daemon to collect metrics over UDP

tcpstat reports certain network interface statistics much like vmstat does for system statistics. tcpstat gets its information by either monitoring a specific interface, or by reading previously saved tcpdump data from a file.

参考：

http://www.frenchfries.net/paul/tcpstat/

系列top

top

top – display Linux tasks

iotop

iotop – simple top-like I/O monitor

iftop

iftop – display bandwidth usage on an interface by host

ntop

ntop – display top network users

atop

atop – Advanced System & Process Monitor

ftop

ftop – show progress of open files and file systems

mytop

mytop – display MySQL server performance info like ‘top’

htop

htop – interactive process viewer

关于安装：如果幸运的话，都可以通过yum install 搞定

tcprstat

tcprstat is a free, open-source TCP analysis tool that watches network traffic and computes the delay between requests and responses.

tcprstat is related to the tcpstat tool, but it focuses on response time measurements, not on the amount and size of the network traffic. This makes it useful for response time analysis, which is needed for techniques such as Goal-Driven Performance Optimization.

The advantages of tcprstat are as follows:

It is lightweight and unobtrusive. No bulky log files need be written and analyzed.
Requests and responses are timed with microsecond resolution.
The output is easy to import into spreadsheets, manipulate with command-line scripts, graph with gnuplot, and so on.
It is protocol-agnostic, and works well for a large variety of client-server protocols that have a simple request-response model.

参考资料：

https://www.percona.com/docs/wiki/tcprstat:start