phpilock技巧_操作系统根本 文件系统分析对象

~# df -hFilesystem Size Used Avail Use% Mounted ondevtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.9G 0 1.9G 0% /dev/shmtmpfs 1.9G 41M 1.9G 3% /runtmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup/dev/sda1 50G 6.6G 44G 14% //dev/sdb 20G 45M 19G 1% /datatmpfs 379M 0 379M 0% /run/user/1000tmpfs 379M 0 379M 0% /run/user/0

可以关注Use%这一列，由于在文件系统利用率较高时会导致文件系统的性能低落，由于资源碎片等缘故原由会将顺序I/O降级为随机I/O。
但这也不是绝对的，须要看详细的文件系统的实现办法。

mout命令可以将文件系统挂载到系统上，也可以列出这些文件系统的类型和挂载参数：

~# mountsysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1915812k,nr_inodes=478953,mode=755)securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls)cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu)cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)configfs on /sys/kernel/config type configfs (rw,relatime)/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)debugfs on /sys/kernel/debug type debugfs (rw,relatime)hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)/dev/sdb on /data type ext4 (rw,noatime,seclabel,discard,nobarrier,data=ordered)tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=388020k,mode=700,uid=1000,gid=1001)systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=33198)binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=388020k,mode=700)free

free命令可以查看buff/cache占用内存的大小，利用-w参数可以将两者分开展示，如下所示

~# free -w total used free shared buffers cache availableMem: 3880176 336416 748160 41480 47996 2747604 3213132Swap: 0

free命令的输出结果中buffers表示的是buff cache大小，cache表示的page cache大小。

top命令中也可以展示buff/cache的大小，但是它们是被放在一起展示的，就像free命令不加-w一样，如下所示：

top - 22:40:39 up 87 days, 1:34, 2 users, load average: 0.00, 0.01, 0.05Tasks: 94 total, 1 running, 93 sleeping, 0 stopped, 0 zombie%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 stKiB Mem : 3880176 total, 747648 free, 336800 used, 2795728 buff/cacheKiB Swap: 0 total, 0 free, 0 used. 3212748 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 602 root 20 0 44168 2012 1668 S 0.3 0.1 87:14.45 qemu-ga 1 root 20 0 54432 6656 4020 S 0.0 0.2 30:02.09 systemdvmstat

vmstat命令也会展示buff/cache的大小，如下所示：

~# vmstat 1procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 748052 47996 2747748 0 0 0 1 1 1 0 0 100 0 0 0 0 0 748052 47996 2747748 0 0 0 0 104 100 0 0 100 0 0 0 0 0 748036 47996 2747764 0 0 0 0 147 131 0 0 100 0 0 0 0 0 748052 47996 2747764 0 0 0 0 87 87 0 0 100 0 0sar

sar工具供应了多种文件系统统计办法。
可以利用-v参数查看文件系统确当前活动：

~# sar -v 1Linux 3.10.0-1160.11.1.el7.x86_64 (xxxxxxx) 07/15/2022 _x86_64_(2 CPU)10:44:52 PM dentunusd file-nr inode-nr pty-nr10:44:53 PM 61343 992 56064 110:44:54 PM 61343 992 56064 110:44:55 PM 61343 992 56064 110:44:56 PM 61343 992 56064 1dentunusd：目录项缓存未利用计数（当前可用目缓存目录数）file-nr：打开的文件数inode-nr：利用的inode数pty-nr：利用伪终端数量

还有一个-r参数，可以展示kbbuffers和kbcached列。
详情可见内存分系工具的先容。

~# sar -r 1Linux 3.10.0-1160.11.1.el7.x86_64 (xxxxxxxx) 07/15/2022 _x86_64_(2 CPU)10:48:50 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty10:48:51 PM 748672 3214008 225316 5.81 47996 2568488 272064 7.01 1338660 1319316 010:48:52 PM 748648 3213984 225340 5.81 47996 2568488 272064 7.01 1338748 1319312 0slabtop

slabtop打印了内核slab缓存，这些缓存有可能是被文件系统缓存利用的:

~# slabtop -o Active / Total Objects (% used) : 1672622 / 1751976 (95.5%) Active / Total Slabs (% used) : 43270 / 43270 (100.0%) Active / Total Caches (% used) : 82 / 115 (71.3%) Active / Total Size (% used) : 265913.80K / 287453.69K (92.5%) Minimum / Average / Maximum Object : 0.01K / 0.16K / 15.25K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME552279 540917 97% 0.10K 14161 39 56644K buffer_head172482 162916 94% 0.04K 1691 102 6764K selinux_inode_security136576 131676 96% 0.03K 1067 128 4268K kmalloc-32136510 136510 100% 0.02K 803 170 3212K fsnotify_mark_connector122560 114275 93% 0.06K 1915 64 7660K kmalloc-64113408 108135 95% 0.02K 443 256 1772K kmalloc-16 90300 79993 88% 0.19K 4300 21 17200K dentry 84480 84480 100% 0.01K 165 512 660K kmalloc-8 75633 72534 95% 0.94K 4449 17 71184K xfs_inode 72266 68579 94% 0.17K 3142 23 12568K xfs_ili 43232 38448 88% 1.00K 2702 16 43232K kmalloc-1024

关心的内容：

文件系统延迟在系统调用接口层面利用strace工具。
strace当前是基于ptrace实现的，会严重影响性能，利用时要把稳。
如下所示，利用strace追踪crond进程查看其读取文件的延时。

~# strace -ttT -p 21621strace: Process 21621 attached[...]23:04:01.944437 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=556, ...}) = 0 <0.000026>[...] <detached ...>perf

perf也是一个强大的工具，可以追中文件系统的tracepoint，利用kprobe查看VFS和文件系统的内部，并且还有一个trace子命令比strace高效的多。

~# perf trace cksum /usr/bin/cksum759126066 33152 /usr/bin/cksum 0.036 ( 0.002 ms): cksum/9147 brk() = 0x1d13000 0.078 ( 0.005 ms): cksum/9147 mmap(len: 4096, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS) = 0x7f42421d5000 0.096 ( 0.008 ms): cksum/9147 access(filename: 0x41fd3cb0, mode: R) = -1 ENOENT (No such file or directory) 0.117 ( 0.006 ms): cksum/9147 open(filename: 0x41fd25a4, flags: RDONLY|CLOEXEC) = 3 0.125 ( 0.003 ms): cksum/9147 fstat(fd: 3, statbuf: 0x7ffd11e0bb70) = 0 0.129 ( 0.004 ms): cksum/9147 mmap(len: 35679, prot: READ, flags: PRIVATE, fd: 3)

可以利用perf list命令列出感兴趣的tracepiont或者kprobe：

~# perf list xfsList of pre-defined events (to be used in -e): xfs:xfs_agf [Tracepoint event] xfs:xfs_agfl_reset [Tracepoint event] xfs:xfs_ail_delete [Tracepoint event] xfs:xfs_ail_flushing [Tracepoint event] xfs:xfs_ail_insert [Tracepoint event] xfs:xfs_ail_locked [Tracepoint event] xfs:xfs_ail_move [Tracepoint event] xfs:xfs_ail_pinned [Tracepoint event]

可以利用perf stat统计xfs trancepoint调用，如下所示：

> perf stat -e 'xfs:' -a^C Performance counter stats for 'system wide': 1 xfs:xfs_ilock 1 xfs:xfs_iunlock 1 xfs:xfs_filemap_fault 1 xfs:xfs_log_force 21.508754970 seconds time elapsed

还可以利用perf record将获取结果存如文件，利用perf script读取文件内容进行剖析。

fatrace利用Linux的fanotify API，示例输出：

~# fatracersyslogd(968876): W /var/log/syslogrsyslogd(968876): W /var/log/auth.loggsd-color(1690): CO /usr/share/zoneinfo/Etc/UTCgsd-color(1690): CO /usr/share/zoneinfo/Etc/UTCgsd-housekeepin(1707): RO /etc/fstabgsd-housekeepin(1707): C /etc/fstab

每一行显示了进程名、PID、事宜类型、全路径和可选的状态。
韶光类型包含：

例如：CW 代表关闭一个写文件，CO关闭一个打开文件

跟踪文件打开事宜

~# opensnoop-bpfcc -TTIME(s) PID COMM FD ERR PATH0.000000000 1700193 hidsagent 4 0 /proc/net/dev0.000075000 1700193 hidsagent 4 0 /proc/stat0.000117000 1700193 hidsagent 4 0 /proc/1700193/stat0.000148000 1700193 hidsagent 4 0 /proc/1700193/task0.000213000 1700193 hidsagent 14 0 /proc/1700193/task/1700193/stat0.000228000 1700193 hidsagent 14 0 /proc/1700193/task/1700194/stat0.000242000 1700193 hidsagent 14 0 /proc/1700193/task/1700196/stat

BCC命令用法

opensnoop [options]

命令选项

bpftrace版本不支持任何参数，其源码如下所示：

BEGIN{ printf("Tracing open syscalls... Hit Ctrl-C to end.\n"); printf("%-6s %-16s %4s %3s %s\n", "PID", "COMM", "FD", "ERR", "PATH");}tracepoint:syscalls:sys_enter_open,tracepoint:syscalls:sys_enter_openat{ @filename[tid] = args->filename;}tracepoint:syscalls:sys_exit_open,tracepoint:syscalls:sys_exit_openat/@filename[tid]/{ $ret = args->ret; $fd = $ret > 0 ? $ret : -1; $errno = $ret > 0 ? 0 : - $ret; printf("%-6d %-16s %4d %3d %s\n", pid, comm, $fd, $errno, str(@filename[tid])); delete(@filename[tid]);}END{ clear(@filename);}

实行结果示例：

~# opensnoop.btAttaching 6 probes...Tracing open syscalls... Hit Ctrl-C to end.PID COMM FD ERR PATH884 snmpd 2 0 /proc/net/dev884 snmpd 2 0 /proc/net/if_inet6884 snmpd 2 0 /sys/class/net/eth0/device/vendor884 snmpd 2 0 /sys/class/net/eth0/device/devicestatsnoop

针对stats系统调用

BCC版本

~# statsnoop-bpfccPID COMM FD ERR PATH2582812 zabbix_agentd 0 0 /var/log/zabbix/zabbix_agentd.log2582812 zabbix_agentd 0 0 /etc/resolv.conf1700214 hidsagent 0 0 /home/sfop/security/hids/log/net_monitor.log1700214 hidsagent 0 0 /home/sfop/security/hids/log/net_monitor.log1700214 hidsagent 0 0 /home/sfop/security/hids/log/net_monitor.log1700214 hidsagent 0 0 /home/sfop/security/hids/log/net_monitor.log2582816 zabbix_agentd 0 0 /var/log/zabbix/zabbix_agentd.log

BCC命令行用法

statsnoop [options]

命令行选项

该版本不支持任何选项，利用示例如下：

~# statsnoop.btAttaching 10 probes...Tracing stat syscalls... Hit Ctrl-C to end.PID COMM ERR PATH925 cron 0 /etc/localtime925 cron 0 crontabs925 cron 0 /etc/crontab925 cron 0 /etc/cron.d925 cron 0 /etc/cron.d/php925 cron 0 /etc/cron.d/popularity-contest925 cron 0 /etc/cron.d/e2scrub_all925 cron 0 /etc/cron.d/sysstat1707 gsd-housekeepin 0 /etc/fstab1707 gsd-housekeepin 0 /proc1707 gsd-housekeepin 0 /proc/self