ablog

不器用で落着きのない技術者のメモ

Linux で huge page を使っているプロセスを調べる方法

調べ方

  • /proc/[pid]/smaps で KernelPageSize がデフォルトの 4kB 以外のプロセスを探す。

調べ方の例

  • /proc/[pid]/smaps で KernelPageSize が 4 kB 以外のプロセスを探す。
[root@yazekats-linux ~]# perl -lane '/KernelPageSize/ and $F[1]!=4 and print qq/$ARGV\t$_/' /proc/*/smaps
/proc/3834/smaps	KernelPageSize:     2048 kB
  • PID:3834 のプロセス名を調べる。
[root@yazekats-linux ~]# ps -fp 3834
UID        PID  PPID  C STIME TTY          TIME CMD
root      3834  3832  0 13:44 pts/0    00:00:01 /home/yazekats/Documents/github/yoheia/misc/hugepage_test/hugepage-shm

おまけ

  • /proc/[pid]/smaps を覗いてみる
[root@yazekats-linux ~]# grep -C 12 'KernelPageSize:     2048 kB' /proc/3834/smaps 
2aaaaac00000-2aaabac00000 rw-s 00000000 00:0b 1277985                    /SYSV00000002 (deleted) <-- huge page を使っているアドレス(2aaaaac00000-2aaabac00000)やshmid(1277985)
Size:             262144 kB <-- huge page を使っている合計メモリサイズ
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:     2048 kB <-- OSカーネルでの論理的なページサイズ
MMUPageSize:        2048 kB <-- MMUでの物理的なページサイズ
Locked:                0 kB
7ffff7fe0000-7ffff7fe3000 rw-p 00000000 00:00 0 
Size:                 12 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
  • pmap コマンドでプロセスが使っているメモリの内訳を確認する
[root@yazekats-linux ~]# pmap -x 3834
3834:   /home/yazekats/Documents/github/yoheia/misc/hugepage_test/hugepage-shm
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000       4       4       4 r-x--  hugepage-shm
0000000000600000       4       4       4 rw---  hugepage-shm
0000003ba6200000     128     108      12 r-x--  ld-2.12.so
0000003ba641f000       4       4       4 r----  ld-2.12.so
0000003ba6420000       4       4       4 rw---  ld-2.12.so
0000003ba6421000       4       4       4 rw---    [ anon ]
0000003ba6600000    1580     236       0 r-x--  libc-2.12.so
0000003ba678b000    2044       0       0 -----  libc-2.12.so
0000003ba698a000      16       8       8 r----  libc-2.12.so
0000003ba698e000       4       4       4 rw---  libc-2.12.so
0000003ba698f000      20      12      12 rw---    [ anon ]
00002aaaaac00000  262144       0       0 rw-s-  SYSV00000002 (deleted) <-- ここが huge page を使っている
00007ffff7fe0000      12      12      12 rw---    [ anon ]
00007ffff7ffc000       8       8       8 rw---    [ anon ]
00007ffff7ffe000       4       4       0 r-x--    [ anon ]
00007ffffffde000     132       8       8 rw---    [ stack ]
ffffffffff600000       4       0       0 r-x--    [ anon ]
----------------  ------  ------  ------
total kB          266116     420      84
  • ipcs コマンドで共有メモリの使用状況を見る
[root@yazekats-linux ~]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 32768      yazekats   600        393216     2          dest         
0x00000000 65537      yazekats   600        393216     2          dest         
0x00000000 98306      yazekats   600        393216     2          dest         
0x00000000 131075     yazekats   600        393216     2          dest         
0x00000000 163844     yazekats   600        393216     2          dest         
0x00000000 196613     yazekats   600        393216     2          dest         
0x00000000 229382     yazekats   600        393216     2          dest         
0x00000000 262151     yazekats   600        393216     2          dest         
0x00000000 294920     yazekats   600        393216     2          dest         
0x00000000 327689     yazekats   600        393216     2          dest         
0x00000000 360458     yazekats   600        393216     2          dest         
0x00000000 393227     yazekats   600        393216     2          dest         
0x00000000 425996     yazekats   600        393216     2          dest         
0x00000000 458765     yazekats   600        393216     2          dest         
0x00000000 491534     yazekats   600        393216     2          dest         
0x00000000 524303     yazekats   600        393216     2          dest         
0x00000000 557072     yazekats   600        393216     2          dest         
0x00000000 589841     yazekats   600        393216     2          dest         
0x00000000 622610     yazekats   600        393216     2          dest         
0x00000000 655379     yazekats   600        393216     2          dest         
0x00000000 688148     yazekats   600        393216     2          dest         
0x00000000 720917     yazekats   600        393216     2          dest         
0x00000000 753686     yazekats   600        393216     2          dest         
0x00000000 786455     yazekats   600        393216     2          dest         
0x00000000 819224     yazekats   600        393216     2          dest         
0x00000000 851993     yazekats   600        393216     2          dest         
0x00000000 884762     yazekats   600        393216     2          dest         
0x00000000 1212443    yazekats   600        393216     2          dest         
0x00000000 1245212    yazekats   600        393216     2          dest         
0x00000000 983069     yazekats   600        393216     2          dest         
0x00000000 1015838    yazekats   600        393216     2          dest         
0x00000000 1048607    yazekats   600        393216     2          dest         
0x00000000 1081376    yazekats   600        393216     2          dest         
0x00000002 1277985    root       600        268435456  1 <-- ココ(shmid が 1277985)                     
0x00000000 1441826    yazekats   600        393216     2          dest         
0x00000000 1474595    yazekats   600        393216     2          dest         
0x00000000 1671204    yazekats   600        123904     2          dest         
0x00000000 2392101    yazekats   600        123904     2          dest         
0x00000000 2588710    yazekats   600        123904     2          dest         

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 163841     root       600        1         
0x74012120 458754     root       644        1         
0x2d012120 557059     root       600        1         

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    
  • shmid:1277985 にアタッチしたプロセスなどを見てみる。
[root@yazekats-linux ~]# ipcs -m -i 1277985

Shared memory Segment shmid=1277985
uid=0	gid=0	cuid=0	cgid=0
mode=0600	access_perms=0600
bytes=268435456	lpid=3834	cpid=3812	nattch=1 <-- PID:3812 が共有メモリセグメントを作成し、PID:3834 が最後にアタッチ。lpid: last-attached PID、cpid: creator PID
att_time=Sat Aug 16 13:44:17 2014  
det_time=Sat Aug 16 13:43:50 2014  
change_time=Sat Aug 16 13:43:07 2014  

再現手順

[root@yazekats-linux ~]#  sysctl -w vm.nr_hugepages=200
vm.nr_hugepages = 200
[root@yazekats-linux hugepage_test]# gdb ./hugepage-shm
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/yazekats/Documents/github/yoheia/misc/hugepage_test/hugepage-shm...done.
(gdb) b 89
Breakpoint 1 at 0x40086f: file hugepage-shm.c, line 89.
(gdb) r
Starting program: /home/yazekats/Documents/github/yoheia/misc/hugepage_test/hugepage-shm 
shmid: 0x138021
shmaddr: 0x2aaaaac00000
Starting the writes:
................................................................................................................................................................................................................................................................

Breakpoint 1, main () at hugepage-shm.c:89
89		dprintf("Done.\n");
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64