事象
perf-map-agent をビルドして
$ sudo yum -y install cmake $ git clone --depth=1 https://github.com/jvm-profiling-tools/perf-map-agent $ cd perf-map-agent $ cmake . $ make $ cd out
/etc/hadoop/conf/ 以下の hadoop-env.sh, yarn-env.sh にexport HADOOP_OPTS="... -XX:+PreserveFramePointer" export YARN_OPTS="... -XX:+PreserveFramePointer" を追加して、hdfs の jvm を kill して再起動。ps で -XX:+PreserveFramePointer が入っていることを確認してから、hdfs のプロセスにアタッチしようとすると "AttachNotSupportedException: Unable to open socket file" と怒られる
$ ps -elf|grep hdfs $ sudo -u hdfs java -cp attach-main.jar:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce 6362 Exception in thread "main" http://com.sun.tools .attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
調査
スモールケースで調べてみた。
カレントディレクトリを perf-map-agent/out にして、
$ cd /home/hadoop/perf-map-agent/out
DoWhileTrueJava.java を作成して
public class DoWhileTrueJava { public static void main(String[] args) throws InterruptedException { System.out.println("Start Processing inside do while loop"); do { Thread.sleep(5 * 1000); } while (true); } }
コンパイルして
$ javac DoWhileTrueJava.java
DoWhileTrueJava を実行して
$ java DoWhileTrueJava & [2] 3995 Start Processing inside do while loop
perf-map-agent で DoWhileTrueJava のJavaプロセスにアタッチしてみると、NoClassDefFoundError で怒られる。
$ java -cp attach-main.jar:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce 3995 Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/tools/attach/AgentInitializationException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.AgentInitializationException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more
あら、$JAVA_HOME/lib/tools.jar がない
$ ls -l $JAVA_HOME/lib/tools.jar ls: cannot access /etc/alternatives/jre/lib/tools.jar: No such file or directory
探すと見つかったので、
$ sudo find /usr -name tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64/lib/tools.jar
お、成功した
$ java -cp /home/hadoop/perf-map-agent/out/attach-main.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64/lib/tools.jar net.virtualvoid.perf.AttachOnce 3995
うまくいってますね
$ head /tmp/perf-3995.map 7fb5ad000060 1d flush_icache_stub 7fb5ad000160 21b get_cpu_info_stub 7fb5ad000420 3f forward exception 7fb5ad00045f e8 call_stub 7fb5ad000547 1f catch_exception 7fb5ad000566 5 atomic_xchg 7fb5ad00056b 7 atomic_xchg_ptr 7fb5ad000572 7 atomic_cmpxchg 7fb5ad000579 9 atomic_cmpxchg_long 7fb5ad000582 9 atomic_add
hdfs のプロセスに対しても成功
$ ps -elf|grep hdfs $ sudo -u hdfs java -cp /home/hadoop/perf-map-agent/out/attach-main.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64/lib/tools.jar net.virtualvoid.perf.AttachOnce 8837
情報とれてますね
$ head /tmp/perf-8837.map 7f1989000060 1d flush_icache_stub 7f1989000160 21b get_cpu_info_stub 7f1989000420 3f forward exception 7f198900045f e8 call_stub 7f1989000547 1f catch_exception 7f1989000566 5 atomic_xchg 7f198900056b 7 atomic_xchg_ptr 7f1989000572 7 atomic_cmpxchg 7f1989000579 9 atomic_cmpxchg_long 7f1989000582 9 atomic_add