ablog

不器用で落着きのない技術者のメモ

perf-map-agent 実行時に "AttachNotSupportedException: Unable to open socket file" と怒られる

事象

perf-map-agent をビルドして

$ sudo yum -y install cmake
$ git clone --depth=1 https://github.com/jvm-profiling-tools/perf-map-agent
$ cd perf-map-agent
$ cmake .
$ make
$ cd out

/etc/hadoop/conf/ 以下の hadoop-env.sh, yarn-env.sh にexport HADOOP_OPTS="... -XX:+PreserveFramePointer" export YARN_OPTS="... -XX:+PreserveFramePointer" を追加して、hdfsjvm を kill して再起動。ps で -XX:+PreserveFramePointer が入っていることを確認してから、hdfs のプロセスにアタッチしようとすると "AttachNotSupportedException: Unable to open socket file" と怒られる

$ ps -elf|grep hdfs
$ sudo -u hdfs java -cp attach-main.jar:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce 6362
Exception in thread "main" http://com.sun.tools .attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

調査

スモールケースで調べてみた。
カレントディレクトリを perf-map-agent/out にして、

$ cd /home/hadoop/perf-map-agent/out

DoWhileTrueJava.java を作成して

public class DoWhileTrueJava {
	public static void main(String[] args) throws InterruptedException {
		System.out.println("Start Processing inside do while loop");
		do {
			Thread.sleep(5 * 1000);
		} while (true);
	}
}

コンパイルして

$ javac DoWhileTrueJava.java

DoWhileTrueJava を実行して

$ java  DoWhileTrueJava &
[2] 3995
Start Processing inside do while loop

perf-map-agent で DoWhileTrueJava のJavaプロセスにアタッチしてみると、NoClassDefFoundError で怒られる。

$ java -cp attach-main.jar:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce 3995
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/tools/attach/AgentInitializationException
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.AgentInitializationException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more

あら、$JAVA_HOME/lib/tools.jar がない

$ ls -l $JAVA_HOME/lib/tools.jar
ls: cannot access /etc/alternatives/jre/lib/tools.jar: No such file or directory

探すと見つかったので、

$ sudo find /usr -name tools.jar
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64/lib/tools.jar

お、成功した

$ java -cp /home/hadoop/perf-map-agent/out/attach-main.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64/lib/tools.jar net.virtualvoid.perf.AttachOnce 3995

うまくいってますね

$ head /tmp/perf-3995.map
7fb5ad000060 1d flush_icache_stub
7fb5ad000160 21b get_cpu_info_stub
7fb5ad000420 3f forward exception
7fb5ad00045f e8 call_stub
7fb5ad000547 1f catch_exception
7fb5ad000566 5 atomic_xchg
7fb5ad00056b 7 atomic_xchg_ptr
7fb5ad000572 7 atomic_cmpxchg
7fb5ad000579 9 atomic_cmpxchg_long
7fb5ad000582 9 atomic_add

hdfs のプロセスに対しても成功

$ ps -elf|grep hdfs
$ sudo -u hdfs java -cp /home/hadoop/perf-map-agent/out/attach-main.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64/lib/tools.jar net.virtualvoid.perf.AttachOnce 8837

情報とれてますね

$ head /tmp/perf-8837.map
7f1989000060 1d flush_icache_stub
7f1989000160 21b get_cpu_info_stub
7f1989000420 3f forward exception
7f198900045f e8 call_stub
7f1989000547 1f catch_exception
7f1989000566 5 atomic_xchg
7f198900056b 7 atomic_xchg_ptr
7f1989000572 7 atomic_cmpxchg
7f1989000579 9 atomic_cmpxchg_long
7f1989000582 9 atomic_add