Jupyter Notebook の sparkmagic は matplotlib でグラフを描画できなかったので、DataFrame をファイルに出力して、conda_python3 で読んでグラフを描画した。
参考
Sparkmagic is a set of tools for interactively working with remote Spark clusters through Livy, a Spark REST server, in Jupyter notebooks. The Sparkmagic project includes a set of magics for interactively running Spark code in multiple languages, as well as some kernels that you can use to turn Jupyter into an integrated Spark environment.
GitHub - jupyter-incubator/sparkmagic: Jupyter magics and kernels for working with remote Spark clusters
（中略）
Sparkmagic uses Livy, a REST server for Spark, to remotely execute all user code. The library then automatically collects the output of your code as plain text or a JSON document, displaying the results to you as formatted text or as a Pandas dataframe as appropriate.
