ablog

不器用で落着きのない技術者のメモ

Spark DataFrame の repartition メソッドは何をするものか

Spark DataFrame の repartition(パーティション数, カラム名) とすると指定したカラムで指定したパーティション数にパーティショニングする。パーティション数を省略するとデフォルト値(Spark 2.3.2 では 200)になる。

repartition(numPartitions, *cols)
Returns a new DataFrame partitioned by the given partitioning expressions. The resulting DataFrame is hash partitioned.

numPartitions can be an int to specify the target number of partitions or a Column. If it is a Column, it will be used as the first partitioning column. If not specified, the default number of partitions is used.

pyspark.sql module — PySpark 2.3.2 documentation

Other Configuration Options
The following options can also be used to tune the performance of query execution. It is possible that these options will be deprecated in future release as more optimizations are performed automatically.

Property Name Default Meaning
spark.sql.shuffle.partitions 200 Configures the number of partitions to use when shuffling data for joins or aggregations.
Spark SQL and DataFrames - Spark 2.3.2 Documentation

以下、実行例。
PySpark を起動して、S3 の JSON を読んで DataFrame を生成する。

$ pyspark

(中略)

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.3.2
      /_/

Using Python version 2.7.14 (default, May  2 2018 18:31:34)
SparkSession available as 'spark'.
>>> from pyspark.context import SparkContext
>>> from pyspark.sql import SQLContext
>>> from pyspark.sql.functions import year, month, date_format
>>>
>>> your_backet_name = "data-bucket"
>>> dataset = "sh10"
>>> in_path = "s3://{your_backet_name}/data/json/{dataset}/sales/*.gz".format(your_backet_name=your_backet_name, dataset=dataset)
>>> out_path = "s3://{your_backet_name}/data/parquet_pyspark/{dataset}/sales/".format(your_backet_name=your_backet_name, dataset=dataset)
>>>
>>> sc._jsc.hadoopConfiguration().set("mapreduce.fileoutputcommitter.algorithm.version", "2")
>>> sqlContext = SQLContext(sc)
>>> sqlContext.setConf("spark.sql.parquet.compression.codec", "snappy")
>>>
>>> df = sqlContext.read.json(in_path)

Dataframe の内容を表示する。

>>> df.show()
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+
|amount_sold|channel_id|courier_org| cust_id|fulfillment_center|prod_id|promo_id|quantity_sold|seller|tax_country|tax_region|            time_id|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+
|       82.0|         3|       1190|11635215|             13260|     62|     256|         44.0| 10691|         NQ|        HA|1995-01-01 00:00:00|
|       66.0|         3|       1459| 7887314|             14912|    102|     120|         84.0| 10944|         CI|        MU|1995-01-01 00:00:00|
|       16.0|         5|       1124| 7245615|             13368|     66|     136|         16.0| 10337|         YH|          |1995-01-01 00:00:00|
|       26.0|         2|       1319| 5363281|             14091|     59|     445|         93.0| 10863|         LH|        CU|1995-01-01 00:00:00|
|       91.0|         2|       1201|14565805|             13145|    126|     422|         21.0| 10568|         PH|        QW|1995-01-01 00:00:00|
|       72.0|         9|       1497|13073124|             14572|     43|     495|         57.0| 10778|         EW|        JB|1995-01-01 00:00:00|
|       83.0|         4|       1419|13223847|             13466|     15|     185|          6.0| 10153|         DW|          |1995-01-01 00:00:00|
|       28.0|         5|       1083| 8831083|             13148|     72|      91|         75.0| 10038|         NP|          |1995-01-01 00:00:00|
|       79.0|         2|       1364| 1122827|             14594|     76|     307|         83.0| 10896|         HR|          |1995-01-01 00:00:00|
|       36.0|         4|       1408|13016476|             14703|     65|      84|         70.0| 10116|         SB|        RV|1995-01-01 00:00:00|
|       65.0|         3|       1183| 6557310|             14384|     89|     362|         65.0| 10765|         CD|        OC|1995-01-01 00:00:00|
|        9.0|         9|       1023|11633931|             13586|     72|     316|         48.0| 10572|         JN|        LU|1995-01-01 00:00:00|
|        1.0|         5|       1515| 7868183|             14906|     97|     244|         98.0| 10605|         HE|        WO|1995-01-01 00:00:00|
|       93.0|         4|       1243| 4513870|             14918|     66|     170|         83.0| 10336|         XG|        US|1995-01-01 00:00:00|
|       13.0|         4|       1456| 9731389|             14339|     21|      90|         74.0| 10326|         TR|        NO|1995-01-01 00:00:00|
|        8.0|         9|       1541|12388703|             13591|    125|     124|         45.0| 10733|         AU|        AA|1995-01-01 00:00:00|
|       39.0|         9|       1221| 4713346|             14190|    111|     400|         43.0| 10979|         PD|          |1995-01-01 00:00:00|
|       48.0|         5|       1554| 4589564|             13727|    121|     331|         70.0| 10626|         AF|        PV|1995-01-01 00:00:00|
|        4.0|         4|       1048| 1051232|             14155|     48|     511|         36.0| 10224|         TF|          |1995-01-01 00:00:00|
|        7.0|         4|       1031|14342970|             14068|    130|     507|         29.0| 10128|         TU|          |1995-01-01 00:00:00|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+
only showing top 20 rows

DataFrame に year カラムを追加して表示する。

>>> yearAddedDf = df.withColumn("year", year(df.time_id))
>>> yearAddedDf.show()
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+
|amount_sold|channel_id|courier_org| cust_id|fulfillment_center|prod_id|promo_id|quantity_sold|seller|tax_country|tax_region|            time_id|year|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+
|       82.0|         3|       1190|11635215|             13260|     62|     256|         44.0| 10691|         NQ|        HA|1995-01-01 00:00:00|1995|
|       66.0|         3|       1459| 7887314|             14912|    102|     120|         84.0| 10944|         CI|        MU|1995-01-01 00:00:00|1995|
|       16.0|         5|       1124| 7245615|             13368|     66|     136|         16.0| 10337|         YH|          |1995-01-01 00:00:00|1995|
|       26.0|         2|       1319| 5363281|             14091|     59|     445|         93.0| 10863|         LH|        CU|1995-01-01 00:00:00|1995|
|       91.0|         2|       1201|14565805|             13145|    126|     422|         21.0| 10568|         PH|        QW|1995-01-01 00:00:00|1995|
|       72.0|         9|       1497|13073124|             14572|     43|     495|         57.0| 10778|         EW|        JB|1995-01-01 00:00:00|1995|
|       83.0|         4|       1419|13223847|             13466|     15|     185|          6.0| 10153|         DW|          |1995-01-01 00:00:00|1995|
|       28.0|         5|       1083| 8831083|             13148|     72|      91|         75.0| 10038|         NP|          |1995-01-01 00:00:00|1995|
|       79.0|         2|       1364| 1122827|             14594|     76|     307|         83.0| 10896|         HR|          |1995-01-01 00:00:00|1995|
|       36.0|         4|       1408|13016476|             14703|     65|      84|         70.0| 10116|         SB|        RV|1995-01-01 00:00:00|1995|
|       65.0|         3|       1183| 6557310|             14384|     89|     362|         65.0| 10765|         CD|        OC|1995-01-01 00:00:00|1995|
|        9.0|         9|       1023|11633931|             13586|     72|     316|         48.0| 10572|         JN|        LU|1995-01-01 00:00:00|1995|
|        1.0|         5|       1515| 7868183|             14906|     97|     244|         98.0| 10605|         HE|        WO|1995-01-01 00:00:00|1995|
|       93.0|         4|       1243| 4513870|             14918|     66|     170|         83.0| 10336|         XG|        US|1995-01-01 00:00:00|1995|
|       13.0|         4|       1456| 9731389|             14339|     21|      90|         74.0| 10326|         TR|        NO|1995-01-01 00:00:00|1995|
|        8.0|         9|       1541|12388703|             13591|    125|     124|         45.0| 10733|         AU|        AA|1995-01-01 00:00:00|1995|
|       39.0|         9|       1221| 4713346|             14190|    111|     400|         43.0| 10979|         PD|          |1995-01-01 00:00:00|1995|
|       48.0|         5|       1554| 4589564|             13727|    121|     331|         70.0| 10626|         AF|        PV|1995-01-01 00:00:00|1995|
|        4.0|         4|       1048| 1051232|             14155|     48|     511|         36.0| 10224|         TF|          |1995-01-01 00:00:00|1995|
|        7.0|         4|       1031|14342970|             14068|    130|     507|         29.0| 10128|         TU|          |1995-01-01 00:00:00|1995|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+
only showing top 20 rows

DataFrame に month を追加して表示する。

>>> monthAddedDf = yearAddedDf.withColumn("month", month(yearAddedDf.time_id))
>>> monthAddedDf.show()
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+
|amount_sold|channel_id|courier_org| cust_id|fulfillment_center|prod_id|promo_id|quantity_sold|seller|tax_country|tax_region|            time_id|year|month|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+
|       82.0|         3|       1190|11635215|             13260|     62|     256|         44.0| 10691|         NQ|        HA|1995-01-01 00:00:00|1995|    1|
|       66.0|         3|       1459| 7887314|             14912|    102|     120|         84.0| 10944|         CI|        MU|1995-01-01 00:00:00|1995|    1|
|       16.0|         5|       1124| 7245615|             13368|     66|     136|         16.0| 10337|         YH|          |1995-01-01 00:00:00|1995|    1|
|       26.0|         2|       1319| 5363281|             14091|     59|     445|         93.0| 10863|         LH|        CU|1995-01-01 00:00:00|1995|    1|
|       91.0|         2|       1201|14565805|             13145|    126|     422|         21.0| 10568|         PH|        QW|1995-01-01 00:00:00|1995|    1|
|       72.0|         9|       1497|13073124|             14572|     43|     495|         57.0| 10778|         EW|        JB|1995-01-01 00:00:00|1995|    1|
|       83.0|         4|       1419|13223847|             13466|     15|     185|          6.0| 10153|         DW|          |1995-01-01 00:00:00|1995|    1|
|       28.0|         5|       1083| 8831083|             13148|     72|      91|         75.0| 10038|         NP|          |1995-01-01 00:00:00|1995|    1|
|       79.0|         2|       1364| 1122827|             14594|     76|     307|         83.0| 10896|         HR|          |1995-01-01 00:00:00|1995|    1|
|       36.0|         4|       1408|13016476|             14703|     65|      84|         70.0| 10116|         SB|        RV|1995-01-01 00:00:00|1995|    1|
|       65.0|         3|       1183| 6557310|             14384|     89|     362|         65.0| 10765|         CD|        OC|1995-01-01 00:00:00|1995|    1|
|        9.0|         9|       1023|11633931|             13586|     72|     316|         48.0| 10572|         JN|        LU|1995-01-01 00:00:00|1995|    1|
|        1.0|         5|       1515| 7868183|             14906|     97|     244|         98.0| 10605|         HE|        WO|1995-01-01 00:00:00|1995|    1|
|       93.0|         4|       1243| 4513870|             14918|     66|     170|         83.0| 10336|         XG|        US|1995-01-01 00:00:00|1995|    1|
|       13.0|         4|       1456| 9731389|             14339|     21|      90|         74.0| 10326|         TR|        NO|1995-01-01 00:00:00|1995|    1|
|        8.0|         9|       1541|12388703|             13591|    125|     124|         45.0| 10733|         AU|        AA|1995-01-01 00:00:00|1995|    1|
|       39.0|         9|       1221| 4713346|             14190|    111|     400|         43.0| 10979|         PD|          |1995-01-01 00:00:00|1995|    1|
|       48.0|         5|       1554| 4589564|             13727|    121|     331|         70.0| 10626|         AF|        PV|1995-01-01 00:00:00|1995|    1|
|        4.0|         4|       1048| 1051232|             14155|     48|     511|         36.0| 10224|         TF|          |1995-01-01 00:00:00|1995|    1|
|        7.0|         4|       1031|14342970|             14068|    130|     507|         29.0| 10128|         TU|          |1995-01-01 00:00:00|1995|    1|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+
only showing top 20 rows

DataFrame に yyyyMM を追加して表示する。

>>> yyyymmAddedDf = monthAddedDf.withColumn("yyyymm", date_format(monthAddedDf.time_id, 'yyyyMM'))
>>> yyyymmAddedDf.show()
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+------+
|amount_sold|channel_id|courier_org| cust_id|fulfillment_center|prod_id|promo_id|quantity_sold|seller|tax_country|tax_region|            time_id|year|month|yyyymm|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+------+
|       82.0|         3|       1190|11635215|             13260|     62|     256|         44.0| 10691|         NQ|        HA|1995-01-01 00:00:00|1995|    1|199501|
|       66.0|         3|       1459| 7887314|             14912|    102|     120|         84.0| 10944|         CI|        MU|1995-01-01 00:00:00|1995|    1|199501|
|       16.0|         5|       1124| 7245615|             13368|     66|     136|         16.0| 10337|         YH|          |1995-01-01 00:00:00|1995|    1|199501|
|       26.0|         2|       1319| 5363281|             14091|     59|     445|         93.0| 10863|         LH|        CU|1995-01-01 00:00:00|1995|    1|199501|
|       91.0|         2|       1201|14565805|             13145|    126|     422|         21.0| 10568|         PH|        QW|1995-01-01 00:00:00|1995|    1|199501|
|       72.0|         9|       1497|13073124|             14572|     43|     495|         57.0| 10778|         EW|        JB|1995-01-01 00:00:00|1995|    1|199501|
|       83.0|         4|       1419|13223847|             13466|     15|     185|          6.0| 10153|         DW|          |1995-01-01 00:00:00|1995|    1|199501|
|       28.0|         5|       1083| 8831083|             13148|     72|      91|         75.0| 10038|         NP|          |1995-01-01 00:00:00|1995|    1|199501|
|       79.0|         2|       1364| 1122827|             14594|     76|     307|         83.0| 10896|         HR|          |1995-01-01 00:00:00|1995|    1|199501|
|       36.0|         4|       1408|13016476|             14703|     65|      84|         70.0| 10116|         SB|        RV|1995-01-01 00:00:00|1995|    1|199501|
|       65.0|         3|       1183| 6557310|             14384|     89|     362|         65.0| 10765|         CD|        OC|1995-01-01 00:00:00|1995|    1|199501|
|        9.0|         9|       1023|11633931|             13586|     72|     316|         48.0| 10572|         JN|        LU|1995-01-01 00:00:00|1995|    1|199501|
|        1.0|         5|       1515| 7868183|             14906|     97|     244|         98.0| 10605|         HE|        WO|1995-01-01 00:00:00|1995|    1|199501|
|       93.0|         4|       1243| 4513870|             14918|     66|     170|         83.0| 10336|         XG|        US|1995-01-01 00:00:00|1995|    1|199501|
|       13.0|         4|       1456| 9731389|             14339|     21|      90|         74.0| 10326|         TR|        NO|1995-01-01 00:00:00|1995|    1|199501|
|        8.0|         9|       1541|12388703|             13591|    125|     124|         45.0| 10733|         AU|        AA|1995-01-01 00:00:00|1995|    1|199501|
|       39.0|         9|       1221| 4713346|             14190|    111|     400|         43.0| 10979|         PD|          |1995-01-01 00:00:00|1995|    1|199501|
|       48.0|         5|       1554| 4589564|             13727|    121|     331|         70.0| 10626|         AF|        PV|1995-01-01 00:00:00|1995|    1|199501|
|        4.0|         4|       1048| 1051232|             14155|     48|     511|         36.0| 10224|         TF|          |1995-01-01 00:00:00|1995|    1|199501|
|        7.0|         4|       1031|14342970|             14068|    130|     507|         29.0| 10128|         TU|          |1995-01-01 00:00:00|1995|    1|199501|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+------+
only showing top 20 rows

yyyymm 列でパーティショニングして表示する。

>>> repartitionedDf = yyyymmAddedDf.repartition("yyyymm")
>>> repartitionedDf.show()
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+------+
|amount_sold|channel_id|courier_org| cust_id|fulfillment_center|prod_id|promo_id|quantity_sold|seller|tax_country|tax_region|            time_id|year|month|yyyymm|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+------+
|       16.0|         3|       1209|10459355|             14599|     87|     532|         28.0| 10706|         RQ|        XO|2000-02-01 00:00:00|2000|    2|200002|
|       77.0|         2|       1412| 5872809|             13794|     17|     286|         10.0| 10566|         LE|        DX|2000-02-01 00:00:00|2000|    2|200002|
|       85.0|         2|       1133| 1736310|             14520|    143|     232|         70.0| 10234|         GK|        RL|2000-02-01 00:00:00|2000|    2|200002|
|       16.0|         4|       1061| 2921916|             13141|    146|     498|         53.0| 10542|         UN|        JT|2000-02-01 00:00:00|2000|    2|200002|
|       74.0|         3|       1169| 7134487|             13887|     55|     198|         67.0| 10557|         XV|          |2000-02-01 00:00:00|2000|    2|200002|
|        1.0|         9|       1465| 9105076|             14640|     48|     363|          9.0| 10349|         LU|        QC|2000-02-01 00:00:00|2000|    2|200002|
|       73.0|         9|       1408| 5426572|             13072|     74|     339|         66.0| 10914|         BQ|        BD|2000-02-01 00:00:00|2000|    2|200002|
|       35.0|         4|       1013|11750382|             14504|    144|      89|         12.0| 10220|         FC|        UY|2000-02-01 00:00:00|2000|    2|200002|
|        3.0|         4|       1467|14292100|             14318|     70|     526|         77.0| 10458|         EW|        HL|2000-02-01 00:00:00|2000|    2|200002|
|       96.0|         4|       1255|14304714|             13334|     77|     460|         34.0| 10409|         EU|          |2000-02-01 00:00:00|2000|    2|200002|
|       74.0|         5|       1373|12033050|             14882|     74|     476|          3.0| 10191|         BC|        US|2000-02-01 00:00:00|2000|    2|200002|
|       81.0|         9|       1363| 5426406|             14751|     84|     195|         14.0| 10506|         RK|        EL|2000-02-01 00:00:00|2000|    2|200002|
|       32.0|         4|       1548| 3004427|             13914|     66|     266|         21.0| 10631|         DD|        TY|2000-02-01 00:00:00|2000|    2|200002|
|       79.0|         3|       1508| 6564626|             14302|     39|     520|         52.0| 10762|         RZ|        CN|2000-02-01 00:00:00|2000|    2|200002|
|       64.0|         3|       1478| 6779800|             13550|    110|     309|         38.0| 10794|         KY|        EN|2000-02-01 00:00:00|2000|    2|200002|
|       67.0|         5|       1542|14438640|             14058|     46|     514|         19.0| 10945|         AY|          |2000-02-01 00:00:00|2000|    2|200002|
|       38.0|         2|       1092|13135799|             13862|     31|     295|         72.0| 10349|         RD|          |2000-02-01 00:00:00|2000|    2|200002|
|       80.0|         5|       1178|  458983|             13180|     99|     526|         64.0| 10931|         IY|        QG|2000-02-01 00:00:00|2000|    2|200002|
|       68.0|         2|       1511| 7818776|             13687|    147|     506|         67.0| 10979|         UY|          |2000-02-01 00:00:00|2000|    2|200002|
|       71.0|         3|       1514|10769032|             14082|    101|     122|         36.0| 10187|         YE|          |2000-02-01 00:00:00|2000|    2|200002|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+------+
only showing top 20 rows

パーティショニングのために追加した yyyymm 列を削除する。

>>> droppedDf = repartitionedDf.drop("yyyymm")
>>> droppedDf.show()
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+
|amount_sold|channel_id|courier_org| cust_id|fulfillment_center|prod_id|promo_id|quantity_sold|seller|tax_country|tax_region|            time_id|year|month|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+
|       64.0|         3|       1238|  240661|             14588|    115|      79|         59.0| 10109|         IC|        QY|2000-02-01 00:00:00|2000|    2|
|       35.0|         5|       1305| 5279056|             13013|    131|      61|         11.0| 10523|         UG|        HU|2000-02-01 00:00:00|2000|    2|
|       18.0|         9|       1149|  562588|             14627|     87|     515|         91.0| 10146|         CE|        CY|2000-02-01 00:00:00|2000|    2|
|       50.0|         4|       1387|12492889|             13649|     73|     289|         67.0| 10447|         ZS|          |2000-02-01 00:00:00|2000|    2|
|       20.0|         4|       1457| 4023446|             14150|    125|     173|         94.0| 10694|         GE|        DB|2000-02-01 00:00:00|2000|    2|
|       24.0|         4|       1212|12139139|             13012|    140|     280|         67.0| 10111|         VV|        DJ|2000-02-01 00:00:00|2000|    2|
|       66.0|         3|       1311| 2633171|             14815|    106|     242|         21.0| 10388|         MJ|        BD|2000-02-01 00:00:00|2000|    2|
|       73.0|         5|       1511|10583767|             14488|    109|     531|         89.0| 10392|         PP|          |2000-02-01 00:00:00|2000|    2|
|       21.0|         3|       1363|  877617|             14501|     29|     311|         28.0| 10962|         GF|        FX|2000-02-01 00:00:00|2000|    2|
|       62.0|         4|       1549| 4429647|             13447|     33|     355|         52.0| 10699|         VC|        BF|2000-02-01 00:00:00|2000|    2|
|       16.0|         3|       1164|11654304|             13910|    125|     274|         25.0| 10337|         JI|        BI|2000-02-01 00:00:00|2000|    2|
|        5.0|         3|       1167| 4927333|             13123|     51|     508|         82.0| 10219|         FB|          |2000-02-01 00:00:00|2000|    2|
|        7.0|         3|       1001| 8434437|             14778|     18|     407|         19.0| 10192|         WD|        OW|2000-02-01 00:00:00|2000|    2|
|       53.0|         3|       1038|14058980|             14742|    111|     160|         92.0| 10527|         BM|        XN|2000-02-01 00:00:00|2000|    2|
|       55.0|         4|       1169| 9633844|             14862|     90|     322|          4.0| 10712|         HJ|        XC|2000-02-01 00:00:00|2000|    2|
|       80.0|         2|       1380| 1516950|             14065|     56|     166|          5.0| 10301|         BF|          |2000-02-01 00:00:00|2000|    2|
|       90.0|         9|       1039| 9921850|             14812|     46|     361|         23.0| 10062|         CY|          |2000-02-01 00:00:00|2000|    2|
|       40.0|         5|       1150| 7263915|             13059|     48|     116|         10.0| 10455|         CL|        OS|2000-02-01 00:00:00|2000|    2|
|       93.0|         4|       1131| 4353455|             13106|    110|     415|         83.0| 10533|         FJ|        AJ|2000-02-01 00:00:00|2000|    2|
|       97.0|         4|       1016|12261020|             14187|     27|     281|          1.0| 10495|         UO|        XY|2000-02-01 00:00:00|2000|    2|
+-----------+----------+-----------+--------+------------------+-------+--------+-------------+------+-----------+----------+-------------------+----+-----+
only showing top 20 rows

パーティション数を確認する。

>>> droppedDf.rdd.getNumPartitions()
200

参考

withColumn(colName, col)
Returns a new DataFrame by adding a column or replacing the existing column that has the same name.

The column expression must be an expression over this DataFrame; attempting to add a column from some other dataframe will raise an error.

  • Parameters:
    • colName – string, name of the new column.
    • col – a Column expression for the new column.
>>> df.withColumn('age2', df.age + 2).collect()
[Row(age=2, name='Alice', age2=4), Row(age=5, name='Bob', age2=7)]
pyspark.sql module — PySpark 2.3.2 documentation

repartition(numPartitions, *cols)
Returns a new DataFrame partitioned by the given partitioning expressions. The resulting DataFrame is hash partitioned.

numPartitions can be an int to specify the target number of partitions or a Column. If it is a Column, it will be used as the first partitioning column. If not specified, the default number of partitions is used.

Changed in version 1.6: Added optional arguments to specify the partitioning columns. Also made numPartitions optional if partitioning columns are specified.

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
pyspark.sql module — PySpark 2.3.2 documentation