Bucketing in PySpark

Bucketing groups rows into a fixed number of buckets using a hash of a column, such as customer_id. Rows with the same key are assigned to the same bucket, which can help reduce data movement when joining compatible bucketed tables.

For a customer-and-orders workload, write both tables using customer_id as the bucketing column and choose a compatible bucket configuration. The example in the infographic uses eight buckets on each side, sorts by customer_id, and saves the results as managed tables before joining them.

Bucketing is most useful for large tables that are joined repeatedly on the same key. It adds preparation and maintenance work, so small datasets, frequently rewritten tables and one-off queries may not benefit. Check the execution plan and measure performance on your workload rather than assuming every join will avoid shuffle.

Partitioning and bucketing solve different problems. Partitioning can reduce the data read when a query filters on partition values. Bucketing organises rows by a hash of the bucket key and can help with compatible joins. The supplied infographic illustrates the two approaches and the bucketed-table workflow.