DBR 7.x / Spark 3.

x
Features &
Migration
Prashanth Babu
Senior Resident Solutions Architect
Created: 25th Nov 2020
Last updated: 20th Jan 2021

Credits to respective contributors

Context

2
Why should you migrate to DBR 7?
❖ Support for DBR 6.4 (Spark 2.4.x) ends on 1st April, 2021.
➢ DBR 6 will not be available post this cutoff

3
Why should you migrate to DBR 7?
❖ Support for DBR 6.4 (Spark 2.4.x) ends on 1st April, 2021.
➢ DBR 6 will not be available post this cutoff
➢ You will not see this version in the Clusters dropdown menu or
using REST API.
■ But your existing jobs on the earlier versions will continue
to run fine.

4
Why should you migrate to DBR 7?
❖ DBR 7.3 (Spark 3.0.1) is the new LTS in 7.x release line

https://docs.microsoft.com/en-us/azure/databricks/release-notes/runtime/releases
https://docs.databricks.com/release-notes/runtime/releases.html
5
Performance SQL Compatibility

Adaptive Query Dynamic Partition 40% Compiler Cache Sync Reserved Keywords Gregorian Store Assignment Overflow
Execution Pruning Time Reduction Minimization in Parser Calendar in INSERT Checking

Feature Connector

Accelerator-aware JDK 11 Built-in Parquet/ORC: Nested New Binary New NOOP CSV Filter
Join Hints Functions
Scheduler Support Column Pruning Data Source Data Source Pushdown

Monitoring PySpark and SparkR

Structured JDBC tab Observable Event log New Pandas UDF Pandas UDF Eager Execution Vectorization in
Streaming UI in SHS Metrics Rollover using Type Hints Enhancements in R Shell SparkR

Usability and Stability Extensibility and Document

Explain Describe Dump Test Data Source APIs Hive 3.x Metastore Hadoop 3 SQL
Formatted Query Plan Coverage + Catalog Support Hive 2.3 Execution Support Reference
6
Agenda
Performance Spark ML new features
3.0 comes with performance improvements to
New features in Spark ML
make Spark faster, cheaper, and more flexible
▪ Adaptive Query Execution Migration and compatibility
▪ Dynamic Partition Pruning Important compatibility / behavior changes for
▪ Join Optimization Hints migration
Usability Spark Ecosystem
New features make Spark even easier to use ▪ New features in Delta
▪ Spark SQL: Explain Plans ▪ Third party connectors

Pandas enhancements References

Enables better usage of Pandas API and improves Links to Summit sessions and blogposts for further
performance reading

7
https://dbricks.co/ebLearningSpark

8
Performance

9
Adaptive Query Execution
(AQE)

10
Adaptive Query Execution (AQE)
Re-optimizes queries based on the most up-to-date runtime statistics

Spark 1.x --> Rule

Spark 2.x --> Rule + Cost

Spark 3.0 --> Rule + Cost + Runtime

11
Optimization in Spark 2.x

12
Adaptive Query Execution

adaptive planning
Based on statistics of the finished plan nodes, re-optimize the execution
plan of the remaining queries
▪ Dynamically switch join strategies
▪ Dynamically coalesce shuffle partitions
▪ Dynamically optimize skew joins
13
Performance Pitfall
Using the wrong join strategy
▪ Choose Broadcast Hash Join?
▪ Increase “ ”?
▪ Use “broadcast” hint?
However
▪ Hard to tune
▪ Hard to maintain over time
▪ OOM…
14
Adaptive Query Execution
Vision: No more manual setting of broadcast hints/thresholds! Capability:
SMJ -> BHJ at runtime

Actual:
8MB

Static size:
15MB

15
Performance Pitfall
Choosing the wrong shuffle partition number
▪ Tuning
▪ Default magic number: 200 !?!
However
▪ Too small: GC pressure; disk spilling
▪ Too large: Inefficient I/O; scheduler pressure
▪ Hard to tune over the whole query plan
▪ Hard to maintain over time
16
Adaptive Query Execution
VISION: No more manual tuning of spark.shuffle.partitions! Capability: Coalesce shuffle
partitions
▪ Set the initial partition number high to accommodate the largest data
size of the entire query execution
▪ Automatically coalesce partitions if needed after each query stage

Execute Optimize

Stage 1 Stage 1 Stage 1

17
Performance Pitfall
Data skew
▪ Symptoms of data skew
▪ Frozen/long-running tasks
▪ Disk spilling
▪ Low resource utilization in most nodes
▪ OOM
▪ Various ways
▪ Find the skew values and rewrite the queries
▪ Adding extra skew keys…
18
Adaptive Query Execution
Data Skew

19
Adaptive Query Execution
VISION: No more manual tuning of skew hints!

20
Skew Join and AQE Demo

21
AQE Configuration Settings
AQE is enabled by default in DBR 7.3 LTS and above: Blogpost

spark.sql.adaptive.
coalescePartitions.
enabled

spark.sql.adaptive.
coalescePartitions.
minPartitionNum

spark.sql.adaptive.
coalescePartitions.
initialPartitionNum

spark.sql.adaptive.
advisoryPartitionSizeInBytes

22
 Key Takeaways - AQE
▪ Adaptive Query Execution is an extensible framework

▪ It’s akin to writing rules for the Catalyst Optimizer

▪ It is enabled by default in DBR 7.3 LTS and above.

▪ For earlier versions, it must be enabled by setting
to

▪ Other AQE features may default to enabled, but are still gated by this master
configuration flag

Link to blogpost: Databricks blogpost

23
Dynamic Partition
Pruning

24
Static Partition Pruning
Most optimizations employ simple static partition pruning 

SELECT * FROM Sales WHERE store_id = 5

Partitioned files with

Basic Data Flow Filter Push-down multi-columnar data
25
A Common Workload
Star Schema Queries

Join

SELECT * FROM Sales JOIN Stores

WHERE Stores.city = 'New York' Scan
Scan Stores
Sales
● Static pruning cannot be applied Filter

● Filter is only acting on the smaller

dimensional table, not the larger fact
table Larger fact table
Small dimensional table

26
Table Denormalization

Join Scan

Filter
Scan Scan city = 'New York'
Sales Stores

27
Dynamic Partition Pruning
Physical Plan Optimization
Broadcast Hash
Join
SCAN Fact Table

File Scan
Broadcast
Exchange
Dynamic Filter
File Scan with
DIM filter

Partitioned files with

multi-columnar data Non-partitioned
dataset

28
Dynamic Partition Pruning
▪ Reads only the data you need
▪ Optimizes queries by pruning
partitions read from a fact table by
identifying the partitions that result
from filtering dimension tables
▪ Significant speedup; shown in many
TPC-DS queries
▪ May help you avoid ETL-ing
denormalized tables

29
 3.0: SQL Engine
Adaptive Query Execution (AQE): change execution plan at runtime to
automatically set # of reducers and join algorithms
Change join
TPC-DS 1TB No-Stats With vs.
algorithm
Without Adaptive Query Execution

Duration (secs)
Accelerates TPC-DS queries
up to 8x
30
SMJ -> BHJ with AQE Demo
Without AQE
With AQE

31
32
Key Takeaways - Dynamic Partition Pruning
▪ It is enabled by default

▪ Spark will produce a query on the “small” table. The result of which is used
to produce a “dynamic filter” similar to our list of

▪ The “dynamic filter” is then broadcast to each executor

▪ At runtime, Spark’s physical plan is adjusted so that our “large” table is

reduced with the “dynamic filter”

▪ And if possible, that filter will employ a predicate pushdown so as to avoid

an InMemoryTableScan
Link to Summit Talk: Spark+AI Summit EU - 2019

33
Join Optimization Hints

34
Join Optimization Hints
▪ Join hints enable the user to override the optimizer to select their own
join strategies.

▪ Spark 3.0 extends the existing BROADCAST join hint by implementing

other join strategy hints:
▪ Shuffle hash
▪ Sort-merge
▪ Cartesian Product

35
Join Strategies
Shuffle Nested
Sort-Merge Broadcast Hash Shuffle Hash
Loop

▪ Most robust ▪ Requires ▪ Needs to ▪ Doesn't

▪ Can handle one side shuffle, but require join
any data size to be no sort keys
▪ Needs to small ▪ Can handle
shuffle and ▪ No shuffle large tables
sort or sort ▪ Will OOM if
▪ Can be slow ▪ Very fast data is
when table skewed
size is small

36
Join Hint Syntax
For Broadcast Joins
* Note the spaces for the hints

SELECT -*+ BROADCAST(t1) -/ * FROM t1 INNER JOIN t2 ON t1.key =

t2.key;

SELECT -*+ BROADCASTJOIN (t1) -/ * FROM t1 LEFT JOIN t2 ON

t1.key = t2.key;

SELECT -*+ MAPJOIN(t2) -/ * FROM t1 RIGHT JOIN t2 ON t1.key =

t2.key;

37
Join Hint Syntax
For Shuffle Sort Merge Joins
* Note the spaces for the hints

SELECT -*+ MERGE(t1) -/ * FROM t1 INNER JOIN t2 ON t1.key =

t2.key;

SELECT -*+ SHUFFLE_MERGE(t1) -/ * FROM t1 INNER JOIN t2 ON

t1.key = t2.key;

SELECT -*+ MERGEJOIN(t2) -/ * FROM t1 INNER JOIN t2 ON t1.key =

t2.key;

38
Join Hint Syntax
For Shuffle Hash Joins and Shuffle-and-Replicate Nested Loop Join
* Note the spaces for the hints

39
Join Hint Syntax
Shuffle Merge

SQL
SELECT -*+ SHUFFLE_MERGE(t1) -/ * FROM t1 INNER JOIN t2 ON
t1.key = t2.key;

Python
df1 = spark.table("t1")
df2 = spark.table("t2")
df1.hint("SHUFFLE_MERGE").join(df2, ["key"]).show()

40
Key Takeaways - Join Optimizations
▪ Join hints enable user to manually override AQE by using join
strategies

▪ With DBR 7.X, solving for skew is easy and automatic:

▪ Enable by setting spark.sql.adaptive.skewedJoin.enabled to true

▪ More join hint strategies are now available as part of DBR 7.X

Link to Summit Talk: Spark+AI Summit - 2020

41
Usability -- Spark SQL:
Explain Plans

42
Spark: Catalyst Optimizer

43
Spark SQL: EXPLAIN Plans

44
Spark SQL: EXPLAIN Plans
Note: Old syntax works in Spark 3.x as well

Spark 2.x Spark 3.x

45
Spark SQL: Old EXPLAIN Plan

46
Spark SQL: New EXPLAIN FORMATTED

Header: Basic operating tree for

the execution plan

Footer: Each operator

with additional attributes

47
Spark SQL: New EXPLAIN FORMATTED
Subqueries are listed separately

48
Key Takeaways - Explain Plans
▪ More features for ease of usability like EXPLAIN FORMATTED for
Plans.

▪ With DBR 7.X, feature to identify costs (statistics) is available.

▪ Old API still works too.

Link to Summit Talk: Spark+AI Summit - 2020

49
Pandas UDFs
(aka Vectorized UDFs)

50
from pyspark.sql.functions import pandas_udf, PandasUDFType
@pandas_udf('long', PandasUDFType.SCALAR) Spark 2.x
def pandas_plus_one(v):
    # `v` is a pandas Series Scalar Pandas UDF
    return v + 1  # outputs a pandas Series [Series to Series]
spark.range(10).select(pandas_plus_one("id")).show()

from pyspark.sql.functions import pandas_udf, PandasUDFType

@pandas_udf('long', PandasUDFType.SCALAR_ITER)
def pandas_plus_one(itr):
    # `iterator` is an iterator of pandas Series. Scalar Iterator Pandas UDF
# outputs an iterator of pandas Series.
    return map(lambda v: v + 1, itr)  [Iterator of Series to Iterator of Series]
spark.range(10).select(pandas_plus_one("id")).show()

from pyspark.sql.functions import pandas_udf, PandasUDFType

@pandas_udf("id long", PandasUDFType.GROUPED_MAP)

def pandas_plus_one(pdf):
    # `pdf` is a pandas DataFrame Grouped Map Pandas UDF
    return pdf + 1  # outputs a pandas DataFrame
[DataFrame to DataFrame]
# `pandas_plus_one` can _only_ be used with
`groupby(--.).apply(--.)`
spark.range(10).groupby('id').apply(pandas_plus_one).show()
 Spark 3.0
Python Type

def pandas_plus_one(v: pd.Series) -> Scalar Pandas UDF Hints

pd.Series: [Series to Series]

    return v + 1

def pandas_plus_one(iter: Iterator[pd.Series]) Scalar Iterator Pandas UDF

-> Iterator[pd.Series]: [Iterator of Series to Iterator of Series]
    return map(lambda v: v + 1, iter)

def pandas_plus_one(pdf: pd.DataFrame) -> Grouped Map Pandas UDF

pd.DataFrame:
    return pdf + 1 [DataFrame to DataFrame]

Link to blogpost: Pyspark and Pandas UDFs in Spark 3.x

Pandas UDFs

Spark 2.3/4

Spark 3.0
Python Type
Hints

53
Pandas UDFs
Pandas Function APIs

Supported function APIs include:

▪ Grouped Map
▪ Map
▪ Co-grouped Map

Link to blogpost: Pyspark and Pandas UDFs in Spark 3.x

54
Pandas UDFs
Pandas Function APIs

Link to blogpost: Pyspark and Pandas UDFs in Spark 3.x 55

Key Takeaways - Pandas UDFs
▪ A new interface for Pandas UDFs that leverages Python type hints to
address the proliferation of Pandas UDF types

▪ More Pythonic and self-descriptive

▪ Easier to read, learn and debug

Link to blogpost: Pyspark and Pandas UDFs in Spark 3.x

56
Spark ML new features

57
Spark ML
▪ ML function parity between Scala and Python.

▪ Many deprecated classes in

package in Spark 2.x have been removed from Spark 3.0

▪ in is deprecated and will be

removed in 3.1.0.
– should be used instead

▪ can handle all numeric types

– Earlier required input column to be Double or
Float Link to migration guide: Spark ML migration guide
Spark ML (notable) additions
▪ Multiple columns support to

– PySpark

▪ Two new evaluators:

–

▪ A new transformer: transformer

▪ Tree-Based Feature Transformation
Link to migration guide: Spark ML migration guide
Spark ML (notable) additions
▪ Classifiers:
– Gaussian Naive Bayes Classifier
– Complement Naive Bayes Classifier
▪ Sample weights support was added in

Link to migration guide: Spark ML migration guide

Key Takeaways - Spark ML
▪ Lot of new additions in Spark 3.0
▪ Many often requested features from the community like
, new evaluators and multiple columns support are
new standout features in 3.0
▪ A few deprecations and removals too

Link to migration guide: Spark ML migration guide

61
Migration &
Compatibility

62
Compiling your code for Spark 3.0
• Only builds with Scala 2.12
• Deprecates Python 2 (already EOL)
• Can build with various Hadoop/Hive versions
– Hadoop 2.7 + Hive 1.2
– Hadoop 2.7 + Hive 2.3 (supports Java 11)  [Default]
– Hadoop 3.2 + Hive 2.3 (supports Java 11)
• Supports the following Hive metastore versions:
– "0.12", "0.13", "0.14", "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3", "3.0", "3.1"
• NOTE: Databricks Runtime 7.x supports only Java 8
Spark Core, Spark SQL, DataFrames, and Datasets
▪ Built with Scala 2.12 (should be backwards compatible)

▪ DBR performs rebasing from the Proleptic Gregorian calendar to

the hybrid calendar (Julian + Gregorian).
– Spark 3.0 uses Java 8 API ( packages) that are based on
ISO chronology.
– For parsing / formatting of / strings in JSON/CSV,
Spak uses under the hood.
Datetime Patterns for Formatting and Parsing
– In Spark <= 2.4.x, was the default.

DBX blogpost on Dates and Timestamps Notes on Azure Databricks Docs Portal
Spark Core, Spark SQL, DataFrames, and Datasets
▪ In Spark 2.x, CSV/JSON datasource convert a malformed string to a
row with all nulls in the mode.
– In Spark 3.0, the returned row can contain non-null fields if some of
CSV/JSON column values were parsed and converted to desired types
successfully.

▪ Since version 3.0.1, the type inference is disabled by default

in JSON datasource and JSON function of .
– Set the JSON option to true to enable the type
inference.
– Note: this option was turned on by default in Spark 3.0.0 - and disabled in
3.0.1 for performance reasons
http://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-24-to-30
Spark Core, Spark SQL, DataFrames, and Datasets
▪ In Spark 3.0, literals are converted to strings using the SQL config
.
– In Spark 2.4 and below, the conversion uses the default time zone of the JVM.

▪ Due to a DateTimeFormatter bug in JDK 8, the pattern

parsing fails with Spark 3.x.
– Workaround in the JDK 8 bug discussion.

▪ For parsing Week number of the year from a timestamp in Spark 3.x EXTRACT or
weekofyear method needs to be used.
– In Spark 2.4 and below this was possible by applying w format option in the
date_format method.

DBX blogpost on Dates and Timestamps Notes on Azure Databricks Docs Portal 66
Spark Core, Spark SQL, DataFrames, and Datasets

▪ The and functions in Spark 3.0 accept only ,

, as the 2nd argument
– Fractional and non-literal strings are not valid anymore.

▪ Spark 3.0’s function does not adjust the resulting date to

a last day of the month if the original date is the last day of the month.

▪ For many changes, you do have the option to restore behavior before
Spark 3.0 using

http://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-24-to-30 67
Spark Core, Spark SQL, DataFrames, and Datasets
▪ Type coercions are performed per ANSI SQL standards when inserting
new values into a column.
– Unreasonable conversions will fail and throw an exception

▪ Two-parameter function signatures are

deprecated.

▪ SparkConf will reject bogus entries

– Code will fail if developers set config keys to invalid values

http://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-24-to-30 68
Spark Core, Spark SQL, DataFrames, and Datasets

▪ is removed

▪ Event log files will be written as UTF-8 encoding, NOT the default charset
of the driver JVM process

http://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-24-to-30 69
Spark Core
Deprecated methods removed/replaced

Removed method Replaced by

TaskContext.isRunningLocally - (Removed)
shuffleBytesWritten bytesWritten
shuffleWriteTime writeTime
shuffleRecordsWritten recordsWritten
AccumulableInfo.apply - (Disallowed)
Accumulator v1 APIs Accumulator v2 APIs

70
ANSI Compliance in Spark SQL
There are two new experimental options available for better compliance with ANSI SQL

Property Name Default Meaning

When true, Spark tries to fit the ANSI

spark.sql.ansi.enabled false
SQL specification.

When inserting values into a column

spark.sql.storeAssignmentPolicy ANSI with a different data type, Spark will
perform a type coercion per ANSI SQL.

71
32 New Built-in Functions

72
PySpark
Library updates may be required

Package Required version

Pandas 1.0.1
PyArrow 1.0.1

https://docs.microsoft.com/en-us/azure/databricks/release-notes/runtime/7.3
https://docs.databricks.com/release-notes/runtime/7.3.html

73
SparkR
Deprecated methods removed/replaced

Deprecated method Replaced by

74
Key Takeaways - Migration & Compatibility
▪ Spark 3.0 is more user-friendly and standardized.
▪ A lot of new built-in functions and HOFs.

▪ Databricks Runtime 7.x supported languages: Java 8 || Scala 2.12 ||

Python 3 (Python 2 Deprecated)

▪ You might have to re-compile your libraries for the upgraded runtime

▪ Follow documentation for details on the deprecated/replaced methods

▪ NOTE: Data/Timestamp parsing changes might be key for your pipelines

Link to migration guide: Apache Spark migration guide
75
Spark Ecosystem

76
Delta
• In addition to many new features of Spark 3.0, DBR 7.x also
comes with a number of optimizations and new Delta features.
– Hilbert Curve (coming soon)
– DBR 7.4 -- CONSTRAINTS, RESTORE
– DBR 7.3 -- Significant reduction in Delta metadata overhead time
– DBR 7.2 -- DEEP / SHALLOW CLONE
– DBR 7.1 -- CONVERT TO DELTA, MERGE INTO performance
improvements
– DBR 7.0 -- Auto Loader, COPY INTO, Dynamic subquery reuse
Third party connectors

78
Compatibility of connectors to external systems
Some 3rd party connectors are still not compatible with Spark 3.0

External system Spark 3 support Notes/Alternatives/Workarounds

Microsoft SQL Server and Not supported, PR is ● Compile yourself using code from the pull request
Azure SQL available in the repo ● Use JDBC connector (possible performance degradation)

Not supported, PR is
Azure Cosmos DB Compile yourself using code from the pull request
available in the repo

Azure Event Hubs Supported Best to use latest version. 2.3.17 had problems

Azure Data Explorer (Azure Supported, since version

Kusto) 2.3.0

Supported, since version For Databricks assembly variant should be used. Some functionality
Apache Cassandra
3.0.0 may not available out of box

Supported, since version

MongoDB
3.0.0

Contact respective organizations / authors if you need support for Spark 3 79

Compatibility of connectors to external systems
Some 3rd party connectors are still not compatible with Spark 3.0

External system Spark 3 support Notes/Alternatives/Workarounds

Not supported, no PR
Neo4j
available

Not supported, no PR
Spark-Redis May work with version compiled with Scala 2.12, not tested
available

Not supported, work in

Couchbase Manually compile available code from repository
progress

Not supported, PR
Elasticsearch Compile yourself using code from the pull request
available

Not supported no PR
Salesforce May work with version compiled with Scala 2.12, not tested
available

Not supported no PR
Apache HBase
available

Contact respective organizations / authors if you need support for Spark 3 80

Compatibility of connectors to external systems
Some 3rd party connectors are still not compatible with Spark 3.0

External system Spark 3 support Notes/Alternatives/Workarounds

Supported via JDBC
IBM DB2 See documentation
connector

SAS data files Supported

Snowflake Supported

Google BigQuery ? Could be compatible in version compiled with Scala 2.12. Needs testing

Apache Kafka Built-in Enhanced version in DB Runtime

AWS Kinesis Supported Enhanced version in DB Runtime

Contact respective organizations / authors if you need support for Spark 3 81

Compatibility of connectors to external systems
Some 3rd party connectors are still not compatible with Spark 3.0

External system Spark 3 support Notes/Alternatives/Workarounds

Exasol ? Could be compatible in version compiled with Scala 2.12. Needs testing

Contact respective organizations / authors if you need support for Spark 3 82

Further references

83
References and further reading

● Azure Databricks Runtime Release notes -- ADB 7.3 release notes

● Databricks Runtime Release notes -- Databricks 7.3 release notes

84
References and further reading
● Spark 3.0 migration guide -- Apache Spark Docs

● Deep dive into Spark 3.0 features -- Spark + AI Summit talk

● Faster SQL: AQE in Databricks -- Databricks Blogpost || Spark+AI Summit talk

● Comprehensive look at Dates & Timestamps in Spark 3.0 -- Databricks Blogpost

|| Spark+AI Summit talk || Notes on ADB Docs Portal

● New Pandas UDFs & Python Type hints in Spark 3.0 -- Databricks Blogpost ||
Spark+AI Summit talk

● Top tuning tips for Spark 3.0 & Delta Lake -- Databricks Tech Talk
85
Thank you

86