Azure Databricks to Azure SQL DB

Recently I got to a stage where I leveraged Databricks to the best of my ability to join couple of CSV files together, play around some aggregations and then output it back to a different mount point ( based on Azure Storage) as a parquet file, I decided that I actually wanted to move this data into Azure SQL DB, which you may want to do one day.

As per the diagram.

sparky1

The documentation is there online, but I wanted to show you the screen shots to do this. Microsoft states that the spark connector should be used  and the connector project uses maven.  Ok great, what does that mean?  Once you have your Databricks up and running on the main workspace menu you need to Import Library.

importlib

Switch to Maven and enter co-ordinate – com.microsoft.azure:azure-sqldb-spark:1.0.2.

mavens

Click create and your clusters will have access to it. As a side note if you ever need to install any JAR files, you can do that below:

JARS'

Now you are ready for some notebook magic.

imports1

Once you have ran that, then below (just a random example) I moved the contents of my data frame to Azure SQL.

valll

You can do the reverse if needed too.

 

1 thought on “Azure Databricks to Azure SQL DB

  1. Pingback: Feeding Databricks Output to Azure SQL Database – Curated SQL

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s