Azure Databricks – Adding Libraries

It is a really common requirement to add specific libraries to databricks. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.

Libraries can be added in 3 scopes. Workspace, Notebook-scoped and cluster. I want to show you have easy it is to add (and search) for a library that you can add to the cluster, so that all notebooks attached to the cluster can leverage the library.

Within the Azure databricks portal – go to your cluster.

Then go to libraries > Install New.

As I mentioned at the start at the blog post, you can add many types, I use the built-in search to find the library I want.

You don’t have to “search” if you know the co-ordinates then you can go that route.

Click Install.

The states will be resolving > Installing > Installed.

Note – worth remembering this. When you install a library on a cluster and you have a notebook already attached to that cluster, it will not immediately see the new library. You must first detach and then reattach the notebook to the cluster.

2 thoughts on “Azure Databricks – Adding Libraries

  1. Pingback: Azure Databricks – Adding Libraries from Blog Posts - SQLServerCentral - Actu Tech

  2. Pingback: Adding Libraries in Databricks – Curated SQL

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s