Azure Databricks – Adding Libraries

Posted on November 3, 2020 by blobeater

It is a really common requirement to add specific libraries to databricks. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.

Libraries can be added in 3 scopes. Workspace, Notebook-scoped and cluster. I want to show you have easy it is to add (and search) for a library that you can add to the cluster, so that all notebooks attached to the cluster can leverage the library.

Within the Azure databricks portal – go to your cluster.

Continue reading →

Azure Databricks and Azure Key Vault

Posted on October 28, 2020 by blobeater

The key vault should always be a core component of your Azure design because we can store keys, secrets, certicates thus abstract / hide the true connection string within files. When working with databricks to mount storage to ingest your data and query it ideally you should be leveraging this to create secrets and secret scopes.

Continue reading →

Azure Databricks – Pin Cluster

Posted on October 22, 2020 by blobeater

Before discussing why you would want to pin a cluster it would be useful to understand the different states of a cluster. We can have:

Continue reading →

Azure Databricks to Power BI

Posted on October 20, 2020 by blobeater

A very common approach is to query data straight from Databricks via Power BI. For this you need Databricks token and the JDBC address URL. This is found within Account settings of the cluster.

Continue reading →

Azure Databricks – The Notebook

Posted on October 14, 2020 by blobeater

Data engineers, pipe line developers, general data enthusiasts will be spending most of their time within a notebook. Here you develop your code, nice visualisations and commentary boxes are possible too, a very rich web-based interface and is best experienced with google chrome ( in my opinion).

Continue reading →