How to install maven library in databricks job cluster

Our databricks applicaiton is using multiple python and maven packages. Using the UI we can install the maven packages without any issue. But as we are going to use job cluster we are finding it difficult to install it.

Our job is schedule via ADF, one option is to add the library in the ADF pipeline but that will be too many changes on ADF side and we don't want to do this.

We would like to call a single notebook which will install all the required libraries. We are able to install the python libraries but having issue while installing maven libraries.

Any help will be really helpfull.

So I'm not aware of a wait to install maven libraries via a notebook but there are other ways to achieve what you need.
There are 3 options how you can do this.

1. A global init script at the workspace level. This impacts all
   clusters in the workspace.
2. An init script localized to a linked service. So all ADF pipelines that need the
   same libraries can call the same linked service if the notebooks are on the same
   ADB workspace.
3. Libraries can be listed out at the individual pipeline level. This provides a
   lot of flexibility but additional effort to include the list of libraries in
   each and every pipeline. This are installed over and above the libraries
   mentioned in the global and Linked service scripts.

Since you want minimum changes to ADF, I would suggest to go with option 1 or 2.
For option 1 :
Download the maven jar from the maven repo.
The admin of the workspace needs to enable dbfs browsing via the admin console like illustrated below. This is done under the Workspace settings within the Admin Settings.

Make sure the DBFS File browsing option is set to enabled as shown below

Next we create a new folder within /dbfs/Filestore/tables/ called jars so the path would be /dbfs/FileStore/tables/jars
Click the 'Data' tab on the left panel, and within that click the 'Browse DBFS' button and then the 'Upload' button.
Make sure the DBFS target directory is set to /FileStore/tables/jars as shown below.
Then drag and drop the jar into the box provided and click on Done.

Our init script should have a .sh suffix (example : init1.sh) and should have the below contents :

#!/bin/bash
pip install msal
cp /dbfs/FileStore/tables/jars/spark_mssql_connector_2_12_1_2_0.jar /databricks/jars/

The above script will install a python library as well as copy the maven jar to the /databricks/jars/ folder.

Load the script as a Global init script . You can refer to this link on how to do that.

Now, ever cluster, interactive or job will use the same set of libraries.