Pyspark - create our python package

In Synpase Notebook I've a separate notebook that contains all the functions that I need to run in another notebooks.

If I create a new notebook, for example, in first command I'll put

%run ETL Functions to use all the functions.

So for example inside that ETL notebook I can have this function:

def calculate_square(x):
    return x ** 2

and I want to create a package and insert for example this function and after I can do this:

from test_package import functions

result = functions.calculate_square(5)
print(result)  # Output: 25

so with that I don't need to run the ETL Function notebook every time I need to use a specific function.

My idea, but I don't know if it is possible, is to keep the ETL Functions notebook, and inside that notebook take the functions and insert them into the package and this package will be allocated to the cluster. That is, if I need to add another function, I just have to go to the notebook, insert the function and run the notebook separately and the function will automatically be available in the package. So I can also define different clusters and use this package in different clusters.

but I'm receptive to other ideas, this idea of ​​mine was more because version control and passing to different environments is easier, I think.

Does anyone have similar experience?

Thanks a lot!

With using only 2 notebook you cannot create package kind of functionality, but what you can do is write your custom functions to some .py file along with these 2 notebooks and import it.

You can follow below steps for it.
First, in ETL notebook add below code blocks.

%%sh
mkdir -p /synfs/user_pkg/
echo '''
    def calculate_square(x):
	    return x ** 2
	    
	def calculate_cube(x):
		return x ** 3
	''' > /synfs/user_pkg/custscript.py

Here, I am adding these functions inside synapse filesystem.
Then by adding this path to system, you can import these functions.
Below is code for adding into system path.

import sys
if  "/synfs/user_pkg/"  not  in sys.path:
    sys.path.append("/synfs/user_pkg/")
print("Done")
print("Import functions from 'custscript'")

Add above code blocks as below in ETL notebook.

Then call it from another notebook only once.

%run ETL

and

from custscript import calculate_square
from custscript import calculate_cube

print(calculate_square(3))
print(calculate_cube(3))

Output:

If you want to insert any other function add that inside
echo command in ETL notebook and run it once.