Dataricks - Error setting certificate verify locations even though file exists

I have an issue with SSL certificates that I cannot wrap my head around, and I am unsure whether it is a problem with Databricks' sometimes confusing file system or with the way curl / certificates works.

Our app uses an API to connect to a company-internal resource. To avoid the well-known "self-signed certificate in certificate chain", we provide the companies publicly-hosted .crt file with the API call. The app is in Python, but to break it down to curl commands, we use the following inside Databricks:

As you can see, this complains about certificate verify locations. However, the file definitely does exist:

On my local PC in an Ubuntu shell, the exact same command (with --cacert providing the .crt file location) works perfectly fine - so I am kind of stuck on what to do here. Can this error be triggered by something else then the file not being in the location? E.g. firewall issues? Does Databricks maybe treat the /tmp/ path in a special way? Any ideas to further debug this?

I found some related threads on stackoverflow, e.g. this one, but I already tried moving the .crt file to a different folder or executing curl as sudo, nothing worked. Happy about any suggestions or leads.

The best way to handle this is to install the certificate into the system chain(s) using the cluster or global init scripts. The only caveat is that you need to put the SSL certificate of your certificate authority on DBFS, and then refer to it as local path /dbfs/<path-on-dbfs>, and this may not work on shared clusters with Unity Catalog or DBSQL warehouses.

I have an example of init script that performs that task and installs CA's SSL certificate into three locations:

• Linux system SSL chain

• Java truststore

• Python's SSL chain provided by certifi package.

So as it turns out, this was an issue with our firewall in combination with a misleading error message of curl and a weird behavior of our certificate download script.

The key takeaway for anyone stumbling on this is that error setting certificate verify location CAN mean that the certificate was not found, but it can also mean that the certificate file is not valid - which we assumed it was, since we expected a different error message in case the certificate was found but not valid.

In our case, the certificate file was actually completely empty - our company some time in the past changed some firewall rules, accidentally blocking connection to our certificate server, and when our script attempted to re-download the certificate last month, it instead created an empty file - which, as mentioned above, raised an error message that led us to believe the file was missing instead of corrupt.