Import dataset from Kaggle to Databricks with Kaggle's API

Here trying to import a dataset from **Kaggle **to **DataBricks **(community) with their Kaggle' API. But I'm falling and lost 3 days. Please a kind soul can help me.

Trying 1:

!pip install kaggle

import os
import kaggle

os.environ['KAGGLE_USERNAME'] = 'xxxxxx'
os.environ['KAGGLE_KEY'] = 'xxxxx'

from kaggle.api.kaggle_api_extended import KaggleApi

api = KaggleApi()
api.authenticate()

api.dataset_download_files('taricov/mobile-wallets-in-egypt-2020')

Trying 1 error:
Error trying 1

Trying 2:

import os
import kaggle

from kaggle.api.kaggle_api_extended import KaggleApi
api = KaggleApi()
api_token_path = '/FileStore/tables/Kaggle_token/kaggle-2.json'
os.environ['KAGGLE_CONFIG_DIR'] = os.path.dirname(api_token_path)

api.dataset_download_files('taricov/mobile-wallets-in-egypt-2020', path='/FileStore/mobile-wallets-in-egypt-2020',unzip=True)

Trying 2 error:
Error trying 2

My kaggle.json credential in Databricks:
kaggle credential

I try two types of connections but its missing something or my credentials are wrong because the error is "Reason: Unauthorized".

The main problem here is that you're trying to reference file on DBFS directly /Filestore/..., but Kaggle API doesn't know anything about that filesystem because it uses Python's local file API. You have two choices:

• Refer to DBFS path directly by prepending the /dbfs to file names. Like, /dbfs/FileStore/tables/Kaggle_token/kaggle-2.json instead of /FileStore/tables/Kaggle_token/kaggle-2.json

• Use DBUtils commands to copy files to local disks or upload downloaded data to DBFS. For example, to copy file to local disk you can use following (not the file: prefix):

import os
conf_dir = "/tmp/kaggle-conf"
os.mkdir(conf_dir)
dbutils.fs.cp('/FileStore/tables/Kaggle_token/kaggle-2.json', 
  f'file:{conf_dir}/kaggle.json')
os.environ['KAGGLE_CONFIG_DIR'] = conf_dir

Try the following:

import os
import kaggle

from kaggle.api.kaggle_api_extended import KaggleApi
api = KaggleApi()
api_token_path = '/FileStore/tables/Kaggle_token/kaggle-2.json'
os.environ['KAGGLE_CONFIG_DIR'] = os.path.dirname(api_token_path)
api.authenticate()

api.dataset_download_files('taricov/mobile-wallets-in-egypt-2020', path='/FileStore/mobile-wallets-in-egypt-2020',unzip=True)

It seems that in your first snippet there is missing the environment variables and in the second one there is no api.authenticate() being called. Inside this method there is the read_config_environment method that is responsible to get those keys.