superset helmchart enable cache

I have superset installed via helmchart in my kubernetes environment, I took everything from official documentation and repository: https://github.com/apache/superset

I'm trying to archive a data autorefresh of the dashboard every 12 hours via helmchart and not via the UI; I read that this can be done enabling superset cache so data will be cached for 12 hours and then dynamically refreshed and everyone that access superset UI can see the same values.

My problem now is one.... I can see the cache configuration on the superset/config.py file:

# Default cache for Superset objects
CACHE_CONFIG: CacheConfig = {"CACHE_TYPE": "NullCache"}

# Cache for datasource metadata and query results
DATA_CACHE_CONFIG: CacheConfig = {"CACHE_TYPE": "NullCache"}

# Cache for dashboard filter state (`CACHE_TYPE` defaults to `SimpleCache` when
#  running in debug mode unless overridden)
FILTER_STATE_CACHE_CONFIG: CacheConfig = {
    "CACHE_DEFAULT_TIMEOUT": int(timedelta(days=90).total_seconds()),
    # should the timeout be reset when retrieving a cached value
    "REFRESH_TIMEOUT_ON_RETRIEVAL": True,
}

# Cache for explore form data state (`CACHE_TYPE` defaults to `SimpleCache` when
#  running in debug mode unless overridden)
EXPLORE_FORM_DATA_CACHE_CONFIG: CacheConfig = {
    "CACHE_DEFAULT_TIMEOUT": int(timedelta(days=7).total_seconds()),
    # should the timeout be reset when retrieving a cached value
    "REFRESH_TIMEOUT_ON_RETRIEVAL": True,
}

As per documentation I'm using the configOverrides section of the helmchart to overwrite the default values and enable the cache of config, data, filter ad explore but I can't find any example of how to do it and everything I try always fail in helmrelease.

I try to read the helmchart but looks that it take all the configOverrides section and I was not able to find where it overwrite those specific values.

Some example of what I try to overwrite, for example enabling some flag works without problem:

configOverrides:
  enable_flags: |
    FEATURE_FLAGS = {
        "DASHBOARD_NATIVE_FILTERS": True,
        "ENABLE_TEMPLATE_PROCESSING": True,
        "DASHBOARD_CROSS_FILTERS": True,
        "DYNAMIC_PLUGINS": True,
        "VERSIONED_EXPORT": True,
        "DASHBOARD_RBAC": True,
    }

But if I try to overwrite one or more cache value it fail (config.py https://github.com/apache/superset/blob/master/superset/config.py), this is one of the different way I try to overwrite checking the helm value file, the template and the superser config.py (and checkign other articles):

configOverrides:
  cache_config: |
    CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_cache_'
      }
  data_cache_config: |
    DATA_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_data_'
    }
  filter_cache_config: |
    FILTER_STATE_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_filter_'
    }
  explore_cache_config: |
    EXPLORE_FORM_DATA_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_explore_'
    }

Any help pls? Or redirect to some good documentation that has example! Ps the redis installation I have it's the default one created by the helmchart, I didn't change anything on it.

TL;DR; your configOverrides should look like this:

configOverrides:
  cache_config: |
    from datetime import timedelta
    from superset.superset_typing import CacheConfig

    CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_cache_'
      }
    
    DATA_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_data_'
    }
    FILTER_STATE_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_filter_'
    }

    EXPLORE_FORM_DATA_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_explore_'
    }

Details:

After running a helm install with your settings, your config file will look a bit like this:

import os
from cachelib.redis import RedisCache
...
CACHE_CONFIG = {
      'CACHE_TYPE': 'redis',
      'CACHE_DEFAULT_TIMEOUT': 300,
      'CACHE_KEY_PREFIX': 'superset_',
      'CACHE_REDIS_HOST': env('REDIS_HOST'),
      'CACHE_REDIS_PORT': env('REDIS_PORT'),
      'CACHE_REDIS_PASSWORD': env('REDIS_PASSWORD'),
      'CACHE_REDIS_DB': env('REDIS_DB', 1),
}
DATA_CACHE_CONFIG = CACHE_CONFIG
...
# Overrides
# cache_config
CACHE_CONFIG: CacheConfig = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
    'CACHE_KEY_PREFIX': 'superset_cache_'
  }

# data_cache_config
DATA_CACHE_CONFIG: CacheConfig = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
    'CACHE_KEY_PREFIX': 'superset_data_'
}

# enable_flags
FEATURE_FLAGS = {
    "DASHBOARD_NATIVE_FILTERS": True,
    "ENABLE_TEMPLATE_PROCESSING": True,
    "DASHBOARD_CROSS_FILTERS": True,
    "DYNAMIC_PLUGINS": True,
    "VERSIONED_EXPORT": True,
    "DASHBOARD_RBAC": True,
}

# explore_cache_config
EXPLORE_FORM_DATA_CACHE_CONFIG: CacheConfig = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
    'CACHE_KEY_PREFIX': 'superset_explore_'
}
# filter_cache_config
FILTER_STATE_CACHE_CONFIG: CacheConfig = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
    'CACHE_KEY_PREFIX': 'superset_filter_'
}

When I looked at the pod logs, there were a lot of errors due to the function timedelta not being defined, here is a sample of the logs I can see:

File "/app/pythonpath/superset_config.py", line 42, in <module>
    'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
    NameError: name 'timedelta' is not defined

The file in question, /app/pythonpath/superset_config.py , is loaded via an import here as mentioned in the comment at the top of the file.

Notice that you're writing a fresh new .py file; which means that you need to add from datetime import timedelta at the top in the configOverrides section.

However, since the doc in the helm chart states the following warning WARNING: the order is not guaranteed Files can be passed as helm --set-file configOverrides.my-override=my-file.py, and you clearly want to use the function timedelta, we must combine all three blocks under the same section like this:

configOverrides:
  cache_config: |
    from datetime import timedelta

    CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_cache_'
      }
    
    DATA_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_data_'
    }
    FILTER_STATE_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_filter_'
    }

    EXPLORE_FORM_DATA_CACHE_CONFIG: CacheConfig = {
        'CACHE_TYPE': 'RedisCache',
        'CACHE_DEFAULT_TIMEOUT': int(timedelta(hours=6).total_seconds()),
        'CACHE_KEY_PREFIX': 'superset_explore_'
    }

Furthermore, you wanted to use the type CacheConfig, so we should also include an import for it at the top.