缺失的列用于ydata-profiling相关性报告

huangapple go评论66阅读模式
英文:

missing columns for ydata-profiling correlation report

问题

I'm using ydata-profiling(pandas-profiling的进化版)来计算大型数据集(例如400411行和27列)中列之间的相关性。
以下是config.yaml中的配置:

correlations:
    pearson:
      calculate: false
      warn_high_correlations: false
      threshold: 0.9
    spearman:
      calculate: true
      warn_high_correlations: false
      threshold: 0.9
    kendall:
      calculate: false
      warn_high_correlations: false
      threshold: 0.9
    phi_k:
      calculate: false
      warn_high_correlations: false
      threshold: 0.9
    cramers:
      calculate: true
      warn_high_correlations: false
      threshold: 0.9
    auto:
       calculate: false
       warn_high_correlations: false
       threshold: 0.9

我只需要对数值数据使用Spearman方法,对分类数据使用Cramers' V方法。
当我执行以下操作时:

tmp_profiler = ydata_profiling.ProfileReport(df, config_file='config.yaml')

它可以正确计算Sperman,但在Cramers中会跳过许多分类列(对于相同大小的其他数据集,会跳过所有列)。

我认为这可能是因为存在大量缺失数据,因此我尝试将这些列中的NaN填充为空字符串,但并没有起作用。
我不认为这是由于某些配置造成的,因为我尝试了扩大所有值:

vars:
    cat:
        length: false
        characters: false
        words: false
        cardinality_threshold: 5000000
        n_obs: 5
        # 设置为零以禁用
        chi_squared_threshold: 0.0
        coerce_str_to_date: false
        redact: false
        histogram_largest: 10
        stop_words: []
...
# 对于分类数据
categorical_maximum_correlation_distinct: 10000000

report:
  precision: 1000

更新:即使我使用

tmp_profiler = ydata_profiling.ProfileReport(df.sample(100000), config_file='config.yaml')

问题依然存在。

有没有人对这种行为有解释和解决方案?

英文:

I'm using ydata-profiling (the evolution of pandas-profiling) to compute correlation among columns of large datasets (e.g. 400411 rows and 27 columns).
These are configurations in config.yaml:

correlations:
    pearson:
      calculate: false
      warn_high_correlations: false
      threshold: 0.9
    spearman:
      calculate: true
      warn_high_correlations: false
      threshold: 0.9
    kendall:
      calculate: false
      warn_high_correlations: false
      threshold: 0.9
    phi_k:
      calculate: false
      warn_high_correlations: false
      threshold: 0.9
    cramers:
      calculate: true
      warn_high_correlations: false
      threshold: 0.9
    auto:
       calculate: false
       warn_high_correlations: false
       threshold: 0.9

I need only Spearman for numerical data and Cramers' V for categorical ones.
When I do

tmp_profiler = ydata_profiling.ProfileReport(df, config_file='config.yaml')

it computes correctly Sperman, but skips a lot of categorical columns in Cramers (with other datasets of same size it skips all of them).

I thought it was due to the presence of a lot of missing data, so I tried to fill Nan with empty string in those columns. It didn't work.
I don't think is due to some configuration, since I tried to enlarge all values:

vars:
    cat:
        length: false
        characters: false
        words: false
        cardinality_threshold: 5000000
        n_obs: 5
        # Set to zero to disable
        chi_squared_threshold: 0.0
        coerce_str_to_date: false
        redact: false
        histogram_largest: 10
        stop_words: []
...
# For categorical
categorical_maximum_correlation_distinct: 10000000

report:
  precision: 1000

UPDATE: even if I use

tmp_profiler = ydata_profiling.ProfileReport(df.sample(100000), config_file='config.yaml')

there is the same issue.

Does someone have some explanation and solutions for this behaviour?

答案1

得分: 0

缺失值的列是否被正确识别为分类列?问题的一个可能原因是类型推断。如果您的分类列具有高基数,它们可能被推断为文本或其他类型。

覆盖推断类型的解决方案:

prof = ProfileReport(
    df,
    config_file="config.yaml",
    type_schema={
        "column_1": "categorical",
        "column_2": "categorical",
    }
)
prof.to_file("profile.html")

如果将这些特征视为分类列,它们应该会出现在相关性中。也有可能您的列有太多缺失数据,以至于样本不返回任何有效数据...

英文:

Are the columns with missing values correctly identified as categorical? One cause of the problem could be type inference. If your categorical columns have high cardinality, they may be inferred as text or another type.

A solution to overwrite inferred types:

prof = ProfileReport(
    df,
    config_file="config.yaml",
    type_schema={
        "column_1": "categorical",
        "column_2": "categorical",
    }
)
prof.to_file("profile.html")

With the features being considered categorical, they should appear on the correlations. It is also possible that your columns have so much missing data that your sample does not return any valid data...

huangapple
  • 本文由 发表于 2023年5月29日 17:04:17
  • 转载请务必保留本文链接:https://go.coder-hub.com/76355985.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定