问题

I am trying to build and a method which takes a dataframe as input and returns the another dataframe as output after checking a certain condition.

If the exception_type column contains FILE_REJECT, it drops the file_name column, otherwise it does not.

I have provided various input and output to the method. Please help me build the method. Thank you.

# 以下是您要翻译的内容
输入
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|   file_name|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAW0001|      SUSPENSE|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0006|    REC_REJECT|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|              |              |

+------------+---------------+-------------+----------------+---------+--------------+--------------+
|   file_name|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|KKAR0523.ccn|       zzzzzzzz|     yyyyyyyy|      2001-12-22|      SHR|              |              |
|KKAR0523.ccn|       zzzzzzzz|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0001|   FILE_REJECT|
+------------+---------------+-------------+----------------+---------+--------------+--------------+

+------------+---------------+-------------+----------------+---------+--------------+--------------+
|   file_name|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0002|   FILE_REJECT|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0001|   FILE_REJECT|
+------------+---------------+-------------+----------------+---------+--------------+--------------+

输出
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|   file_name|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAW0001|      SUSPENSE|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0006|    REC_REJECT|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|              |              |
+------------+---------------+-------------+----------------+---------+--------------+--------------+

+---------------+-------------+----------------+---------+--------------+--------------+
|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+---------------+-------------+----------------+---------+--------------+--------------+
|       zzzzzzzz|     yyyyyyyy|      2001-12-22|      SHR|              |              |
|       zzzzzzzz|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0001|   FILE_REJECT|
+---------------+-------------+----------------+---------+--------------+--------------+

+---------------+-------------+----------------+---------+--------------+--------------+
|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+---------------+-------------+----------------+---------+--------------+--------------+
|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0002|   FILE_REJECT|
|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0001|   FILE_REJECT|
+---------------+-------------+----------------+---------+--------------+--------------+

谢谢。

英文:

I am trying to build and a method which takes a dataframe as input and returns the another dataframe as output after checking a certain condition.

If the exception_type column contains FILE_REJECT, it drops the file_name column, otherwise it does not.

I have provided various input and output to the method. Please help me build the method. Thank you.

Input
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|   file_name|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAW0001|      SUSPENSE|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0006|    REC_REJECT|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|              |              |
+------------+---------------+-------------+----------------+---------+--------------+--------------+

+------------+---------------+-------------+----------------+---------+--------------+--------------+
|   file_name|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|KKAR0523.ccn|       zzzzzzzz|     yyyyyyyy|      2001-12-22|      SHR|              |              |
|KKAR0523.ccn|       zzzzzzzz|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0001|   FILE_REJECT|
+------------+---------------+-------------+----------------+---------+--------------+--------------+

+------------+---------------+-------------+----------------+---------+--------------+--------------+
|   file_name|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0002|   FILE_REJECT|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0001|   FILE_REJECT|
+------------+---------------+-------------+----------------+---------+--------------+--------------+

Output
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|   file_name|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+------------+---------------+-------------+----------------+---------+--------------+--------------+
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAW0001|      SUSPENSE|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0006|    REC_REJECT|
|KKAR0523.ccn|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|              |              |
+------------+---------------+-------------+----------------+---------+--------------+--------------+

+---------------+-------------+----------------+---------+--------------+--------------+
|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+---------------+-------------+----------------+---------+--------------+--------------+
|       zzzzzzzz|     yyyyyyyy|      2001-12-22|      SHR|              |              |
|       zzzzzzzz|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0001|   FILE_REJECT|
+---------------+-------------+----------------+---------+--------------+--------------+

+---------------+-------------+----------------+---------+--------------+--------------+
|register_number|jacket_number|annual_return_dt|data_type|exception_code|exception_type|
+---------------+-------------+----------------+---------+--------------+--------------+
|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0002|   FILE_REJECT|
|       xxxxxxxx|     yyyyyyyy|      2001-12-22|      SHR|      CHAR0001|   FILE_REJECT|
+---------------+-------------+----------------+---------+--------------+--------------+

Thank you.

答案1

得分: 1

您可以使用以下方法来实现：

from pyspark.sql.functions import col

def process_dataframe(input_df):
    if 'file_name' in input_df.columns and 'exception_type' in input_df.columns:
        if input_df.filter(col('exception_type') == 'FILE_REJECT').count() > 0:
            output_df = input_df.drop('file_name')
        else:
            output_df = input_df
    else:
        output_df = input_df
    
    return output_df

英文:

You can achieve that using such method:

from pyspark.sql.functions import col

def process_dataframe(input_df):
    if &#39;file_name&#39; in input_df.columns and &#39;exception_type&#39; in input_df.columns:
        if input_df.filter(col(&#39;exception_type&#39;) == &#39;FILE_REJECT&#39;).count() &gt; 0:
            output_df = input_df.drop(&#39;file_name&#39;)
        else:
            output_df = input_df
    else:
        output_df = input_df
    
    return output_df

通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。

删除基于另一个pyspark的值的列。

问题

答案1

Create subplot, by overlapping two dataframes of different shapes and column names, for every group/id,

创建基于另一个数据集的作者的数据集。

可以使用一个输入将多个整数放入一个列表吗？

如何正确使用boto3的if语句

What's the correct way to type hint an empty list as a literal in python?

如何在Highcharts Gantt中更改本地化的星期名称

如何在同一个流中使用多个过滤器和映射函数？

如何使用Map/Set来将代码优化到O(n)？

.NET MAUI Android在GitHub Actions上构建失败，错误代码为1。

如何在Playwright视觉比较中屏蔽多个定位器？

在C++中，可以使用可变模板参数来检索类型的内部类型。

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found

Creating and opening a URL to log in to Website via Basic Auth with Robot Framework/Selenium (Python)

AG Grid 在上下文菜单中以大文本形式打开

发表评论