List of lists of dictionaries with the same keys but different values

Firs off, let me preface this by asking you not to downvote the post. I've tried other posts with a 'minimal reproducible example' but it didn't work. The log is too complex. So far no one's been able to help.

I need to collect certain key/value pairs from logs from the antivirus. I’ve been able to collect all key/value pairs I need except for one, the action taken by the antivirus.
Everything revolves around the ‘indicators’ key, which contains a list of dicts with each containing a certain piece of information about the virus/malware found. Each of these dicts starts with an id number (1,2,3,4…) which varies depending on the virus/malware. While these dicts have the exact same structure (and same key names), their values differ. Take a gander at the excerpt below:

"indicators": [
            {
                "id": 1,
                "type": "detection_name",
                "field": "malName",
                "value": "HKTL_CAIN",
                "relatedEntities": [
                    "C888A5B2"
                ],
                "filterIds": [
                    "a665ee2c"
                ],
                "provenance": [
                    "Alert"
                ]
            },
            {
                "id": 2,
                "type": "file_sha1",
                "field": "fileHash",
                "value": "",
                "relatedEntities": [
                    "C888A5B2"
                ],
                "filterIds": [
                    "a665ee2c"
                ],
                "provenance": [
                    "Alert"
                ]
            },
            {
                "id": 3,
                "type": "filename",
                "field": "fileName",
                "value": "D:\\ECIH",
                "relatedEntities": [
                    "C888A5B2"
                ],
                "filterIds": [
                    "a665ee2c"
                ],
                "provenance": [
                    "Alert"
                ]
            },
            {
                "id": 4,
                "type": "fullpath",
                "field": "fullPath",
                "value": "D:\\ECIH",
                "relatedEntities": [
                    "C888A5B2"
                ],
                "filterIds": [
                    "a665ee2c"
                ],
                "provenance": [
                    "Alert"
                ]
            },
            {
                "id": 5,
                "type": "text",
                "field": "actResult",
                "value": "File cleaned",
                "relatedEntities": [
                    "C888A5B2"
                ],
                "filterIds": [
                    "a665ee2c"
                ],
                "provenance": [
                    "Alert"
                ]
            },
            {
                "id": 6,
                "type": "text",
                "field": "scanType",
                "value": "Scheduled Scan",
                "relatedEntities": [
                    "C888A5B2"
                ],
                "filterIds": [
                    "a665ee2c"
                ],
                "provenance": [
                    "Alert"
                ]
            }
        ]

Note that id 1 relates to the malware name. Id 2 is the hash value, id 3 is the file name, etc. What I'm interested in is in id 5, which contains the action taken by the antivirus. There is a long list of possible actions, but for exemplification purposes, the actions are 'File cleaned' and 'File quarantined.' The action is always found in the key 'value', but the problem is that 'value' appears everywhere. I noticed that the 'value' I need (server action) is always paired with the 'actResult' value in the 'field', which also appears everywhere.

        {
            "id": 5,
            "type": "text",
            "field": "actResult",
            "value": "File cleaned",
            "relatedEntities": [
                "C888A5B2"
            ],
            "filterIds": [
                "a665ee2c"
            ],
            "provenance": [
                "Alert"
            ]
        }

Another issue is that these logs aren't always the same length, so some have id1, id2,..id5 whereas others might have 11 ids. Nothing is consistent. Regardless, what I need to do is to capture the values I need and put them into a dataframe, but given that the 'value' key appears everywhere, the script is faulty.

In the sample I provide, there are 10 records total, so I'll have 10 IDs. But since the info on the virus changes based on the virus type, so do the 'indicators'. Hence, where a record is missing, I replace it with a ' - '. But since the 'value' key appears everywhere, I end up with an uneven number of ID/Action.

Please refer to the log here (https://codeshare.io/j0yX1A). The script is below:

actions = ['File cleaned', 'File deleted', 'File quarantined']
actions_list = []
action_list = []

id = [id['id'] for id in logs]
print(id)

for log in logs:
        for indicator in log['indicators']:
                if indicator['value'] in actions:
                        action_list.append(indicator['value'])
                    else:
                                action_list.append('-')
print(action_list)

Current output:

As you can see, the current script picks up all 'value' keys rather than just the ones whose values are in the actions list.

Expected Output

If there is no key/value, replace it with a ' - '.

['WB-13273-20230604-00000', 'WB-13273-20230603-00000', 'WB-13273-20230601-00001', 'WB-13273-20230601-00000', 'WB-13273-20230529-00000', 'WB-13273-20230526-00000', 'WB-13273-20230523-00001', 'WB-13273-20230523-00000', 'WB-13273-20230510-00002', 'WB-13273-20230510-00003']
[' - ',' - ','File cleaned', 'File cleaned', ' - ','File cleaned', ' - ','File quarantined', 'File quarantined', 'File quarantined']

Same number of IDs and actions.

So how can I collect the action value from the "field" key and replace the missing records with a ' - ' while ignoring the other 'value' keys that aren't needed?

It looks like you might be looking for the else clause of the for loop to account for the cases where you have not "actResults". I call this the no break clause as the else is what happens in the event that the for loop did not do a break.

Given your data:

import json

with open("log.json", "r") as file_in:
    log_data = json.load(file_in)

action_list = []
for log_entry in log_data:
    for indicator in log_entry["indicators"]:
        if indicator.get("field") == "actResult":
            action_list.append((log_entry["id"], indicator["value"]))
            break
    else:
        action_list.append((log_entry["id"], "--"))

for action in action_list:
    print(action)

Will give you back your list of 10:

('WB-13273-20230601-00001', '--')
('WB-13273-20230601-00000', '--')
('WB-13273-20230529-00000', '--')
('WB-13273-20230526-00000', 'File cleaned')
('WB-13273-20230523-00001', '--')
('WB-13273-20230523-00000', '--')
('WB-13273-20230510-00002', 'File quarantined')
('WB-13273-20230510-00003', 'File quarantined')
('WB-13273-20230510-00001', 'File quarantined')
('WB-13273-20230510-00000', 'File quarantined')

Since you're still a relatively new user and you've shown some amount of effort at asking your (somewhat incomplete) question, I'll give you the benefit of the doubt. I think what you're looking for is:

actions_to_track = ['File cleaned', 'File deleted', 'File quarantined']

def get_actresult(log):
    for ind in log['indicators']:
        if ind.get('field') == 'actResult':  # an action indicator
            if ind.get('value') in actions_to_track:
                return ind.get('value')

    return '-'

log_ids_to_actresult = {log['id']: get_actresult(log) for log in logs}
# => {'example_id_0': 'File cleaned'}

Running against logs from https://codeshare.io/j0yX1A, this produces:

{
  'WB-13273-20230510-00000': 'File quarantined',
  'WB-13273-20230510-00001': 'File quarantined',
  'WB-13273-20230510-00002': 'File quarantined',
  'WB-13273-20230510-00003': 'File quarantined',
  'WB-13273-20230523-00000': '-',
  'WB-13273-20230523-00001': '-',
  'WB-13273-20230526-00000': 'File cleaned',
  'WB-13273-20230529-00000': '-',
  'WB-13273-20230601-00000': '-',
  'WB-13273-20230601-00001': '-',
}

However, I'm not sure because your question is unclear and missing key information. My answer is based on the following guesses I've had to make that you should've clarified in your question:

1. You've got a list of log dictionaries, each with "id" and "indicators" keys
2. There are a variable number of indicators per log. They are all dict of the same format
3. An "action indicator" will have a "field" of "actResult"
4. For each log, you want to extract the "value" from an action indicator if it exists and if the value is one of an explicit list of actions you care about, otherwise use "-"
5. The actResult values you care about are ['File cleaned', 'File deleted', 'File quarantined']

A complete but minimal-ish example of the logs and indicator data is:

logs = [
    {
        "id": "example_id_0",
        "indicators": [
            {
                "id": 1,
                "type": "detection_name",
                "field": "malName",
                "value": "HKTL_CAIN",
            },
            {
                "id": 5,
                "type": "text",
                "field": "actResult",
                "value": "File cleaned",
            },
        ],
    },
]

It was good that you included your attempt code. It would've helped if you included annotations explaining what you were trying to do (even if you don't know how) and omitted extraneous information (e.g. about antiviruses or something).

Don't simply dismiss all the users giving you critique as "pedantic". People want to answer your question, but they can only do so if you include relevant information and exclude irrelevant information. Please take the above as an example of how you can ask a better question next time.