Optimize the code to sum scores for the tokens in noun phrase list

Requirement

• The below code looks complicated in time complexity, I am looking for solutions to optimize the below code.

Aim of the below code is to sum up the scores of tokens if the token is in nounphrase_list and also all_tokens (detailed explanation below)

full_string = ["Mitchell is using machine learning and software engineering on his mobile phone to generate songs . On top of machine learning he is also using deep learning"

# the token list are words from above full_string
all_tokens = ["Mitchell", "is", "using", "machine", "learning", 
              "and", "software", "engineering", "on", "his", 
              "mobile", "phone", "to", "generate", "songs", 
              "On", "top", "of", "machine", "learning", 
              "he", "is", "also", "using", "deep", "learning"]

all_scores = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]

# These noun phrases extracted from the full_string
nounphrase_list = ["Mitchell", "machine learning", "software engineering", "machine learning", "deep learning"]

updated_scores = all_scores.copy()

seen_indices = set()
for np_item in nounphrase_list:
    np_words = np_item.split()
    indices = []
    for word in np_words:
        for idx, token in enumerate(all_tokens):
            if word == token:
                if idx not in seen_indices:
                    indices.append(idx)
                    seen_indices.add(idx)
                    break
    sum_score = sum(all_scores[idx] for idx in indices)
    for idx in indices:
        updated_scores[idx] = sum_score

Explanation:

• Mitchell is nounphrase_list and the score of Mitchell is 1, Hence the updated score is also 1
• The next phrase in noun phrase list is the first occurrence of machine learning has scores [4, 5], Hence the updated scores for machine is 9 and learning is 9
• The next phrase in noun phrase list is the first occurrence of software engineering has scores [7, 8], Hence the updated scores for software is 15 and learning is 15
• The next phrase in noun phrase list is the first occurrence of machine learning has scores [19, 20], Hence the updated scores for machine is 39 and learning is 39
• The next phrase in noun phrase list is the first occurrence of deep learning has scores [25, 26], Hence the updated scores for machine is 51 and learning is 51

Note:

• The words machine & learning should occur together in same order
• The words software & engineering should occur together in same order
• The words machine & learning should occur together in same order
• The words deep & learning should occur together in same order

So your question is somewhat complicated, but perhaps we should look at it a different way.

Single-word 'noun phrases' are essentially red herrings for this problem. Since their scores don't change, we do not need to worry about them.

So, we only need to look at the two-word noun phrases:

for i, word in enumerate(all_tokens[:-1]): #loop through all but last word (remember single-words are red herrings!)
    if (value := word + " " + all_tokens[i+1]) in nounphrase_list: #use walrus operator to remove from the list so no accidental duplicates.
        nounphrase_list.remove(value)
        total = all_scores[i] + all_scores[i+1] #combined score
        all_scores[i] = all_scores[i+1] = total #set score

Result:

[1, 2, 3, 9, 9, 6, 15, 15, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 39, 39, 21, 22, 23, 24, 51, 51]

Time Complexity

Not the best at calculating this, but I can calculate runtime the old-fashioned way.

Your code (ran 10,000 times): .2439 seconds.

My code (ran 10,000 times): .1208 seconds.

Your code (ran 100,000 times): 3.352 seconds.

My code (ran 100,000 times): 1.294 seconds.

Your code (ran 1,000,000 times): 26.1021 seconds

My code (ran 1,000,000 times): 12.4637 seconds.

Note: I used the timeit module. Also clearly only works for two-word 'noun phrases'. It's easy to modify for longer phrases, just add an elif statement.

Hope this helps!