How do I set the number of extracted words with the Rake algorithm in python?

When extracting keywords with Rake, the algorithm creates its candidate phrases and ranks them based on their score and returns the phrases with at least a certain score.

How can I set this minimum score, or set the minimum number of extracted keywords, or at least get all the candidate phrases ?

from rake-nltk import rake 

r = Rake()

# Extraction given the text.
r.extract_keywords_from_text(text)

keywords=r.get_ranked_phrases()
print(keywords)

this is the standard procedure, but I would like to know if there is a different function, or a parameter I can set to get not just the most significant keywords, but all of them or at least a bigger amount of them

Lets try to walk through the code https://csurfer.github.io/rake-nltk/_build/html/_modules/rake_nltk/rake.html:

We start with .extract_keywords_from_text() function that is a wrapper over .extract_keywords_from_sentences()

    def extract_keywords_from_text(self, text: str):
        sentences: List[Sentence] = self._tokenize_text_to_sentences(text)
        self.extract_keywords_from_sentences(sentences)


    def extract_keywords_from_sentences(self, sentences: List[Sentence]):
        phrase_list: List[Phrase] = self._generate_phrases(sentences)
        self._build_frequency_dist(phrase_list)
        self._build_word_co_occurance_graph(phrase_list)
        self._build_ranklist(phrase_list)

The first step of the Rake algorithm in .extract_keywords_from_sentences() is to generate possible phrases using the ._generate_phrases() function, i.e.

    def _generate_phrases(self, sentences: List[Sentence]) -> List[Phrase]:
        phrase_list: List[Phrase] = []
        # Create contender phrases from sentences.
        for sentence in sentences:
            word_list: List[Word] = [word.lower() for word in self._tokenize_sentence_to_words(sentence)]
            phrase_list.extend(self._get_phrase_list_from_words(word_list))

        # Based on user's choice to include or not include repeated phrases
        # we compute the phrase list and return it. If not including repeated
        # phrases, we only include the first occurance of the phrase and drop
        # the rest.
        if not self.include_repeated_phrases:
            unique_phrase_tracker: Set[Phrase] = set()
            non_repeated_phrase_list: List[Phrase] = []
            for phrase in phrase_list:
                if phrase not in unique_phrase_tracker:
                    unique_phrase_tracker.add(phrase)
                    non_repeated_phrase_list.append(phrase)
            return non_repeated_phrase_list

        return phrase_list

which calls the ._get_phrase_list_from_words() function to extract the phrases, note the docstring:

> Method to create contender phrases from the list of words that form a
> sentence by dropping stopwords and punctuations and grouping the left
> words into phrases. Only phrases in the given length range (both
> limits inclusive) would be considered to build co-occurrence matrix.

Ex:
 
Sentence: Red apples, are good in flavour. 

List of words: ['red', 'apples', ",", 'are', 'good', 'in', 'flavour'] 

List after dropping punctuations and stopwords. 

List of words: ['red', 'apples', *, *, good, *, 'flavour'] 
List of phrases: [('red', 'apples'), ('good',), ('flavour',)]

Short conclusion

There is some sort of upperbound to the number of phrases, since the first step already created "contender phrases" and the upperbound number of phrases you can extract using rake-nltk is determined by your dataset + rules in ._get_phrase_list_from_words().

TL;DR

While you cannot add to the max no. of phrases, you can access the full list with:

from rake_nltk import Rake

from nltk.corpus import reuters

text = '\n'.join([' '.join(s) for s in reuters.sents()[:1000]])


r = Rake()
r.extract_keywords_from_text(text)
print(r.rank_list)

Note:

• The rank_list returns a list of tuples where the 1st item in the tuple is the rake score that skews towards longer "phrases" have higher score, and the 2nd item is the "phrase" itself.

• It might be used to re-normalize the rake score by the length of the output, e.g.

from rake_nltk import Rake
from collections import Counter

from nltk.corpus import reuters

text = '\n'.join([' '.join(s) for s in reuters.sents()[:1000]])


r = Rake()
r.extract_keywords_from_text(text)

normalized_rank_phrases = Counter()

for score, phrase in r.rank_list:
    if len(phrase.split()) > 1:
        normalized_rank_phrases[phrase] = score/len(phrase.split())**2  # Penalize by dividing length
    
normalized_rank_phrases.most_common()

[out]:

from rake_nltk import Rake
from collections import Counter

from nltk.corpus import reuters

text = '\n'.join([' '.join(s) for s in reuters.sents()[:1000]])


r = Rake()
r.extract_keywords_from_text(text)

normalized_rank_phrases = Counter()

for score, phrase in r.rank_list:
    if len(phrase.split()) > 1:
        normalized_rank_phrases[phrase] = score/len(phrase.split())**2  # Penalize by dividing length
    
normalized_rank_phrases.most_common()

[out]:

[('qtrly div', 7.318181818181818),
 ('73 cts', 7.050699300699301),
 ('33 cts', 5.225699300699301),
 ('independent chairman', 4.845833333333333),
 ('60 cts', 4.741875771287536),
 ('42 cts', 4.550699300699301),
 ('30 cts', 4.498615967365968),
 ('nine mths', 4.453571428571428),
 ('73 pct', 4.4229452054794525),
 ('net profit', 4.156236497191416),
 ('fleet financial', 4.125),
 ('net loss', 4.056598347798808),
 ('hikes dividend', 4.015151515151516),
 ('65 cts', 3.988199300699301),
 ('declares one', 3.9166666666666665),
 ('50 cts', 3.868881118881119),
... ]