Python - turns string into dictionary where keys are subheadings and values are links

In the middle of some text I have the following.

Some random text before.

----CAPITAL WORDS:
first subheading
https://link1
https://link2

second subheading
https://link3

third subheading
https://link4
https://link5
https://link6
https://link7

----MORE CAPITAL WORDS:
Some random text after.

I would like to extract the string between ----CAPITAL WORDS: and ----MORE CAPITAL WORDS and store it in a dictionary as follows

{
    'first subheading': ["https://link1", "https://link2"],
    'second subheading': ["https://link3"]
    'third subheading': ["https://link4", "https://link5", "https://link6", "https://link7"]
}

Attempt

pattern = r"CAPITAL WORDS:(.*?)(?:\n----MORE CAPITAL WORDS:|$)"
matches = re.search(pattern, descriptions[0], re.DOTALL)
lines = matches.group(1).strip().split("\n")

link_dict = {}
for line in lines:
    if line:
         pass # unsure how to continue

example = """Some random text before.

----CAPITAL WORDS:
first subheading
https://link1
https://link2

second subheading
https://link3

third subheading
https://link4
https://link5
https://link6
https://link7

----MORE CAPITAL WORDS:
Some random text after."""


out = {}
current_heading = None
for line in example.splitlines():
    if line.startswith('----'):
        pass
    elif line.startswith('http'):
        out[current_heading].append(line)
    elif line.islower():
        current_heading = line
        out[current_heading] = []

Output:

{'first subheading': ['https://link1', 'https://link2'], 
'second subheading': ['https://link3'], 
'third subheading': ['https://link4', 'https://link5', 'https://link6', 'https://link7']}

To match ----CAPITAL WORDS: and process all following parts without crossing either ----CAPITAL WORDS: or ----MORE CAPITAL WORDS: you could make use of the PyPi regex module and captures() for repeated capture groups.

(?:^----CAPITAL WORDS:\n|\G)(?!(----(?:MORE )?CAPITAL WORDS:))(?P<sub>\S.*)(?:\n(?!(?1))(?P<val>\S.*))+\s*

The pattern matches:

• (?: Non capture group
  • ^----CAPITAL WORDS:\n Match literally from the start of the string followed by a newline
  • | Or
  • \G Assert the current position at the end of the previous match
• ) Close the non capture group
• (?! Negative lookahead, assert that what is directly to the right is not
  • (----(?:MORE )?CAPITAL WORDS:) Capture in group 1, matching CAPITAL WORDS: with an optional leading MORE part
• ) Close the negative lookahead
• (?P<sub>\S.*) Capture group sub, match the single lined subheading (starting with at least a single non whitespace char to prevent matching empty lines)
• (?: Non capture group to repeat as a whole part
  • \n Match a newline
  • (?!(?1)) Assert that the pattern of group 1 is not directly to the right
  • (?P<val>\S.*) Capture in group val the single lines values like "https://link1"
• )+ Close the non capture group and repeat it 1+ times
• \s* Match optional whitespace chars

See a regex demo and a Python demo.

import regex
pattern = r"(?:^----CAPITAL WORDS:\n|\G)(?!(----(?:MORE )?CAPITAL WORDS:))(?P<sub>\S.*)(?:\n(?!(?1))(?P<val>\S.*))+\s*"
s = ("Some random text before.\n\n"
"----CAPITAL WORDS:\n"
"first subheading\n"
"https://link1\n"
"https://link2\n\n"
"second subheading\n"
"https://link3\n\n"
"third subheading\n"
"https://link4\n"
"https://link5\n"
"https://link6\n"
"https://link7\n\n"
"----MORE CAPITAL WORDS:\n"
"Some random text after.")
matches = regex.finditer(pattern, s, regex.MULTILINE)
dct = {}
for _, m in enumerate(matches):
dct[m.captures("sub")[0]] = m.captures("val")
print(dct)

Output

{
'first subheading': ['https://link1', 'https://link2'],
'second subheading': ['https://link3'],
'third subheading': ['https://link4', 'https://link5', 'https://link6', 'https://link7']
}