Grab groups of matches based on strings that start a line

I am trying to build a regex that will capture groups of lines up to and including lines from ^INS through ^DMG. I am able to exclude INS*Y*G8 through DMG. However, I keep getting catastrophic backtracking.

INS*Y*G8*030**A***AC~
REF*0F*XXXXXXXX~
NM1*IL*1*JWHWWWW*TWDD****34*XXXXXXXXX~
N3*45874 WHYYWYW WTWYXW~
N4*DYXWHXVYW*NY*88980~
DMG*D8*19700101*M~
**INS*Y*01*030**A***AC~**
REF*0F*XXXXXXXXX~
NM1*IL*1*JWHWWWW*RRWTW****34*XXXXXXXXX~
PER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~
N3*45874 WHYYWYW WTWYXW~
N4*DYXWHXVYW*NY*88980~
**DMG*D8*20000101*F~**
**INS*Y*19*030**A***AC~**
REF*0F*XXXXXXXXX~
NM1*IL*1*JWHWWWW*GGDFS****34*XXXXXXXXX~
PER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~
N3*45874 WHYYWYW WTWYXW~
N4*DYXWHXVYW*NY*88980~
**DMG*D8*20000101*F~**
AMT*D2*100~
AMT*FK*100~
AMT*R*50~
AMT*C1*30~
AMT*P3*31~
AMT*B9*32~
NM1*31*1~
N3*45874 WHYYWYW WTWYXW~
N4*DYXWHXVYW*NY*88980~

In general, I need a regex that can capture groups of lines given a string that starts a line, up to and including the line that ends the capture group based on another given start of a line.

I have tried this (INS\*Y\*[^G8]+.*)(.*?)(?=DMG) and other variations unsuccessfully. What I am expecting is...

Group 1 should be:

**INS*Y*01*030**A***AC~**
REF*0F*XXXXXXXXX~
NM1*IL*1*JWHWWWW*RRWTW****34*XXXXXXXXX~
PER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~
N3*45874 WHYYWYW WTWYXW~
N4*DYXWHXVYW*NY*88980~
**DMG*D8*20000101*F~**

Group 2 should be:

**INS*Y*19*030**A***AC~**
REF*0F*XXXXXXXXX~
NM1*IL*1*JWHWWWW*GGDFS****34*XXXXXXXXX~
PER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~
N3*45874 WHYYWYW WTWYXW~
N4*DYXWHXVYW*NY*88980~
**DMG*D8*20000101*F~**

Thank you for your help.

With your shown samples and attempts please try following regex

(?:^|\n)(INS\*(?!Y\*G8)(?:.*\n)+?DMG\S+)

Here is the Online Demo for used Regex.

Here is the Complete python3 code written and tested in Python3 using re.findall module of it.

re.findall(r"(?m)(?:^|\n)(INS\*(?!Y\*G8)(?:.*\n)+?DMG\S+)",var)

Output will be as follows with your shown samples:

['INS*Y*01*030**A***AC~**\nREF*0F*XXXXXXXXX~\nNM1*IL*1*JWHWWWW*RRWTW****34*XXXXXXXXX~\nPER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~\nN3*45874 WHYYWYW WTWYXW~\nN4*DYXWHXVYW*NY*88980~\nDMG*D8*20000101*F~**', 'INS*Y*19*030**A***AC~**\nREF*0F*XXXXXXXXX~\nNM1*IL*1*JWHWWWW*GGDFS****34*XXXXXXXXX~\nPER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~\nN3*45874 WHYYWYW WTWYXW~\nN4*DYXWHXVYW*NY*88980~\nDMG*D8*20000101*F~**']

Explanation of Used regex:

(?:^|\n)     ##In a non-capturing group match starting of value OR new line.
(            ##Starting a capturing group from here.
  INS\*      ##Matching INS followed by literal * here.
  (?!Y\*G8)  ##Using negative look ahead to make sure Y*G8 is not present.
  (?:.*\n)+? ##In a non-capturing group matching till new line greedy match with 1 or more matches.
  DMG\S+     ##Matching DMG followed by continuous non-spaces.
)            ##Closing capturing group here.

Edit

If the leading ** in the example for **INS text are markers for bold text in the question, then you could write the pattern as:

^INS\*(?!Y\*G8).*(?:\n(?!INS\*|DMG\*).*)*\nDMG\*.*

See a regex101 demo

<hr>

You could use a single match starting with **INS and asserting that it is not followed by Y*G8

Then match all following lines that do not start with either **INS or **DMG

^\*\*INS(?!Y\*G8).*(?:\n(?!\*\*(?:INS|DMG)).*)*\n\*\*DMG.*

Explanation

• ^ Start of string
• \*\*INS(?!Y\*G8) Match **INS and assert that it is not directly followed by Y*G8
• .* Match the whole line
• (?: Non capture group to repeat as a whole part
  • \n Match a newline
  • (?!\*\*(?:INS|DMG)) Negative lookahead, assert not **INS or **DMG directly to the right
  • .* Match the whole line
• )* Close the non capture group and optionally repeat it to match all lines
• \n\*\*DMG.* Match a newline, then **DMG and the rest of th eline

Regex demo101.

Example code

import re

pattern = r"^\*\*INS(?!Y\*G8).*(?:\n(?!\*\*(?:INS|DMG)).*)*\n\*\*DMG.*"

s = ("INS*Y*G8*030**A***AC~\n"
	"REF*0F*XXXXXXXX~\n"
	"NM1*IL*1*JWHWWWW*TWDD****34*XXXXXXXXX~\n"
	"N3*45874 WHYYWYW WTWYXW~\n"
	"N4*DYXWHXVYW*NY*88980~\n"
	"DMG*D8*19700101*M~\n"
	"**INS*Y*01*030**A***AC~**\n"
	"REF*0F*XXXXXXXXX~\n"
	"NM1*IL*1*JWHWWWW*RRWTW****34*XXXXXXXXX~\n"
	"PER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~\n"
	"N3*45874 WHYYWYW WTWYXW~\n"
	"N4*DYXWHXVYW*NY*88980~\n"
	"**DMG*D8*20000101*F~**\n"
	"**INS*Y*19*030**A***AC~**\n"
	"REF*0F*XXXXXXXXX~\n"
	"NM1*IL*1*JWHWWWW*GGDFS****34*XXXXXXXXX~\n"
	"PER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~\n"
	"N3*45874 WHYYWYW WTWYXW~\n"
	"N4*DYXWHXVYW*NY*88980~\n"
	"**DMG*D8*20000101*F~**\n"
	"AMT*D2*100~\n"
	"AMT*FK*100~\n"
	"AMT*R*50~\n"
	"AMT*C1*30~\n"
	"AMT*P3*31~\n"
	"AMT*B9*32~\n"
	"NM1*31*1~\n"
	"N3*45874 WHYYWYW WTWYXW~\n"
	"N4*DYXWHXVYW*NY*88980~")

print(re.findall(pattern, s, re.M))

Output

[
'**INS*Y*01*030**A***AC~**\nREF*0F*XXXXXXXXX~\nNM1*IL*1*JWHWWWW*RRWTW****34*XXXXXXXXX~\nPER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~\nN3*45874 WHYYWYW WTWYXW~\nN4*DYXWHXVYW*NY*88980~\n**DMG*D8*20000101*F~**',
'**INS*Y*19*030**A***AC~**\nREF*0F*XXXXXXXXX~\nNM1*IL*1*JWHWWWW*GGDFS****34*XXXXXXXXX~\nPER*IP**TE*XXXXXXXXX*AP*XXXXXXXXX~\nN3*45874 WHYYWYW WTWYXW~\nN4*DYXWHXVYW*NY*88980~\n**DMG*D8*20000101*F~**'
]

You were a little vague on just what your desired stop & start patterns should be, so I assumed a start pattern of <begin line> **INS and an end pattern of <begin line> **DMG. If that's not exactly correct you can adjust the regex.

The key to this solution, is to set the right set of flags to scan multiple lines correctly. They are:

• "g" - global - continue to scan for all matches, not just the first
• "s" - single line - .matches newline, so multiple lines can be matched with .+
• "m" - mult-line - ^ & $ match <newline> as well as begin & end of string.
• "i" - ignore case - May not be necessary, but it can make the patterns shorter.

So, this solution is

r"^\*\*INS.*?^\*\*DMG.*?$"gmsi

...pretty simple. Find a **INS at the beginning of a line, continue to scan any character across multiple lines until you see a **DMG at the beginning of a line, then scan everything up to the next line end.

Here's s a screenprint in Regex101: