Regex for terminating string sequences syntax analysis

I am reading a code source file line by line and am attempting to detect whether or not there is a non-terminating string on a given line (which implies it continues to the next line). To accomplish this, I want to use regex to try and match non-terminating strings, but am running into an issue where strings can begin with " or ' and it's unclear to me how to reference my prior match within the regex itself so that it doesn't try to match the other token (the one not being used as the definition).

It is worth noting in my test strings that I am treating doubling up a character as escaping it, which is why there's an exception for "" or '' baked in.

So far I have this (Python flavored regex, so " is escaped via \"):

(?:\"(?:[^\"]|\"\")+$)|(?:'(?:[^']|'')+$)

Which does a good job on most non-terminated strings, however this test case revealed the flaw:

"this i""s a '  '' nnoying"

This matches from character 13-27, since the regex is unaware that we are currently within another string.

So I am trying to condense the regex into a single match case like so:

(?:(\"|')(?:[^\"]|\"\")+$)

Which almost works, but it will fail on this case:

'test string which " is totally invalid  "

I need a way to match either " or ', and then in the context of the rest of the regex, know which one I did match to be able to understand if the string is terminating or not. Otherwise, nesting a fake string will cause the detection to fail.

There's a bunch of potential gotchas between your first attempt here and what I think you need.

First, when trying to ignore escaped character sequences, I always put all the "ignore" items earlier in the list of alternates, so they are matched and treated as a 'single' character:

\"(\"\"|[^\"])+\"|\'(\'\'|[^\'])+\'

Also, it may work for your purposes, but you should also consider zero characters between a quote and end of line character... but that is getting away from a better solution, as it seems you want to ignore any valid quotes if they appear in the line before an invalid quote:

w = 'hi, ' + "hello
x = 'hi,  + "hello"
y = "hi, " + "hello"
z = "hi, " + "hello 'Bubba'"

should trigger on x and y, I believe, but y and z should not be matched

so you need to match the following, with positive look-behind:

a beginning of line, followed by
(   one or more non-quote characters
       OR
    any valid quotations
) -- any number of times
any invalid quotation
end of line

If we define these as follows:

one or more non-quote characters:
   [^\"']+

any valid quotation:
   (\"|')(?:|\\\1|(?!).)+

an invalid quotation:
   (\"|')(?:|$|(?!).)*

inside the structure:
   (?>^(?:<non-quote-chars>|<valid quote>)*<invalid quote>$

And if we adjust the back references for the second type of quote, we get:

   (?>^(?:[^\"']+|(\"|')(?:|\\\1|(?!).)+)*)(\"|')(?:|$|(?!).)*$

The (?> structure specifies atomic grouping which is fairly important in reducing unnecessary (and eventually even catastrophic) backtracking - python should support it after version 3.11, but if you make it a non-matching group, it can still work for very small lines (might just be slow). you didn't specify version of Python

I can try to add some more detail at some point if you like

One more potential improvement if you're searching large amounts of text instead of line by line:

(?>^(?:(?:(?!$|\"|').)+|(\"|')(?:|\\\1|(?!).)+)*)(\"|')(?:|$|(?!).)*$