Regex with m flag in Perl vs. Python

I'm trying to automatically translate some simple Perl code with a regex to Python, and I'm having an issue. Here is the Perl code:

$stamp='[stamp]';
$message = "message\n";
$message =~ s/^/$stamp/gm;
print "$message";
[stamp]message

Here is my Python equivalent:

>>> import re
>>> re.sub(re.compile("^", re.M), "[stamp]", "message\n", count=0)
'[stamp]message\n[stamp]'

Note the answer is different (it has an extra [stamp] at the end). How do I generate code that has the same behavior for the regex?

Perl and Python's regex engines differ slightly on the definition of a "line"; Perl does not consider the empty string following a trailing newline in the input string to be a line, Python does.

Best solution I can come up with is to change "^" to r"^(?=.|\n)" (note r prefix on string to make it a raw literal; all regex should use raw literals). You can also simplify a bit by just calling methods on the compiled regex or call re.sub with the uncompiled pattern, and since count=0 is already the default, you can omit it. Thus, the final code would be either:

re.compile(r"^(?=.|\n)", re.M).sub("[stamp]", "message\n")

or:

re.sub(r"^(?=.|\n)", "[stamp]", "message\n", flags=re.M)

Even better would be:

start_of_line = re.compile(r"^(?=.|\n)", re.M)  # Done once up front

start_of_line.sub("[stamp]", "message\n")  # Done on demand

avoiding recompiling/rechecking compiled regex cache each time, by creating the compiled regex just once and reusing it.

Alternative solutions:

1. Split up the lines in a way that will match Perl's definition of a line, then use the non-re.MULTILINE version of the regex per line, then shove them back together, e.g.:

   start_of_line = re.compile(r"^")  # Compile once up front without re.M
   
   # Split lines, keeping ends, in a way that matches Perl's definition of a line
   # then substitute on line-by-line basis
   ''.join([start_of_line.sub("[stamp]", line) for line in "message\n".splitlines(keepends=True)])
   

2. Strip a single trailing newline, if it exists, up-front, perform regex substitution, add back newline (if applicable):

   message = '...'
   if message.endswith('\n'):
       result = start_of_line.sub("[stamp]", message[:-1]) + '\n'
   else:
       result = start_of_line.sub("[stamp]", message)
   

Neither option is as succinct/efficient as trying to tweak the regex, but if arbitrary user-supplied regex must be handled, there's always going to be a corner case, and pre-processing to something that removes the Perl/Python incompatibility is a lot safer.

Perl's multiline mode doesn't consider an empty string after the last newline to be a line of its own. That is, it treats A\nB and A\nB\n as both being two lines, while A\nB\nC as being three lines. This differs from Python's multine mode, which treats every newline as starting a new line:

> re.M: When specified, the pattern character '^' matches at the beginning of the string and at the beginning of each line (immediately following each newline)

You can mimic the behavior of Perl's multiline mode by adding a lookahead assertion for at least one character at the start of the line:

(?=.|\n)

Note that we need to explicitly permit \n with an alternative |'d pattern since by default . does not match \n. Without this, the pattern would fail to match starts of lines that are immediately followed by a \n.

Here's how this pattern behaves in your example:

>>> re.sub(re.compile(r"^(?=.|\n)", re.M), "[stamp]", "message\n", count=0)
'[stamp]message\n'
>>> re.sub(re.compile(r"^(?=.|\n)", re.M), "[stamp]", "message\nmessage", count=0)
'[stamp]message\n[stamp]message'
>>> re.sub(re.compile(r"^(?=.|\n)", re.M), "[stamp]", "message\nmessage\n", count=0)
'[stamp]message\n[stamp]message\n'
>>> re.sub(re.compile(r"^(?=.|\n)", re.M), "[stamp]", "message\n\n", count=0)
'message\n[stamp]\n'

The problem is that Perl doesn't consider the empty string after the \n to be a line of text, but Python does. So the Perl's RegEx code sees "message\n" as one line of text, but Python's RegEx code sees it as two.

You can resolve this difference by having the Python code check for a final \n. If it detects one, remove that \n before running the regular expression, and then add it back in after.

You will probably want to check all edge cases, too. For example, how do the Perl and Python code behave if the entire message itself is an empty string? (Will the Perl code do anything in that case? How about the Python code?)

If all your messages are guaranteed to be non-zero-length text ending in a newline, then you can probably get away with just removing the final \n, applying the Python regex code, and then appending that \n back in. But it would still be good form to consider all edge cases.

ADDITION:

Although it's tempting to come up with a regular expression in Python that exactly mimics the one in Perl, I wouldn't necessarily recommend it -- especially if the Python regex is so complicated that it's not easy to tell what it's doing at first or second glance.

Regular expressions don't always handle algorithmic logic very gracefully. So if you can eliminate regexp complexity by introducing some simple algorithmic logic, I would recommend doing that instead.

In other words, you won't win any awards by using overly-complicated regular expressions, so why not use a simple regex paired with simple non-regex algorithmic logic instead? (The future maintainers of your code will thank you!)

Here's a recommendation:

import re
message = "message\n"
message = re.sub(r"(?m:^)", "[stamp]", message)
# Remove the final "[stamp]" if it appears alone at the end:
message = message.removesuffix("[stamp]")

It's simple and easy to follow, and the only part of it that might be confusing is if you've never seen the m flag used inside of a regular expression like (?m:...) before.

Just be sure to test this out on edge cases (like empty messages) before you decide to use it. You don't want to leave your program's logic to behavior you didn't realize existed.