Replace invalid html tags with < and > to &lt; and &gt;

When sending a message to telegram with the help of a bot, html formatting is used, which means when trying to send message with <argument> there will be an error, and you need to replace these arrows with < and > but writing them every time in a pile of text is just not convenient as hell, I would like to try to make a regular that will replace such things automatically but do not touch valid html tags for example:

Example of a valid tag that does not need to be replaced

<a href="tg://user?id=1">Bot</a>

Invalid tag that needs to be replaced

<argument>

Here is the code I tried to make, but it doesn't work in the end

import re

def replace_invalid_tags(html_string):
    invalid_tag_pattern = r'<[^a-zA-Z/!?](.*?)>'
    fixed_html = re.sub(invalid_tag_pattern, r'<>', html_string)
    return fixed_html

html_string = '<a href="#">Link</a> <argument1> <argument2>'
fixed_html = replace_invalid_tags(html_string)
print(fixed_html)

I would personally suggest you use a Python HTML parsing or sanitizing
library to do the job. Regular expressions are great and I love them. But
in some cases I prefer using well-tested libraries which are especially
built for the purpose of the problem.

I'm not a Python programmer, but mostly a PHP programmer. In good
CMS projects, you can add some sanitizing libraries such as
HTMLPurifier and define rules.

In your case, some tags should be transformed to HTML entities, in order
to display as normal text and in some other cases the tags have to be
left as they are. Certainly some attributes and specific tags should be
also dropped (ex: <img onload="alert('xss attack')" or
<script>alert('bad')</script>. This is where parsers or sanitizing
libs will do a safer job.

Let's say that these tags are allowed:

• <a> with the href attribute. Probably no other attributes should
  be allowed. Typically, I would drop style="font-size: 100px".
• <strong> and <em> without attributes. What about the old <b> and
  <i> tags? I would transform them to respectively <strong> and <em> as they might be useful for readability but not allowed in
  Telegram.

All other tags should be converted to <var> (if allowed) with the content
converted to HTML special chars (< to < and > to >) as you
mentionned. Maybe it would be safe to handle other conversions if needed.

In Python, I see that you can use the
html-sanitizer library.

I see that one can define some preprocessor functions, typically
to convert some tags if needed. This means that you could create a
function that will convert all unauthorized tags to a <var> or
<pre> tag and fill its content with the escaped equivalent HTML of
the found tag. Some pre-built preprocessor functions already exist,
such as bold_span_to_strong() so there are some examples to solve your problem.

A pure regex solution to find invalid tags could be done with this:

<\s*/?\s*(?!(?:a|strong|em)\b)\w+[^>]*>

Demo here: https://regex101.com/r/xiZl1n/1

I'm accepting optional spaces, the slash for the closing tag and then
using a negative lookahead to avoid matching the tags you want to accept.
I added the word-boundary \b after the valid tags to avoid that
it accepts <argument> which starts with an "a" char. I want to match
the full word only.

You can then decide what to do with all your matches. If you directly
want to replace < by < then you can do this:

https://regex101.com/r/xiZl1n/3

EDIT: handle ->, >.<, =>, etc

I'm still convinced that a parser is the best choice. But you asked
if the regex could be changed to handle more cases. I personally
don't think one regex can do it. It will certainly be unsafe too.

But as I commented, you could try something in several steps:

1. Convert <i> tags to <em> and <b> tags to <strong> if
   you think it's worth it.

2. Find all valid tags, such as <a>, <strong>, <em> and
   replace them by [a], [strong] and respectively [em].
   This can be done with a pattern such as:

   <\s*(/?)\s*(?=(?:a|strong|em)\b)(\w+[^>]*)>
   

   and replace it by [\1\2]. In action:
   https://regex101.com/r/xiZl1n/4

3. Now you can replace < by < and > by >:

   • < becomes < : https://regex101.com/r/xiZl1n/5
   • > becomes > : https://regex101.com/r/xiZl1n/6

4. Transform back the valid tags to correct HTML. It will be
   the same regex as in step 1, but with brackets instead:

   \[\s*(/?)\s*((?:a|strong|em)\b)([^\]]*)\]
   

   and replace it by <\1\2\3>. In action:
   https://regex101.com/r/xiZl1n/8

   In this step, the capturing group n°3 contains all the
   tag attributes. This is where you could filter it and only
   accept some specific ones, such as href, id, title
   but remove all the others (ex: class, style, onclick).

It may be important to enable the i flag to be case-insensitive.
This is what it could look like in JavaScript:

<!-- begin snippet: js hide: false console: true babel: false -->

<!-- language: lang-js -->

const input = `<a href="https://www.example.com/page">Example page</a> is a <strong>valid</strong> tag.
< EM class="something"> is also allowed</em>
But <param> or <argument> should be escaped.
This also: <br/> <
br /> <img onload="alert('xss')" src="http://images.com/nice-cat.jpg" alt="Nice cat" />
<script type="text/javascript">alert('Bad JS code')</` + `script>
<a href="javascript:alert('XSS attack')" onclick="alert('xss')">A bad link</a>
<a href = http://test.com title="This is just a test">test.com</a>

Turn left <- or turn right ->
Also, accept this => or a smiley >.<

Accept <B>bold</B> and <i style="color:green">italic without style</i> converted to new tags.
Also strip <b href="https://www.google.com">wrong attributes</b>`;

// Attributes to drop are all design changes done with classes or style
// and all attributes such as onload, onclick, etc.
const regexAttributeToDrop = /^(?:style|class|on.*)$/i;
// The attributes which can have a URL.
const regexAttributeWithURL = /^(?:href|xlink:href|src)$/i;
// Only accept absolute URLs and not bad stuff like javascript:alert('xss')
const regexValidURL = /^(https?|ftp|mailto|tel):/i;

/**
 * Filter tag attributes, based on the tag name, if provided.
 *
 * @param string attributes All the attributes of the tag.
 * @param string tagName Optional tag name (in lowercase).
 * @return string The filtered string of attributes.
 */
function filterAttributes(attributes, tagName = '') {
  // Regex for attribute: $1 = attribute name, $2 = value, $3 = simple/double quote or nothing.
  const regexAttribute = /\s*([\w-]+)\s*=\s*((?:(["']).*?|[^"'=<>\s]+))/g;
  
  attributes = attributes.replace(regexAttribute, function (attrMatch, attrName, attrValue, quote) {
    // Don't keep attributes that can change the rendering or run JavaScript.
    if (name.match(regexAttributeToDrop)) {
      return '';
    }
    // Not an attribute to drop.
    else {
      // If the attribute is "href" or "xlink:href" then only accept full URLs
      // with the correct protocols and only for <a> tags.
      if (attrName.match(/^(?:xlink:)?href$/i)) {
        // If it's not a link then drop the attribute.
        if (tagName !== 'a') {
          return '';
        }
        // The attribute value can be quoted or not so we'll remove them.
        const url = attrValue.replace(/^['"]|['"]$/g, '');
        // If the URL is valid.
        if (url.match(regexValidURL)) {
          return ` ${attrName}="${url}"`;
        }
        // Invalid URL: drop href and notify visually.
        else {
          return ' class="invalid-url" title="Invalid URL"';
        }
      }
      // All other attributes: nothing special to do.
      else {
        return ` ${attrName}=${attrValue}`;
      }
    }
  });

  // Clean up: trim spaces around. If it's not empty then just add a space before.
  attributes = attributes.trim();
  if (attributes.length) {
    attributes = ' ' + attributes;
  }
  
  return attributes;
}

const steps = [
  {
    // Replace <b> by <strong>.
    search: /<\s*(\/?)\s*(b\b)([^>]*)>/gi,
    replace: '<$1strong>' // or '<$1strong$3>' to keep the attributes.
  },
  {
    // Replace <i> by <em>.
    search: /<\s*(\/?)\s*(i\b)([^>]*)>/gi,
    replace: '<$1em>' // or '<$1em$3>' to keep the attributes.
  },
  {
    // Transform accepted HTML tags into bracket tags.
    search: /<\s*(\/?)\s*(?=(?:a|strong|em)\b)(\w+[^>]*)>/gi,
    replace: '[$1$2]'
  },
  {
    // Replace "<" by "<".
    search: /</g,
    replace: '<'
  },
  {
    // Replace ">" by ">".
    search: />/g,
    replace: '>'
  },
  {
    // Transform the accepted tags back to correct HTML.
    search: /\[\s*(\/?)\s*((?:a|strong|em)\b)([^\]]*)\]/gi,
    // For the replacement, we'll provide a callback function
    // so that we can alter the attributes if needed.
    replace: function (fullMatch, slash, tagName, attributes) {
      // Convert the tag name to lowercase.
      tagName = tagName.toLowerCase();
      // If the slash group is empty then it's the opening tag.
      if (slash === '') {
        attributes = filterAttributes(attributes, tagName);
        return '<' + tagName + attributes + '>';
      }
      // The closing tag.
      else {
        return '</' + tagName + '>';
      }
    }
  },
  {
    // Optional: inject <br /> tags at each new lines to preserve user input.
    search: /(\r?\n)/gm,
    replace: '<br />$1'
  }
];

let output = input;

steps.forEach((step, i) => {
  output = output.replace(step.search, step.replace);
});

document.getElementById('input').innerText = input;
document.getElementById('output').innerText = output;
document.getElementById('rendered-output').innerHTML = output;

<!-- language: lang-css -->

body {
  font-family: Georgia, "Times New Roman", serif;
  margin: 0;
  padding: 0 1em 1em 1em;
}

p:first-child { margin-top: 0; }

pre {
  background: #f8f8f8;
  border: 1px solid gray;
  padding: .5em;
  overflow-x: scroll;
}

.box {
  background: white;
  box-shadow: 0 0 .5em rgba(0, 0, 0, 0.1);
  padding: 1em;
}

.invalid-url {
  color: darkred;
}

<!-- language: lang-html -->

<h1>Quick & dirty HTML filtering with regular expressions</h1>

<p>Input:</p>
<pre id="input"></pre>

<p>Output:</p>
<pre id="output"></pre>

<p>Rendered output:</p>
<div id="rendered-output" class="box"></div>

<!-- end snippet -->