Unique row no matter the column

Is there a way to make an entire row unique no matter what value the column is in?

So, if 1 is in col a, and 2 is in col b, then if you were to insert 2 into col a and 1 into col b an error would be thrown.

+---+---+
| a | b |
+---+---+
| 1 | 2 |
| 2 | 1 | <- Would throw error on insert since this already exists above
+---+---+

Define 2 virtual generated columns with the least and the greatest of the 2 values and set a unique constraint on them:

CREATE TABLE tablename (
  a INT NOT NULL, 
  b INT NOT NULL,
  x INT GENERATED ALWAYS AS (LEAST(a, b)),
  y INT GENERATED ALWAYS AS (GREATEST(a, b)),
  UNIQUE (x, y)
);

See the demo.<br/>

Or, for MySql 8.0+:

CREATE TABLE tablename (
  a INT NOT NULL, 
  b INT NOT NULL,
  UNIQUE ((LEAST(a, b)), (GREATEST(a, b))) -- don't miss the parentheses 
);

See the demo.<br/>

You could enforce it with a trigger. But I can't think of a good way to do this with a UNIQUE KEY constraint.

It would be easier if you change the way you store values, so they are in a single column, on multiple rows of an additional table. Then you could easy use UNIQUE KEY to enforce uniqueness.

If you're trying to represent related items:

CREATE TABLE item_group (
  group_id INT AUTO_INCREMENT PRIMARY KEY
);

CREATE TABLE item_group_members (
  group_id INT NOT NULL,
  member_id INT NOT NULL,
  PRIMARY KEY (group_id, member_id)
);

INSERT INTO item_group_members VALUES (1, 1), (1, 2);

If you need each member to appear in only one group:

ALTER TABLE item_group_members ADD UNIQUE KEY (member_id);

Have third column hash VARCHAR NOT NULL UNIQUE, where you would place sorted hash.
Sort it to always get same hash for values, no matter of their order: 1, 2, 2, 3 will have same hash as 3, 2, 1, 2

E.g. in PHP:

sort($columns);
$hash = md5(implode('|', $columns));

$columns['hash'] = $hash;

$sql = "INSERT INTO my_tbl VALUES (?, ?, ?)";
$stmt = $pdo->prepare($sql);
$stmt->execute($columns);

When inserting data check for duplicates using before-insert trigger like so:

if (exists(
  select 1
  from t
  where (least(a, b), greatest(a, b)) = (least(new.a, new.b), greatest(new.a, new.b))
)) then
  signal sqlstate value '45000' set message_text = 'cannot insert duplicate pair';
end if;

A simpler solution is to add CHECK constraint along with UNIQUE on your table:

alter table t add constraint unique (a, b)
alter table t add constraint check (a < b)

Now, you must store data so that a < b. If you try to insert 2, 1 the check fails. If you insert 1, 2 the unique constraint fails.

DB<>Fiddle