What is the influence of `use utf8` on `Encode`?

So I am hunting a wide-character problem in our DB routines, and stumbled upon the following oddity:

use utf8;

use Encode;

my $x = "ö";

my $decoded  = Encode::decode('UTF-8', $x);

my $encoded = Encode::encode('UTF-8', $decoded);

my $redecoded = Encode::decode('UTF-8', $encoded);

{
    use bytes;
    printf "original: %vx\n", $x;
    printf "decoded: %vx\n", $decoded;
    printf "encoded: %vx\n", $encoded;
    printf "redecoded: %vx\n", $redecoded;
}

Executing this script gives:

original: c3.b6
decoded: ef.bf.bd
encoded: ef.bf.bd
redecoded: ef.bf.bd

I do not think that 0xef 0xbf 0xbd is valid UTF-8. If I remove the use utf8 statement, the output is, as expected:

original: c3.b6
decoded: c3.b6
encoded: c3.b6
redecoded: c3.b6

So why is use utf8; changing the decoding semantics?

use utf8 tells Perl to decode the script using UTF-8 instead of ASCII. It has no effect on Encode. You are getting different results because you are passing different strings to decode.

You failed to notice this because of your incorrect use of use bytes;. (To be clear, using use bytes; is always incorrect.) Let's remove that and run your program again.

Specifically, I'll use the following:

printf "original: %vx\n", $x;
printf "decode('UTF-8', %vx): %vx\n", $x,       $decoded;
printf "encode('UTF-8', %vx): %vx\n", $decoded, $encoded;
printf "decode('UTF-8', %vx): %vx\n", $encoded, $redecoded;

With use utf8;:

original: f6
decode('UTF-8', f6): fffd
encode('UTF-8', fffd): ef.bf.bd
decode('UTF-8', ef.bf.bd): fffd

With use utf8;, Perl expects your program to be encoded using UTF-8, so your program has the equivalent of my $x = "\N{LATIN SMALL LETTER O WITH DIAERESIS}";. Put differently, your string contains a single char with value 0xf6. This is not valid UTF-8, so you get garbage (U+FFFD REPLACEMENT CHARACTER) when you pass it to decode 'UTF-8'.

Replace my $decoded = Encode::decode('UTF-8', $x); with my $decoded = $x; to get the expected results.

original: f6
decoded: f6
encode('UTF-8', f6): c3.b6
decode('UTF-8', c3.b6): f6

Without use utf8;:

original: c3.b6
decode('UTF-8', c3.b6): f6
encode('UTF-8', f6): c3.b6
decode('UTF-8', c3.b6): f6

Without use utf8;, Perl expects your program to be encoded using ASCII. It can't possibly contain ö as that is not an ASCII character. Instead, you have the equivalent of my $x = "\xc3\xb6"; (because string literals are "8-bit clean"). This is valid UTF-8, and can thus be decoded.