Why don't certain VS Code themes have semantic highlighting, and how can I get semantic highlighting for them?

In Visual Studio Code, I want to use the Tokyo Night colour theme. However, it sets variables, modules, class identifiers and properties to the same colour. I find this a bit confusing so I try to customise them to different colours. However, the Tokyo Night colour extension replaces the semantic token analyser with a TextMate-based one.

See images below. Compare VS Code's dark+ colour theme for Python, which recognises np's semantic token type as a module. Tokyo Night does not recognise this.

• darkplus_image
• tokyonight_image

How do I change the colour theme without changing the semantic analyser?

It might just be that the tokyo night theme hasn't enabled semantic highlighting for itself. See also https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#enablement-of-semantic-highlighting. If you look at the builtin theme JSON files of VS Code, you'll see that they have "semanticHighlighting": true. Ex. in the Solarized Dark Theme. You can see the commit where VS Code opted in to semantic highlighting for its builtin themes here: 41a4adb.

I currently don't see a line like that in in https://github.com/enkia/tokyo-night-vscode-theme/blob/master/themes/tokyo-night-color-theme.json. Actually, upon further digging (where I found this related discussion), Tokyo Night actually did at one point in its past opt in to semantic highlighting. See commit ce2b1f1, but decided to opt back out in commit 2dbedf6.

You can force semantic highlighting to be enabled for that theme by setting your editor.semanticHighlighting.enabled setting to true, or doing something like the following in settings.json:

"editor.semanticTokenColorCustomizations": {
    "[Tokyo Night]": {
        "enabled": true,
    },
    "[Tokyo Night Storm]": {
        "enabled": true,
    },
    "[Tokyo Night Light]": {
        "enabled": true,
    },
},