Why are strings considered tokens in C while arrays aren't?

This is a quite basic theoritical question. I started learning the C language. I came across the topic Tokens in C.

Quoting from geeksforgeeks.org,

> A token is the smallest element of a program that is meaningful to the compiler.Tokens can be classified as follows:
>
> 1. Keywords
> 2. Identifiers
> 3. Constants
> 4. Strings
> 5. Special Symbols
> 6. Operators

Why strings are considered as a token while arrays aren't?

Geeksforgeeks is almost as bad a source for learning as ChatGPT.

It is true that strings in C consists of null-terminated character arrays. But what it means to say is string literals, "these things". That is, a constant used to initialize character arrays or to use as a read-only string.

Similarly, "constants" does not refer to things like const int x=1; but rather just the number part 1 - this is what formal C means when it refers to an integer constant (sometimes also called "integer literal" although that term is strictly speaking not correct).

Note that tokens is mostly a concept that matters when writing macros, it's not a concept that beginners usually have to worry about. The formal grammar (C17 6.4), "lexical elements", groups everything in C in these groups/sub-chapters:

• Keywords
• Identifiers
• Universal character names
• Constants
• String literals
• Punctuators
• Header names
• Preprocessing numbers
• Comments

A token is an indivisible parsing unit.

• ; is a token.
• + is a token.
• == is a token.
• Decimal numeric literal 4 is a token.
• Decimal numeric literal 12 is a token.
• String literal "abc" is a token.
• Identifier foo is a token.
• Identifier int is a token.

However,

• Strings aren't tokens because strings aren't pieces of code. (But see string literals above.)
• Arrays aren't tokens because array aren't pieces of code.
• Array declarations (e.g. int a[4];) aren't tokens because they are made of multiple other tokens.
• Array initializers (e.g. { 4, 5, i+2 }) aren't tokens because they are made of multiple other tokens.

You can generally put spaces between tokens, but never within.

• 12 is not the same as 1 2
• "abc" is not the same as "a b c".
• foo is not the same as f o o.
• i+2 is the same as i + 2.
• {4,5,i+2} is the same as { 4, 5, i + 2 }.

When the compiler processes source code, it first splits them into tokens. Example:

printf("%d", 4 << 2);

This is turned into the following tokens:

• printf
• (
• "%d" -- a string literal
• ,
• 4
• <<
• 2
• )
• ;

An array declaration like int a[] = {1, 2, 3}; consists of multiple tokens, therefore it's not a token itself. The a here is a token though, but it's not specifically an array-token but more generally an identifier ("variable name").

Side note on printf(): That function itself will also kind-of tokenize the string it receives as first argument. The only distinction is whether a character is a % placeholder or not, so it's a much simpler. The principle stays the same though.

It is often better to go directly to the source. The C language syntax is described semi-formally in the C Standard (C17), Annex A. The first paragraph (A.1.1) states:

  token:
      keyword
      identifier
      constant
      string-literal
      punctuator

Notice that "string-literal" is specially mentioned as a token.

As to why: the C language is built up from the same layers as a natural language: letters, words and sentences. When the compiler reads a program file, it reads the characters of the file and groups them into tokens the same way we group letters into words when reading a book. It then interprets the program as a sequence of tokens the same way as we interpret a text as a sequence of words.

A string-literal being a token is simply a decision taken by the designers of the C language because it makes sense when describing the syntax and semantics of the language.