Transformers always only use a single Linear layer for classification head?

For example, in the class BertForSequenceClassification definition, only one Linear layer is used for the classifier. If just one Linear layer is used, doesn’t it just do linear projection for pooled_out? Will such a classifier produce good predictions? Why not use multiple Linear layers? Does transformers offer any option for using multiple Linear layers as the classification head?

I looked at several other classes. They all use a single Linear layer as the classification head.

To add onto the previous answer,

1. Embedding layers (self.bert = BertModel(config) in your case) transform the original data (a sentence, an image etc.) into some semantic-aware vector spaces. This is where all the architecture designs come in (e.g. attention, cnn, lstm etc.), which are all far more superior than a simple FC for their chosen tasks. So if you have the capacity of adding multiple FCs, why not just add another attention block? On the other hand, the embeddings from a decent model should have large inter-class distance and small intra-class variance, which could easily be projected to their corresponding classes in a linear fashion, and a FC is more than enough.

2. It would be ideal to have the pretrained portion as big as possible such that, as a downstream user, I just have to train/finetune a tiny bit of the model (e.g. the fc classification layer)

Since there is an infinite number of downstream scenarios and not a one-fits-all head for a task, the transformers library only adds a head that is sufficient enough to perform the task. You need to play around with your own data and different architectures in case the performance doesn't meet your expectations.

The library is fully compatible with PyTorch and you can use the base class of each model as a module in your own NN:

import torch
import torch.nn as nn
from transformers import BertModel, BertTokenizer

class MyOwnBert(nn.Module):
    def __init__(self, model_id, num_labels):
        super(MyOwnBert, self).__init__()
        self.bert = BertModel.from_pretrained(model_id)
        
        self.my_fancy_outputLayer = nn.Sequential(
            nn.GELU(),
            nn.Dropout(0.1),
            nn.Linear(self.bert.config.hidden_size, self.bert.config.hidden_size),
            nn.GELU(),
            nn.Linear(self.bert.config.hidden_size, num_labels),
        )
        
    def forward(
        self,
        input_ids = None,
        attention_mask = None,
        token_type_ids = None,
        position_ids = None,
        labels = None,
    ):
        outputs = self.bert(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
        )

        pooled_output = outputs.pooler_output

        logits = self.my_fancy_outputLayer(pooled_output)

        return logits

model_id = "bert-base-uncased"
t = BertTokenizer.from_pretrained(model_id)
m = MyOwnBert(model_id, 4)
m(**t("this is just an example", return_tensors="pt"))

Output:

tensor([[ 0.0291, -0.0370,  0.0255,  0.0234]], grad_fn=<AddmmBackward0>)