TypeError: 不支持的操作类型：’Tensor’ 和 ‘NoneType’

huangapple
go评论75阅读模式
英文:

TypeError: unsupported operand type(s) for +: 'Tensor' and 'NoneType'

问题

这是您提供的代码的翻译：

我正在尝试构建一个Transformer模型，但出现以下错误：

```none
TypeError: 不支持的操作类型 +: 'Tensor' 和 'NoneType'

错误发生在这段代码中：

xe = np.random.randint(0, 20_000, size=(8,512)) # 批大小为8 ;; 序列长度为512
xe_t = torch.tensor(xe).to(device)

xd = np.random.randint(0, 10_000, size=(8, 256))
xd_t = torch.tensor(xd).to(device)

maske = np.ones((8, 512))
maske[:, 256:] = 0
maske_t = torch.tensor(maske).to(device)

maskd = np.ones((8, 256))
maskd[:, 128:] = 0
maskd_t = torch.tensor(maskd).to(device)

out = transformer(xe_t, xd_t, maske_t, maskd_t)
out.shape

我正在尝试理解这个错误。以下是错误消息：

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[11], line 15
     12 maskd[:, 128:] = 0
     13 maskd_t = torch.tensor(maskd).to(device)
---> 15 out = transformer(xe_t, xd_t, maske_t, maskd_t)
     16 out.shape

File ~\anaconda3\envs\ml\lib\site-packages\torch\nn\modules\module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # 如果没有任何挂钩，我们希望跳过此函数中的其余逻辑，直接调用forward。
   1191 # 当我们使用jit时，如果没有任何挂钩，我们希望跳过此函数中的其余逻辑，只需调用forward。
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # 使用jit时不要调用函数
   1196 full_backward_hooks, non_full_backward_hooks = [], []

Cell In[8], line 8, in Transformer.forward(self, enc_input, dec_input, enc_mask, dec_mask)
      7 def forward(self, enc_input, dec_input, enc_mask, dec_mask):
----> 8     enc_output = self.encoder(enc_input, enc_mask)
      9     dec_output = self.decoder(enc_output, dec_input, enc_mask, dec_mask)
     10     return dec_output

File ~\anaconda3\envs\ml\lib\site-packages\torch\nn\modules\module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # 如果没有任何挂钩，我们希望跳过此函数中的其余逻辑，直接调用forward。
   1191 # 当我们使用jit时，如果没有任何挂钩，我们希望跳过此函数中的其余逻辑，只需调用forward。
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # 使用jit时不要调用函数
   1196 full_backward_hooks, non_full_backward_hooks = [], []

Cell In[6], line 30, in Encoder.forward(self, x, pad_mask)
     28 x = self.pos_encoding(x)
     29 for block in self.transformer_blocks:
---> 30     x = block(x, pad_mask)
     32 # 多对一（x的形状为N x T x D）
     33 # x = x[:, 0, :]
     35 x = self.ln(x)

File ~\anaconda3\envs\ml\lib\site-packages\torch\nn\modules\module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # 如果没有任何挂钩，我们希望跳过此函数中的其余逻辑，直接调用forward。
   1191 # 当我们使用jit时，如果没有任何挂钩，我们希望跳过此函数中的其余逻辑，只需调用forward。
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # 使用jit时不要调用函数
   1196 full_backward_hooks, non_full_backward_hooks = [], []

Cell In[3], line 17, in EncoderBlock.forward(self, x, pad_mask)
     16 def forward(self, x, pad_mask=None):
---> 17     x = self.ln1(x + self.mha(x, x, x, pad_mask))
     18     x = self.ln2(x + self.ann(x))
     19     x = self.dropout(x)

TypeError: 不支持的操作类型 +: 'Tensor' 和 'NoneType'`

我找不到代码中的任何问题，我尝试寻找解决方案。


<details>
<summary>英文:</summary>

I&#39;m trying to build a Transformer model, but it produces an error of

```none
TypeError: unsupported operand type(s) for +: &#39;Tensor&#39; and &#39;NoneType&#39;

It happens in this code:

xe = np.random.randint(0, 20_000, size=(8,512)) # batch size of 8 ;; sequence length 512
xe_t = torch.tensor(xe).to(device)

xd = np.random.randint(0, 10_000, size=(8, 256))
xd_t = torch.tensor(xd).to(device)

maske = np.ones((8, 512))
maske[:, 256:] = 0
maske_t = torch.tensor(maske).to(device)

maskd = np.ones((8, 256))
maskd[:, 128:] = 0
maskd_t = torch.tensor(maskd).to(device)

out = transformer(xe_t, xd_t, maske_t, maskd_t)
out.shape

I'm trying to understand the error.
Here is the error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[11], line 15
     12 maskd[:, 128:] = 0
     13 maskd_t = torch.tensor(maskd).to(device)
---&gt; 15 out = transformer(xe_t, xd_t, maske_t, maskd_t)
     16 out.shape

File ~\anaconda3\envs\ml\lib\site-packages\torch\nn\modules\module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don&#39;t have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

Cell In[8], line 8, in Transformer.forward(self, enc_input, dec_input, enc_mask, dec_mask)
      7 def forward(self, enc_input, dec_input, enc_mask, dec_mask):
----&gt; 8     enc_output = self.encoder(enc_input, enc_mask)
      9     dec_output = self.decoder(enc_output, dec_input, enc_mask, dec_mask)
     10     return dec_output

File ~\anaconda3\envs\ml\lib\site-packages\torch\nn\modules\module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don&#39;t have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

Cell In[6], line 30, in Encoder.forward(self, x, pad_mask)
     28 x = self.pos_encoding(x)
     29 for block in self.transformer_blocks:
---&gt; 30     x = block(x, pad_mask)
     32 # many-to-one (x has the shape N x T x D)
     33 # x = x[:, 0, :]
     35 x = self.ln(x)

File ~\anaconda3\envs\ml\lib\site-packages\torch\nn\modules\module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don&#39;t have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

Cell In[3], line 17, in EncoderBlock.forward(self, x, pad_mask)
     16 def forward(self, x, pad_mask=None):
---&gt; 17     x = self.ln1(x + self.mha(x, x, x, pad_mask))
     18     x = self.ln2(x + self.ann(x))
     19     x = self.dropout(x)

TypeError: unsupported operand type(s) for +: &#39;Tensor&#39; and &#39;NoneType&#39;`

I can't find anything off in my code, I tried looking for solutions.

答案1

得分: 3

错误提示表明您正在尝试将一个Tensor对象添加到一个NoneType对象中，这是不受支持的。这个问题发生在EncoderBlock类内的这一行：x = self.ln1(x + self.mha(x, x, x, pad_mask))

所以您必须确保传递给self.mha函数的pad_mask参数不是None，您需要修改代码，尝试这个：

class EncoderBlock(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout_rate):
        super(EncoderBlock, self).__init__()
        self.mha = MultiheadAttention(d_model, num_heads)
        self.ln1 = nn.LayerNorm(d_model)
        self.ann = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model),
        )
        self.ln2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout_rate)
        
    def forward(self, x, pad_mask=None):
        attn_output = self.mha(x, x, x, pad_mask)
        x = self.ln1(x + attn_output)
        x = self.ln2(x + self.ann(x))
        x = self.dropout(x)
        return x

不要忘记实现MultiheadAttention类或正确导入它，因为它没有包含在您提供的代码片段中。如果这个解决方案不起作用，请提供您正在使用的MultiheadAttention类或模块的实现，以便我更好地理解它。

英文:

bro The error you're encountering indicates that you are trying to add a Tensor object to a NoneType object, which is not supported. This issue is occurring in the line x = self.ln1(x + self.mha(x, x, x, pad_mask)) inside the EncoderBlock class
so you have to ensure that the pad_mask parameter passed to the self.mha function is not None you have to midfy the code try this one

class EncoderBlock(nn.Module): def init(self, d_model, num_heads, d_ff, dropout_rate): super(EncoderBlock, self).init() self.mha = MultiheadAttention(d_model, num_heads) self.ln1 = nn.LayerNorm(d_model) self.ann = nn.Sequential( nn.Linear(d_model, d_ff), nn.ReLU(), nn.Linear(d_ff, d_model), ) self.ln2 = nn.LayerNorm(d_model) self.dropout = nn.Dropout(dropout_rate def forward(self, x, pad_mask=None): attn_output = self.mha(x, x, x, pad_mask) x = self.ln1(x + attn_output) x = self.ln2(x + self.ann(x)) x = self.dropout(x) return x

dont forget to implemented the MultiheadAttention class or imported it correctly because it's is not included in the code snippet you provided and if this solution didn't work with you just ge me the implementation of the MultiheadAttention class or module you are using so ican understand it more

huangapple
  • 本文由 发表于 2023年6月8日 03:36:40
  • 转载请务必保留本文链接：https://go.coder-hub.com/76426603.html
  • neural-network
  • python
  • pytorch
  • torch
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定