Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu – Unable to figure out which tensors are the problem

huangapple
go评论86阅读模式
英文:

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu - Unable to figure out which tensors are the problem

问题

It looks like you're encountering a "Expected all tensors to be on the same device" error, which means that some of your tensors are on the CPU while others are on the GPU. To fix this issue, you need to ensure that all tensors involved in operations are on the same device, either CPU or GPU.

Here are some steps you can take to resolve this issue:

  1. Ensure that your model (detector), as well as all tensors related to it (including inputs and targets), are on the same device. You can move tensors to the GPU using the .to(device) method.

  2. Specifically, make sure that the gt_bboxes_proj, gt_classes, and anc_boxes_all tensors in the get_req_anchors function are on the same device as your model and inputs.

  3. If you're using any custom functions or modules within your model or training loop that work with tensors, ensure that they also operate on tensors on the same device.

  4. Double-check that you haven't inadvertently mixed CPU and GPU operations within your code.

  5. It might also be helpful to explicitly set the device for any tensor that you create to avoid device mismatches.

Here's an example of how you can move a tensor to a specific device:

  1. # Move a tensor to the GPU
  2. tensor_on_gpu = tensor_on_cpu.to(device)
  4. # Move a tensor to the CPU
  5. tensor_on_cpu = tensor_on_gpu.to('cpu')

By ensuring that all tensors are on the same device (either CPU or GPU), you should be able to resolve this issue.

英文:

I am trying to train an object detection model on a GPU. The code is written in Pytorch.
There are existing questions with a query about the same error but unfortunately none of them worked out for me.

The GPU device declaration is as follows:

  1. device = torch.device("cuda:0")

My training loop is as follows:

  1. detector = TwoStageDetector(img_size, out_size, out_c, n_classes, roi_size)
  2. detector=detector.to(device)
  3. #detector.eval()
  4. #total_loss = detector(img_batch, gt_bboxes_batch, gt_classes_batch)
  5. #proposals_final, conf_scores_final, classes_final = detector.inference(img_batch)
  7. print("STARTING TRAINING")
  8. def training_loop(model, learning_rate, train_dataloader, n_epochs):
  9.     #model=model.to(device)
  10.     optimizer = optim.Adam(model.parameters(), lr=learning_rate)
  12.     model.train()
  13.     loss_list = []
  15.     for i in tqdm(range(n_epochs)):
  16.         total_loss = 0
  17.         for img_batch, gt_bboxes_batch, gt_classes_batch in train_dataloader:
  18.             img_batch=img_batch.to(device)
  19.             gt_bboxes_batch=gt_bboxes_batch.to(device)
  20.             gt_classes_batch=gt_classes_batch.to(device)
  21.             # forward pass
  22.             loss = model(img_batch, gt_bboxes_batch, gt_classes_batch)
  23.             # backpropagation
  24.             optimizer.zero_grad()
  25.             loss.backward()
  26.             optimizer.step()
  28.             total_loss += loss.item()
  30.         loss_list.append(total_loss)
  32.     return loss_list
  34. learning_rate = 1e-3
  35. n_epochs = 1000
  37. loss_list = training_loop(detector, learning_rate, od_dataloader, n_epochs)

The relevant model class from the model.py file is as follows:

  1. class TwoStageDetector(nn.Module):
  2.     def __init__(self, img_size, out_size, out_channels, n_classes, roi_size):
  3.         super().__init__() 
  4.         self.rpn = RegionProposalNetwork(img_size, out_size, out_channels)
  5.         self.classifier = ClassificationModule(out_channels, n_classes, roi_size)
  6.         
  7.     def forward(self, images, gt_bboxes, gt_classes):
  8.         total_rpn_loss, feature_map, proposals, \
  9.         positive_anc_ind_sep, GT_class_pos = self.rpn(images, gt_bboxes, gt_classes)
  10.         
  11.         # get separate proposals for each sample
  12.         pos_proposals_list = []
  13.         batch_size = images.size(dim=0)
  14.         for idx in range(batch_size):
  15.             proposal_idxs = torch.where(positive_anc_ind_sep == idx)[0]
  16.             proposals_sep = proposals[proposal_idxs].detach().clone()
  17.             pos_proposals_list.append(proposals_sep)
  18.         
  19.         cls_loss = self.classifier(feature_map, pos_proposals_list, GT_class_pos)
  20.         total_loss = cls_loss + total_rpn_loss
  21.         
  22.         return total_loss
  23.     
  24.     def inference(self, images, conf_thresh=0.5, nms_thresh=0.7):
  25.         batch_size = images.size(dim=0)
  26.         proposals_final, conf_scores_final, feature_map = self.rpn.inference(images, conf_thresh, nms_thresh)
  27.         cls_scores = self.classifier(feature_map, proposals_final)
  28.         
  29.         # convert scores into probability
  30.         cls_probs = F.softmax(cls_scores, dim=-1)
  31.         # get classes with highest probability
  32.         classes_all = torch.argmax(cls_probs, dim=-1)
  33.         
  34.         classes_final = []
  35.         # slice classes to map to their corresponding image
  36.         c = 0
  37.         for i in range(batch_size):
  38.             n_proposals = len(proposals_final[i]) # get the number of proposals for each image
  39.             classes_final.append(classes_all[c: c+n_proposals])
  40.             c += n_proposals
  41.             
  42.         return proposals_final, conf_scores_final, classes_final
  44. class RegionProposalNetwork(nn.Module):
  45.     def __init__(self, img_size, out_size, out_channels):
  46.         super().__init__()
  48.         self.img_height, self.img_width = img_size
  49.         self.out_h, self.out_w = out_size
  51.         # downsampling scale factor
  52.         self.width_scale_factor = self.img_width // self.out_w
  53.         self.height_scale_factor = self.img_height // self.out_h
  55.         # scales and ratios for anchor boxes
  56.         self.anc_scales = [2, 4, 6]
  57.         self.anc_ratios = [0.5, 1, 1.5]
  58.         self.n_anc_boxes = len(self.anc_scales) * len(self.anc_ratios)
  60.         # IoU thresholds for +ve and -ve anchors
  61.         self.pos_thresh = 0.7
  62.         self.neg_thresh = 0.3
  64.         # weights for loss
  65.         self.w_conf = 1
  66.         self.w_reg = 5
  68.         self.feature_extractor = FeatureExtractor()
  69.         self.proposal_module = ProposalModule(out_channels, n_anchors=self.n_anc_boxes)
  71.     def forward(self, images, gt_bboxes, gt_classes):
  72.         batch_size = images.size(dim=0)
  73.         feature_map = self.feature_extractor(images)
  75.         # generate anchors
  76.         anc_pts_x, anc_pts_y = gen_anc_centers(out_size=(self.out_h, self.out_w))
  77.         anc_base = gen_anc_base(anc_pts_x, anc_pts_y, self.anc_scales, self.anc_ratios, (self.out_h, self.out_w))
  78.         anc_boxes_all = anc_base.repeat(batch_size, 1, 1, 1, 1)
  80.         # get positive and negative anchors amongst other things
  81.         gt_bboxes_proj = project_bboxes(gt_bboxes, self.width_scale_factor, self.height_scale_factor, mode='p2a')
  83.         positive_anc_ind, negative_anc_ind, GT_conf_scores, \
  84.         GT_offsets, GT_class_pos, positive_anc_coords, \
  85.         negative_anc_coords, positive_anc_ind_sep = get_req_anchors(anc_boxes_all, gt_bboxes_proj, gt_classes)
  87.         # pass through the proposal module
  88.         conf_scores_pos, conf_scores_neg, offsets_pos, proposals = self.proposal_module(feature_map, positive_anc_ind, \
  89.                                                                                         negative_anc_ind,
  90.                                                                                         positive_anc_coords)
  92.         cls_loss = calc_cls_loss(conf_scores_pos, conf_scores_neg, batch_size)
  93.         reg_loss = calc_bbox_reg_loss(GT_offsets, offsets_pos, batch_size)
  95.         total_rpn_loss = self.w_conf * cls_loss + self.w_reg * reg_loss
  97.         return total_rpn_loss, feature_map, proposals, positive_anc_ind_sep, GT_class_pos

The get_iou_mat() function and get_req_anchors() function from the utils.py file are as follows:

  1. def get_iou_mat(batch_size, anc_boxes_all, gt_bboxes_all):
  2.     # flatten anchor boxes
  3.     anc_boxes_flat = anc_boxes_all.reshape(batch_size, -1, 4)
  4.     # get total anchor boxes for a single image
  5.     tot_anc_boxes = anc_boxes_flat.size(dim=1)
  6.     # create a placeholder to compute IoUs amongst the boxes
  7.     ious_mat = torch.zeros((batch_size, tot_anc_boxes, gt_bboxes_all.size(dim=1)))
  9.     # compute IoU of the anc boxes with the gt boxes for all the images
  10.     for i in range(batch_size):
  11.         gt_bboxes = gt_bboxes_all[i]
  12.         #gt_bboxes = gt_bboxes[None, :]
  13.         anc_boxes = anc_boxes_flat[i]
  14.         ious_mat[i, :] = ops.box_iou(anc_boxes, gt_bboxes)
  15.     return ious_mat
  18. def get_req_anchors(anc_boxes_all, gt_bboxes_all, gt_classes_all, pos_thresh=0.7, neg_thresh=0.2):
  19.     '''
  20.     Prepare necessary data required for training
  22.     Input
  23.     ------
  24.     anc_boxes_all - torch.Tensor of shape (B, w_amap, h_amap, n_anchor_boxes, 4)
  25.         all anchor boxes for a batch of images
  26.     gt_bboxes_all - torch.Tensor of shape (B, max_objects, 4)
  27.         padded ground truth boxes for a batch of images
  28.     gt_classes_all - torch.Tensor of shape (B, max_objects)
  29.         padded ground truth classes for a batch of images
  31.     Returns
  32.     ---------
  33.     positive_anc_ind -  torch.Tensor of shape (n_pos,)
  34.         flattened positive indices for all the images in the batch
  35.     negative_anc_ind - torch.Tensor of shape (n_pos,)
  36.         flattened positive indices for all the images in the batch
  37.     GT_conf_scores - torch.Tensor of shape (n_pos,), IoU scores of +ve anchors
  38.     GT_offsets -  torch.Tensor of shape (n_pos, 4),
  39.         offsets between +ve anchors and their corresponding ground truth boxes
  40.     GT_class_pos - torch.Tensor of shape (n_pos,)
  41.         mapped classes of +ve anchors
  42.     positive_anc_coords - (n_pos, 4) coords of +ve anchors (for visualization)
  43.     negative_anc_coords - (n_pos, 4) coords of -ve anchors (for visualization)
  44.     positive_anc_ind_sep - list of indices to keep track of +ve anchors
  45.     '''
  46.     # get the size and shape parameters
  47.     B, w_amap, h_amap, A, _ = anc_boxes_all.shape
  48.     N = gt_bboxes_all.shape[1]  # max number of groundtruth bboxes in a batch
  50.     # get total number of anchor boxes in a single image
  51.     tot_anc_boxes = A * w_amap * h_amap
  53.     # get the iou matrix which contains iou of every anchor box
  54.     # against all the groundtruth bboxes in an image
  55.     iou_mat = get_iou_mat(B, anc_boxes_all, gt_bboxes_all)
  56.     #print(iou_mat.shape)
  57.     # for every groundtruth bbox in an image, find the iou
  58.     # with the anchor box which it overlaps the most
  59.     max_iou_per_gt_box, _ = iou_mat.max(dim=1, keepdim=True)
  60.     #print(max_iou_per_gt_box.shape)
  61.     #print(max_iou_per_gt_box)
  62.     # get positive anchor boxes
  64.     # condition 1: the anchor box with the max iou for every gt bbox
  65.     #print(max_iou_per_gt_box > 0)
  66.     positive_anc_mask = torch.logical_and(iou_mat == max_iou_per_gt_box, max_iou_per_gt_box > 0)
  67.     #print(positive_anc_mask.shape)
  68.     # condition 2: anchor boxes with iou above a threshold with any of the gt bboxes
  69.     positive_anc_mask = torch.logical_or(positive_anc_mask, iou_mat > pos_thresh)
  70.     #print(positive_anc_mask.shape)
  72.     positive_anc_ind_sep = torch.where(positive_anc_mask)[0]  # get separate indices in the batch
  73.     # combine all the batches and get the idxs of the +ve anchor boxes
  74.     positive_anc_mask = positive_anc_mask.flatten(start_dim=0, end_dim=1)
  75.     positive_anc_ind = torch.where(positive_anc_mask)[0]
  77.     # for every anchor box, get the iou and the idx of the
  78.     # gt bbox it overlaps with the most
  79.     max_iou_per_anc, max_iou_per_anc_ind = iou_mat.max(dim=-1)
  80.     max_iou_per_anc = max_iou_per_anc.flatten(start_dim=0, end_dim=1)
  82.     # get iou scores of the +ve anchor boxes
  83.     GT_conf_scores = max_iou_per_anc[positive_anc_ind]
  85.     # get gt classes of the +ve anchor boxes
  87.     # expand gt classes to map against every anchor box
  88.     #print(gt_classes_all.shape)
  89.     gt_classes_expand = gt_classes_all.view(B, 1, N).expand(B, tot_anc_boxes, N)
  90.     # for every anchor box, consider only the class of the gt bbox it overlaps with the most
  91.     GT_class = torch.gather(gt_classes_expand, -1, max_iou_per_anc_ind.unsqueeze(-1)).squeeze(-1)
  92.     # combine all the batches and get the mapped classes of the +ve anchor boxes
  93.     GT_class = GT_class.flatten(start_dim=0, end_dim=1)
  94.     GT_class_pos = GT_class[positive_anc_ind]
  96.     # get gt bbox coordinates of the +ve anchor boxes
  98.     # expand all the gt bboxes to map against every anchor box
  99.     gt_bboxes_expand = gt_bboxes_all.view(B, 1, N, 4).expand(B, tot_anc_boxes, N, 4)
  100.     # for every anchor box, consider only the coordinates of the gt bbox it overlaps with the most
  101.     GT_bboxes = torch.gather(gt_bboxes_expand, -2,
  102.                              max_iou_per_anc_ind.reshape(B, tot_anc_boxes, 1, 1).repeat(1, 1, 1, 4))
  103.     # combine all the batches and get the mapped gt bbox coordinates of the +ve anchor boxes
  104.     GT_bboxes = GT_bboxes.flatten(start_dim=0, end_dim=2)
  105.     GT_bboxes_pos = GT_bboxes[positive_anc_ind]
  107.     # get coordinates of +ve anc boxes
  108.     anc_boxes_flat = anc_boxes_all.flatten(start_dim=0, end_dim=-2)  # flatten all the anchor boxes
  109.     positive_anc_coords = anc_boxes_flat[positive_anc_ind]
  111.     # calculate gt offsets
  112.     GT_offsets = calc_gt_offsets(positive_anc_coords, GT_bboxes_pos)
  114.     # get -ve anchors
  116.     # condition: select the anchor boxes with max iou less than the threshold
  117.     negative_anc_mask = (max_iou_per_anc < neg_thresh)
  118.     negative_anc_ind = torch.where(negative_anc_mask)[0]
  119.     # sample -ve samples to match the +ve samples
  120.     negative_anc_ind = negative_anc_ind[torch.randint(0, negative_anc_ind.shape[0], (positive_anc_ind.shape[0],))]
  121.     negative_anc_coords = anc_boxes_flat[negative_anc_ind]
  123.     return positive_anc_ind, negative_anc_ind, GT_conf_scores, GT_offsets, GT_class_pos, \
  124.            positive_anc_coords, negative_anc_coords, positive_anc_ind_sep

I have not placed any tensors from the utils.py file in the GPU explicitly.

The error stack trace is as follows:

  1. File "/home/main.py", line 353, in <module>
  2.     loss_list = training_loop(detector, learning_rate, od_dataloader, n_epochs)
  3.   File "/home/main.py", line 336, in training_loop
  4.     loss = model(img_batch, gt_bboxes_batch, gt_classes_batch)
  5.   File "/home/miniconda3/envs/pytor/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  6.     return forward_call(*args, **kwargs)
  7.   File "/home/model.py", line 215, in forward
  8.     positive_anc_ind_sep, GT_class_pos = self.rpn(images, gt_bboxes, gt_classes)
  9.   File "/home/miniconda3/envs/pytor/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  10.     return forward_call(*args, **kwargs)
  11.   File "/home/model.py", line 104, in forward
  12.     negative_anc_coords, positive_anc_ind_sep = get_req_anchors(anc_boxes_all, gt_bboxes_proj, gt_classes)
  13.   File "/home/utils.py", line 222, in get_req_anchors
  14.     iou_mat = get_iou_mat(B, anc_boxes_all, gt_bboxes_all)
  15.   File "/home/utils.py", line 181, in get_iou_mat
  16.     ious_mat[i, :] = ops.box_iou(anc_boxes, gt_bboxes)
  17.   File "/home/miniconda3/envs/pytor/lib/python3.9/site-packages/torchvision/ops/boxes.py", line 271, in box_iou
  18.     inter, union = _box_inter_union(boxes1, boxes2)
  19.   File "/home/miniconda3/envs/pytor/lib/python3.9/site-packages/torchvision/ops/boxes.py", line 244, in _box_inter_union
  20.     lt = torch.max(boxes1[:, None, :2], boxes2[:, :2])  # [N,M,2]
  21. RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

I have tried sending different tensors to the GPU in an attempt to get rid of this error, but without success. It would be great if someone could indicate what I am doing wrong. Thanks

UPDATE
I sent the anc_boxes_all variable in RegionProposalNetwork to the GPU, and this fixed the above error. But is now giving me the same error for something else:

  1. Traceback (most recent call last):
  2.   File "/home/main.py", line 352, in <module>
  3.     loss_list = training_loop(detector, learning_rate, od_dataloader, n_epochs)
  4.   File "/home/main.py", line 336, in training_loop
  5.     loss = model(img_batch, gt_bboxes_batch, gt_classes_batch)
  6.   File "/home/miniconda3/envs/pytor/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  7.     return forward_call(*args, **kwargs)
  8.   File "/home/model.py", line 207, in forward
  9.     positive_anc_ind_sep, GT_class_pos = self.rpn(images, gt_bboxes, gt_classes)
  10.   File "/home/miniconda3/envs/pytor/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  11.     return forward_call(*args, **kwargs)
  12.   File "/home/model.py", line 104, in forward
  13.     negative_anc_coords, positive_anc_ind_sep = get_req_anchors(anc_boxes_all, gt_bboxes_proj, gt_classes)
  14.   File "/home/utils.py", line 258, in get_req_anchors
  15.     GT_class = torch.gather(gt_classes_expand, -1, max_iou_per_anc_ind.unsqueeze(-1)).squeeze(-1)
  16. RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)

答案1

得分: 1

lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2]

boxes1 在 GPU 0 上，boxes2 在 CPU 上。

你可以展示一下你是如何定义你的模型、boxes1 和 boxes2 的吗？

你还可以展示一下你是在哪里定义 device 吗？

英文: 
  1.  lt = torch.max(boxes1[:, None, :2], boxes2[:, :2])  # [N,M,2]

boxes1 is on GPU zero and boxes2 is on CPU.

Can you show how you define your model, boxes1 and boxes2?

Can you also show where you define device?

huangapple
  • 本文由 发表于 2023年5月10日 22:19:13
  • 转载请务必保留本文链接：https://go.coder-hub.com/76219530.html
  • gpu
  • pytorch
  • tensor
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定