如何使用交错的Hugging Face数据集创建一个PyTorch数据加载器？

huangapple
go评论156阅读模式
英文:

How does one create a pytoch data loader using an interleaved hugging face dataset?

问题

当我交错数据集、获取一个经过标记的批次，并将批次提供给PyTorch数据加载器时，我收到错误消息：

  1. ---------------------------------------------------------------------------
  2. TypeError                                 Traceback (most recent call last)
  3. /usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)
  4.     126         try:
  5. --> 127             return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  6.     128         except TypeError:
  8. 9 frames
  9. TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>
  11. During handling of the above exception, another exception occurred:
  13. TypeError                                 Traceback (most recent call last)
  14. /usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)
  15.     148                 return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed]
  16.     149 
  17. --> 150     raise TypeError(default_collate_err_msg_format.format(elem_type))
  18.     151 
  19.     152 
  21. TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>

为什么？为什么单独的数据集c4和wiki-text没有出现这个错误？只有交错的数据集会出现这个错误吗？

理想情况下，我不想编写自己的collate_function。

英文:

When I interleave data sets, get a tokenized batch, feed the batch to the pytorch data loader, I get errors:

  1. # -*- coding: utf-8 -*-
  2. """issues with dataloader and custom data sets
  4. Automatically generated by Colaboratory.
  6. Original file is located at
  7.     https://colab.research.google.com/drive/1sbs95as_66mtK9VK_vbaE9gLE-Tjof1-
  8. """
  10. !pip install datasets
  11. !pip install pytorch
  12. !pip install transformers
  14. token = None
  15. batch_size = 10
  16. from datasets import load_dataset
  17. import torch
  18. from transformers import GPT2Tokenizer, GPT2LMHeadModel
  20. tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
  21. if tokenizer.pad_token_id is None:
  22.   tokenizer.pad_token = tokenizer.eos_token
  23. probe_network = GPT2LMHeadModel.from_pretrained("gpt2")
  24. device = torch.device(f"cuda:{0}" if torch.cuda.is_available() else "cpu")
  25. probe_network = probe_network.to(device)
  27. # -- Get batch from dataset
  28. from datasets import load_dataset
  29. # path, name = 'brando/debug1_af', 'debug1_af'
  30. path, name = 'brando/debug0_af', 'debug0_af'
  31. remove_columns = []
  32. dataset = load_dataset(path, name, streaming=True, split="train", token=token).with_format("torch")
  33. print(f'{dataset=}')
  34. batch = dataset.take(batch_size)
  35. # print(f'{next(iter(batch))=}')
  37. # - Prepare functions to tokenize batch
  38. def preprocess(examples):  # gets the raw text batch according to the specific names in table in data set & tokenize
  39.     return tokenizer(examples["link"], padding="max_length", max_length=128, truncation=True, return_tensors="pt")
  40. def map(batch):  # apply preprocess to batch to all examples in batch represented as a dataset
  41.     return batch.map(preprocess, batched=True, remove_columns=remove_columns)
  42. tokenized_batch = batch.map(preprocess, batched=True, remove_columns=remove_columns)
  43. tokenized_batch = map(batch)
  44. # print(f'{next(iter(tokenized_batch))=}')
  46. from torch.utils.data import Dataset, DataLoader, SequentialSampler
  47. dataset = tokenized_batch
  48. print(f'{type(dataset)=}')
  49. print(f'{dataset.__class__=}')
  50. print(f'{isinstance(dataset, Dataset)=}')
  51. # for i, d in enumerate(dataset):
  52. #     assert isinstance(d, dict)
  53. #     # dd = dataset[i]
  54. #     # assert isinstance(dd, dict)
  55. loader_opts = {}
  56. classifier_opts = {}
  57. # data_loader = DataLoader(dataset, shuffle=False, batch_size=loader_opts.get('batch_size', 1),
  58. #                         num_workers=loader_opts.get('num_workers', 0), drop_last=False, sampler=SequentialSampler(range(512))  )
  59. data_loader = DataLoader(dataset, shuffle=False, batch_size=loader_opts.get('batch_size', 1),
  60.                     num_workers=loader_opts.get('num_workers', 0), drop_last=False, sampler=None)
  61. print(f'{iter(data_loader)=}')
  62. print(f'{next(iter(data_loader))=}')
  63. print('Done\a')

with error:

  1. ---------------------------------------------------------------------------
  2. TypeError                                 Traceback (most recent call last)
  3. /usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)
  4.     126         try:
  5. --> 127             return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  6.     128         except TypeError:
  8. 9 frames
  9. TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>
  11. During handling of the above exception, another exception occurred:
  13. TypeError                                 Traceback (most recent call last)
  14. /usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)
  15.     148                 return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed]
  16.     149 
  17. --> 150     raise TypeError(default_collate_err_msg_format.format(elem_type))
  18.     151 
  19.     152 
  21. TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>

why? And why doesn't the single data set c4 and wiki-text give this error? Only interleaved data sets?

Ideally I don't want to write my own collate_function.

答案1

得分: 0

以下是您提供的代码的翻译部分：

  1. For some reason when the data sets are intersected the collate function gets confused because there are extra rows so it doesn't know how to merge things? The way I fixed it is by only keeping the columns I want:

由于数据集交集时存在额外的行，导致整理函数感到困惑，因此不知道如何合并这些数据？我解决这个问题的方法是只保留我想要的列：

  2. but also doing this in the collate works if you know the text field you want (assuming `"text"` due to how common it is):

但如果你知道你想要的文本字段（假设为 "text"，因为它很常见），在整理中这样做也有效：

  2. full code:
  3. ```python
  4. 完整代码：

希望这对您有所帮助。如果您需要进一步的翻译或有其他问题，请随时提出。

英文:

For some reason when the data sets are intersected the collate function gets confused because there are extra rows so it doesn't know how to merge things? The way I fixed it is by only keeping the columns I want:

  1.     # -- Get data set
  2.     # remove_columns = ['text', 'timestamp', 'url']
  3.     keep_col = ['text']
  4.     # keep the strings in dataaset.column_names that intersect with keep_col str list, one liner
  5.     print('-- interleaving datasets')
  6.     datasets = [load_dataset(path, name, streaming=True, split="train").with_format("torch") for path, name in zip(path, name)]
  7.     [print(f'{dataset.description=}') for dataset in datasets]
  8.     dataset = interleave_datasets(datasets, probabilities)
  9.     remove_columns = [col for col in dataset.column_names if col not in keep_col]
  10.     print(f'{dataset=}')
  11.     batch = dataset.take(batch_size)

but also doing this in the collate works if you know the text field you want (assuming "text" due to how common it is):

  1.     def collate_tokenize(data):
  2.         print(f'{data[0]=}')
  3.         text_batch = [element["text"] for element in data]
  4.         tokenized = tokenizer(text_batch, padding='longest', truncation=True, return_tensors='pt')
  5.         return tokenized
  6.     data_loader = DataLoader(tokenized_batch, shuffle=False, batch_size=8, num_workers=0, drop_last=False, collate_fn=collate_tokenize)
  7.     # data_loader = DataLoader(tokenized_batch, shuffle=False, batch_size=8, num_workers=0, drop_last=False)
  8.     # num_batches = len(list(data_loader))
  9.     batch = next(iter(data_loader))
  10.     print(f'{batch=}')
  11.     print('Done!\a')

full code:

  1. def test_interleaved_data_set_2_data_loader():
  2.     """ https://colab.research.google.com/drive/1QWDhA6Q64qijXYnwIGn63Aq9Eg5qt8tQ#scrollTo=Wjyy6QYimvIm """
  3.     remove_columns = []
  4.     # -- Get probe network
  5.     from datasets import load_dataset
  6.     import torch
  7.     from transformers import GPT2Tokenizer, GPT2LMHeadModel
  9.     tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
  10.     if tokenizer.pad_token_id is None:
  11.         tokenizer.pad_token = tokenizer.eos_token
  12.     probe_network = GPT2LMHeadModel.from_pretrained("gpt2")
  13.     device = torch.device(f"cuda:{0}" if torch.cuda.is_available() else "cpu")
  14.     probe_network = probe_network.to(device)
  16.     from datasets import interleave_datasets
  18.     path, name = ['c4', 'wikitext'], ['en', 'wikitext-103-v1']
  19.     probabilities = [1.0/len(path)] * len(path)
  20.     batch_size = 512
  22.     # -- Get data set
  23.     # remove_columns = ['text', 'timestamp', 'url']
  24.     keep_col = ['text']
  25.     # keep the strings in dataaset.column_names that intersect with keep_col str list, one liner
  26.     print('-- interleaving datasets')
  27.     datasets = [load_dataset(path, name, streaming=True, split="train").with_format("torch") for path, name in zip(path, name)]
  28.     [print(f'{dataset.description=}') for dataset in datasets]
  29.     dataset = interleave_datasets(datasets, probabilities)
  30.     remove_columns = [col for col in dataset.column_names if col not in keep_col]
  31.     print(f'{dataset=}')
  32.     batch = dataset.take(batch_size)
  34.     # - Prepare functions to tokenize batch
  35.     def preprocess(examples):
  36.         return tokenizer(examples["text"], padding="max_length", max_length=128, truncation=True, return_tensors="pt")
  37.     def map(batch):
  38.         return batch.map(preprocess, batched=True, remove_columns=remove_columns)
  39.     # tokenized_batch = batch.map(preprocess, batched=True, remove_columns=remove_columns)
  40.     tokenized_batch = map(batch)
  41.     print(f'{next(iter(tokenized_batch))=}')
  43.     # -- Get data loader
  44.     from torch.utils.data import DataLoader, Dataset
  46.     # def collate_tokenize(data):
  47.     #     print(f'{data[0]=}')
  48.     #     text_batch = [element["text"] for element in data]
  49.     #     tokenized = tokenizer(text_batch, padding='longest', truncation=True, return_tensors='pt')
  50.     #     return tokenized
  51.     # data_loader = DataLoader(tokenized_batch, shuffle=False, batch_size=8, num_workers=0, drop_last=False, collate_fn=collate_tokenize)
  52.     data_loader = DataLoader(tokenized_batch, shuffle=False, batch_size=8, num_workers=0, drop_last=False)
  53.     # num_batches = len(list(data_loader))
  54.     batch = next(iter(data_loader))
  55.     print(f'{batch=}')
  56.     print('Done!\a')

huangapple
  • 本文由 发表于 2023年8月11日 02:21:06
  • 转载请务必保留本文链接：https://go.coder-hub.com/76878387.html
  • huggingface
  • huggingface-datasets
  • python
  • pytorch
  • pytorch-dataloader
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定