英文:
How to skip weights init when loading pretrained transformers model?
问题
-
我需要找出如何在开始时不初始化权重的情况下加载预训练的转换器模型（以节省时间和内存）？
saved_inits = torch.nn.init.kaiming_uniform_, torch.nn.init.uniform_, torch.nn.init.normal_ # 保留 torch.nn.init.kaiming_uniform_ = skip torch.nn.init.uniform_ = skip torch.nn.init.normal_ = skip model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path=args.model_path) torch.nn.init.kaiming_uniform_, torch.nn.init.uniform_, torch.nn.init.normal_ = saved_inits # 恢复 -
对于
nn.module子类，有torch.nn.utils.skip_init，但它不适用于AutoModelForCausalLM。
Quest: 找到一种在AutoModelForCausalLM（或任何类似的变换器类）中跳过权重初始化的方法，可以使用一些标准包装器或参数。
英文:
I need to find out how to load a pretrained transformer model without initializing weights in the beginning (to save time and memory)?
- I saw this code example, but this is not elegant:
saved_inits = torch.nn.init.kaiming_uniform_, torch.nn.init.uniform_, torch.nn.init.normal_ # preserving torch.nn.init.kaiming_uniform_ = skip torch.nn.init.uniform_ = skip torch.nn.init.normal_ = skip model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path=args.model_path) torch.nn.init.kaiming_uniform_, torch.nn.init.uniform_, torch.nn.init.normal_ = saved_inits # restoring - for
nn.modulesubclasses there istorch.nn.utils.skip_init, but it won't work withAutoModelForCausalLM
Quest: find a way to skip weights initialization in AutoModelForCausalLM (or any similar transformers class) either using some standard wrapper or parameter.
答案1
得分: 1
以下是翻译好的内容：
"答案在cronoik的评论中提到了："
"我使用Llama 30B进行了测试，发现加载时间加快了3倍，但内存使用没有增加。"
英文:
The answer was suggested in the comment by cronoik:
model = AutoModelForCausalLM.from_pretrained(
pretrained_model_name_or_path=args.model_path,
low_cpu_mem_usage=True
)
I tested it with Llama 30B and found 3x acceleration in loading time, though no gain in memory use.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论