如何使用Huggingface transformers加载基于llama的fine-tuned peft/lora模型？

huangapple
go评论209阅读模式
英文:

How to load a fine-tuned peft/lora model based on llama with Huggingface transformers?

问题

我已经按你的要求翻译了以下内容：

尝试加载本地保存的模型

  1. model = AutoModelForCausalLM.from_pretrained("finetuned_model")

结果为 Killed。

尝试从hub加载模型：

  1. import torch
  2. from peft import PeftModel, PeftConfig
  3. from transformers import AutoModelForCausalLM, AutoTokenizer
  5. peft_model_id = "lucas0/empath-llama-7b"
  6. config = PeftConfig.from_pretrained(peft_model_id)
  7. model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
  8. tokenizer = AutoTokenizer.from_pretrained(cwd+"/tokenizer.model")
  10. # 加载Lora模型
  11. model = PeftModel.from_pretrained(model, peft_model_id)

结果为：

  1. AttributeError: /home/ubuntu/empath/lora/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

完整的堆栈跟踪

模型创建：

我使用PEFT和LoRa进行了微调：

  1. model = AutoModelForCausalLM.from_pretrained(
  2.     "decapoda-research/llama-7b-hf",
  3.     torch_dtype=torch.float16,
  4.     device_map='auto',
  5. )

我不得不下载并手动指定llama的分词器。

  1. tokenizer = LlamaTokenizer(cwd+"/tokenizer.model")
  2. tokenizer.pad_token = tokenizer.eos_token

至于训练：

  1. from peft import LoraConfig, get_peft_model
  3. config = LoraConfig(
  4.     r=8,
  5.     lora_alpha=16,
  6.     target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
  7.     lora_dropout=0.05,
  8.     bias="none",
  9.     task_type="CAUSAL_LM"
  10. )
  12. model = get_peft_model(model, config)
  14. data = pd.read_csv("my_csv.csv")
  15. dataset = Dataset.from_pandas(data)
  16. tokenized_dataset = dataset.map(lambda samples: tokenizer(samples["text"]))
  18. trainer = transformers.Trainer(
  19.     model=model,
  20.     train_dataset=tokenized_dataset,
  21.     args=transformers.TrainingArguments(
  22.         per_device_train_batch_size=4,
  23.         gradient_accumulation_steps=4,
  24.         warmup_steps=100,
  25.         max_steps=100,
  26.         learning_rate=1e-3,
  27.         fp16=True,
  28.         logging_steps=1,
  29.         output_dir='outputs',
  30.     ),
  31.     data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
  32. )
  34. model.config.use_cache = True  # 消除警告，请在推断时重新启用！
  35. trainer.train()

并且本地保存了模型：

  1. trainer.save_model(cwd+"/finetuned_model")
  2. print("saved trainer locally")

以及推送到hub：

  1. model.push_to_hub("lucas0/empath-llama-7b", create_pr=1)

如何加载我微调的模型？

英文:

I've followed this tutorial (colab notebook) in order to finetune my model.

Trying to load my locally saved model

  1. model = AutoModelForCausalLM.from_pretrained("finetuned_model")

yields Killed.

Trying to load model from hub:

yields

  1. import torch
  2. from peft import PeftModel, PeftConfig
  3. from transformers import AutoModelForCausalLM, AutoTokenizer
  5. peft_model_id = "lucas0/empath-llama-7b"
  6. config = PeftConfig.from_pretrained(peft_model_id)
  7. model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
  8. tokenizer = AutoTokenizer.from_pretrained(cwd+"/tokenizer.model")
  10. # Load the Lora model
  11. model = PeftModel.from_pretrained(model, peft_model_id)

yields

  1. AttributeError: /home/ubuntu/empath/lora/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

full stacktrace

Model Creation:

I have finetuned a model using PEFT and LoRa:

  1. model = AutoModelForCausalLM.from_pretrained(
  2. "decapoda-research/llama-7b-hf",
  3. torch_dtype=torch.float16,
  4. device_map='auto',
  5. )

I had to download and manually specify the llama tokenizer.

  1. tokenizer = LlamaTokenizer(cwd+"/tokenizer.model")
  2. tokenizer.pad_token = tokenizer.eos_token

to the training:

  1. from peft import LoraConfig, get_peft_model
  3. config = LoraConfig(
  4.     r=8,
  5.     lora_alpha=16,
  6.     target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
  7.     lora_dropout=0.05,
  8.     bias="none",
  9.     task_type="CAUSAL_LM"
  10. )
  12. model = get_peft_model(model, config)
  14. data = pd.read_csv("my_csv.csv")
  15. dataset = Dataset.from_pandas(data)
  16. tokenized_dataset = dataset.map(lambda samples: tokenizer(samples["text"]))
  18. trainer = transformers.Trainer(
  19.     model=model,
  20.     train_dataset=tokenized_dataset,
  21.     args=transformers.TrainingArguments(
  22.         per_device_train_batch_size=4,
  23.         gradient_accumulation_steps=4,
  24.         warmup_steps=100,
  25.         max_steps=100,
  26.         learning_rate=1e-3,
  27.         fp16=True,
  28.         logging_steps=1,
  29.         output_dir='outputs',
  30.     ),
  31.     data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
  32. )
  33. model.config.use_cache = True  # silence the warnings. Please re-enable for inference!
  34. trainer.train()

and saved it locally with:

  1. trainer.save_model(cwd+"/finetuned_model")
  2. print("saved trainer locally")

as well as to the hub:

  1. model.push_to_hub("lucas0/empath-llama-7b", create_pr=1)

How can I load my finetuned model?

答案1

得分: 3

要加载经过微调的 peft/lora 模型，请查看 guanco 示例，https://stackoverflow.com/a/76372390/610569

  1. import torch
  2. from peft import PeftModel    
  3. from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer
  5. model_name = "decapoda-research/llama-7b-hf"
  6. adapters_name = "lucas0/empath-llama-7b"
  8. print(f"开始加载模型 {model_name} 到内存中")
  10. m = AutoModelForCausalLM.from_pretrained(
  11.     model_name,
  12.     #load_in_4bit=True,
  13.     torch_dtype=torch.bfloat16,
  14.     device_map={"": 0}
  15. )
  16. m = PeftModel.from_pretrained(m, adapters_name)
  17. m = m.merge_and_unload()
  18. tok = LlamaTokenizer.from_pretrained(model_name)
  19. tok.bos_token_id = 1
  21. stop_token_ids = [0]
  23. print(f"成功加载模型 {model_name} 到内存中")

为了正确加载模型，您至少需要一个 A10g GPU 运行时。

有关更多详细信息，请参阅：

英文:

To load a fine-tuned peft/lora model, take a look at the guanco example, https://stackoverflow.com/a/76372390/610569

  1. import torch
  2. from peft import PeftModel    
  3. from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer
  5. model_name = "decapoda-research/llama-7b-hf"
  6. adapters_name = "lucas0/empath-llama-7b"
  8. print(f"Starting to load the model {model_name} into memory")
  10. m = AutoModelForCausalLM.from_pretrained(
  11.     model_name,
  12.     #load_in_4bit=True,
  13.     torch_dtype=torch.bfloat16,
  14.     device_map={"": 0}
  15. )
  16. m = PeftModel.from_pretrained(m, adapters_name)
  17. m = m.merge_and_unload()
  18. tok = LlamaTokenizer.from_pretrained(model_name)
  19. tok.bos_token_id = 1
  21. stop_token_ids = [0]
  23. print(f"Successfully loaded the model {model_name} into memory")

You will need an A10g GPU runtime minimally to load the model properly.

For more details see

答案2

得分: 0

你可以在推送后像这样加载。我成功地使用以下代码片段完成了这个操作：

  1. # 安装所需的库
  2. # pip install peft transformers
  4. import torch
  5. from peft import PeftModel, PeftConfig
  6. from transformers import LlamaTokenizer, LlamaForCausalLM
  7. from accelerate import infer_auto_device_map, init_empty_weights
  9. peft_model_id = "--path--"  # 替换为你的模型路径
  11. # 从预训练模型配置中加载配置
  12. config = PeftConfig.from_pretrained(peft_model_id)
  14. # 使用LlamaForCausalLM加载预训练模型
  15. model1 = LlamaForCausalLM.from_pretrained(
  16.     config.base_model_name_or_path,
  17.     torch_dtype='auto',
  18.     device_map='auto',
  19.     offload_folder="offload",
  20.     offload_state_dict=True
  21. )
  23. # 使用LlamaTokenizer加载tokenizer
  24. tokenizer = LlamaTokenizer.from_pretrained(config.base_model_name_or_path)
  26. # 加载Lora模型
  27. model1 = PeftModel.from_pretrained(model1, peft_model_id)

注意：请将peft_model_id替换为你实际的模型路径。

英文:

You can load like this after pushing. I did using the following snippet successfully .

  1. # pip install peft transformers
  2. import torch
  3. from peft import PeftModel, PeftConfig
  4. from transformers import LlamaTokenizer, LlamaForCausalLM
  5. from accelerate import infer_auto_device_map, init_empty_weights
  7. peft_model_id = "--path--"
  9. config = PeftConfig.from_pretrained(peft_model_id)
  11. model1 = LlamaForCausalLM.from_pretrained(
  12.     config.base_model_name_or_path,
  13.     torch_dtype='auto',
  14.     device_map='auto',
  15.     offload_folder="offload", offload_state_dict = True
  16. )
  17. tokenizer = LlamaTokenizer.from_pretrained(config.base_model_name_or_path)
  19. # Load the Lora model
  20. model1 = PeftModel.from_pretrained(model, peft_model_id)

huangapple
  • 本文由 发表于 2023年6月13日 01:34:46
  • 转载请务必保留本文链接：https://go.coder-hub.com/76459034.html
  • huggingface-transformers
  • llama-index
  • peft
  • python
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定