英文:
Haystack: PromptNode takes too much time to load the model
问题
我使用了来自Haystack教程的以下代码:
lfqa_prompt = PromptTemplate("deepset/question-answering-with-references", output_parser=AnswerParser(reference_pattern=r"Document\[(\d+)\]"))
prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", default_prompt_template=lfqa_prompt)
pipe = Pipeline()
pipe.add_node(component=retriever, name="retriever", inputs=["Query"])
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["retriever"])
output = pipe.run(query="A question?")
print(output["answers"][0].answer)
第一次运行时,下面这行代码会花费时间,因为它会将模型下载到缓存中:
prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", default_prompt_template=lfqa_prompt)
我假设下一次运行时会使用缓存中的模型。正如预期的那样,它没有下载,但仍然需要很长时间。
我们能否通过保存已处理的模型来缩短这个时间?
英文:
I use the below code based on the tutorials from Haystack:
lfqa_prompt = PromptTemplate("deepset/question-answering-with-references", output_parser=AnswerParser(reference_pattern=r"Document\[(\d+)\]"))
prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", default_prompt_template=lfqa_prompt)
pipe = Pipeline()
pipe.add_node(component=retriever, name="retriever", inputs=["Query"])
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["retriever"])
output = pipe.run(query="A question?")
print(output["answers"][0].answer)
The very first time when I ran this the below line took time as it downloaded the model to my cache:
prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", default_prompt_template=lfqa_prompt)
My Assumption was for the next run it will use the cached model. As expected it's not downloading, but it's still taking a lot of time.
Can we reduce this time by saving the already processed model?
答案1
得分: 2
我无法完全复现你的示例,因为你还有一个检索器,但我使用了下面的最小示例:
from haystack import Pipeline, Document
from haystack.nodes import PromptTemplate, PromptNode
prompt_template = PromptTemplate("deepset/question-answering")
prompt_node = PromptNode(model_name_or_path="google/flan-t5-base", default_prompt_template=prompt_template)
pipe = Pipeline()
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["Query"])
output = pipe.run(query="What's the capital of Germany?", documents=[Document("Berlin is the capital of Germany.")])
print(output["results"])
我将你的模型更改为google/flan-t5-base
,并且每次运行脚本时都能立即产生正确的响应。我猜测你的检索器可能在进行一些耗时的设置。请尝试上面的示例。
英文:
I couldn't reproduce exactly your example as you have a retriever as well but I used a minimal example below:
from haystack import Pipeline, Document
from haystack.nodes import PromptTemplate, PromptNode
prompt_template = PromptTemplate("deepset/question-answering")
prompt_node = PromptNode(model_name_or_path="google/flan-t5-base", default_prompt_template=prompt_template)
pipe = Pipeline()
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["Query"])
output = pipe.run(query="What's the capital of Germany?", documents=[Document("Berlin is the capital of Germany.")])
print(output["results"])
I changed your model to google/flan-t5-base
, and the script instantly produced the correct response every time I ran it. My best guess is that your retriever might be doing some time-consuming setup. Please try the example above.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论