Haystack:PromptNode加载模型的时间太长了。

huangapple go评论76阅读模式
英文:

Haystack: PromptNode takes too much time to load the model

问题

我使用了来自Haystack教程的以下代码:

lfqa_prompt = PromptTemplate("deepset/question-answering-with-references", output_parser=AnswerParser(reference_pattern=r"Document\[(\d+)\]"))

prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", default_prompt_template=lfqa_prompt)

pipe = Pipeline()
pipe.add_node(component=retriever, name="retriever", inputs=["Query"])
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["retriever"])

output = pipe.run(query="A question?")
print(output["answers"][0].answer)

第一次运行时,下面这行代码会花费时间,因为它会将模型下载到缓存中:

prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", default_prompt_template=lfqa_prompt)

我假设下一次运行时会使用缓存中的模型。正如预期的那样,它没有下载,但仍然需要很长时间。

我们能否通过保存已处理的模型来缩短这个时间?

英文:

I use the below code based on the tutorials from Haystack:

    lfqa_prompt = PromptTemplate("deepset/question-answering-with-references", output_parser=AnswerParser(reference_pattern=r"Document\[(\d+)\]"))

    prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", default_prompt_template=lfqa_prompt)

    pipe = Pipeline()
    pipe.add_node(component=retriever, name="retriever", inputs=["Query"])
    pipe.add_node(component=prompt_node, name="prompt_node", inputs=["retriever"])
    
    output = pipe.run(query="A question?")
    print(output["answers"][0].answer)

The very first time when I ran this the below line took time as it downloaded the model to my cache:

prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", default_prompt_template=lfqa_prompt)

My Assumption was for the next run it will use the cached model. As expected it's not downloading, but it's still taking a lot of time.

Can we reduce this time by saving the already processed model?

答案1

得分: 2

我无法完全复现你的示例,因为你还有一个检索器,但我使用了下面的最小示例:

from haystack import Pipeline, Document
from haystack.nodes import PromptTemplate, PromptNode

prompt_template = PromptTemplate("deepset/question-answering")

prompt_node = PromptNode(model_name_or_path="google/flan-t5-base", default_prompt_template=prompt_template)

pipe = Pipeline()
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["Query"])
output = pipe.run(query="What's the capital of Germany?", documents=[Document("Berlin is the capital of Germany.")])
print(output["results"])

我将你的模型更改为google/flan-t5-base,并且每次运行脚本时都能立即产生正确的响应。我猜测你的检索器可能在进行一些耗时的设置。请尝试上面的示例。

英文:

I couldn't reproduce exactly your example as you have a retriever as well but I used a minimal example below:

from haystack import Pipeline, Document
from haystack.nodes import PromptTemplate, PromptNode

prompt_template = PromptTemplate("deepset/question-answering")

prompt_node = PromptNode(model_name_or_path="google/flan-t5-base", default_prompt_template=prompt_template)

pipe = Pipeline()
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["Query"])
output = pipe.run(query="What's the capital of Germany?", documents=[Document("Berlin is the capital of Germany.")])
print(output["results"])

I changed your model to google/flan-t5-base, and the script instantly produced the correct response every time I ran it. My best guess is that your retriever might be doing some time-consuming setup. Please try the example above.

huangapple
  • 本文由 发表于 2023年7月27日 14:40:01
  • 转载请务必保留本文链接:https://go.coder-hub.com/76777086.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定