如何查看保存在Lang Chain中的文档的嵌入(或任何其他DB)?

huangapple go评论158阅读模式
英文:

How to see the Embedding of the documents with Chroma (or any other DB) saved in Lang Chain?

问题

I can see everything but the Embedding of the documents when I used Chroma with Langchain and OpenAI embeddings. It always shows me None for that.

Here is the code:

for db_collection_name in tqdm(["class1-sub2-chap3", "class2-sub3-chap4"]):
    documents = []
    doc_ids = []

    for doc_index in range(3):
        cl, sub, chap = db_collection_name.split("-")
        content = f"This is {db_collection_name}-doc{doc_index}"
        doc = Document(page_content=content, metadata={"chunk_num": doc_index, "chapter":chap, "class":cl, "subject":sub})
        documents.append(doc)
        doc_ids.append(str(doc_index))

    # Initialize a Chroma instance with the original document
    db = Chroma.from_documents(
         collection_name=db_collection_name,
         documents=documents, ids=doc_ids,
         embedding=embeddings, 
         persist_directory="./data")
    
     db.persist()

When I do db.get(), I see everything as expected except embedding is None.

{'ids': ['0', '1', '2'],
 'embeddings': None,
 'documents': ['This is class1-sub2-chap3-doc0',
  'This is class1-sub2-chap3-doc1',
  'This is class1-sub2-chap3-doc2'],
 'metadatas': [{'chunk_num': 0,
   'chapter': 'chap3',
   'class': 'class1',
   'subject': 'sub2'},
  {'chunk_num': 1, 'chapter': 'chap3', 'class': 'class1', 'subject': 'sub2'},
  {'chunk_num': 2, 'chapter': 'chap3', 'class': 'class1', 'subject': 'sub2'}]}

My embeddings are also working fine as it returns:

len(embeddings.embed_documents(["EMBED THIS"])[0])
>> 1536

Also, in my ./data directory, I have an Embedding file as chroma-embeddings.parquet.

I tried the example with example given in the document but it also shows None too.

# Import Document class
from langchain.docstore.document import Document

# Initial document content and id
initial_content = "This is an initial document content"
document_id = "doc1"

# Create an instance of Document with initial content and metadata
original_doc = Document(page_content=initial_content, metadata={"page": "0"})

# Initialize a Chroma instance with the original document
new_db = Chroma.from_documents(
    collection_name="test_collection",
    documents=[original_doc],
    embedding=OpenAIEmbeddings(),  # using the same embeddings as before
    ids=[document_id],
)

Here also new_db.get() gives me None.

英文:

I can see everything but the Embedding of the documents when I used Chroma with Langchain and OpenAI embeddings. It always show me None for that

Here is the code:

for db_collection_name in tqdm(["class1-sub2-chap3", "class2-sub3-chap4"]):
    documents = []
    doc_ids = []

    for doc_index in range(3):
        cl, sub, chap = db_collection_name.split("-")
        content = f"This is {db_collection_name}-doc{doc_index}"
        doc = Document(page_content=content, metadata={"chunk_num": doc_index, "chapter":chap, "class":cl, "subject":sub})
        documents.append(doc)
        doc_ids.append(str(doc_index))


    # # Initialize a Chroma instance with the original document
    db = Chroma.from_documents(
         collection_name=db_collection_name,
         documents=documents, ids=doc_ids,
         embedding=embeddings, 
         persist_directory="./data")
    
     db.persist()

when I do db.get(), I see everything as expected except embedding is None.

{'ids': ['0', '1', '2'],
 'embeddings': None,
 'documents': ['This is class1-sub2-chap3-doc0',
  'This is class1-sub2-chap3-doc1',
  'This is class1-sub2-chap3-doc2'],
 'metadatas': [{'chunk_num': 0,
   'chapter': 'chap3',
   'class': 'class1',
   'subject': 'sub2'},
  {'chunk_num': 1, 'chapter': 'chap3', 'class': 'class1', 'subject': 'sub2'},
  {'chunk_num': 2, 'chapter': 'chap3', 'class': 'class1', 'subject': 'sub2'}]}

My embeddings is also working fine as it returns:

len(embeddings.embed_documents(["EMBED THIS"])[0])
>> 1536

also, in my ./data directory I have Embedding file as chroma-embeddings.parquet


I tried the example with example given in document but it shows None too

# Import Document class
from langchain.docstore.document import Document

# Initial document content and id
initial_content = "This is an initial document content"
document_id = "doc1"

# Create an instance of Document with initial content and metadata
original_doc = Document(page_content=initial_content, metadata={"page": "0"})

# Initialize a Chroma instance with the original document
new_db = Chroma.from_documents(
    collection_name="test_collection",
    documents=[original_doc],
    embedding=OpenAIEmbeddings(),  # using the same embeddings as before
    ids=[document_id],
)

Here also new_db.get() gives me None

答案1

得分: 7

# 获取所有嵌入
db._collection.get(include=['embeddings'])

# 根据文档ID获取嵌入
db._collection.get(ids=['doc0', ..., 'docN'], include=['embeddings'])
英文:

You just need to specify that you want the embeddings as well when using .get

# Get all embeddings
db._collection.get(include=['embeddings'])

# Get embeddings by document_id
db._collection.get(ids=['doc0', ..., 'docN'], include=['embeddings'])

huangapple
  • 本文由 发表于 2023年6月1日 15:07:40
  • 转载请务必保留本文链接:https://go.coder-hub.com/76379440.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定