Question and answer over multiple csv files in langchain

I've a folder with multiple csv files, I'm trying to figure out a way to load them all into langchain and ask questions over all of them.

Here's what I have so far.

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain import OpenAI, VectorDBQA
from langchain.document_loaders import DirectoryLoader
from langchain.document_loaders.csv_loader import CSVLoader
import magic
import os
import nltk

os.environ['OPENAI_API_KEY'] = '...'

loader = DirectoryLoader('../data/', glob='**/*.csv', loader_cls=CSVLoader)

documents = loader.load()

text_splitter = CharacterTextSplitter(chunk_size=400, chunk_overlap=0)

texts = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings(openai_api_key=os.environ['OPENAI_API_KEY'])

docsearch = Chroma.from_documents(texts, embeddings)

qa = VectorDBQA.from_chain_type(llm=OpenAI(), chain_type="stuff", vectorstore=docsearch)

query = "how many females are present?"
qa.run(query)

You should load them all into a vectorstore such as Pinecone or Metal. Then use a RetrievalQAChain or ConversationalRetrievalChain depending on if you want memory or not.

Not sure whether you want to integrate multiple csv files for your query or compare among them. Here is the link if you want to compare/see the differences among multiple csv files using similar approach with querying one file. https://python.langchain.com/en/latest/modules/agents/toolkits/examples/csv.html

agent = create_csv_agent(OpenAI(temperature=0), ['titanic.csv', 'titanic_age_fillna.csv'], verbose=True)
agent.run("how many rows in the age column are different?")

screenshot for more details

You can also consider using open source local LLM like llama2 for this purpose. Try localGPT. https://github.com/PromtEngineer/localGPT