Skip to main content
TF-IDF means term-frequency times inverse document-frequency.
This notebook goes over how to use a retriever that under the hood uses TF-IDF using scikit-learn package. For more information on the details of TF-IDF see this blog post. 
pip install -qU  scikit-learn

from langchain_community.retrievers import TFIDFRetriever

​
Create New Retriever with Texts

retriever = TFIDFRetriever.from_texts(["foo", "bar", "world", "hello", "foo bar"])

​
Create a New Retriever with Documents

You can now create a new retriever with the documents you created. 
from langchain_core.documents import Document

retriever = TFIDFRetriever.from_documents(
    [
        Document(page_content="foo"),
        Document(page_content="bar"),
        Document(page_content="world"),
        Document(page_content="hello"),
        Document(page_content="foo bar"),
    ]
)

​
Use Retriever

We can now use the retriever! 
result = retriever.invoke("foo")

result

[Document(page_content='foo', metadata={}),
 Document(page_content='foo bar', metadata={}),
 Document(page_content='hello', metadata={}),
 Document(page_content='world', metadata={})]

​
Save and load

You can easily save and load this retriever, making it handy for local development! 
retriever.save_local("testing.pkl")

retriever_copy = TFIDFRetriever.load_local("testing.pkl")

retriever_copy.invoke("foo")

[Document(page_content='foo', metadata={}),
 Document(page_content='foo bar', metadata={}),
 Document(page_content='hello', metadata={}),
 Document(page_content='world', metadata={})]
Edit the source of this page on GitHub.
Connect these docs programmatically to Claude, VSCode, and more via MCP for real-time answers.
⌘I