Skip to main content
This doc helps you get started with Fireworks AI chat models. For detailed documentation of all ChatFireworks features and configurations head to the API reference. Fireworks AI is an AI inference platform to run and customize models. For a list of all models served by Fireworks see the Fireworks docs.

​
Overview

​
Integration details

ClassPackageLocalSerializableJS supportDownloadsVersion
ChatFireworkslangchain-fireworks❌beta✅PyPI - DownloadsPyPI - Version

​
Model features

Tool callingStructured outputJSON modeImage inputAudio inputVideo inputToken-level streamingNative asyncToken usageLogprobs
✅✅✅❌❌❌✅✅✅✅

​
Setup

To access Fireworks models you’ll need to create a Fireworks account, get an API key, and install the langchain-fireworks integration package.

​
Credentials

Head to (fireworks.ai/login to sign up to Fireworks and generate an API key. Once you’ve done this set the FIREWORKS_API_KEY environment variable: 
import getpass
import os

if "FIREWORKS_API_KEY" not in os.environ:
    os.environ["FIREWORKS_API_KEY"] = getpass.getpass("Enter your Fireworks API key: ")
To enable automated tracing of your model calls, set your LangSmith API key: 
os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")
os.environ["LANGSMITH_TRACING"] = "true"

​
Installation

The LangChain Fireworks integration lives in the langchain-fireworks package: 
pip install -qU langchain-fireworks

​
Instantiation

Now we can instantiate our model object and generate chat completions:
  • TODO: Update model instantiation with relevant params.
from langchain_fireworks import ChatFireworks

llm = ChatFireworks(
    model="accounts/fireworks/models/llama-v3-70b-instruct",
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2,
    # other params...
)

​
Invocation

messages = [
    (
        "system",
        "You are a helpful assistant that translates English to French. Translate the user sentence.",
    ),
    ("human", "I love programming."),
]
ai_msg = llm.invoke(messages)
ai_msg

AIMessage(content="J'adore la programmation.", response_metadata={'token_usage': {'prompt_tokens': 35, 'total_tokens': 44, 'completion_tokens': 9}, 'model_name': 'accounts/fireworks/models/llama-v3-70b-instruct', 'system_fingerprint': '', 'finish_reason': 'stop', 'logprobs': None}, id='run-df28e69a-ff30-457e-a743-06eb14d01cb0-0', usage_metadata={'input_tokens': 35, 'output_tokens': 9, 'total_tokens': 44})

print(ai_msg.content)

J'adore la programmation.

​
API reference

For detailed documentation of all ChatFireworks features and configurations head to the API reference: python.langchain.com/api_reference/fireworks/chat_models/langchain_fireworks.chat_models.ChatFireworks.html To use the langchain-fireworks package, follow these installation steps: 
pip install langchain-fireworks

​
Basic usage

​
Setting up

  1. Sign in to Fireworks AI to obtain an API Key to access the models, and make sure it is set as the FIREWORKS_API_KEY environment variable. Once you’ve signed in and obtained an API key, follow these steps to set the FIREWORKS_API_KEY environment variable:
    • Linux/macOS: Open your terminal and execute the following command:
    export FIREWORKS_API_KEY='your_api_key'
    Note: To make this environment variable persistent across terminal sessions, add the above line to your ~/.bashrc, ~/.bash_profile, or ~/.zshrc file.
    • Windows: For Command Prompt, use:
    set FIREWORKS_API_KEY=your_api_key
  2. Set up your model using a model id. If the model is not set, the default model is fireworks-llama-v2-7b-chat. See the full, most up-to-date model list on fireworks.ai. 
import getpass
import os

# Initialize a Fireworks model
llm = Fireworks(
    model="accounts/fireworks/models/llama-v3p1-8b-instruct",
    base_url="https://api.fireworks.ai/inference/v1/completions",
)

​
Calling the Model Directly

You can call the model directly with string prompts to get completions. 
# Single prompt
output = llm.invoke("Who's the best quarterback in the NFL?")
print(output)

# Calling multiple prompts
output = llm.generate(
    [
        "Who's the best cricket player in 2016?",
        "Who's the best basketball player in the league?",
    ]
)
print(output.generations)

​
Advanced usage

​
Tool use: LangChain Agent + Fireworks function calling model

Please checkout how to teach Fireworks function calling model to use a calculator in this notebook. Fireworks focus on delivering the best experience for fast model inference as well as tool use. You can check out our blog for more details on how it compares to GPT-4, the punchline is that it is on par with GPT-4 in terms of function calling use cases, but it is way faster and much cheaper.

​
RAG: LangChain agent + Fireworks function calling model + MongoDB + Nomic AI embeddings

Please check out the cookbook here for an end to end flow
Edit the source of this page on GitHub.
Connect these docs programmatically to Claude, VSCode, and more via MCP for real-time answers.
⌘I