Running LlamaIndex with DeepSeek and other Chinese AI models needs no
custom integration. LlamaIndex ships an OpenAILike LLM class specifically for
OpenAI-compatible endpoints, so DeepSeek, GLM, Kimi and Qwen slot into every index, retriever and query
engine you already use. This guide builds a working RAG pipeline end to end, explains the two
configuration mistakes that account for most failures, and does the cost arithmetic that makes long
context affordable in the first place.
LlamaIndex’s default OpenAI class validates model names against a hard-coded
registry of OpenAI models. Pass deepseek-v4-pro and it raises before it ever makes a
request. The
OpenAILike
class exists precisely for this case: it skips the registry, sends the request as-is, and lets you
declare the model’s properties yourself.
This is the single most common stumbling block, and it produces a confusing error message, so it is
worth stating plainly: use OpenAILike, declare is_chat_model=True, and
set context_window to the real value.
pip install llama-index llama-index-llms-openai-like llama-index-embeddings-huggingfaceimport os
from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(
model="deepseek-v4-pro",
api_base="https://aiwave.live/v1", # note: api_base, not base_url
api_key=os.environ["AIWAVE_API_KEY"],
is_chat_model=True, # required, or completions format is used
context_window=128000, # declare it; LlamaIndex uses this for chunking
temperature=0.1,
)
print(llm.complete("Summarise mixture-of-experts routing in one sentence."))Get a key from the console — email or GitHub, no Chinese phone number. The endpoint contract is documented in the API reference, and authentication covers key handling.
RAG has two model decisions, not one. The LLM generates; the embedding model decides what the LLM ever sees. A weak retriever cannot be rescued by a strong generator — if the right chunk is not in the top-k, no amount of reasoning recovers it.
Local embeddings are often the right default. They cost nothing per query, keep your corpus on your own machines, and remove a network round trip from every retrieval:
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import Settings
Settings.llm = llm
Settings.embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5" # 384-dim, fast, runs on CPU
)For Chinese or mixed-language corpora, BAAI publishes multilingual variants in the same family; see the BAAI model collection. If you prefer a hosted embedding endpoint instead, the embeddings API documents the server-side option.
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)
index.storage_context.persist("./storage") # build once, reusePersist the index. Rebuilding embeddings on every process start is the most common source of “why is startup so slow” in LlamaIndex projects, and with local embeddings it is pure wasted CPU.
from llama_index.core import StorageContext, load_index_from_storage
storage = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage)engine = index.as_query_engine(similarity_top_k=5)
response = engine.query("What changed in the v4 release?")
print(response)
for node in response.source_nodes:
print(f"{node.score:.3f} {node.metadata.get('file_name')}")Always print the source nodes during development. Most “the model hallucinated” reports turn out to be retrieval failures: the answer was never in the context. Inspecting scores tells you immediately whether the problem is the retriever or the generator.
RAG answers are slow because the prompt is long. Streaming hides most of that latency behind the first token.
engine = index.as_query_engine(streaming=True, similarity_top_k=5)
streaming_response = engine.query("Explain the migration path.")
streaming_response.print_response_stream()The chunk format is the standard server-sent-event stream documented in streaming responses.
A 128K context window is a budget, not a target. Every retrieved chunk is input tokens you pay for on
every single query, so similarity_top_k is a cost dial as much as a quality dial.
Current rates for models commonly used as the generator in a RAG stack:
| Model ID | Input / 1M tokens | Output / 1M tokens |
|---|---|---|
deepseek-v4-flash | $0.206 | $0.412 |
deepseek-v4-pro | $1.09 | $2.17 |
glm-4.7-flash | free | free |
glm-5 | $1.55 | $4.96 |
kimi-k2.5 | $0.66 | $3.30 |
qwen3-coder-480b-a35b-instruct | $0.12 | $0.36 |
Rates read from the AIWave pricing endpoint on 2026-07-26. Check live pricing before budgeting — providers revise rates.
Work the arithmetic for your own pipeline. A query that retrieves 5 chunks of 800 tokens plus a 200
token question sends roughly 4,200 input tokens. At deepseek-v4-flash rates that is a
fraction of a cent per query; at 100,000 queries a month it becomes a line item worth optimising. The
multiplication is: (input tokens ÷ 1,000,000) × input price, plus the same for
output. Nothing about that is a benchmark claim — it is the published rate times your volume.
from llama_index.core.postprocessor import SentenceTransformerRerank
reranker = SentenceTransformerRerank(
model="cross-encoder/ms-marco-MiniLM-L-6-v2",
top_n=3,
)
engine = index.as_query_engine(
similarity_top_k=20, # retrieve wide, cheaply
node_postprocessors=[reranker], # then narrow before paying for tokens
)The in-memory store is fine for prototypes and wrong for production. LlamaIndex supports the usual backends; Chroma is the lowest-friction step up:
import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import StorageContext, VectorStoreIndex
client = chromadb.PersistentClient(path="./chroma")
collection = client.get_or_create_collection("docs")
vector_store = ChromaVectorStore(chroma_collection=collection)
storage = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, storage_context=storage)The LLM configuration does not change when you swap the vector store — that separation is the main reason to use a framework here at all.
RAG pipelines fail in two places: retrieval returns nothing useful, or the LLM call errors. Handle both explicitly.
from openai import APIStatusError
RETRYABLE = {408, 429, 500, 502, 503}
def safe_query(engine, question, fallback_llm=None):
try:
resp = engine.query(question)
except APIStatusError as e:
if e.status_code in RETRYABLE and fallback_llm:
Settings.llm = fallback_llm # same request body, different model
resp = engine.query(question)
else:
raise
if not resp.source_nodes:
return "No relevant documents found." # do not let the model improvise
return str(resp)Which status codes justify a retry is covered in error codes; rate limits has a backoff implementation with jitter.
The OpenAI class validates against a model registry and rejects unknown names before
sending anything. The error mentions an unknown model, which reads like a server problem but is entirely
client-side.
LlamaIndex uses context_window to decide how much retrieved text fits. Leave it at a
conservative default and you silently truncate context on a model that could have handled far more
— producing worse answers with no error at all.
Do not pick a model from a leaderboard. Build a small evaluation set from real questions and run the
same index against several generators — the index does not change, only Settings.llm
does.
for name in ["deepseek-v4-flash", "deepseek-v4-pro", "glm-5"]:
Settings.llm = OpenAILike(
model=name, api_base="https://aiwave.live/v1",
api_key=os.environ["AIWAVE_API_KEY"],
is_chat_model=True, context_window=128000, temperature=0,
)
engine = index.as_query_engine(similarity_top_k=5)
for q in eval_questions:
print(name, q, str(engine.query(q))[:200])Judge the outputs against your own ground truth. A cheaper model that answers your questions correctly is the right answer regardless of how it ranks publicly.
Vector similarity alone ignores everything you already know about a document. If a question is about the 2026 pricing policy, there is no reason to search 2023 documents at all. Attaching metadata at ingestion and filtering at query time removes whole categories of wrong retrieval for zero token cost.
from llama_index.core import Document
from llama_index.core.vector_stores import MetadataFilters, MetadataFilter
docs = [
Document(text=body, metadata={"year": 2026, "team": "platform", "kind": "runbook"})
for body in raw_bodies
]
index = VectorStoreIndex.from_documents(docs)
engine = index.as_query_engine(
similarity_top_k=5,
filters=MetadataFilters(filters=[
MetadataFilter(key="year", value=2026),
MetadataFilter(key="kind", value="runbook"),
]),
)A practical rule: any field you would use in a WHERE clause belongs in metadata. Source
file, date, product area, access level and language are the usual suspects. Access level in particular
is worth getting right early — retrieval that ignores permissions is a data leak waiting for its
first user.
LlamaIndex’s node parsers control how documents become retrievable units, and the choice materially changes answer quality. The sentence splitter respects sentence boundaries rather than slicing mid-thought:
from llama_index.core.node_parser import SentenceSplitter
Settings.node_parser = SentenceSplitter(
chunk_size=768,
chunk_overlap=96,
)For structured documentation, splitting on headings preserves the hierarchy the author already created, which usually beats any fixed-size window:
from llama_index.core.node_parser import MarkdownNodeParser
Settings.node_parser = MarkdownNodeParser()Two failure modes to watch for. Chunks that are too small lose the context that makes a passage meaningful — a table row without its header is noise. Chunks that are too large dilute the embedding, so a document about ten topics matches poorly on all ten. If retrieval quality is disappointing, adjust chunking before you reach for a bigger model; it is free and it is more often the actual cause.
Questions of the form “how does X compare to Y” retrieve badly, because no single chunk discusses both. The sub-question query engine splits the question, retrieves for each part separately, then synthesises:
from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata
tools = [
QueryEngineTool(
query_engine=index.as_query_engine(similarity_top_k=4),
metadata=ToolMetadata(
name="product_docs",
description="Product documentation, runbooks and release notes",
),
),
]
engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=tools)
print(engine.query("How does the v4 rollout differ from the v3 rollout?"))This costs more — each sub-question is its own LLM call plus retrieval. It is worth it for genuinely comparative questions and wasteful for simple lookups, which is a good argument for routing: detect the question type cheaply, then choose the engine. A cheap fast model is entirely adequate for that classification step.
The single most useful debugging habit in RAG work is looking at the final assembled prompt. What the model receives is rarely what you imagined.
import llama_index.core
llama_index.core.set_global_handler("simple") # prints prompts and responses
response = engine.query("What is the rollback procedure?")Once you can see the prompt, most quality problems become obvious: the relevant chunk is missing, or it is present but buried beneath four irrelevant ones, or the instruction template is fighting the retrieved context. Each of those has a different fix, and guessing between them wastes days.
Treating RAG as one black box makes failures unattributable. Measure the retriever on its own first — if the correct document is not in the top-k, the generator never had a chance.
def hit_rate(index, cases, k=5):
retriever = index.as_retriever(similarity_top_k=k)
hits = 0
for c in cases: # c = {"question": ..., "source": "file.md"}
nodes = retriever.retrieve(c["question"])
files = {n.metadata.get("file_name") for n in nodes}
hits += c["source"] in files
return hits / len(cases)
print("retrieval hit rate:", hit_rate(index, eval_cases))Fifty labelled questions are enough to be informative. If hit rate is below roughly 0.8, work on chunking, metadata and reranking — not on the LLM. Only once retrieval is reliable does comparing generators tell you anything.
context_window honestly so LlamaIndex packs context correctly.similarity_top_k — it is the main lever on per-query cost.If your pipeline currently points at OpenAI, the change is confined to the LLM object. Indexes, retrievers, node parsers, postprocessors and evaluation code are untouched.
# before
from llama_index.llms.openai import OpenAI
Settings.llm = OpenAI(model="gpt-4o", api_key=os.environ["OPENAI_API_KEY"])
# after
from llama_index.llms.openai_like import OpenAILike
Settings.llm = OpenAILike(
model="deepseek-v4-pro",
api_base="https://aiwave.live/v1",
api_key=os.environ["AIWAVE_API_KEY"],
is_chat_model=True,
context_window=128000,
)Note that embeddings and the LLM are independent choices. Many teams keep an existing embedding setup, swap only the generator, and re-run their evaluation set before committing. Keep the previous configuration behind an environment variable so rollback is one deploy rather than one revert. The non-framework-specific parts of the move are covered in migrating an OpenAI app to Chinese models.
Yes, through OpenAILike pointed at an OpenAI-compatible endpoint. No DeepSeek-specific
package is required, and every LlamaIndex abstraction built on the LLM interface keeps working.
You are almost certainly using the OpenAI class, which validates names against a
registry of OpenAI models. Switch to OpenAILike.
Local embedding models such as the BGE family are free per query and keep your corpus on your own infrastructure, which suits most RAG workloads. Hosted embeddings make sense when you cannot run a model locally or need a specific hosted model.
Multiply your per-query token count by the published rate. A typical query sending around 4,000 input tokens costs a small fraction of a cent on the cheaper models — see the live pricing page for current rates.
Every claim above traces back to one of these. Pricing figures come from the AIWave pricing endpoint on 2026-07-26; capability claims come from the vendors’ own documentation.