Skip to main content

Multi-turn Chat: Pass Tool-Call History Correctly

End-to-end walkthrough — create a Knowledge Base, then pass tool-call history through multi-turn API requests to avoid redundant retrieval

This guide shows how to pass tool-call history across turns so the model answers from prior retrieval results instead of re-running searches. The same sample document drives every step, so the results are reproducible.

Why context gets lost

The RAG Agent is stateless — each request stands alone and the server keeps no conversation history. A single response usually moves through several stages:
If the next turn sends only the text history (the user question and the assistant's final answer), the model loses sight of the tool calls in between. It does not know what has already been retrieved, so it searches the same content again — wasting tokens and risking inconsistent answers. The fix is to pass the full tool-call chain (assistant.tool_calls plus the matching tool results) as history messages, so the model can answer from what it has already retrieved.

Prepare the sample document

This guide uses the Model Studio docs sample bundle — 92 official Model Studio technical documents covering API usage, model references, and best practices. Download, unzip, and upload them to a Knowledge Base. The walkthrough uses First call to the Qwen API from the bundle to demonstrate follow-up questions:
  • Turn 1: Ask "How do I call the Qwen API?" (triggers multiple tool calls)
  • Turn 2: Follow up with "How do I set the API Key as an environment variable?" (verifies that history is reused and no extra retrieval runs)
You can substitute your own documents — the steps and outcome are the same.

End-to-end walkthrough

1. Create a Knowledge Base and upload the sample

  1. Go to Data Ingestion → Knowledge Management and click Create Knowledge Base.
  2. Choose Document Search as the type and Basic Document Q&A as the use case.
  3. Upload the sample documents. Wait for chunking and vectorization to finish — the status should change to Parsed.
For details, see Create a Knowledge Base and Document Management.

2. Create a Knowledge Q&A service and bind the Knowledge Base

  1. Go to Knowledge Service → Knowledge Q&A and click Create Q&A Service.
  2. Enter a service name, then open the configuration page.
  3. Click + Add and bind the Knowledge Base from the previous step.
  4. Pick a generation model and a retrieval mode (use Multi-turn Agentic Mode for multi-turn scenarios), then click Publish.
For details, see Knowledge Q&A.

3. Get the API parameters

In the Q&A service list, click API Debugging on the target service and collect the following parameters:
ParameterWhere to find itExample
endpointEndpoint shown on the API Debugging page. {workspaceId} is your workspace ID.https://{workspaceId}.cn-beijing.maas.aliyuncs.com/api/v2/apps/knowledge/chat
agent_idRequest parameter shown on the API Debugging page — the Q&A service ID.aid-xxxxxxxx
API KeyCreate it on Settings → API Key.sk-xxxxxxxx
agent_id uniquely identifies a Q&A service — each service has its own agent_id.

4. First turn

Send the first request with the parameters you collected: 
curl -X POST 'https://{workspaceId}.cn-beijing.maas.aliyuncs.com/api/v2/apps/knowledge/chat' \
  -H 'Authorization: Bearer $API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": {
      "messages": [{"role": "user", "content": "How do I call the Qwen API"}]
    },
    "parameters": {
      "agent_options": {"agent_id": "aid-xxxxxxxx"}
    },
    "stream": true
  }'
The Agent runs several rounds of tool calls against the Knowledge Base, then streams back the final answer. You need to extract the tool-call history from the SSE stream — see Extract tool history from the SSE stream below.

5. Second turn: include tool history

For the follow-up "How do I set the API Key as an environment variable?", pass the full history from turn 1 (including the tool-call chain): 
curl -X POST 'https://{workspaceId}.cn-beijing.maas.aliyuncs.com/api/v2/apps/knowledge/chat' \
  -H 'Authorization: Bearer $API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": {
      "messages": [
        {"role": "user", "content": "How do I call the Qwen API"},
        {"role": "assistant", "content": "", "tool_calls": [{"id":"call_956...","function":{"name":"semantic_search",...}}]},
        {"role": "tool", "tool_call_id": "call_956...", "content": "Retrieved 5 relevant chunks..."},
        {"role": "assistant", "content": "Based on the retrieved First call to the Qwen API content..."},
        {"role": "user", "content": "How do I set the API Key as an environment variable?"}
      ]
    },
    "parameters": {"agent_options": {"agent_id": "aid-xxxxxxxx"}},
    "stream": true
  }'
tool_call_id must be the exact value returned by the first turn's stream — do not invent one. The model uses it to match each call to its result.

6. Comparing the two approaches

Measured against First call to the Qwen API: after turn 1 asks "How do I call the Qwen API", turn 2 asks "How do I set the API Key as an environment variable?". Compare what happens with each history strategy:
Full tool historyText-only history
Tool calls in turn 20 — answers directly from prior resultsRe-runs retrieval (2 semantic_search calls observed)
Response speedFast — no retrieval round tripSlow — retrieval runs again
Answer consistencySame retrieval results power both turnsNew retrieval may hit different chunks, leading to drift
Measured numbers: turn 1 ran 2 rounds of tool calls (semantic_search → obtain_file), producing 6 history messages. Turn 2 with full history (7 messages) answered with 0 tool calls. Turn 2 with text-only history (3 messages) re-ran semantic_search twice. The exact counts depend on your document structure and how well the question matches, but the takeaway holds: full history eliminates redundant retrieval.
Full history makes the input longer (it carries tool results), but skips the retrieval round trip. Text-only history sends a smaller input but pays for retrieval again. In multi-turn follow-up scenarios, full history wins on both response speed and answer consistency.

Extract tool history from the SSE stream

The Agent streams output as SSE — each frame is data:{...}, with the message at obj.output.choices[0].message. message.extra.step_change marks state-change boundaries. You only need to handle three event types:
EventDetection conditionData to extract
Tool callstep_change == "tool_calling"The tool_calls array (all parallel calls in this round)
Tool returnstep_change == "tool_return" and role == "tool"content plus tool_call_id
Final answerBetween generation_start and generation_endConcatenate every content chunk

State flow

The Agent may run several rounds of tool calls before it has enough information to generate the final answer.

Message format and key rules

Organize the extracted history messages by role, in this order:
RoleRequired fieldsNotes
userrole, contentUser input
assistant (tool call)role, content (empty), tool_callsAll parallel tool calls dispatched in this round
toolrole, content, tool_call_idTool result. tool_call_id must match the id in the corresponding tool_calls[] entry.
assistant (final answer)role, contentThe model's final answer
Messages must stay in chronological order:
tool_calls[].function.arguments is a JSON string, not an object. Keep it as a string when you send it back.

Full Python implementation

The code below ties together "extract history from SSE → manage multi-turn history automatically → compare both approaches" in a single runnable script. The numbers shown were measured against the sample document and prompts on this page.

Configuration

import json
import requests

# Get these from the API Debugging page of your Q&A service in the console.
API_URL = "https://{workspaceId}.cn-beijing.maas.aliyuncs.com/api/v2/apps/knowledge/chat"
API_KEY = "sk-xxxxxxxx"      # Workspace-scoped API Key (sk-ws- prefix)
AGENT_ID = "aid-xxxxxxxx"    # Q&A service ID

Extract tool history from the SSE stream

class ConversationTracker:
    """Track a single Agent response and extract the full message sequence from the stream."""

    def __init__(self):
        self.messages = []          # Extracted message sequence (tool calls + tool returns)
        self.tool_call_rounds = 0   # Number of tool-call rounds (used for comparison)
        self.answer = ""            # Final answer
        self._generating = False

    def process_sse_line(self, line: str):
        if not line.startswith("data:"):
            return
        obj = json.loads(line[len("data:"):].strip())
        choices = obj.get("output", {}).get("choices", [])
        if not choices:
            return

        msg = choices[0].get("message", {})
        step_change = msg.get("extra", {}).get("step_change", "")
        role = msg.get("role", "")
        content = msg.get("content", "") or ""

        # Event 1: tool-call dispatch (step_change == "tool_calling") -> extract tool_calls
        if step_change == "tool_calling" and msg.get("tool_calls"):
            self.tool_call_rounds += 1
            self.messages.append({
                "role": "assistant", "content": "",
                "tool_calls": [{"id": tc["id"], "type": "function",
                                "function": {"name": tc["function"]["name"],
                                             "arguments": tc["function"]["arguments"]}}
                               for tc in msg["tool_calls"]],
            })
        # Event 2: tool return (step_change == "tool_return", role == "tool") -> extract content + tool_call_id
        elif step_change == "tool_return" and role == "tool":
            self.messages.append({"role": "tool",
                                  "tool_call_id": msg.get("tool_call_id", ""),
                                  "content": content})
        # Event 3: final answer (generation_start -> content stream -> generation_end) -> accumulate content
        elif step_change == "generation_start":
            self._generating = True
            self.answer = content
        elif self._generating and content:
            self.answer += content
        elif step_change == "generation_end":
            self.answer += content
            self._generating = False

    def get_response_messages(self):
        """Return the full message sequence extracted from this turn, including the final answer."""
        result = list(self.messages)
        if self.answer:
            result.append({"role": "assistant", "content": self.answer})
        return result

Multi-turn chat manager

class MultiTurnChat:
    """Manage multi-turn conversation history and carry the tool-call chain automatically."""

    def __init__(self):
        self.history = []

    def ask(self, question: str) -> str:
        """Send a question, automatically assemble full history (including the tool-call chain), return the final answer."""
        self.history.append({"role": "user", "content": question})
        sent = len(self.history)
        tracker = self._call_agent(self.history)
        # Append this turn's tool-call chain and final answer to the history.
        self.history.extend(tracker.get_response_messages())
        print(f"  Sent {sent} messages, {tracker.tool_call_rounds} tool-call round(s)")
        return tracker.answer

    def _call_agent(self, messages):
        headers = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
        body = {
            "input": {"messages": messages},
            "parameters": {"agent_options": {"agent_id": AGENT_ID}},
            "stream": True,
        }
        tracker = ConversationTracker()
        resp = requests.post(API_URL, headers=headers, json=body, stream=True, timeout=180)
        for chunk in resp.iter_lines():
            if chunk:
                tracker.process_sse_line(chunk.decode("utf-8").strip())
        return tracker

Compare full history vs text-only history

chat = MultiTurnChat()

# Turn 1
print("[Turn 1] How do I call the Qwen API")
chat.ask("How do I call the Qwen API")

# Turn 2 — full tool history (carried automatically)
print("\n[Turn 2 - full tool history]")
chat_full = MultiTurnChat()
chat_full.history = list(chat.history)
chat_full.ask("How do I set the API Key as an environment variable?")

# Turn 2 — text-only history (drop tool calls, keep only user + assistant final answers)
print("\n[Turn 2 - text-only history]")
text_only = [m for m in chat.history
             if m["role"] == "user" or (m["role"] == "assistant" and not m.get("tool_calls"))]
chat_text = MultiTurnChat()
chat_text.history = text_only
chat_text.ask("How do I set the API Key as an environment variable?")
Measured output: 
[Turn 1] How do I call the Qwen API
  Sent 1 messages, 2 tool-call round(s)

[Turn 2 - full tool history]
  Sent 7 messages, 0 tool-call round(s)      <- reused prior results, no extra retrieval

[Turn 2 - text-only history]
  Sent 3 messages, 2 tool-call round(s)      <- retrieval ran again

Caveats

Manage history length

Tool returns can be long. After several turns the total may exceed the context limit. Recommendations:
  • Truncate earlier tool returns: keep full tool content only for the most recent K turns. For older turns, truncate the tool message to the first N characters.
  • Sliding window: beyond 3 turns, replace older tool results with summaries.
def truncate_history(messages, max_tool_len=2000, keep_recent=2):
    """Truncate tool returns from earlier turns."""
    user_indices = [i for i, m in enumerate(messages) if m["role"] == "user"]
    if len(user_indices) <= keep_recent:
        return messages
    cutoff = user_indices[-keep_recent]
    result = []
    for i, msg in enumerate(messages):
        if i < cutoff and msg["role"] == "tool" and len(msg["content"]) > max_tool_len:
            msg = {**msg, "content": msg["content"][:max_tool_len] + "\n...(truncated)"}
        result.append(msg)
    return result

Content you do not need to send back

ContentWhy
reasoning_contentThe model's reasoning trace — display it to the user, no need to echo it back.
extra fieldsStreaming protocol state markers, used only for client-side rendering.
usageToken-usage metrics, used only for monitoring and billing.
tool_call_chunksStreaming fragments — already merged into tool_calls.

FAQ

What happens if I do not send tool-call history? The model loses the retrieval context, re-searches the same content, and answer quality drops. Measured: with full history, the follow-up runs 0 tool calls and answers immediately; without it, retrieval runs again. Can I skip the intermediate tool calls and send only the final answer? It works, but it is not recommended. Sending only user + assistant(final answer) strips the model's memory of the Knowledge Base content, and follow-up quality noticeably drops. Does the order of parallel tool returns matter? The order does not affect function — the model matches by tool_call_id. But every tool message must immediately follow its corresponding assistant(tool_calls) message.
Getting Started
Playground
Data Ingestion
Knowledge Service
Cookbook
Integration
Settings, Billing & Limits