openai_harmony.HarmonyError: unexpected tokens remaining in message header

[iseesaw]

I encountered the following problem while using vllm (https://cookbook.openai.com/articles/gpt-oss/run-vllm#using-vllm-for-direct-sampling).
It appears that the text has been truncated beyond the maximum length during generation, resulting in it being unparseable.

Traceback (most recent call last):                                                                                                                                           
  File "/mnt/public/kyzhang/MARTI/tests/run_eval_aime.py", line 66, in <module>                                                                                              
    entries = encoding.parse_messages_from_completion_tokens(output_tokens, Role.ASSISTANT)                                                                                  
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                  
  File "/root/miniconda3/envs/gpt-oss/lib/python3.12/site-packages/openai_harmony/__init__.py", line 525, in parse_messages_from_completion_tokens                           
    raw_json: str = self._inner.parse_messages_from_completion_tokens(                                                                                                       
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                       
openai_harmony.HarmonyError: unexpected tokens remaining in message header: ["The", "task:", "Count", "rectangles", "with", "sides", "along", "edges", "or", "diagonals", "of
", "regular", "dodecagon.", "Vertices", "of", "rectangle", "must", "be", "among", "12", "vertices?", "Not", "necessarily,", "sides", "lie", "along", "sides", "or", "diagonal
s,", "so", "rectangle's", "sides", "may", "be", "segments", "connecting", "two", "vertices", "that", "are", "either", "adjacent", "or", "any", "other", "pair", "forming", "a
", "segment", "(diagonal).", "The", "rectangle's", "vertices", "are", "intersection", "points", "of", "such", "segments?", "Wait", "rectangle", "sides", "along", "sides", "o
r", "diagonals:", "meaning", "each", "side", "of", "rectangle", "coincides", "with", "a", "chord", "connecting", "two", "polygon", "vertices", "(either", "side", "or", "diag
onal).", "So", "rectangle", "vertices", "are", "among", "the", "12", "vertices", "of", "polygon.", "Because", "sides", "are", "subsets", "of", "sides", "or", "diagonals", "s
o", "endpoints", "at", "vertices.", "So", "rectangle", "is", "a", "sub-graph", "of", "complete", "graph", "on", "12", "vertices", "with", "edges", "being", "sides", "or", "d
iagonals,", "forming", "a", "rectangle", "(4-cycle", "with", "right", "angles", "in", "Euclidean", "geometry).", "Need", "to", "count", "all", "rectangles", "whose", "vertic
es", "among", "12", "vertices", "and", "whose", "edges", "are", "chords", "of", "the", "regular", "dodecagon.", "We", "need", "to", "find", "all", "quadrilaterals", "with", 
"vertices", "among", "12", "that", "are", "rectangles", "in", "the", "plane", "of", "the", "regular", "dodecagon", "(regular", "12-gon).", "In", "regular", "polygon,", "rect
angle", "can", "be", "symmetric.", "In", "a", "regular", "polygon", "of", "even", "number,", "inscribed", "rectangles", "correspond", "to", "selecting", "two", "opposite", "
vertices", "(16?).", "Let's", "analyze:", "In", "a", "circle,", "any", "rectangle", "inscribed", "must", "have", "opposite", "vertices", "opposite", "on", "circle?", "Not:",
 "Inscribed", "rectangle", "in", "circle:", "its", "vertices", "lie", "on", "circle;", "rectangle", "sides", "chords;", "Opposite", "vertices", "are", "diametrically", "oppo
site?", "For", "any", "inscribed", "rectangle,", "it's", "cyclic", "where", "opposite", "angles", "are", "90", "degrees", "=>", "arcs", "between", "adjacent", "vertices", "a
re", "180-?", "Wait", "property:", "In", "a", "circle,", "a", "rectangle", "is", "a", "cyclic", "quadrilateral", "with", "opposite", "angles", "90°,", "so", "each", "side", 
"chord", "corresponds", "to", "arc", "180°?", "Actually", "rectangle:", "opposite", "angles", "are", "90°,", "so", "arcs", "subtended", "by", 
"opposite", "sides", "sum", "torilateral", "with", "right", "angles", "at", "all", "vertices", "is", "a", "rectangle", "only", "if", "it's", "a", "cyclic", 
"rectangle.", "For", "a", "rectangle", "inscribed", "in", "a", "circle,", "its", "diagonals", "are", "diameters", "of", "the", "circle?", "In", "general,", 
"a", "rectangle", "inscribed", "in", "a", "circle", "must", "have", "its", "diagonals", "be", "diameters?", "Wait:", "In", "a", "rectangle,", 
"diagonals", "are", "equal", "but", "not", "necessarily", "diameters.", "For", "rectangle", "inscribed", "in", "a", "circle,", "the", "center", "of", 
"circle", "is", "intersection", "of", "diagonals.", "So", "diagonal", "length", "=", "diameter?", "Actually", "the", "rectangle's", "circumcircle", 
"passes", "through", "its", "vertices,", "center", "at", "intersection.", "For", "rectangle,", "intersection", "of", "diagonals", "at", "center", "leads", 
"that", "diagonals", "equal", "in", "length", "=", "diameter", "of", "circle.", "Indeed", "for", "a", "rectangle", "inscribed", "in", "circle,", "the", 
"diagonals", "serve", "as", "diameters:", "because", "they", "both", "go", "through", "center,", "equal", "to", "diameter.", "So", "given", "circle,", 
"rectangle", "vertices", "must", "be", "pairs", "of", "opposite", "points", "of", "circle", "coordinates", "such", "that", "XOR?", "But", "not", 
"necessarily", "that", "each", "vertex", "is", "exactly", "diameter", "endpoints.", "The", "rectangle", "requires", "that", "the", "four", "vertices", 
"are", "arranged", "such", "that", "opposite", "vertices", "are", "diametrically", "opposite.", "Let's", "test:", "In", "a", "square", "inscribed", "in", 
"circle,", "vertices", "are", "90°", "apart;", "opposite", "vertices", "180°.", "For", "rectangle", "that", "is", "not", "square,", "still", "opposite", 
"vertices", "are", "180°,", "so", "rectangle", "vertices", "come", "in", "two", "antipodal", "pairs.", "So", "polygon", "vertices", "must", "be", 
"arranged", "in", "such", "that", "we", "choose", "two", "pairs", "of", "opposite", "vertices", "in", "the", "regular", "dodecagon", "that", "are", 
"each", "separated", "by", "6", "steps", "(180°).", "But", "we", "also", "need", "adjacent", "vertices", "of", "rectangle", "correspond", "to", 
"chords", "that", "may", "not", "be", "sides", "or", "long", "diagonals?", "But", "any", "chord", "between", "two", "vertices", "qualifies.", "So", 
"rectangle", "determined", "by", "choosing", "two", "diameters:", "pick", "an", "ordered", "pair", "of", "diameters", "such", "that", "the", 
"quadrilateral", "connecting", "these", "four", "vertices", "in", "order", "yields", "a", "rectangle.", "Given", "a", "12-gon,", "diameters", 
"correspond", "to", "pairs", "of", "opposite", "vertices", "(difference", "6).", "There", "are", "6", "such", "diameters.", "Choose", "any", "two", 
"distinct", "diameters", "that", "are", "not", "parallel?", "But", "rectangle", "sides", "come", "from", "chords", "between", "the", "chosen", 
"vertices.", "Let's", "list", "vertices:", "number", "them", "0..11", "around", "circle.", "Opposite", "pairs:", "(0,6),", "(1,7),", "(2,8),", "(3,9),", "
(4,10),", "(5,11).", "Choose", "two", "diameters:", "say", "(0,6)", "and", "(1,7).", "Connect", "vertices", "in", "sequence", "0,1,6,7?", "But", 
"forming", "rectangle?", "Let's", "order", "them", "properly", "around", "circle:", "we", "need", "quadrilateral", "with", "vertices", "alternating", 
"between", "the", "two", "diameters.", "For", "diameters", "a-b", "and", "c-d", "with", "a", "opposite", "b,", "c", "opposite", "d.", "If", "we", 
"take", "vertices", "a,", "c,", "b,", "d", "in", "that", "order,", "we", "get", "crossing?", "Actually", "quadrilateral", "with", "vertices", "a,c,b,d", 
"around", "circle", "gives", "a", "crossing", "shape", "-", "not", "simple?", "Let's", "think:", "The", "rectangle", "must", "be", "non-self-
intersecting.", "The", "rectangle's", "vertices", "arranged", "such", "that", "sides", "connect", "a", "to", "c,", "c", "to", "b,", "b", "to", "d,", "d", 
"to", "a.", "So", "the", "quadrilateral", "uses", "diagonals", "across", "circle?", "But", "side", "a-c", "is", "chord", "connecting", "a", "to", "c", "
(not", "necessarily", "diameter).", "Similarly", "c-b", "etc.", "But", "does", "this", "produce", "rectangle?", "Since", "a", "and", "b", "are", 
"opposite,", "difference", "6.", "Similarly", "c", "and", "d", "opposite", "difference", "6.", "The", "quadrilateral", "a-c-b-d", "forms", "a", 
"rectangle", "when", "the", "angle", "between", "chords", "a-c", "and", "c-b", "is", "90", "degrees,", "etc.", "We", "can", "brute", "logic:", "For", 
"regular", "12-gon,", "rectangle", "inscribed", "must", "have", "vertices", "symmetric", "about", "center:", "pairs", "opposite.", "That", "yields", 
"all", "rectangles?", "There", "might", "be", "also", "rectangles", "whose", "vertices", "not", "all", "diametrically", "opposite", "but", "still", 
"rectangles?", "Suppose", "vertices", "0,2,6,8?", "Are", "opposite?", "0-6", "opposite", "yes,", "2-8", "opposite", "yes.", "This", "is", "same", 
"pattern", "as", "above.", "So", "any", "selection", "of", "two", "distinct", "diameter", "pairs", "yields", "a", "rectangle?", "Check", 
"coordinates:", "coordinates", "at", "12", "equally", "spaced", "points", "on", "unit", "circle.", "Let", "a", "angle", "0°,", "b", "180°,", "c", "30°,", 
"d", "210°.", "Actually", "30°", "is", "30°,", "210°", "is", "210°,", "difference", "180.", "Quadrilateral", "0°,30°,180°,210°?", "coordinates?", "Does", 
"that", "shape", "have", "right", "angles?", "Compute", "slopes?", "Might", "produce", "rectangle.", "I", "think", "any", "pair", "of", "diameters", 
"gives", "rectangle.", "Number", "of", "ways", "to", "choose", "2", "diameters", "from", "6", "is", "C(6,2)=15.", "For", "each", "pair,", "but", 
"ordering", "of", "vertices", "maybe", "gives", "same", "rectangle?", "Does", "pair", "of", "diameters", "define", "a", "unique", "rectangle", 
"independent", "of", "which", "side", "chosen?", "The", "rectangle", "vertices", "are", "the", "four", "points.", "Reordering", "doesn't", 
"change", "rectangle.", "So", "number", "rectangles", "=", "15.", "But", "also", "consider", "rectangles", "with", "sides", "along", "polygon", 
"sides", "or", "diagonals", "but", "not", "necessarily", "using", "diameters?", "Is", "it", "possible", "that", "rectangle", "vertices", "are", "not", 
"diametric", "pairs?", "Example:", "consider", "rectangle", "with", "vertices", "0,1,6,7:", "0-1", "are", "adjacent,", "6-7", "also", "adjacent,", "0-
6", "diametric,", "1-7", "diametric.", "This", "still", "two", "diameters", "(0-6,1-7).", "So", "all", "rectangles", "indeed", "come", "from", "two", 
"diameters.", "Thus", "answer", "15?", "Double-check", "if", "any", "other", "combinations", "produce", "rectangle", "but", "not", "using", 
"diameters?", "Suppose", "choose", "vertices", "0,3,6,9:", "3-9", "diff", "6", "diam.", "0-6", "diff6,", "yes.", "So", "again", "diameters.", "So", 
"seems", "all", "rectangles", "use", "diametric", "pairs.", "Thus"]

[cszhz]

I also meet this issue on B200 during test GPT-OSS

(APIServer pid=27525) ERROR 08-06 23:25:57 [serving_chat.py:1001] Error in chat completion stream generator.
(APIServer pid=27525) ERROR 08-06 23:25:57 [serving_chat.py:1001] Traceback (most recent call last):
(APIServer pid=27525) ERROR 08-06 23:25:57 [serving_chat.py:1001]   File "/home/ubuntu/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 642, in chat_completion_stream_generator
(APIServer pid=27525) ERROR 08-06 23:25:57 [serving_chat.py:1001]     harmony_parser.process(token_id)
(APIServer pid=27525) ERROR 08-06 23:25:57 [serving_chat.py:1001]   File "/home/ubuntu/.venv/lib/python3.12/site-packages/openai_harmony/__init__.py", line 627, in process
(APIServer pid=27525) ERROR 08-06 23:25:57 [serving_chat.py:1001]     self._inner.process(token)
(APIServer pid=27525) ERROR 08-06 23:25:57 [serving_chat.py:1001] openai_harmony.HarmonyError: unexpected tokens remaining in message header: ["––commentary"]
(APIServer pid=27525) INFO:     127.0.0.1:34650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=27525) INFO 08-06 23:26:05 [loggers.py:123] Engine 000: Avg prompt throughput: 2119.3 tokens/s, Avg generation throughput: 25.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 73.8%
(APIServer pid=27525) INFO 08-06 23:26:15 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 73.8%

[charlotte12l]

Encountered the same issue, how did you resolve it?

[MoellerAI]

I encounter the same issue while running gpt-oss-120b using the vllm v0.10.1 docker image. It doesn't appear to be an issue with the 20b model.

Seems like the issue arises only in multi-turn conversations and it appears to happen when the model is done with it's reasoning tokens and is about to start it's final response.

But it is not a consistent problem tho, however happens a lot

@dkundel-openai

[darshanj03]

Hi guys,

were you able to resolve this error ?
I have served gpt-oss-120b using vllm and built an application using langgraph - this erroring out happens quite often.
Any help would be greatly appreciated.

Thanks,
Darshan

[yongho-chang]

hello did you call with "ignore_eos":true ?
when i get same error using this parameter

[MoellerAI]

hello did you call with "ignore_eos":true ?
when i get same error using this parameter

No I don't. I just use a classic call with the python OpenAI sdk. It both happens when using the chat.completions endpoint and the responses endpoint. In multi-turn conversations and with streaming on. It is a really annoying bug I don't know how to solve. @dkundel-openai

It can be reproduced by just appending new messages to a conversation in a loop e.g. range(20) - then it always happens for me.

[aleksanderhan]

I get the same issue.

[IsaacRe]

I've been seeing this fairly frequently in vLLM when the 20b model fails to predict a valid header completion. Usually during a tool call where the model can predict something like ... Let's use the tool.<|end|><|start|>assistant to=functions.name<|channel|>commentary json<|message|> referencing the function name inside the header, but gets confused and outputs something invalid like to=...? Let's call the function with ... <|end|> without closing the header.

I think technically this can happen anytime the model generates multiple messages at once (eg. analysis -> final response or preamble -> tool call) regardless of which model or sampling config you're using.

The problem is that, unless guided decoding is used, you can't be sure that the model output will be parseable with harmony, and inference frameworks aren't handling the parsing errors so they show up downstream.

I think you need to treat outputs from gpt-oss the way you would treat any other structured output. Either catch parsing errors and retry or implement guided decoding around the output format.

[levunet]

@IsaacRe
It seems like that PR probably resolved it.

#76

[pduy395]

@IsaacRe It seems like that PR probably resolved it.

#76

I have tested it after pulling the latest code, but the error still occurs.

[levunet]

Could you provide reproducible test code? Or if you describe the situation, I'll look into it in more detail.

[pduy395]

import requests
import json

url = "http://192.168.51.250:9876/v1/chat/completions" 
headers = {"Content-Type": "application/json"}

body = {
    "model": "GPT-OSS-20B",
      'messages': [
    {
      'role': 'system',
      'content': 'You are GPT-OSS, a large language model trained by OpenAI.\nYou act as a mature traffic signal control assistant who gives accurate advice for complex signal scenarios.\n\nTOOLS:\n------\nYou have access to the following tools:\n\n\nGet Intersection Layout: Useful when you want to know the structure of the intersection. This tool provides the description of the intersection layout. The input to this tool should always be the str, \'None\'.\nGet Signal Phase Structure: Useful when you want to know the structure of the signal phase. This tool provides the description of the signal phase structure, including the `Phase ID` in this intersection and `Movement ID` in each signal phase. The input to this tool should always be the str, \'None\'.\nGet Current Occupancy: Useful when you want to get the congestion situation of each traffic movement at the **current** moment. The input to this tool should be a string, `junction_id`.\nGet Previous Occupancy: Useful when you want to get the congestion situation of each traffic movement at the **previous** moment. The input to this tool should be a string, `junction_id`.\nGet Traditional Decision: Useful when you want to obtain the decision made by traditional methods under the current environment. The result of traditional methods is applicable for minimizing queue length. However, in long-tail problems, such as the presence of an ambulance or sudden unavailability of certain movements, traditional methods are not the best solution. The input to this tool should always be the str, \'None\'.\nGet Available Actions: Useful before you make the decisions, this tool let you know what are your available actions in this situation step. The input to this tool should always be the str, \'None\'.\nGet Junction Situation: Useful when you want to determine whether the environment is a long-tail problem in traffic signal control. When you want to judge whether there is an ambulance in the environment, check whether each movement is passable. The input to this tool should always be the str, \'None\'.\n\nThe way you use the tools is by specifying a json blob.\nSpecifically, this json should have a `action` key (with the name of the tool to use) and a `action_input` key (with the input to the tool going here).\n\nThe only values that should be in the "action" field are: Get Intersection Layout, Get Signal Phase Structure, Get Current Occupancy, Get Previous Occupancy, Get Traditional Decision, Get Available Actions, Get Junction Situation\n\nThe $JSON_BLOB should only contain a SINGLE action, do NOT return a list of multiple actions. Here is an example of a valid $JSON_BLOB:\n\n```\n{\n  "action": $TOOL_NAME,\n  "action_input": $INPUT\n}\n```\n\nALWAYS use the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction:\n```\n$JSON_BLOB\n```\nObservation: the result of the action\n... (this Thought/Action/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nThe traffic signal control task often involves multiple steps. Break it down and complete step by step.\nThere is no rush to give a final answer unless you are confident it is correct.\n\nRules:\n- Use EXACTLY the two-line tool-call format:\n  Action: TOOL_NAME\n  Action Input: INPUT\n- Use only ONE tool per turn. Do NOT wrap the tool call in markdown fences.\n- If a tool requires no argument, write: Action Input: None\n\nReminder: you MUST use the EXACT characters `Final Answer` when returning the final decision.\nTake a deep breath and work through the problem step-by-step.\n'
    },
    {
      'role': 'user',
	'content': 'As the \'traffic signal light\', you are tasked with controlling the traffic signal at an intersection. You\'ve been in control for 158.0 seconds. The last decision you made was 1, with the explanation Phase 1 is chosen to give pedestrians priority while ensuring vehicles yield, providing a safe crossing for the guard.. Now, you need to assess the current situation and make a decision for the next step.\n\nTo do this, you must describe the Static State and Dynamic State of the traffic light, including the Intersection Layout, Signal Phase Structure, and Current Occupancy. Determine if you are facing a long-tail problem, such as the presence of an ambulance, impassable movements or the detectors are not work well. \n\nIf it\'s a standard situation, refer to the Traditional Decision and justify your decision based on the observed scene. If it\'s a long-tail scenario, analyze the possible actions, make a judgment, and output your decision.\n\nRemember to prioritize public transportation and emergency vehicles, follow the signal phase durations, and do not give a green light to impassable movements.\n\nHere are your attentions points:\n1. DONOT finish the task until you have a final answer. You must output a decision when you finish this task. Your final output decision must be unique and not ambiguous. For example you cannot say "I can either keep lane or accelerate at current time".\n2. You can only use tools mentioned before to help you make decision. DONOT fabricate any other tool name not mentioned.\n3. Remember what tools you have used, DONOT use the same tool repeatedly.\n\nLet\'s take a deep breath and think step by step. Once you made a final decision, output it in the following format: \n\n```\nFinal Answer: \n    "decision":{"traffic signal light decision, ONE of the available actions"},\n    "expalanations":{"your explaination about your decision, described your suggestions to the Crossing Guard"}\n```\n\n'
    }
  ],
    "temperature": 0.2,
    "stream": False
}

response = requests.post(url, headers=headers, data=json.dumps(body))
print(response.json())

I’ve tried running this code a few times, and I noticed that it often throws an error during the first few runs — either right after restarting the LLM, or even when the LLM is already running, it sometimes fails within the first 2–3 runs.

(APIServer pid=2032681) Traceback (most recent call last):
(APIServer pid=2032681)   File "/home/sysadmin/miniconda3/envs/duypt_llm/lib/python3.10/site-packages/vllm/entrypoints/harmony_utils.py", line 414, in parse_output_into_messages
(APIServer pid=2032681)     parser.process(token_id)
(APIServer pid=2032681)   File "/home/sysadmin/miniconda3/envs/duypt_llm/lib/python3.10/site-packages/openai_harmony/__init__.py", line 637, in process
(APIServer pid=2032681)     self._inner.process(token)
(APIServer pid=2032681) openai_harmony.HarmonyError: unexpected tokens remaining in message header: Some("to=Get Intersection")
(APIServer pid=2032681) INFO:     192.168.51.250:34618 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error