Vals study finds Chinese AI answers shift with language and context

Language changes the answer

Vals measured agreement with the Chinese government’s position on a scale from −1 to +1, calling the score “pro-China bias.” Positive scores indicate greater agreement with the official position or support for the government; negative scores indicate more critical answers.

Chinese models were, overall, less supportive of the government’s position than the researchers expected, though results varied considerably by model. GLM-5.3 Flash, for example, directly acknowledged that human rights abuses in China are well documented. Some models from other countries, meanwhile, sometimes agreed with the official Chinese position on political questions.

For most tested models, answers in Simplified Chinese were more supportive of the government than answers in English, Russian or Spanish. The effect appeared in both Chinese and American models, but was much stronger in the Chinese ones. Vals also found a smaller but consistent difference between Simplified Chinese, common in mainland China, and Traditional Chinese, used in Taiwan and Hong Kong.

The effect extended beyond language. Some models gave more pro-government answers when a system prompt said they were being used inside a company in mainland China rather than in the United States. Vals suggests models may infer an expected viewpoint from the language or setting of a prompt.

Refusal is not the only response

Vals found that the idea that Chinese models mostly block sensitive questions held up only for some DeepSeek models. Most other Chinese models rarely refused. Instead, some gave answers aligned with the government’s position. Kimi K3, for instance, denied the existence of Uyghur camps in Xinjiang and described the facilities as vocational training centers established to combat extremism and terrorism.

Answer format mattered too. When asked for only “yes” or “no” rather than a detailed response, GLM-5.3 Flash and MiMo V2.6 Pro were notably less likely to support the government’s position.

Several models also received lower pro-China bias scores after Vals gave them system instructions and tool descriptions for coding assistants. The models barely used the tools. Access to web search produced a similar shift toward less pro-government answers, even when the models did not search the internet.

The benchmark has limits

Vals says its political questions do not reflect typical user requests. Political preferences were harder to detect in more realistic work tasks, though the company found examples there too.

When models translated a list of country codes for a travel app into Simplified Chinese, most rendered “Taiwan” without qualification. Qwen3.8 Max added “Taiwan, China,” a formulation used by the Chinese government.

In an example published by Vals, OpenAI’s image-generation model GPT Image 2.5 Sunburst also produced different maps depending on the prompt language. The Chinese-language result labeled Taiwan “Taiwan Province” and included the disputed nine-dash line used by Beijing to mark its territorial claims in the South China Sea. Those elements were absent from the English-language result.

Fine-tuning could change the pattern substantially. Thomson 1.0 Small, a model fine-tuned by Thomson Reuters, gave answers aligned with the official Chinese position far less often than its base model, Qwen3.6 35B: the share fell from 84.6% to 15.0%. It still defended the “one China” principle and the idea of “peaceful reunification” with Taiwan in some answers.

The political slant of responses depends on the prompt language. The gap between English and Simplified Chinese is especially wide for Chinese models. | Image: Vals.ai

Source: the-decoder.com

I think the clearest lesson is not that Chinese models are either compliant or independent. It is that a benchmark score can shift with the prompt, the format and the tools on offer. Vals’s results also need to be read with two caveats: refusals counted as neutral, and many Chinese models were tested through third-party services rather than their makers’ official platforms, where additional filters may apply. Earlier research by Aleph Alpha and NewsGuard found Chinese models often repeated official accounts on sensitive topics, while Chinese rules require public AI services to uphold “core socialist values.” The unresolved issue is how much those constraints shape the ordinary answers people actually see.