- A study by Cisco Foundation AI and Carnegie Mellon University conducted 325,000 experiments across 13 models from four AI families, including OpenAI, Anthropic, Google and Qwen.
- Eight of the 13 models tended to recommend more expensive options to wealthier user profiles, even when users made exactly the same requests.
- Claude Opus 4.8 showed the largest disparities: an average of $198 more for airline tickets and $284 more per month for health insurance for wealthy users.
- Gemini 2.5 Flash produced corresponding price gaps of $177 for airline tickets and $217 for health insurance. GPT-5 showed a $107 difference in airline ticket prices.
- Notably, the effect persisted even when users explicitly requested “the cheapest option.” Gemini 2.5 Flash still recommended airline tickets costing an average of $208 more for wealthy profiles.
- AI could also infer wealth from emails. When allowed to read just two emails, Gemini 2.5 Flash generated a $175 price gap and examined financial emails first in 97% of experiments.
- The researchers call this phenomenon “adversarial delegation”: personal data can help AI agents serve users better, while also potentially causing them to act against users’ stated interests.
- Blocking employment or demographic data does not consistently solve the problem. The disparities fell substantially only when financial attributes were directly restricted.
- The study is currently a preprint and has not undergone peer review. OpenAI also stated that the tested version of ChatGPT differs from its current consumer shopping system.
📌 This is an important warning for the age of AI agents: personalization does not automatically mean acting in the user’s best interests. When an agent has access to income, emails and personal circumstances, data intended to “understand the customer” can inadvertently become a basis for discriminating according to purchasing power. The findings from 325,000 experiments suggest a new governance principle: AI agents should follow users’ explicitly stated objectives before optimizing recommendations based on personal profiles, and should use only the data genuinely necessary for the task.
