
Does ‘eval’ in the workspace path change what your agent does?
Microsoft for Developers microsoft.com
Listen to this summary
Your eval might be telling the agent it’s a test before the agent reads a single instruction. Eval tools need a fresh folder for every run, and the easiest way to get one is a temp directory with a random name like /tmp/eval-7f3a2c. The agent sees that path on every turn, in the system prompt as its working directory and in the output of every tool call. If models can tell when they’re being tested, a folder literally called eval is about the easiest giveaway there is.
So we checked whether the folder name changes what agents do. Across about 900 runs and four models, it made them more convinced they were being tested, but it didn’t change their behavior in any way we could measure. They didn’t cheat less, cut fewer corners, report more honestly or work harder.
More ways to use this
Open on another device
Scan to open this page. Your saved notes and account access are not included.
How Tenable brought multi-language support to security detectors with agentic translation