Discover · microsoft.com

Does ‘eval’ in the workspace path change what your agent does?

Microsoft for Developers microsoft.com

Your eval might be telling the agent it’s a test before the agent reads a single instruction. Eval tools need a fresh folder for every run, and the easiest way to get one is a temp directory with a random name like /tmp/eval-7f3a2c. The agent sees that path on every turn, in the system prompt as its working directory and in the output of every tool call. If models can tell when they’re being tested, a folder literally called eval is about the easiest giveaway there is.

So we checked whether the folder name changes what agents do. Across about 900 runs and four models, it made them more convinced they were being tested, but it didn’t change their behavior in any way we could measure. They didn’t cheat less, cut fewer corners, report more honestly or work harder.

More ways to use this
Open on another device
Scan this QR code to open the same public page

Scan to open this page. Your saved notes and account access are not included.

Download QR code

Open the page link

Names and references

  1. Change — person
  2. Path — person

In the piece

Keep going

Each of these is read from what this estate already kept: the feed a piece came through, the publisher behind it, the credits and topics the publisher printed. None of it is a recommendation, an identity or a record kept anywhere else.

More like this

Also from Microsoft for Developers

  1. Do AI coding agents care about SemVer?Waldek Mastykarz microsoft.com
  2. Your best AX fixes are hiding in agent trajectoriesWaldek Mastykarz microsoft.com
  3. Publish Visual Studio Marketplace extensions securely with Trusted PublishingLynn Dai microsoft.com
  4. PostgreSQL Like A Pro: Build, query, and troubleshoot without leaving VS CodeCynthia Zanoni microsoft.com
  5. Can a local git remote replace GitHub in your agent eval?Waldek Mastykarz microsoft.com
  6. Introducing the Agent Experience (AX) Practitioner PlaybookWaldek Mastykarz microsoft.com