Discover · microsoft.com

No picture published by microsoft.com

How to test agent skills without hitting real APIs

Microsoft for Developers microsoft.com

This is what was read here when it was published. microsoft.com no longer carries it in what it publishes now.

When an agent eval involves pushing, pulling or keeping a git branch in sync, the fixture needs a remote repository. The easiest one to set up is a bare repository next to the fixture. It takes one git init --bare, works offline and resets with the rest of the workspace. A real GitHub repository in comparison, needs a token, an API call to create it and cleanup after every run. Run scenarios in parallel, and it gets even worse. Every run needs its own repository with a unique name that still looks like something a developer would create, and every failed teardown leaves one behind. Multiply that by hundreds of runs, and you’re also putting load on GitHub’s services for traffic that…

So we wanted to know whether a local remote is a good enough stand-in for the real thing. Mostly, it is. The agent rebased just as well and pushed just as safely against both. But when the agent had to decide on its own whether to push, the local remote changed the answer. It made the agent push less. If your eval checks whether an agent changes the remote without being asked, a local remote will make the agent look more careful than it is in real use.

More ways to use this
Open on another device
Scan this QR code to open the same public page

Scan to open this page. Your saved notes and account access are not included.

Download QR code

Open the page link

Names and references

  1. Test — person

In the piece

Keep going

Each of these is read from what this estate already kept: the feed a piece came through, the publisher behind it, the credits and topics the publisher printed. None of it is a recommendation, an identity or a record kept anywhere else.

More like this

Also from Microsoft for Developers

  1. Do AI coding agents care about SemVer?Waldek Mastykarz microsoft.com
  2. Your best AX fixes are hiding in agent trajectoriesWaldek Mastykarz microsoft.com
  3. Does ‘eval’ in the workspace path change what your agent does?Waldek Mastykarz microsoft.com
  4. Publish Visual Studio Marketplace extensions securely with Trusted PublishingLynn Dai microsoft.com
  5. PostgreSQL Like A Pro: Build, query, and troubleshoot without leaving VS CodeCynthia Zanoni microsoft.com
  6. Can a local git remote replace GitHub in your agent eval?Waldek Mastykarz microsoft.com