How to test agent skills without hitting real APIs
Microsoft for Developers microsoft.com
This is what was read here when it was published. microsoft.com no longer carries it in what it publishes now.
Listen to this summary
When an agent eval involves pushing, pulling or keeping a git branch in sync, the fixture needs a remote repository. The easiest one to set up is a bare repository next to the fixture. It takes one git init --bare, works offline and resets with the rest of the workspace. A real GitHub repository in comparison, needs a token, an API call to create it and cleanup after every run. Run scenarios in parallel, and it gets even worse. Every run needs its own repository with a unique name that still looks like something a developer would create, and every failed teardown leaves one behind. Multiply that by hundreds of runs, and you’re also putting load on GitHub’s services for traffic that…
So we wanted to know whether a local remote is a good enough stand-in for the real thing. Mostly, it is. The agent rebased just as well and pushed just as safely against both. But when the agent had to decide on its own whether to push, the local remote changed the answer. It made the agent push less. If your eval checks whether an agent changes the remote without being asked, a local remote will make the agent look more careful than it is in real use.
More ways to use this
Open on another device
Scan to open this page. Your saved notes and account access are not included.
How Tenable brought multi-language support to security detectors with agentic translation