Foundry Agents/Copilot agents and Playwright

[faint depot]

Hello folks!

I am looking to create two AI agents, 1st Agent to generate automated tests requirements based on Confluence, and 2nd Agent to generate the Playwright automation script for each test case and execute the tests together with report

Hard requirement: app is hosted on pre-mises

Questions:

1. Is this something you have seen being done?
2. Would you recommend this with Foundry Agents? (here with a VNet we will solve the hybrid connectivity)
   3.I saw an article for Copilot Agents: https://github.blog/ai-and-ml/github-copilot/debugging-ui-with-ai-github-copilot-agent-mode-meets-mcp-servers/
   The question here would be, how do we sort the connection to on-premises app

Thank you very much in advance
#1291681390836056074

[cerulean sparrow]

@faint depot
Yes, this pattern absolutely makes sense, it’s already emerging in the wild, and you can do it with either Foundry Agents or Copilot Agents + MCP. The key is where your tools (Confluence access + Playwright runner) live relative to your on‑prem app, and how you bridge that network boundary.

1. Is this something you have seen being done?

Conceptually, yes, this is a very natural pattern and I’ve seen variations of it:

• Agent 1: Test-requirements generator

  • Inputs: Confluence pages (requirements, specs, user stories).
  • Outputs: Structured test cases (title, steps, preconditions, expected results, tags, priority).
  • Implementation pattern:
    • A tool that queries Confluence via REST/GraphQL, filtered by space/page labels.
    • The agent turns the content into a strict schema (JSON) for test cases.
    • Those test cases can be written back to Confluence, a test-management system (e.g., Azure DevOps, Jira XRay), or a repo.

• Agent 2: Test-automation generator & executor

  • Inputs: Structured test cases from Agent 1.
  • Outputs: Playwright test files + execution results + report.
  • Implementation pattern:
    • Tooling to:
      • Create/patch Playwright spec files in a repo.
      • Trigger a test runner (self-hosted agent, container, or MCP server) that has Playwright + browser dependencies installed.
      • Collect JUnit/HTML reports and push them to a storage location or test dashboard.

The agent → tools → CI/test-runner pattern is now pretty standard in enterprise AI setups; your twist is that the system under test is on‑prem, which just means your test runner and/or tools must live inside or be securely connected to that network.

2. Would you recommend this with Foundry Agents (with VNet for hybrid)?

Yes, Foundry Agents are a good fit, especially if you’re already in Azure and want enterprise-grade governance.

Core pattern with Foundry Agents

1. Host the agent in Foundry

   • The “brain” of both agents (test-requirement generator and Playwright generator) runs as Foundry Agents in Foundry.

2. Use a VNet-secured Agent environment

   • Foundry Agent Service supports “Standard Setup with private networking”, where the agent runs inside a private virtual network you control.
   • This lets your agent securely access private resources with no public egress, using “container injection” into your VNet.

3. Hybrid connectivity to on‑prem

   • From that VNet, you use an enterprise pattern:
     • VPN or ExpressRoute from Azure VNet to your on‑prem network, or
     • Use private endpoints to reach other Azure-hosted services that themselves bridge to on‑prem.
   • Once the VNet is connected to on‑prem, the agent’s tools can talk to:
     • Confluence (if self-hosted on-prem).
     • Your web app under test.
     • Self-hosted test runners or MCP servers.

4. Tools for this scenario

   • Confluence tool: REST client tool that lives in the same VNet, authenticated via OAuth/API token.
   • Test-repo tool: Git/DevOps tool to commit Playwright tests (e.g., via Azure DevOps or GitHub, using Foundry Connections patterns).
   • Execution tool: A tool that calls:
     • A self-hosted CI runner (Azure DevOps, GitHub Actions, Jenkins) living in the VNet/on‑prem network, or
     • A custom HTTP API/MCP server that wraps npx playwright test and returns results.

Foundry Connections samples show how to integrate agents with external services using secure Connections, and you can reuse that pattern for Confluence, CI, and storage.

Use a VNet-secured Agent environment

• Foundry Agent Service supports “Standard Setup with private networking”, where the agent runs inside a private virtual network you control.
• This lets your agent securely access private resources with no public egress, using “container injection” into your VNet.

Hybrid connectivity to on‑prem

• From that VNet, you use an enterprise pattern:
  • VPN or ExpressRoute from Azure VNet to your on‑prem network, or
  • Use private endpoints to reach other Azure-hosted services that themselves bridge to on‑prem.
• Once the VNet is connected to on‑prem, the agent’s tools can talk to:
  • Confluence (if self-hosted on-prem).
  • Your web app under test.
  • Self-hosted test runners or MCP servers.

Tools for this scenario

• Confluence tool: REST client tool that lives in the same VNet, authenticated via OAuth/API token.
• Test-repo tool: Git/DevOps tool to commit Playwright tests (e.g., via Azure DevOps or GitHub, using Foundry Connections patterns).
• Execution tool: A tool that calls:
  • A self-hosted CI runner (Azure DevOps, GitHub Actions, Jenkins) living in the VNet/on‑prem network, or
  • A custom HTTP API/MCP server that wraps npx playwright test and returns results.

Foundry Connections samples show how to integrate agents with external services using secure Connections, and you can reuse that pattern for Confluence, CI, and storage.

So I would recommend Foundry Agents for this, using:

• VNet-secured Foundry Agent environment
• VNet ↔ on‑prem connectivity (VPN/ExpressRoute)
• Tools in that VNet for Confluence + Playwright execution

Copilot Agents + MCP and on‑prem connectivity

The GitHub article you referenced (Copilot Agent mode + MCP servers) essentially shows this pattern:

• Copilot Agent = reasoning/orchestration, running in GitHub/IDE.
• MCP servers = tool providers (e.g., inspect UI, run tests, query systems).
• The agent calls MCP servers over HTTP/WebSockets, using a simple protocol, and they can live anywhere as long as the agent can reach them.

How to reach an on‑prem app with Copilot Agents

You solve this by placing the MCP server next to the on‑prem app:

1. Host an MCP server inside your on‑prem network
   • The MCP server exposes tools like:
     • getconfluencepage, listrequirements, updatetest_status
     • generateplaywrighttests
     • runplaywrightsuite, gettestreport
   • The MCP server has direct network access to:
     • On‑prem Confluence
     • The web app under test
     • Local Playwright runtime

Establish connectivity from Copilot Agent to MCP
Options:

• VPN from dev environment (if agent runs in a context that has network into your on‑prem).
• Secure reverse tunnel (like dev-tunnel pattern described for hybrid AI, where a cloud agent talks back to a local MCP server via a tunnel).
• Reverse proxy that publishes the MCP server through a locked-down HTTPS endpoint, protected with mTLS/API gateway.