Advice on Adding Web Scraper Plugin to .NET Semantic Kernel App

[grave ether]

Hey everyone! I’m building a .NET AI app using Semantic Kernel and the progress framework. I want to add a web scraper plugin/service that visits a webpage, finds a recipe, and returns structured data. Any advice on how to approach this? Should I use one of the MCP web scraper services? Thanks in advance!

[stiff fractal]

Hi @grave ether Have you seen the Playwright MCP server?

To achieve your goal of building a web scraper plugin/service for your .NET AI app using Semantic Kernel, the Playwright MCP server is a great choice.

Here's how you can approach it:

1. Set Up the Playwright MCP Server:

   • Install the Playwright MCP package using npm install @playwright/mcp.
   • Start the MCP server with a script in your package.json, such as:

     {
       "scripts": {
         "server": "npx @playwright/mcp --port 8931"
       }
     }
     

   • Run the server using npm run server. This will launch the Playwright MCP server and display the port and endpoints.

2. Integrate MCP with Semantic Kernel:

   • Install the MCP client NuGet package in your .NET project:

     dotnet add package ModelContextProtocol --prerelease
     

   • Connect to the Playwright MCP server and retrieve tools:

     var mcpClient = await McpClientFactory.CreateAsync(new McpServerConfig {
         Id = "playwright",
         Name = "Playwright",
         TransportType = TransportTypes.Sse,
         Location = "http://localhost:8931"
     });
     var tools = await mcpClient.ListToolsAsync();
     

3. Configure Semantic Kernel:

   • Add the MCP tools to Semantic Kernel as functions:

     var kernelBuilder = Kernel.CreateBuilder();
     kernelBuilder.Plugins.AddFromFunctions(
         pluginName: "playwright",
         functions: tools.Select(x => x.AsKernelFunction())
     );
     var kernel = kernelBuilder.Build();
     

4. Create a Recipe Scraper Function:

   • Use Semantic Kernel to define a prompt that instructs the MCP server to navigate to a webpage, locate a recipe, and extract structured data. For example:

     var result = await kernel.InvokePromptAsync(
         "Visit the webpage, find a recipe, and return the ingredients and steps as structured data.",
         new KernelArguments()
     );
     

5. Test and Refine:

   • Test the integration to ensure the scraper correctly identifies and extracts recipe data. You may need to tweak the prompt or MCP server configuration for optimal results.

For more details, you can explore this GitHub repository and this guide that demonstrate integrating Playwright MCP with Semantic Kernel.

These resources provide examples and code snippets to help you get started.