Copilot Studio Kit

Copilot Studio Kit, published by the Microsoft Power CAT team, is a comprehensive set of capabilities designed to augment Microsoft Copilot Studio. The kit helps makers test custom agents, use a large language model to validate AI-generated content, and track aggregated key performance indicators.

Testing capabilities

Makers can configure and run test sets against the Copilot Studio APIs to verify that the agent responses are as expected. Test results can be enriched with additional data points from Azure Application Insights and from conversation transcripts in Dataverse. For AI-generated answers, AI Builder prompts are used to compare the generated answer with a sample answer or validation instructions. Now also support multi-turn testing that allows validating core scenarios end-to-end and helps with testing custom agents with generative orchestration enabled.

Conversation KPIs

Conversation KPIs are designed to help makers track and analyze the performance of their custom agents. This feature complements the existing built-in analytics of the Copilot Studio and simplifies the process of understanding conversation outcomes by providing aggregated data in Dataverse rather than requiring you to analyze the complex conversation transcripts.

SharePoint synchronization

SharePoint synchronization allows makers to configure selective content synchronization from SharePoint locations to a custom agent knowledge base as files. Compared to using SharePoint site as a knowledge source, larger files and more file types are supported. Additionally, PDF file indexing is improved, and SharePoint pages (.aspx) are supported as well.

Webchat Playground

Webchat Playground simplifies customizing the appearance and behavior of the copilot agent webchat, including colors, fonts, thumbnails and initials. Easy to use UI allows makers to define the look and feel of their webchat, and HTML is generated with the specified styles.

Adaptive Cards Gallery

Adaptive Cards Gallery provides makers with a dozen built-in adaptive card templates for different scenarios. They demonstrate the extensibility of the adaptive card visuals and behavior, and provide examples on the agent side implementation as well on things like dynamic data binding.

Agent Inventory

Agent Inventory can be used to get a tenant-wide view to all Copilot Studio custom agents in the organization, including detailed information on the features they are using, authentication mode, knowledge sources, orchestration type and more. Agent Inventory ships with a dashboard and the data can be exported for use in other applications.

Agent Review Tool

Agent Review Tool is a solution analysis tool that can be used to review your agents for any potential issues or anti-patterns that might have negative impact on the performance or the security of your agent. After analysing the solution, Agent Review Tool presents the findings in easy to interpret format, with severity and details on how to address the issue.