Overview Research Questions Method Findings Improvements Impact Next Steps Reflection

UX Research · Usability Testing · AI Platform

Microsoft
Foundry

End-to-end usability testing and research driving strategic design improvements for Microsoft's AI developer platform — uncovering critical blockers in the agent-creation flow.

Microsoft Foundry is a publicly available platform. All information shared in this case study is based on publicly accessible features. Participant identities and internal data remain confidential and fully anonymized.

My Role
UX Designer & Researcher
Team
4 researchers · Microsoft CoreAI
Timeline
Jan – Mar 2026 (10 weeks)
Methods
Moderated Usability Testing · Think Aloud
Context
Microsoft CoreAI
Developer Platform
Focus
Agent Creation Flow
& Usability
Participants
8 moderated
usability sessions
Study Design
Within-subject
Think Aloud

Introduction

How might we make AI agent creation more intuitive for new developers?

Microsoft Foundry is a platform built for students, developers, and startup founders who want to create AI agents for their websites or apps. It helps users build smart assistants that work with their data and automate tasks — even if they're still learning or exploring AI.

I led usability testing on Foundry's CoreAI platform, specifically the Knowledge, Data, and FoundryIQ features used in agent creation. My goal was to identify challenges in this flow and make the experience more successful for users, so they are more inclined to use and adopt the platform.

I focused on students with technical backgrounds and startup founders — people who are curious about AI and looking to leverage it for real projects, but may not be experts in the platform itself.

Research Questions

What we set out
to understand.

I shaped the research questions to investigate participants' mental models when interacting with Microsoft Foundry — where confusion begins, what makes it feel like a blocker, and what users actually want to build.

01
At what point in the Foundry agent-creation flow do users first feel confused or overwhelmed?
02
Which concepts or terms make users feel like they need external help to continue?
03
Do users perceive confusion as a temporary learning curve — or a hard blocker that makes the tool unusable?
04
What data types do users want to work with, and what do they want their agents to do?

Method

How we conducted
the study.

I conducted moderated usability testing utilizing a within-subject design. This was the most appropriate method because the research questions are designed to investigate participants' mental models. With moderated sessions, I used the Think Aloud protocol, which helped me understand exactly which points of the user flow were most confusing.

🎙️
Moderated Sessions
I facilitated one-on-one moderated usability tests with real tasks in the live Foundry platform, using Think Aloud protocol to capture real-time reasoning and confusion points.
n = 8 participants
📋
Task-Based Scenarios
Participants completed structured tasks including agent creation, configuring knowledge sources, uploading data, and publishing agents — mirroring real use cases.
Within-subject design
📊
Likert Ratings & Quotes
Post-task ease-of-use ratings on a 1–5 scale combined with qualitative Think Aloud data and direct participant quotes for triangulated findings.
Mixed methods

Participant Criteria

I recruited participants who matched Foundry's target audience — students and early-career professionals with technical backgrounds who are curious about AI and looking to leverage it for projects.

Critical Criteria
Active student status CS or technical background Curious about AI Looking to leverage AI for a project Experience with data sets

Findings

What worked well.

Before diving into areas for improvement, it's important to highlight features where participants had a smooth, positive experience — signaling strong existing design decisions.

✅ Publishing an Agent
5 out of 8 participants rated the ease of publishing their agent as a 1 out of 5 (very easy). The publishing flow aligned with familiar patterns from other tools, making it intuitive and frictionless.
"Publishing the agent was pretty straightforward and aligns with what I would expect it to do because it's very similar UI elements to what other tools do right now."
✅ Overall Task Completion
4 out of 8 participants rated the overall ease of use as a 2 out of 5 (easy). While the platform had areas of confusion, participants were ultimately able to navigate and complete tasks — suggesting a solid foundation to build on.

Areas of Improvement

Opportunities to improve the Foundry experience.

I prioritized issues by severity (highest first) using Norman's severity scale. Each finding includes the interface problem, supporting data, and actionable recommendations.

Severity Scale (Norman's)
4 Usability catastrophe — Prevents task completion; imperative to fix
3 Major problem — Causes significant confusion; important to fix
2 Minor problem — Adds friction but doesn't block completion; should be addressed
1 Cosmetic — Surface-level issue; fix if time permits
Severity 4 Guardrail Blocks Agent Creation
8 out of 8 participants were unable to proceed with creating an agent because the interface blocked interactions due to an unassigned or mismanaged guardrail. The system leaves new agents in an ambiguous "inheriting" state instead of automatically assigning Microsoft's default guardrail, causing the interface to prevent users from completing the task.
The error message further compounds confusion by prompting users to "create guardrail," when the actual resolution is to reassign to an existing default guardrail. Participants lacked contextual guidance on why the interaction was blocked, were unclear about the differences between guardrail versions, and could not see the active guardrail status — all of which increased frustration and wasted time.
Recommendations
Automatically assign Microsoft's default guardrail during agent creation. Update error messages to direct users to "Reassign Guardrail" rather than "Create Guardrail." Provide inline explanations about why interactions are blocked and how to resolve them. Clarify guardrail purposes and version differences through tooltips or descriptions. Make guardrail status more visible in the interface.
Severity 3 Confusing Terminology & Labeling
7 out of 8 participants were confused by overlapping or unclear terminology in the platform. Key points of confusion included the distinction between "Tools" and "Knowledge" (and why file uploads appeared under Tools), as well as the difference between "Agent Instructions" and "Message Agent." While this didn't fully prevent task completion, it was the most relevant usability friction to the overall experience.
Recommendations
Audit and simplify terminology across the platform to ensure labels are distinct, descriptive, and consistent. Add contextual definitions (tooltips or inline descriptions) for key concepts like "Tools," "Knowledge," and "Agent Instructions." Consider renaming overlapping terms to reduce cognitive load for first-time users.
Severity 2 Cluttered Landing Page & Visual Hierarchy
4 out of 8 participants experienced discoverability issues on the landing page. The "Start Building" button lacked visual prominence due to its relatively small size and the presence of multiple competing visual elements. The "Coding Quick Start" bar was significantly larger and attracted users' attention first. Additionally, the similarity in terminology between these two options created confusion regarding the appropriate starting point.
"I see a couple of places I could go to. Do I go to Start Building? Do I go to the Coding Quick Start part?"
Recommendations
Rebalance the page layout to reduce visual competition among elements and enhance hierarchical clarity. Increase the visual prominence of the "Start Building" button by enlarging its size, contrast, and positioning it within a primary focal area.
Severity 2 Unclear Platform Navigation Flow
8 out of 8 participants demonstrated a different flow for navigating Microsoft Foundry. After creating an agent, 2 out of 8 participants interacted with the navigation sidebar tabs to gain more understanding about the platform's terminology. While participants were ultimately able to navigate, the lack of guided structure forced exploratory behavior.
"I'll start by looking at the nav bar because there was no clear instruction of how like the different steps that will be involved in making the AI agents, so I'll have to explore the software on my own."
Recommendations
Introduce onboarding assistance or a guided tutorial for first-time users. 3 out of 8 participants specifically suggested this. As one shared: "I feel like if I looked up a tutorial or if the platform gave me some info when I created my account, it would be pretty easy to figure out as you go."
Severity 2 Misleading Error Messages on File Upload
5 out of 8 participants experienced confusion when files were successfully uploaded but an error message appeared. The upload process itself was straightforward, but the false error introduced unnecessary doubt and broke user confidence in the system.
"At least uploading [files] was straightforward. But that error message was a little bit confusing."
Recommendations
Investigate and resolve the underlying bug causing false error messages on successful uploads. Ensure confirmation states clearly communicate success and distinguish between warnings, errors, and informational messages.

Study at a Glance

The numbers behind
the research.

Key metrics from the moderated usability sessions I conducted that shaped findings and recommendations for the Foundry team.

8
moderated usability sessions conducted with technical students and founders
5
distinct usability issues identified and prioritized by severity for the Foundry team
100%
of participants encountered the critical guardrail blocker during agent creation

Next Steps

Where this research
goes from here.

My study surfaced both immediate fixes and broader areas to explore in future research cycles.

01
Resolve the Guardrail Blocker
The guardrail issue emerged mid-study and affected every single participant. Addressing this is the highest priority, as it completely blocks the core agent-creation workflow.
02
Test Agent Integration End-to-End
Future studies should explore how easily users can take a published agent and actually integrate it (via code or embed) into a product or website — a critical step my study didn't cover.
03
Redesigning Foundry — UX Design Co-op
As of April 2026, I've taken on a User Experience Design co-op under the same Microsoft CoreAI team to redesign Foundry — turning the research insights from this study into tangible design solutions.

Reflection

What I learned &
what comes next.

Conducting a usability study for a live Microsoft product was an incredible learning experience — from navigating stakeholder relationships to adapting on the fly when the guardrail issue caught the team off guard.

My biweekly check-ins with the Microsoft sponsor were invaluable for resolving technical issues in a timely manner and keeping the study on track. The combination of strong team collaboration and productive usability sessions meant that every test taught me something new.

What Went Well
+ Timely and sufficient participant recruitment — I met the minimum of 6–8 participants
+ Biweekly 1-hour check-ins with the Microsoft sponsor helped me resolve issues in real time
+ Strong team organization with weekly internal syncs
+ Each usability session was productive and surfaced new insights
What I'd Do Differently
~ Prepare workarounds for known platform issues before testing begins — the guardrail blocker was so new that it caught the team off guard too
~ Otherwise, the study ran smoothly — a testament to strong planning and team dynamics
Next Project
Nox+ — Secure Digital Identity
View case study →
My Work About Resume