Foundry
Agent Builder

Introduction

How might we make AI agent creation more intuitive for new developers?

Foundry set out to democratize agent creation, but new developers kept stalling before they finished.

Microsoft Foundry lets students, developers, and startup founders build AI agents. The vision is to democratize agent creation for people still learning, but vision only matters if users can get through the flow.

As a UX Researcher on Microsoft's CoreAI team, I led 8 moderated usability sessions on the Knowledge, Data, and FoundryIQ features: I defined the research questions, designed the protocol, ran the sessions, and owned team communication. In the very first session, and every one after, we found a critical blocker that 100% of participants hit, one the team hadn't anticipated. The study went from "let's see how intuitive the flow is" to "we need to fix this before anything else matters."

Note: Due to the confidential nature of this product and user privacy, visuals in this case study are limited.

Research Questions

What we set out
to understand.

The question that framed everything: when does confusion stop being a learning curve and become a dead end?

I shaped the questions to go past surface usability and find where confusion turns from "learning curve" into "I'm done." That distinction, temporary confusion versus a hard blocker, became the most important framing of the study.

Method

How we conducted
the study.

Eight moderated Think-Aloud sessions, chosen to catch not just where users failed but why.

I chose moderated usability testing because the research centered on mental models. I needed to be there when confusion happened. Remote unmoderated testing would capture task failure, but not why. The Think Aloud protocol revealed participants' reasoning in real time, and their silences were often most revealing.

Participant Criteria

I recruited participants matching Foundry's target audience: students and early-career professionals with technical backgrounds, curious about AI and looking to leverage it for projects.

What I tested, and what I left out

Eight sessions surface pattern-level problems, not statistics. I scoped the study to the first-run agent-creation path (create, add knowledge and data, configure, publish) and left three things out of frame on purpose.

Naming these limits up front kept the readout honest: when I reported that every participant hit the guardrail block, the team could trust it as a qualitative certainty, not a statistic stretched past what eight sessions can carry.

Findings

What worked well.

Some patterns were already landing, and naming them tells the team what to protect.

Not everything was broken, and that matters. Naming what works tells the team which patterns to protect as they iterate.

Areas of Improvement

Five issues. One that stops everything.

One issue was a Severity-4 blocker that stopped all eight participants; the rest, I ranked behind it.

I prioritized findings using Nielsen's severity scale. A confusing label is a different problem than a blocker preventing every user from completing the core task. Clear severity framing gave the Foundry team an actionable roadmap, not just a list of complaints.

From findings to something the team could act on

A finding a team cannot act on is just an opinion. I structured every issue the same way, so design and engineering could pick up their part without a translation step: the observed behavior, how many of the eight it hit, a severity rating, and a specific recommendation. I handed the set to the Foundry design team with the guardrail fix flagged as blocking, and the terminology audit and landing-page hierarchy sequenced behind it for later sprints.