Thryv · AI product design case study

SMB Reporting

Three AI reporting dashboards, tested with 53 small business owners

I planned the study and designed three versions of a reporting dashboard. I built them as working prototypes with Claude Code, along with the live AI service behind them, and 53 owners each tested one version.

Role
Lead Product DesignerResearch plan to working code
Timeline
Jul to Sep 2026About 2 months, discovery to report
Research method
53 owners, unmoderatedThree versions, one per owner
Usability score (SUS)
92.7, the top versionAll three above the 68 average

The problem

Owners said they check 3 to 5 separate tools to see how the business is doing.

About 70% named QuickBooks, with spreadsheets filling the gaps, and none had one view of revenue, leads and customers together. Their tools showed what a number did, but rarely why it moved.

I have MailChimp, QuickBooks, and a spreadsheet that I made three years ago. They do not talk to each other. I have to be the connection.

Study participant, Version A
The question

I wanted to find out how much owners want the AI to set up their dashboard, and how much they would rather arrange it themselves.

What I did

I designed Version A, where the AI sets up the dashboard, and Version B, where the owner arranges it. Another designer asked me to add Version C, a chat-first dashboard, because it would affect his product area. I built all three.

This study ran alongside Ask Thryv. I built and tested all three versions at the same time. Version A's chat panel reused Ask Thryv's logic and its chart library with simpler options on each chart, since the study tested whether owners would add a chart from the chat to their dashboard, not the charts' own features.

The 30-second read

Owners scored the dashboard they arranged themselves highest.

My research and prototype went to product for its dashboard redesign.

01 · What I built

I built three working versions on one live AI service.

I used Claude Code for all three and for the AI service behind them. Trace an AI answer

02 · What we learned

Owners still wanted a template to start from.

B, the version owners arranged themselves, scored 92.7 against a 68 benchmark. Nobody wanted a blank canvas, so I recommended a hybrid built on B.

My role: Lead Product Designer and PM, end to end. I owned the research plan, screener and test script, both waves of unmoderated testing, the analysis and final report, and the design, plus the working prototypes and AI service I built for the study.

  • AI product design
  • Research planning
  • Unmoderated testing
  • Coded prototypes
  • Prompt design
  • LLM tool calling
  • Claude Code
  • Design system MCP

See all three working · 40 seconds

Watch how each version gets a new metric onto the dashboard.

The research plan

I wrote the research plan before building anything.

Then I designed the screener and test script to keep three very different versions comparable, with an addendum for Version C.

13

research areas, written before any prototype existed

8

screener questions, each with at least one disqualifying answer

4

versions of the test script

Plan, script, screener and session details

How the plan split the research areas between versions

Research areaVersion A · AI-forwardVersion B · User-controlled
Which metrics actually matterObserved, through what people accept or change from the AI's defaultsTested directly through the metrics people pick themselves
Appetite for customizationObserved, through whether people try to change the AI-built layoutCore focus drag-and-drop comfort and template use
Where AI belongsCore focus whether a prominent AI feels helpful or intrusiveTested directly whether people seek out an optional AI or ignore it
How deep people goTested directly through AI-assisted deeper explorationObserved, through whether people click into widgets
What brings people backTested directly whether AI setup creates enough value to returnTested directly whether building it yourself creates ownership
Trust in AI answersCore focus reactions to AI insights and errorsObserved, as trust when the AI is optional

Six of the 13 areas are shown. Version C's mapping is in the addendum.

What changed in the test script, and why

ChangedFromToWhy
Response typesWritten open textVerbal, multiple choice and 1 to 7 ease ratingsTyped answers in unmoderated sessions come back short and flat
Think-aloudSeparate question listsFolded into the taskAsked afterward, people reconstruct their reaction from memory
Platform surveyPlatform questionnaireRemovedIt duplicated the closing questions and cost minutes the tasks needed
Usability score1 to 5 ease and likelihood ratingsStandard 10-item scaleOne benchmarked score that compares across all three versions
Intro framingA fictional company, applied to your own businessYou run Hartwell, plus what your version shows firstEveryone judges the same business as its owner and knows what their version shows first

I made these changes over the script's four versions. I also rewrote two task premises after checking them against the code, so that confusion caused by the prototype would not look like a design problem in the results. One premise claimed a metric was visible when the layout logic excluded it. The other asked participants to imagine returning tomorrow, in a prototype with no session persistence.

The session

One session, start to finish

  • AAI-forward
  • BUser-controlled
  • CChat-first
  1. Before the prototype

    Screener

    8 questions: role, vertical, size, tools

    Same for every owner. No prototype on screen yet. Each owner will see one version only.

    Before any version is shown

    Qualifies owners
  2. Scenario intro

    You run Hartwell. Think out loud.

    Each owner sees one version only

    Think-aloud from here on
  3. Background

    How do you track the business today?

    No prototype on screen yet

    verbal
  4. In the prototype

    T0First look

    Look, no clicks yet. What is it for?

    AWelcome screen, one Get started button

    BChoose a starting point: goal cards

    CKey metrics, suggested questions, chat box

    verbal
  5. T1Getting started

    First time opening it. Talk it through.

    AWelcome, 3 questions, layout pick, AI builds

    BGoal template picker, short load

    CNo setup, straight to the welcome screen

    1 to 7 ease
  6. T2Read the dashboard

    How is Hartwell doing? What is missing?

    AAI insight banner, Vitals, Key metrics

    BAt a glance cards, Key metrics

    COverview metric cards

    1 to 7 ease
  7. T3Investigate a decline

    Leads may be down. Find out why.

    ANew Leads card → Detail modal, or Ask AI panel

    BNew Leads card → Detail modal, or wand popover

    C“Why are my leads down?” chip → answer card

    1 to 7 easeverbal
  8. T4Make it your own

    Change, add or rearrange anything.

    ADrag and resize, and AI adds metrics and charts

    BEdit Dashboard + Add widget drawer

    CSave to Dashboard → Your metrics

    verbal
  9. T5Use the AI

    Find the AI. Revenue vs last month?

    AHeader Ask AI side panel

    BWand on a revenue widget, no global button

    CThe chat itself, already home

    1 to 7 ease
  10. T6Thirty-second check

    Hectic morning: the one thing you’d check

    Their own pick, on the version they had

    verbal
  11. After the prototype

    Closing

    Most useful alerts, and where you check

    Same for every owner, after the prototype.

    Same for all versions

    multiple choice
  12. Usability scale

    System Usability Scale (SUS), 10 items

    Same for all versions

    usability scale
Session structure from the final test script. Version rows show what each build offered at that step, mapped from the code, not what each participant did.

What the screener filters for

  1. Decision authority, not just employment at a small business
  2. Reviews their own numbers, at some real cadence
  3. Business stage, from first hires to expansion, with solo operators screened out
  4. Service vertical, weighted toward home services
  5. Headcount and revenue band
  6. Tools already used, which doubles as context for the findings
  7. No one who had used the company's own product, so nobody judged a familiar tool

The testing platform needs at least one disqualifying answer per question, so every question has to filter. When I merged my screener with the organization's template, I dropped one of its questions as redundant. It was not, and it went back in verbatim.

The three versions

Each version gives control of the dashboard to someone different.

I kept everything else the same across the three versions, with one fictional business (Hartwell Home Services), the same numbers, the same visual system and identical rating questions.

Version A dashboard: an AI insight banner above the fold, company vitals and key metrics
Version B in edit mode: cards with chart-type tabs, layout presets and an Add widget button
Version C chat answer to why leads are down, with an insight card and Save to Dashboard
See the nine key screens of the three builds, wireframed
AAI-forwardSwipe
  • A1Setup, step 2 of 4: a tap answers and moves on

  • A2Dashboard: AI insight above the fold

  • A3Ask AI panel: the dashboard narrows, answers can be added

  1. 1Setup is three questions plus a layout pick. One tap answers and moves on. There is no Back or Continue.
  2. 2The AI Insight banner follows the chosen priority. It is passive, not clickable, and never about leads.
  3. 3One global Ask AI button. Its side panel opens beside the dashboard, which narrows to make room.
  4. 4Adding a metric or changing a chart runs through the AI. Answer cards carry a chart pill and Add to dashboard.
BUser-controlledSwipe
  • B1Choose a starting point: four goal templates

  • B2Dashboard view: a wand on every card, no global AI

  • B3Edit mode with Add a widget open, plus the wand popover inset

  1. 1Start from one of four goal templates. One click builds the dashboard. There is no confirm step.
  2. 2Edit Dashboard is the only header control, with no global Ask AI and no global period. Each card has its own period pill.
  3. 3AI is an icon-only wand on each card. Its popover anchors to that card and answers an automatic first question.
  4. 4Edit mode adds chart-type tabs, layout presets and a trash in the wand's corner. Add a widget lists every metric. Those on the canvas are dimmed and tagged Added.
CChat-firstSwipe
  • C1Welcome: no setup, chat is the home screen

  • C2Chat answer: a paragraph plus a live insight card

  • C3Your metrics: saved answers become the dashboard

  1. 1No setup. Chat is the home screen, with five overview cards, four starter questions and one input.
  2. 2Answers come back as a paragraph plus a live insight card with period, segment and chart pills.
  3. 3Save to Dashboard is how the dashboard gets built. Saved cards collect in Your metrics.
  4. 4One period picker drives every card in Your metrics. Back to chat returns to the thread.
ABCShared · Detail modal

ModalOpened from a card or panel in A, a card body in B, an overview card in C

  1. 1One drill-down for all three shows value, change against the prior period, and a Worth a look note when a metric declines.
  2. 2Period tabs plus Pin to top, which adds a pinned row in A and B.
  3. 3Ask AI about this… is scoped to the open metric and period.
Wireframes mapped from the tested prototype's code (final build, September 2026), not design-phase artifacts.
Who can change what, version by version
ControlA · AI-forwardB · User-controlledC · Chat-first
ReorderDrag, always onDrag, in Edit modeDrag, in Your metrics
ResizeDrag an edgeDrag an edge, in Edit modeDrag an edge, in Your metrics
Chart typeOnly by asking the AIDirect chart-type tabs on each card, in Edit modeA chart pill, or a "Show as" chip
PeriodOne period bar for the pageA pill on each cardA pill on each card, plus one master picker
Add a metricFrom an AI answer cardDirect an Add widget drawerSave to Dashboard on an answer
Remove a metricOnly metrics added through chatTrash, with UndoRemove on a saved card
DensityChosen once, in setupCompact, Comfortable or ExpandedNone

From the tested prototype's code. Control moves from the AI in A, to the owner in B, to the conversation in C.

The build

I built all three versions as one React app on a live AI service.

In the first round of Ask Thryv, my AI chat project that ran alongside this one, one wrong scripted reply was enough to change how people felt about the feature. Every question an owner typed or tapped went to a live model, Claude, through a service I built and deployed for the study.

Architecture of the study build: the dashboard's shared data layer holds the numbers every widget shows. A seeded dataset with 2 years of synthetic history was calibrated to match it and feeds the live AI service. An audit checks every figure the AI's tools return against every dashboard figure, and the build was held to zero direction flips. The data layer and the AI service both feed three working prototypes, built on Thryv's design system components through its MCP server and served from one hosted build, which 53 owners used in study sessions on UserTesting.com, one version each.

Built end to end with Claude Code: three versions on the design system's MCP server, the AI service and the audit

Seeded dataset

2 years of synthetic history for one fictional business

Shared data layer

The numbers every dashboard widget shows

Audit: every number the AI's tools return vs every dashboard figure, 4 periods: up or down must match, after every change

0 direction flips

Live AI service

Claude · 21 lookup tools

Express · TypeScript

System prompt: Told to call a tool before stating any number

Scripted fallback if the model stalls

Hosting: Deployed to Thryv's cloud sandbox through the company's own pipeline · infrastructure managed by IT

Three working prototypes

One hosted build, switched by ?s=1, 2, 3

  • AVersion A: AI-forward

    Setup questions, then a dashboard built around the answers

  • BVersion B: User-controlled

    Pick a goal template, then drag, resize and switch charts

  • CVersion C: Chat-first

    Ask in chat, then save answers as cards

Thryv's design system components, via its MCP server

One chart renderer · 18 chart types

Study sessions

on UserTesting.com

53

owners, one version each

About 2 months, discovery to final report

  1. First prototype commit
  2. AI service begins
  3. Two unmoderated study waves
  4. Final report
From the prototype and AI service code. Dates from the project's commit history and the final report.
The hardest problem

I had to make the AI agree with the dashboard.

In the first pass, the AI confidently reported a metric rising while the widget beside it showed the same metric falling. A reporting tool can't have that bug. They read different data. With no real customer data, and Thryv's data team still building one trusted source, the dashboard showed sample numbers and the AI read a sample dataset I generated, with 2 years of history so it could answer for any period.

I wrote an audit script that compares every number the AI's lookup tools return with the matching dashboard figure, across 4 periods, and I ran it after every change. It had to find zero cases where one said up and the other said down. I checked its written answers by hand.

How I used AI
  • Research and the report. Deep research to start, and a first draft of the final report, which I edited through a few rounds.
  • Building. I built the three versions, the AI service and the audit script in Claude Code, with components from the design system's MCP server.
  • Checking. AI read the plan, script and code together and flagged two task premises the build could not support. I confirmed both and rewrote them.
What stayed mine

I set the hypothesis that the real variable is who decides what the dashboard shows. I decided what each version was built to test and wrote every task premise. AI let me build and check more than I could alone.

What deploying it taught me
  1. A 200 response did not mean it worked. The service falls back to a polite reply by design, so it returned that fallback for six straight tries while a key was misconfigured.
  2. The browser path needed its own test. The endpoint worked from the command line while every real browser call was blocked by a CORS setting nobody had set.
  3. A request with no timeout could stall a session. In the tested build, an AI request that never came back left an endless spinner in a dimmed detail view. After the first wave, I made every call give up after 9 seconds and fall back to a scripted answer.

Testing

53 owners each tested one version, in two unmoderated waves.

Owners used the dashboard on UserTesting.com and thought out loud.

Participants

Who took part

  1. 9service verticalsOwners screened from home services to dental offices and fitness studios
  2. 60recruited8-question screener, two waves
  3. 7excluded3 by a Version B screen freeze
  4. 53analyzedEach saw one version
  5. 43completed the usability scaleA18B13C12
Participant counts from the final study report.
The main task

Every owner got the same task, to find out why leads might be down.

Each version offered its own routes to the cause. Owners often missed A's drill-down and found B's AI wand late, so those two features were hard to judge.

See every path through the leads task

Task flow diagram. From the prompt that leads might be down vs last month, it maps every route each version of the tested prototype offered, ending at a cause named, or not.

Leads might be down vs last month

AAI-forward

DashboardAI Insight,Vitals,Key metrics,Performance

In the codeThe AI Insight banner is about the chosen priority. It never mentions leads.

Where do they look?
Key metrics
New Leads card477.8% down
click card
Detail modalChange vs last month, Worth a look, period tabs, Trend, By service typeAsk AI about this…

Drill-down: appreciated when found, not consistently discovered.

scroll down
Leads by SourceAI note: 51 to 47
click panel
Detail modalThe same modal as above
Ask AI
Ask AI panelSuggested chips, or type a question
tap chip or type
AnswerNew Leads card47
Follow-up chipsUnder the latest reply

BUser-controlled

Dashboard viewAt a glance,Key metrics,Performance

In the codeNo global Ask AI button. Each card has an icon-only wand.

Where do they look?
Key metrics
New Leads card477.8% downPeriod pill, and a wand in the corner
click card body
Detail modalTrend, By service typeAsk AI about this…
wand icon on the card
About New LeadsAuto asks “What's driving the change in New Leads?”
tap chip or type
Follow-up answerChips, or Ask a follow-up…

The wand: discovered late, and read as an edit control.

scroll down
Leads by SourcePanel with its own wand
wand icon
About Leads by Source“Which lead source is performing best?”
Edit Dashboard
Edit modeEditing bar, chart-type tabs, trash
chart-type tabs
Chart type switchedSame New Leads data, new view

CChat-first

WelcomeNo setupNew Leads477.8% downWhy are my leads down?
Where do they look?

Suggested questions reduced blank-slate anxiety.

tap chip or type
AnswerNew Leads card47

Naming the source was the highest-trust moment.

follow-up chip
Next answer card“How does New Leads compare to last month?” or “Show as a column chart”
Save to Dashboard
SavedToast, plus a badge on Dashboard
click overview card
Detail modalTrend, By service typeAsk AI about this…
  • Screen
  • AI surface
  • Step to a screen
  • Step to an AI surface
  • Decision
  • Note from the study report
  • Note from the code

A cause named, or not

Routes mapped from the tested prototype's code. Notes from the study report.
Between test waves

After the first wave, I fixed the AI so it could explain why leads dropped.

In the first wave, the AI could not say why leads fell, because its lookup tool returned this period only. Between waves I gave the tool last period's counts for every source and told the model to name the part that changed most as the likely cause. In the second wave, it named Google.

One question, traced through the live AI

claude-sonnet-521 tools

  1. The owner asksReact prototype

    Why are my leads down?

    Version C · suggested question

  2. Claude picks a toolExpress service + Claude

    tools 21

    • get_revenue
    • get_open_jobs
    • get_leads_by_source
    • get_avg_job_value
    • get_outstanding_invoices
    • +16 more+20 more

    tool_use

    Tool call: get_leads_by_source, period This Month.

    21 lookup tools. The model is told to call one before stating any number.

  3. The service answers from dataExpress service, seeded data

    tool_result excerpt

    Tool result excerpt: 51 leads this month, 69 the month before, down 26.1 percent. Google 26, down from 41 (51 percent of leads). Referral 17, up from 15. Repeat 6, down from 12. Social 2, up from 1.
  4. The answer, groundedReact prototype
    AI

    Figures cited in the answer: 51 new leadsdown 26% vs last monthGoogle: 41 to 26

    • Looked it up before stating it
    • Uses the time range on screen
    • From a widget? Assumes that metric
    • Out of scope? It says so and offers what it can
    • No reply in 9 s? A polite fallback
    • Likely driver, not proven cause
The real tool call and the This Month values it returned in the study's second wave, from the AI service's code and dataset. The New Leads card showed 47, down 7.8%. I cover that size drift in Looking back. Answer wording varied by session, so it is shown as placeholder lines.

Findings

Owners wanted a sensible dashboard to start from, then to move things around.

B led by only about 5 points, so I leaned on what owners said and did.

The finding that set the direction

Owners who moved widgets around started talking about the dashboard as their own, and arranging it themselves produced the strongest intent to return. Nobody wanted to start from a blank canvas, though.

I do not want to design a dashboard. I want to approve one. Give me something sensible and let me move the pieces.

Study participant, Version B
A · AI-forward

87.4"Excellent" band18 people

Worked
It felt like a business partner.
Broke
Some found the setup prompts ambiguous, and the drill-down was often missed.
Risk
People took its numbers on trust, so a confident wrong number would go unnoticed.
B · User-controlled

92.7"Best imaginable" band13 people

Worked
It made people proud of their dashboard.
Broke
Several people read its icon-only AI button as an edit control, and it was often found late.
Risk
Customization costs effort, so B leaned on its goal templates.
C · Chat-first

87.7"Excellent" band12 people

Worked
The study's biggest highs. Suggested questions meant nobody faced a blank page.
Broke
It did not always honor formatting requests, and several people found saving by accident.
Standout
"I knew Google was important but not that important. That number changes how I allocate my budget."

Usability (SUS) scores from 43 of the 53 owners, with 12 to 18 people per version. The line on each bar is the industry average of 68. Every session ran on desktop, so mobile was never tested.

  1. 01

    Version A's insight strip was the most trusted AI in the study.

    In Version A, the AI generated an insight for each owner's chosen priority and showed it on the dashboard, before they asked anything. Its placement and specificity earned owners' trust. Owners forgave a gap in accuracy more readily than vagueness, which made them doubt the data. Ignored formatting requests and repeated replies also hurt trust. As one Version C owner said: "It felt like it was not really listening."

  2. 02

    Owners preferred simple line and bar charts.

    Owners read line charts without hesitation, and bar charts worked for comparisons. Donut charts caused consistent hesitation. One Version C owner said: "Put the number in the middle, put the label outside. Do not make me guess." The highest-rated customization was simply changing the chart type.

My takeaway

Owners trusted AI that was specific and showed up where they were already looking, so in the hybrid each insight stays on its card, with a See breakdown link to the numbers behind it.

What came next

For the next round, I recommended a hybrid built on B.

B scored highest, but the strongest parts of A and C held up too. Larger operations leaned toward B and smaller teams toward A.

A rough concept wireframe of the study's recommendation for the next round: a hybrid built on B.

  • Foundation from B: layout control and chart-type switchingCards you drag, resize and switch chart type on, with a Lock layout toggle. Every card has a labeled Ask AI button and a See breakdown link, and a card with a big change offers "Why is this changing?"
  • The way in from A: setup questions and the insight stripGuided setup you can re-trigger, then the AI insight strip at the top with a persistent Ask AI text entry under it.
  • Depth from C: chat, when you want itAn optional side panel with suggested questions that follow the data, such as "Why are my leads down?", a year-over-year prompt, and a choice of text, table or chart answers.

Needs from the report

  1. 1Year over year, the most requested missing feature
  2. 2Let people choose text, table or chart answers
  3. 3A clearer save cue (the report found saving was often discovered by accident in C)
  4. 4An AI entry that is not mistaken for an edit control (open problem, not solved)
Concept sketch of the study's recommendation for the next round, drawn for this case study.

The redesign starts from what owners told us. They want a familiar dashboard they control, AI they can call up when they need it, and insights in place on the screen.

  1. Study53 owners, Jul to Sep 2026
  2. Final reportSeptember 2026
  3. HandoffReport and prototype, where it stands now
  4. RedesignProduct's next step
Product
Thryv's Head of Product asked me for this research. The prototype build lives in the Product team's GitHub repository, and the tests and final report are in Confluence, with links to every doc.
Chart library
I kept building the same charts for these prototypes, so I turned them into a shared chart library. Other designers pulled it into their own prototypes, and engineering used it for Ask Thryv instead of the outdated charts.See the chart library (opens in a new tab)

Looking back

What held up

Building three separate versions was worth the extra work.

Three distinct products cost far more to build than one product with a feature toggled, and they are the reason the comparison means anything. It also let the hybrid I recommended keep what worked in each, with setup from A, layout control from B and chat from C.

What I would change

Next time I'd check how much each number changed, not just which way.

My audit caught any case where the AI and the dashboard disagreed on up or down, but not on how much. For this month's leads, the card said down 7.8% and the AI said down 26%. I'd also have both read from one data source.

SMB Reporting · Thryv

Thanks for reading.