Thryv · Conversational AI case study

Ask Thryv

Ask how your business is doing, and get the answer and the why.

Thryv's small business owners had to click through dashboards and reports to see how the business was doing. I designed Ask Thryv so they can ask in plain words, built both prototypes in Claude Code and led both rounds of testing. Round two scored 82.4 for usability, and I handed it off to engineering for a beta.

Role
Lead Product DesignerUI, both prototypes, testing, handoff
Timeline
Apr to Aug 2026Round one in July, round two in August
Research method
Two rounds, 9 then 20 peopleBoth unmoderated, one screener
Usability score (SUS)
82.4 on a real LLMHanded off for a beta of about 150 users

The problem

Owners dug for answers in their dashboards.

They opened a dashboard, hunted for the right widget and worked out the why on their own. In round one, 7 of 9 people checked their data daily. Ask Thryv, the first AI feature a Thryv product team added, started as AI insights on each reporting page. Users responded well, but a new metric only reached owners once a whole report around it was built. So the business turned to an AI chat, where owners could ask for any single metric. Another team had started one, and my team ended up taking over with a focus on reporting.

So it gives you insights by asking questions instead of waiting for the report.

Round one participant

My role: Lead Product Designer, working with a PM. I owned the discovery research, the interaction design, both prototypes and round two's AI service, and the test plan, script and final report. Our lead researcher helped run round one from my test plan, and I ran round two on my own. The PM wrote the original epic. I created design tasks for work outside its scope, handed the prototype and its repo to engineering and stayed on for redlines.

Ask Thryv started first. My SMB Reporting study ran alongside it, and one of its three versions reused Ask Thryv's chat logic. It finished before Ask Thryv's beta.

  • Conversational AI design
  • LLM prototyping
  • Prompt design
  • LLM tool calling
  • Coded prototypes
  • Claude Code
  • Design system components
  • Unmoderated testing
  • Research planning

The 30-second read

From scripted answers to a real LLM.

Round one's keyword rules couldn't keep up with how people asked.

01 · What I did

Owners ask in plain words and get a chart.

Each answer says what the numbers mean and what to do about it, with chips for a follow-up. I designed it, built both prototypes in Claude Code and led both rounds of testing.See how it reads a question

02 · What changed

Round two scored higher than round one.

For round two I named the timeframe on comparisons and had a real LLM write the answers and chips. Then I handed it off to engineering for a beta with Thryv users.

Walkthrough · 39 seconds

Controls update the card in place, and each question gets a new reply.

Discovery and the first design

I designed two versions and tested the whole flow.

Research and Thryv's own widgets disagreed on where the controls go.

Round one

What the two versions tested

Both versions ran the same 5 leads tasks. I wanted to see whether people knew what to do with a chat box, and whether they tapped a control or typed.

Early research suggested people would want controls near the chat box, where most tapping happens, so A put them below the chart and B kept them above, as on Thryv's dashboard widgets. Ease was within 0.33 on every task, and most people preferred controls below.

Version A

Version A in round one: a Leads card shown as an area chart, with the period, segment and chart pills and a View report button in the card footer, under the chart

Controls below

Kept in round two

Version B

Version B in round one: the same Leads card as an area chart, with the period, segment and chart pills above the chart and no View report button

Controls above

Version A

Version A in round one: the leads breakdown shows three rows, then a Show more button

Three rows, then Show more

Version B

Version B in round one: the leads breakdown lists more rows and pages through them with numbered pages

Numbered pages

Both opened the breakdown on its data, under the card's chart and totals.

Version A

EmailTom BradleyJun 6Proposal97$5.3KShow moreLast 7 daysAll sourcesAll statusesView reportAsk Thryv 10:30 amWhat percentage of leads are converting?Are my leads booking appointments?Ask anything

Type a request

Version B

Leadsvs. previous periodLast 7 daysAll sourcesAll statusesNo chartTOTAL LEADS14212.6%CONVERSION RATE13%2.1%AVG LEAD SCORE785.4%SourceNameDateStatusScoreValueEmailOlivia GrantJun 12Proposal91$4.3KEmailCarlos RiveraJun 8Contacted68$7.4K

A chart menu, or type

Round two opened breakdowns with no chart until you picked one from the chart menu in the card footer.

Control by control: Version A, Version B and round two
  • Both versions · The startBuilt to test whether people know what to do with a chat box, or hesitate.
  • Version A · Controls belowBuilt to test whether people find controls under the data, and how they ask for a chart with no menu.
  • Version B · Controls aboveBuilt to test whether controls above are found sooner, and whether people reach for a menu or type.

Round one: Version A and Version B

ControlA · Controls belowB · Controls above
Card controlsIn the card footer, under the data, with menus that open upwardAbove the content, with menus that open downward
Long breakdownsThree rows, then Show moreNumbered pages
Chart on a cardLine, Column, Area, Bar, Donut, Hero or Table for leadsIdentical
Chart on a breakdownA chart menu only when the card already has a chart, otherwise type a requestMenu on every breakdown (No chart on a new one), or type
View reportA button on each card, which did nothingNo button on leads cards, just an item in the card's menu, which also did nothing
Everything elseIdentical welcome, suggested questions, sample data and scripted answersIdentical

From the final test script and the round one prototype's code. The Version A and B stills above are from its last build, made after the sessions, with the leads card switched to Area from its chart menu. Layout, controls and chip labels match the July sessions.

Round one to round two

ControlRound oneRound two
Answers and chipsKeyword rules and scripted answers, with chips set by hand for each metricA real LLM wrote the answers from my AI service's data, with chips that follow the conversation
ComparisonsOnly when asked, labeled vs. previous period, with the timeframe only in the date range pillOnly when asked, with the timeframe named next to the metric to match the date range, such as vs. previous 7 days
View reportA button (A) or a menu item (B) that did nothingRemoved, and the card's menu stayed
When it couldn't answerText only, with a general reason and chips about other metricsText only, saying why and what to try, with chips it can answer
Card controlsIn the footer (A) or above the chart (B)Kept In the footer, as in A
BreakdownsOpened on the data in both, and you typed to add a chart (B also had a chart menu)Refined Open on the data with no chart until you pick one from the chart menu in the footer

Round one from its code and the July sessions. Everything else, from the welcome screen to the layout, stayed the same.

Round one · July

In July, 9 people tried both scripted versions.

I built round one in Claude Code as one React app with both versions. Five keyword rules and scripted answers meant the same words always got the same reply. Our lead researcher helped run it, unmoderated, with small business owners and decision makers, and each did the 5 leads tasks in both versions.

What we learned

People started with the suggestions.

All 9 started with a suggested question, even after an intro that told them to type like a search engine.

This is so good because I don't even have to type in.

Round one participant
  • The period3 of 9 said, unprompted, that a comparison didn't say which period it was compared with.
  • View report4 of 9 clicked View report while looking for the rest of the data. It was a placeholder and did nothing.
  • When it couldn't answerA typed question about lead sources got a general reply, even though the data was there.
The hardest problem

Using a dropdown on the card and typing the same request didn't always match.

In Version A, about half changed the period by typing and half with the card's control, so both had to give the same answer. They didn't always. In B, one person said typing gave "a more detailed answer" than switching the period. In A, another lost the summary after switching to 90 days.

In the design tasks I created afterwards, I set one rule. A control updates the card in place, a question adds a new card, and changing the chart type never rewrites the text.

How the two rounds were set up

Round one · July

9 people

each used both versions

5 started with A 4 started with B

Dental, retail, construction and trades, field service and more

Round two · August

20 people

each used one prototype on a real LLM

Held the same

Same screener · same 5 tasks · unmoderated · desktop

Screener

Small business owners and decision makers, from the final screener

Same in both rounds
Setting

Unmoderated, on the same testing platform, on desktop

Same in both rounds
5 leads tasks
The same 5 leads tasks
  1. 1Check this week's leads
  2. 2Compare to the previous period
  3. 3Pull up the last 90 days
  4. 4Break leads down by source
  5. 5Explore the rest of the data
Same in both rounds
Measures

A 30-second first look, and a 1 to 7 ease rating after each task

Same in both rounds

Changed for round two

People

Round one: 9, each using both versions

Round two: 20, each using one prototype

Answers

Round one: Rules and scripted replies

Round two: A real LLM and my AI service

Usability scale

Round one: After each version, with 8 completing it for A and 7 for B

Round two: Once, completed by all 20

New questions

Round one: None added Three comparison questions closed the session.

Round two: Was the comparison clear? asked after the compare taskAn error task then, did the reply help?

Round one's order and counts from the testing platform's export and the usability questionnaires.

Round one's crossover, plan and script

Diagram: in round one, every participant used both versions. 9 participants split into two groups: 5 used Version A first and 4 used Version B first. Each version session was a 30 second first look with no clicking, 5 tasks with a 1 to 7 ease rating after each, then the usability scale, before switching to the other version for the same steps. Three comparison questions closed the session. 8 people completed the usability scale for Version A and 7 for Version B. The five tasks: check this week's leads, compare to the previous period, pull up the last 90 days, break leads down by source, and explore the rest of the data. 7 of 9 check their data daily. 3 are main decision makers, 5 share decisions with an owner and 1 leads one area.

Round one study design, from the final test script. Order and counts from the testing platform's export and the usability questionnaires.

The four research questions

  1. Do people know how to explore their data through chat, or do they hesitate?
  2. When they want to change the period, segment or chart, do they reach for the controls or type?
  3. Do they expect a breakdown to show a chart by default, or is the table enough?
  4. To add a chart, do they go to a menu or type the request, and which do they prefer?

How round one's script changed before it ran

ChangedFirst draftFinal script
Tasks7, with revenue tasks in Version B5 leads tasks, identical in both versions
Date rangeThe last 30 daysThe last 90 days
Chart taskTurn the breakdown into a donut chartRemoved, and asked as a comparison question after both versions
RatingsFollow-up questions after each taskA 1 to 7 ease rating after each task, and the usability scale after each version, before seeing the other
Round one, task by task (Versions A and B)

Chart, round one, task by task: mean ease out of 7 and mean time on task, Version A then Version B. Task 1, check this week's leads: ease 6.89 and 7.00, 99.6 and 55.0 seconds. Task 2, compare to the previous period: 6.56 and 6.44, 63.4 and 52.3 seconds. Task 3, pull up the last 90 days: 6.67 and 6.44, 68.9 and 62.2 seconds. Task 4, break leads down by source: 5.33 and 5.22, 110.0 and 92.4 seconds, the slowest task in both versions. Task 5, explore the rest of the data: 4.44 and 4.11, 78.6 and 65.8 seconds, the lowest rated in both versions. The two versions were within 0.33 on every task.

Round one means from the testing platform's export. Times include thinking aloud and first-use learning, and 5 of 9 saw Version A first. Usability (SUS) from 8 and 7 completed questionnaires. The medians were 76.25 and 82.5, and one score of 27.5 pulls B's average down, so neither version won. Ease ratings (1 to 7) came from all 9 after every task. With 9 people in round one, all on desktop, the gaps between A and B are only suggestive.

People clicked View report, and it did nothing.

It was a placeholder, but 4 of 9 in round one clicked it while looking for the rest of the data, and 3 of them rated the explore task 2 or lower in both versions. Round two removed it and left the report view for a later phase.

People couldn't tell which period a comparison used.

Round one's compare chart showed only this week's dates. 3 of 9 raised it unprompted, and one asked, "is that the previous month? ... or is that the previous year? Oh, is it year over year?" Round two named the timeframe next to the metric.

Close-ups use sample data.

What changed for round two

I made four changes, each tied to a finding.

The biggest change was a real LLM on the AI service I built. The LLM wrote the chips and the reply when it couldn't answer. The rest were small changes to the interface.

One Ask Thryv window, both rounds, with sample data: the Leads comparison card, then the typed question show me my churn rate. 1, the chips. Round one: all 9 opened with a suggested question, and its chips were set by hand, here Revenue across all segments and Bookings year to date. Round two: praised again, the chips the LLM wrote, here How are recurring appointments trending? and What can I ask you about? 2, the period. Round one: the tag read vs. previous period, and 3 of 9 raised, unprompted, that it didn't say which period. One asked "is that the previous month?". Round two: vs. previous 7 days, next to the metric, and it was clear to most of the 20 I asked. 3, View report. Round one: 4 of 9 clicked View report, and it did nothing. It showed a real need for a report view, left for a later phase. Round two: removed. 4, the reply. Round one, a general reply when stuck: I wasn't able to find "churn rate" in the available metrics. This could mean it hasn't been connected yet, or it may live in a different data source... Round two: I don't have churn in your data yet, so I can't chart it. If you mean clients who stopped booking, I can show your recurring appointments. Most said the error task's reply helped. It gave a reason, and its chips showed what to try next or another way to ask.

Kept and refined from round one. Card controls stayed in the footer, as in Version A. Breakdowns still open on the data, now with no chart until you pick one from the chart menu in that footer.

Round two · August

In August, 20 people tried a prototype on a real LLM.

Round one's session videos showed why it had to be real. One wrong scripted reply could shift how an excited participant felt about the feature. I went beyond the project's requirements, built round two in Claude Code on a real LLM and ran it on my own. The screener and the 5 tasks stayed the same, and I added questions about the comparison and an error.

One question, two rounds. The same typed question in both rounds: "which sources are driving this number". Sample data.

Round one, rules: five keyword rules read it (Out of scope?, Period, segment, chart, Metric, Typos, Last turn). The first rule caught it as out of scope, though the data was there, behind a chip labeled Break that down by period. It gave a general reply: I wasn't able to find "Which sources" in the available metrics. This could mean it hasn't been connected yet, or it may live in a different data source.

Round two, a real LLM: Sonnet 5 read the question as written, following rules and guidelines I wrote, including Thryv's voice and tone. It looked up the answer through tool calls to an AI service I built, on a test environment, with sample data that had no gaps. It answered with a new card, Leads by source for the last 7 days, compared with the previous 7 days: Paid search first with 38, up 5, then Organic 32, Social 28, Referral 23 and Email 21.

How it reads a question, step by step

Before it answers, Ask Thryv works out which number, which dates and whether to compare. In round one, five keyword rules did that, and anything outside them got a general reply with chips. In round two, a real LLM read the question.

With sample data, the typed question which sources are driving this number, in both rounds. Round one, rules and scripted answers: a text-only reply that it could not find Which sources in the available metrics, with a general reason and chips about revenue and bookings. Round two, a real LLM: a new card of leads by source for the last 7 days, still compared with the previous 7 days, paid search first with 38, up 5, then organic 32, social 28, referral 23 and email 21, with chips that follow the thread. In the card spec, round two added split: By source, from the word sources.

Each window shows a reply while it was the newest, so its chips show. In both rounds, chips clear when the next reply arrives.

With sample data, two more questions in both rounds, each shown while its reply was the newest. The tapped chip Compare to prior period. Round one repeated the answer and tip before it, Your leads grew 16.2% in last 7 days, your leads are better quality, up 12% year-over-year, over a new Leads card labeled vs. previous period, with footer controls Last 7 days, All segments, Line and View report, and chips set by hand for leads. Round two said that against the previous 7 days, leads are up 13%, 142 against 126, with a tip, over a new Leads card labeled vs. previous 7 days next to the metric, footer controls without View report, and chips written for this answer. The typed question show me my churn rate. Round one gave the same text-only reply and chips as for sources. Round two replied in text only that churn is not in its data yet, pointed to recurring appointments instead, and offered two chips it can answer.

Diagram of round two: a real LLM read all three questions as written. It named the compared period, added a card by source, and said churn was not in the data yet. In round one, the rules caught both typed questions as out of scope.

Round oneFive keyword rules read each question, with no model

  1. Out of scope?Metrics it didn't have, over a year, regions, off topic
  2. Period, segment, chartWords like "this week" or "bar chart"
  3. MetricLeads, revenue, reviews and more
  4. TyposClose spellings like "leeds"
  5. Last turnWhat the last card used

Round two reached the breakdown from the word "sources". Typing adds a new card. A card's own footer controls update it in place.

Round oneThe leads breakdown by source was already there, behind a chip labeledBreak that down by period

Round one follows the prototype's code and the July sessions. Question 3 is from the prototype's own test screens. Both rounds here show sample data.

How I used AI

  • Research, Figma and the report. Deep research to start, then some of round two's Figma files and the report's first draft.
  • Building both prototypes. Round one as one app that switches version by URL, round two on the AI service I built.
  • Moving round one's app onto the design system's components, most of it in one afternoon, with a snapshot before each swap, then a token pass.

What I wrote and decided myself

  • Rules over a model in round one, so behavior was testable.
  • The system prompt and guidelines the LLM followed in round two, including Thryv's voice and tone.
  • What each version tested, and the design tasks that followed.

What I told the model in round two

  • Look up a number before stating it. Never estimate or make one up.
  • Use This Month when no period is named.
  • If it can't answer, say so plainly, then suggest related questions it can answer, so no one hits a dead end.
  • Suggest follow-ups they'd likely ask next, following the thread so far, and never repeat one already asked.

How I checked its answers

  • I tested it by hand, checking its answers against the data.
  • I ran a check script on the numbers the AI service returned, after every change.
How I built the two prototypes
Both built in Claude Code, on the design system's components
Round one · JulyScripted, so every session behaved the same Round two · AugustA real LLM reads each question
Screens One React app holding both versionsWorking One prototype, updated after round oneWorking
Reading the question A rule-based parser with five rulesRules Changed: A real LLM, Sonnet 5Real LLM
Answers and chips Scripted, chips set by handScripted Changed: Written by the LLM, within my guidelinesReal LLM
Data and backend Reporting data from a Thryv test environment. I used Claude to find its gaps and filled them with realistic sample data, so no one got an empty chart. Changed: An AI service I built, with sample data that had no gaps. My SMB Reporting study ran on it too.

What carries over between turns

A new question about something else resets the period and segment, unless the question names a date range or filter, or the tapped suggestion does (for example, one that says last week).

In round two, the chips the LLM wrote built on the latest question and related earlier ones.

The final design

The final design, as tested in round two.

It's the version I handed to engineering, built on the design system.

Results

People rated round two higher on usability.

Round one sat in the Good band, and round two reached Excellent.

Chart: usability (SUS) on a 0 to 100 scale. Round two, one prototype on a real LLM: 82.4 from 20 people, in the Excellent band, 80.3 and up. Round one, scripted: Version A 78.4 from 8 of its 9 people, and Version B 76.4 from 7 of 9, both in the Good band, 68 to 80.2. A line marks the 68 average.

In round one, 9 people each rated both scripted versions. In round two, 20 people rated one prototype on a real LLM. The two rounds were set up differently, so the higher score is a good sign but doesn't prove the changes caused it.

It's so cool to ask a simple question and get a detailed visual answer that shows how my business is doing, tells me why, and suggests what to do about anything that isn't performing well. And I love having quick controls to tweak the chart right there in the chat.

Round two participant

New in round two

Saying what it couldn't answer built trust.

In round two I asked people to trigger an error on purpose.

Most said the reply helped.

What people valued

  • Said plainly what it didn't have
  • Explained why
  • Offered alternatives to try
About these scores

The bands come from the round one report. Good runs from 68 to 80.2, and Excellent from 80.3.

In round one, 8 of the 9 completed the usability scale for Version A and 7 for Version B. In round two, all 20 completed it.

Round one's order and counts come from the testing platform's export and the usability questionnaires.

What came next

I handed it off to engineers for a beta.

Engineering chose my chart library over the outdated charts.

Timeline: round one in July, with 9 people and two scripted versions. I built the changes into one prototype in Claude Code. Round two in August, 2 to 3 weeks after round one, with 20 people and one prototype on a real LLM. The prototype and its repo handed off to engineering. Then engineering's beta with Thryv users, starting in November 2026.

What engineering got

  • The prototype and its repoRound two's working prototype, built in Claude Code on the design system's components.
  • User stories in 7 feature areasIncluding the rule that a control updates the card in place and a question adds a new card.
  • RedlinesI stayed on with the engineers for redlines, new items and feedback.

About150Thryv users in the beta

Engineering
Engineering used my repo as the working reference and pulled code snippets from it. It gave them a fully working example and documentation of how a chart should behave, including responsiveness, axes, load animations, hover tooltips, the logic for which chart type fits which data, and color and hierarchy with multiple segments.
Chart library
I built a shared chart library while designing Ask Thryv and SMB Reporting. Engineering had planned to reuse the outdated charts for Ask Thryv and used my library instead.See the chart library (opens in a new tab)

Looking back

What held up

Testing the interaction first paid off.

Round one's rules behaved the same way every time, so its findings were about the interface, such as the comparison labels, the chips, a placeholder button and where the controls go. I acted on each one in round two.

What I would change

I would test with a real LLM sooner.

Scripted answers kept round one controlled, but they could not follow what people asked or how they asked it, and that cost trust. Reporting was the first part of the chat to roll out, and in both rounds a few people asked for things outside it, so I'd make that scope clear from the first screen. I would also test labels like "source" before round one, since its report found the word unclear.

Ask Thryv · Thryv

Thanks for reading.