Year
Feb '25 – Apr '25
Tools
.webp){kind=small}
Deliverables
Comparative Evaluation, Heuristic Evaluation,
Usability Testing,
Design Recommendations
Poe by Quora –
Mixed-Method UX Evaluation
Poe is Quora's AI chatbot aggregator that brings models like ChatGPT, Claude, and DeepSeek into one platform. As part of The Vanguard Collective, our team conducted a thorough UX evaluation using mixed methods : a comparative evaluation against 5 competitors, a heuristic audit based on Nielsen's principles, and usability testing with graduate students.
Our study revealed key usability issues in onboarding, bot comparison, discoverability, and error messaging. I led the synthesis of qualitative and quantitative findings into actionable design improvements that aimed to reduce user drop-off, enhance discoverability, and streamline bot interactions. This project showcases my strength in structured UX research, critical insight synthesis, and evidence-based design thinking, especially in AI-integrated consumer apps.
Poe is a unique platform combining the power of multiple AI models under one roof, but it poses complex UX challenges for first-time users on the mobile app.
🧠 Feature | 🧭 What It Does | 👥 Why It Matters to Users |
|---|---|---|
Explore Trending Prompts/ Chatbots | Browse trending conversations & sample inputs | Reduces writer’s block, inspires new tasks |
Create Custom Bots | Users can build and publish personalized bots | Encourages creativity and repeated use |
Compare Bot Responses | Lets users send the same prompt to multiple bots | Helps find the most useful AI output quickly |
The Problem: Powerful Features, But Poor First-Time Experience
While Poe offers powerful AI tools and customization features, first-time users struggle to understand what it does, how to choose a bot, and how to start. As a result, critical features like bot comparison and creation often go undiscovered.
“I didn’t even realize I could create my own bot until the end.”
“There are so many bots. I don’t know where to begin.”
How might we improve Poe’s feature discoverability and first-time experience to help users get value faster?
.png)
No onboarding
Too many bots,
unclear categories
No tooltip or CTA for bot creation
Phase 1 : Comparative Evaluation
Understanding how Poe performs against similar and analogous tools in terms of first-time user experience, navigation, and feature accessibility. We focused on six key features to see how Poe compares to competitors and where Poe has room to improve.
-
AI Models: Organization & PricingUI:
-
Readability, Layout & NavigationUX
-
Learnability/Intuitiveness & Onboarding
-
Customizability
-
Response Accuracy
-
Cross-Platform Availability
Logo | Competitor | Type | Strength | Weakness |
|---|---|---|---|---|
Fiverr | Analogous | Clear outcomes, task-based interface | Human-driven, expensive, not instant | |
ChatGPT | Parallel | Strong model output, continuous threads | Single-bot interface, no comparison UX | |
There's an AI for That | Partial | Vast directory, easy search filters | Static links, no real interaction UX | |
You.com | Partial | Search integration, personal assistant UX | Lacks bot creation/customization | |
Thinkbuddy | Direct | Clean onboarding, fewer bots | Limited personalization |
Finding 1 : No Onboarding Support for First-Time Users
Poe does not provide any onboarding experience or tooltips, unlike competitors like ThinkBuddy or You.com. New users are left to figure things out on their own, which makes the app harder to navigate from the start.
.png)
Poe’s onboarding
vs
.png)
you.com’s onboarding explanation card deck
ThinkBuddy’s onboarding has a video explaining how the tool works
Finding 2 : Key Feature Discoverability is Poor
Unlike tools like “There’s an AI for That” that use clear task categories, Poe relies on abstract terms like “Reasoning” and “Official.” This makes it hard for users to know which bot to choose or where to start.
Finding 3 : Homepage Layout is overwhelming
Poe’s homepage displays a dense mix of bots, feeds, and subscription prompts. In contrast, competitors keep their home screen focused and clean. This clutter in Poe leads to cognitive overload and hesitation to engage.
.png)
Poe’s homepage and categories
vs
.png)
You.com’s homepage and categories
ThinkBuddy’s category dropdown
What This Means for Poe: Powerful, But Unapproachable
Poe offers a rich set of features but lacks the onboarding, structure, and task guidance needed to help new users succeed.
✅ No Guided Entry Points
Unlike competitors, Poe doesn’t help users take a first step. There’s no walkthrough or quick-start.
✅ Overloaded Homepage
Mixing trending content, all bots, and subscriptions creates confusion instead of clarity.
✅ Missed Opportunities for Personalization
Competitors tailor content by task (e.g., “Write an email”); Poe assumes users know which bot to use and why.
Phase 2 : Heuristic Evaluation : 7 Core Task Areas, 10 Heuristics
The UX Researchers on the team find Poe’s usability issues by judging its compliance with recognized usability principles. ( Nielsen, 1994)
-
The task categories to be evaluated were identified.
-
These categories were broken down into smaller tasks or steps.
-
Each step was evaluated for violations to usability heuristics and then assigned a numerical rating for it’s usability.
Here, one of the Category is Home Screen Navigation, the steps are as follows -
-
Land on the homepage after opening the app
-
Scan through bot cards, trending prompts, and UI sections
-
Find important links like “Create a Bot,” “Help,” or “Explore”
-
Understand what each section represents
-
Choose a logical next step from the homepage
(higher score = worse usability)
Our heuristic audit focused on evaluating usability across these high-impact areas:
-
Chatbot Interaction – 9 steps
-
Chatbot Browsing & Selection – 6 steps
-
Home Screen Navigation – 6 steps
-
Feed Exploration – 2 steps
-
Settings & Customization – 2 steps
-
Bot Creation Process – 1 step
-
Help Resources & Onboarding – 3 steps
Each task was rated using Nielsen’s 10 Heuristics with severity from 0 (no issue) to 4 (catastrophic). 60% of all evaluated tasks had minor or major usability issues. The most friction-heavy areas were home navigation, bot creation, and chatbot discovery.
Top 3 Heuristic Violations
Finding 1: Users can delete entire chats, bots, or even their accounts without any warning or option to undo.
Heuristics Violated: Error Prevention, User Control & Freedom, Help Users Recognize, Diagnose, and Recover from Errors
Severity: 3 – Major usability issue
Why it Matters: Users fear making mistakes that cannot be undone. This breaks trust and increases anxiety while using the product.
Recommendation: Add confirmation dialogs and undo options for all destructive actions.
Finding 2: Users struggle to browse and differentiate between bots due to vague labels (e.g., “Reasoning,” “Official”) and an overwhelming, unfiltered list.
Heuristics Violated: Match Between System and Real World, Consistency and Standards, Aesthetic and Minimalist Design
Severity: 3 – Major usability issue
Why it Matters: If users can’t identify which bot to choose, they’ll abandon the experience before getting value from the app.
Recommendation: Use task-based labels (e.g., “Write Emails,” “Explain Concepts”) and group bots by user goals.
Finding 3: The process of creating a new bot lacks validation, guidance, and feedback. Users often abandon it midway or make errors without knowing why.
Heuristics Violated: Error Prevention, Help and Documentation, Visibility of System Status
Severity: 3 – Major usability issue
Why it Matters: This feature is a differentiator for Poe. If users can’t complete it confidently, the platform loses one of its key engagement drivers.
Recommendation: Add step-by-step creation flow, field validation, and bot preview before publish.
Phase 3 : Usability Testing - Real Tasks, Real Friction
To observe how new users interact with Poe’s main features, uncover friction points, and validate earlier findings from heuristic and comparative evaluation.
Total users : 6 graduate students
Context : In-person moderated, with Think-Aloud protocol
Tools : Poe Mobile app, Zoom, screen recording, Observation sheet
Observation Focus : Task completion, hesitation, confusion, navigation strategies
The Process
-
Observed pre-selected tasks involving onboarding, chatbot selection, and feature usage.
-
Collected performance metrics (completion time, success rates) and observed user behaviors.
-
Analyzed findings through group synthesis.
Task Overview
Task | Completion Rate | Main Friction Point |
|---|---|---|
7. Create a personalized bot | ❌ 2/6 | No field validation, unclear expectations |
6. Evaluate subscription tiers | ❌ 2/6 | Confusing model access vs. pricing info |
5. Return to previous chat | ⚠️ 3/6 | Generic chat names; no “continue” indicator |
4. Find a bot for writing task | ⚠️ 4/6 | No filter/sorting, vague bot titles |
3. Compare bots | ❌ 1/6 | @botname feature hidden, no visual cue |
2. Ask a question | ✅ 6/6 | Input field worked, but unclear model limitations |
1. Select a chatbot | ✅ 6/6 | Overwhelming layout, unclear categories |
Out of 7 tasks, only 2 were completed smoothly by all participants. The most problematic areas included bot comparison, bot creation, and subscription understanding, all of which affected discoverability and confidence.
3 Critical Failures That Stopped Users in Their Tracks
🔍 1. Compare Bots (1/6 success)
Observed Behavior: Most users typed the same prompt into different bots manually. Only one knew about @botname.
“I didn’t realize I could compare. There’s no button or guide.”
Issue: Feature discoverability is
nearly zero.
🔍 2. Create a Bot (2/6 success)
Observed Behavior: Users didn’t know what to input. Some bots failed silently.
“What do I write here? Is there a format?”
Issue: Lack of instructions, examples, or form validation.
🔍 3. Evaluate Subscription Tiers (2/6 success)
Observed Behavior: Users couldn’t understand which tier unlocked which bot.
“Why is this bot greyed out? What do I need to upgrade to?”
Issue: Poor mapping of access levels to pricing.
Design Recommendations : Effort vs Impact to Improve First-Time UX and Feature Discoverability
🚨 High Impact & High Effort
-
Redesign homepage layout to group bots by task type, reduce cognitive load
-
Preview of paid bots
-
Feature tour/onboarding modal to guide first-time users
-
Create filter/sort functionality for bots by use case or tone
✅ High Impact & Low Effort
-
Add Help Center in bottom nav for easier access
-
Surface Compare earlier (before first reply) or a floating
Compare button -
Move language settings out of Help
-
Add subscription feature-access legend below pricing tiers
-
Use tooltips for premium bots
and points
🧱 Low Impact & High Effort
-
Redesign subscription plan page with icons, features list, and simplified pricing
-
Create bot templates gallery to inspire use cases
-
Breadcrumb-style navigation
🎯 Low Hanging Fruit
-
Show real-time name errors in
bot creation -
Relabel vague bot types with
plain language -
Gray out or locked premium bots
-
Exit confirmation when
creating bot
Project Impact & Strategic Value
Metric/Insight | Potential Impact |
|---|---|
📚 Help Center discoverability | Faster self-resolution, reduced frustration |
🔐 Better error prevention | Increased trust & task confidence |
🔍 Surfaced comparison feature | +60% likelihood of users trying more bots |
🤖 Higher bot creation success rate | +50% more completed custom bots |
🧭 Improved feature discoverability | +30–40% increase in first-week engagement |
Poe’s core strength, AI model diversity was being underleveraged due to discoverability issues. Our research revealed that small UX changes could unlock major engagement drivers without altering the product's core.