A bit of background
Wolfram|Alpha, a computational knowledge engine website, is one of Wolfram Research's flagship products. Here, users submit queries to Wolfram|Alpha, which then computes answers and relevant visualizations from externally sourced curated data. Wolfram|Alpha paid service, Wolfram|Alpha Pro, provides additional convenient features such as Step-by-step solution and has more than 100,000 active subscriptions at the time.
Why we were interested in collecting these data?
We knew from the existing data that Wolfram|Alpha has two groups of users. The first group is more-or-less a Wolfram fan group: they are enthusiastic about Wolfram technology and are heavy users of many other Wolfram products. The second group consists of students and friends of the users from the first group: they'd hear Wolfram|Alpha from the classroom or private conversation, then try it out by themselves. As a result, we were curious about the communication between the first group ("fan group") and the second group ("common group"). How would fan group users introduce Wolfram|Alpha to the common group? How does it impact the first impression the common group users have about Wolfram|Alpha? Will they keep using it? We were expecting that the data we collected would help us understand how people perceive Wolfram|Alpha, and in turn, find opportunities to attract potential users from the common group.
How we recruited users
Wolfram Research holds an annual conference to introduce its latest technology. The team decided to recruit users and interview them on-site. We used the following criteria to select our ideal participants:
Using the conference registration info, we emailed out invitations to conference attendees. We ended up recruiting seven participants: three of them were either Math or Statistic professors; one was a private math tutor; one was a high school student, and two people worked in the financial industry. All of them were male.
The interview process, and how we analyzed the data
Each interview session was 45 mins. A colleague of mine and I took turns to moderate and take notes. After each interview, we'd have a 15-min discussion to exchange our thoughts about the participant and key points we had learned.
All participants fell into the fan user group. It may not be surprising since we were recruiting at a conference that only people who were enthusiastic about Wolfram technology would attend. How can we get the common user data from them? I asked each participant whether they had introduced Wolfram|Alpha to anyone else and if yes, how. I then used this info to picture how common users receive information about Wolfram|Alpha. Additionally, I've borrowed some data from previous user studies to help complete the common group user persona and the journey map.
What were some opportunities we found
Most of the cases, people knew Wolfram|Alpha from their math teachers, so they naturally assume that math is all Wolfram|Alpha can do. We observed that users often left the site when they ran out of math questions. If we could teach new users what else Wolfram|Alpha can do, they may stay at Wolfram|Alpha longer.
We recruited only one student participant. While he did not fit our common user profile, he shed some light on how he and his friends used Wolfram|Alpha. Most of his friends and classmates used Wolfram|Alpha without Pro because they felt Pro is expensive. Some of them would buy Pro for one semester if they needed to use the step-by-step solution at a course and would cancel the plan after the semester ended.
All participants we've interviewed were Mathematica users. They see Wolfram|Alpha as a less powerful version of Mathematica, and would only use it when they don't have access to Mathematica. As a result, they don't use Wolfram|Alpha as frequently as we'd like.
How we improved Wolfram|Alpha based on the findings
Since the study, we've made several improvements to make Wolfram|Alpha onboarding experience superior. We've changed the homepage design to show a list of example queries that Wolfram|Alpha is capable of answering. We've added a student pro plan to offer more discounts to this user group. We've also redesigned the pro sign up popup for step-by-step solution to make it concise.
How this study could have been approved
When the participants do not match the ideal criteria, how to make the most out of what you get? The moment that it became clear we wouldn't be able to interview common users, I reconstructed my research plan. Instead of asking the common users how they know Wolfram|Alpha, I asked fan users how they introduce Wolfram|Alpha to others. When abandoning and rerunning the study is not an option, we need to be creative. This project provided me a unique experience of adjusting the research plan to compromise for the imperfect participant sampling.