A bit of background

Wolfram|Alpha, a computational knowledge engine website, is one of Wolfram Research's flagship products. Here, users submit queries to Wolfram|Alpha, which then computes answers and relevant visualizations from externally sourced curated data. Wolfram|Alpha paid service, Wolfram|Alpha Pro, provides additional convenient features such as Step-by-step solution and has more than 100,000 active subscriptions at the time.

Why we were interested in collecting these data?

We knew from the existing data that Wolfram|Alpha has two groups of users. The first group is more-or-less a Wolfram fan group: they are enthusiastic about Wolfram technology and are heavy users of many other Wolfram products. The second group consists of students and friends of the users from the first group: they'd hear Wolfram|Alpha from the classroom or private conversation, then try it out by themselves. As a result, we were curious about the communication between the first group ("fan group") and the second group ("common group"). How would fan group users introduce Wolfram|Alpha to the common group? How does it impact the first impression the common group users have about Wolfram|Alpha? Will they keep using it? We were expecting that the data we collected would help us understand how people perceive Wolfram|Alpha, and in turn, find opportunities to attract potential users from the common group.

My responsibilities

• created user interview plan
• recruited participants
• led user interviews and follow-up discussions with my colleagues
• identified two user personas and drew one user journey map

How we recruited users

Wolfram Research holds an annual conference to introduce its latest technology. The team decided to recruit users and interview them on-site. We used the following criteria to select our ideal participants:

• 50% of the participants would be professors/high school teachers
• 50% would be high school/college students
• one or two participants with a non-educational background

Using the conference registration info, we emailed out invitations to conference attendees. We ended up recruiting seven participants: three of them were either Math or Statistic professors; one was a private math tutor; one was a high school student, and two people worked in the financial industry. All of them were male.

The interview process, and how we analyzed the data

Each interview session was 45 mins. A colleague of mine and I took turns to moderate and take notes. After each interview, we'd have a 15-min discussion to exchange our thoughts about the participant and key points we had learned.

• First, we asked the participant to briefly introduce himself and describe how he would use Wolfram|Alpha in his daily life. If the participant had introduced Wolfram|Alpha to other people, we then asked him, "how did you introduce Wolfram|Alpha to others?"
• While we were most interested in the participant's first interaction with Wolfram|Alpha, not everyone can have a clear memory of what happened a long time ago. Therefore, we asked the participant to recall his most recent interaction with Wolfram|Alpha, then use this data to cross-reference his first interaction with Wolfram|Alpha
• We then asked the participant to play with Wolfram|Alpha for 10 mins, asking any questions that they think they could ask Wolfram|Alpha
• We then asked the participant to recall how he first knew Wolfram|Alpha and his first interaction with Wolfram|Alpha
• Finally, we asked the participant if he had any suggestion to make Wolfram|Alpha better

All participants fell into the fan user group. It may not be surprising since we were recruiting at a conference that only people who were enthusiastic about Wolfram technology would attend. How can we get the common user data from them? I asked each participant whether they had introduced Wolfram|Alpha to anyone else and if yes, how. I then used this info to picture how common users receive information about Wolfram|Alpha. Additionally, I've borrowed some data from previous user studies to help complete the common group user persona and the journey map.

What were some opportunities we found

• People often come to Wolfram|Alpha without a full understanding of what Wolfram|Alpha is and what it can do

Most of the cases, people knew Wolfram|Alpha from their math teachers, so they naturally assume that math is all Wolfram|Alpha can do. We observed that users often left the site when they ran out of math questions. If we could teach new users what else Wolfram|Alpha can do, they may stay at Wolfram|Alpha longer.

• Students are price sensitive

We recruited only one student participant. While he did not fit our common user profile, he shed some light on how he and his friends used Wolfram|Alpha. Most of his friends and classmates used Wolfram|Alpha without Pro because they felt Pro is expensive. Some of them would buy Pro for one semester if they needed to use the step-by-step solution at a course and would cancel the plan after the semester ended.

• Fan users often have access to more powerful Wolfram Technology tools

All participants we've interviewed were Mathematica users. They see Wolfram|Alpha as a less powerful version of Mathematica, and would only use it when they don't have access to Mathematica. As a result, they don't use Wolfram|Alpha as frequently as we'd like.

How we improved Wolfram|Alpha based on the findings

Since the study, we've made several improvements to make Wolfram|Alpha onboarding experience superior. We've changed the homepage design to show a list of example queries that Wolfram|Alpha is capable of answering. We've added a student pro plan to offer more discounts to this user group. We've also redesigned the pro sign up popup for step-by-step solution to make it concise.

How this study could have been approved

• We could have found our targeted users from a different setting: the conference is not the ideal location if we want to talk to novice users. Instead, we can go to high schools, colleges, and universities to find student users.
• A follow-up study to understand Wolfram|Alpha onboarding experience of novice users would be beneficial: we've known how fan group users introduce Wolfram|Alpha to their family and friends. By adding a successor study for first time Wolfram|Alpha users, we could have the other half of the picture.
• There might be some user groups we've missed: for example, we've known that high school teachers often introduce Wolfram|Alpha to their students. However, it is not likely that every high school teacher would be a Mathematica user. We might be able to create another user persona if we talk to people who are familiar with Wolfram|Alpha but not other Wolfram products.

Final thoughts

When the participants do not match the ideal criteria, how to make the most out of what you get? The moment that it became clear we wouldn't be able to interview common users, I reconstructed my research plan. Instead of asking the common users how they know Wolfram|Alpha, I asked fan users how they introduce Wolfram|Alpha to others. When abandoning and rerunning the study is not an option, we need to be creative. This project provided me a unique experience of adjusting the research plan to compromise for the imperfect participant sampling.