Learn Together Part One: User Study
UX Team: Wenwen Song, Nina Zhou, Morgan Evans
Time: February 2016 to April 2016
A user study on soft skill learning in social (peer-to-peer) contexts regarding the problems, practices, constraints, and potentials.
The study, consisting of 12 sessions in total, was conducted with employees from Alaska Airline and Costco to validate product ideas, understand user needs and goals in terms of soft skill learning.
My Contribution:
DEFINE
-
Understand Product Ideas
-
Define study goals
DESIGN
-
4 Sessions per Person
-
Survey
-
User Interview
-
Wireframe/Prototype
PRACTICE
-
12 user sessions
-
10 participants
-
Alaska Airline
-
Costco
RESULT
-
Affinity Diagram
-
Findings
I. DEFINE
Understand Product Ideas
As I joined the team, the founder of the project already developed a clear image of the product concepts. They were stated in LearnTogether Whitepaper and further explained by the founder as below.
Bob Boiko
Founder, LearnTogether
LearnTogether is a social learning system that helps people improve their soft skills:
-
Learners learning by example and then performing
-
Mentors engaging in short and targeted interactions
-
Managers linking learning to business value
-
A system that connects people, analyzes and serves their needs and evolves a crowdsourced learning library.
Define Study Goals
-
Know Research Subject. Soft skill learning in social (peer-to-peer) contexts regarding the problems, practices, constraints, and potentials.
-
Product Ideas Validation. Learn about the social, cultural and technical fit of our ideas to their lives and model our UI and process with them.
II. DESIGN
Overview
User sessions were designed based on the study goals previously stated and the participant profiles: 5 of them are the employees of Alaska Airlines (AA), and the other 5 Costco. Typically, we defined objectives for each session and drafted the script. Adjustments were made to the plan later based on feedback from the founder, and how it went out during the previous sessions.
Alaska Airlines
-
5 participants divided into 2 groups
-
4 user sessions were scheduled for each participant every other week.
-
Each session lasted for an hour
SESSION 1
SESSION 2
SESSION 3
SESSION 4
Who Are We? Who Are You?
Let's Discuss Mentoring and Rating!
Try The Process: Mentoring and Rating
Get onboard and get engaged!
Note
The user study with Costco was revised based on how it went out with AA since the plan with AA was always 2 sessions ahead of it with Costco.
Costco
-
5 participants in one user group
-
4 user sessions were scheduled for each participant every other week.
-
Each session lasted for an hour
Session 1
Session 2
Session 3
Session 4
Who Are We? Who Are You?
Let's Discuss Mentoring and Rating!
Let's Discuss Problem Solving and Learning Community!
Test The Process: Problem, Interact, and LearnTogether.
Session 1. Who Are We? Who Are You?
Since we were going to conduct a series of user sessions with the same group of participants, we were expecting to establish a good relationship with the participants. For the 1st session, we decided not to overwhelm them with too many questions but set up the context instead.
Objectives
1. Introduce LearnTogether and set up shared knowledge of social learning.
2. Know about our participants: demographics, tech-savviness, current learning/training model, and challenges.
Examples
The attached was the plan for Costco User Session 1, which shows a revised and improved design compared to AA's as previously mentioned.
Costco User Session 1
March 7, 2016
I. Introduction (10 min)
Introducing LearnTogether and the concept of Social Learning.
Method: A Brief Presentation
To introduce LearnTogether and set up shared knowledge of social learning.
II. Warm-up/Ice-breaker (5 min)
What's your job? And what soft skill would you contribute to the team?
III. Take the Survey (10 min)
The survey was designed to get information about the following 5 aspects: Personal Info, Tech-Savvy Measurement, System Usage, Training Environment, and Social Environment.
Method: Survey (5 Sections)
A. Personal Info
To know participants’ job responsibility, position, age, gender, years of experience in this field, working duration in this company.
B. Tech-Savvy Measurements
To gauge how comfortable the participant is with technology, device usage, application usage, and frequency.
C. System Usage
To see what systems ours might interact with at work and be compared to outside of work.
D. Training Environment
To know participants' opinions towards training sessions, supportiveness, and social environment.
E. Social Environment
To know the social environment in which training happens.
IV. Discussion (35 min)
By conducting the user interview, to know about the participants' current staff training: pain points, needs, and current solutions.
Method: Group User Interview
We broke 5 participants into 3 groups of 2, 2, 1 and conducted the interview separately with the same set of questions:
A. How do people usually get trained in the company? (10 min)
-
What skills do you get trained on now?
-
Can anyone share the experience of a good training session you were in?
-
What are the most effective things in the current model? Why?
-
What could be improved?
B. How do people improve their soft skills? (10 min)
-
How does soft skill training work at Costco?
-
How have you learned soft skills?
-
Could you judge the soft skills of other people?
-
How might you help someone improve on that skill?
-
How do people reach out for help with soft skills?
C. Suppose your company had the best soft skill development in the industry. (15 min)
-
How does it work?
-
Why does it work?
V. CLOSING (5 min)
Any last comments/questions, thank you, and wrap-up.
Note: why introduction?
By introducing, we would also like to convey the message that they were not only the tester as they had presumed, but the essential part of our design process with ownership. "Let's build a great product together!"
Note: why on-site survey?
Why Survey? Given limited time we had for each participant and the limited facilitator we could bring to the user session, we would like to spend more time on more important topics and questions.
Why on site? The survey was requested to be done during the session because it would be easier to get honest answers in a time they had promised you. We didn't want it to be a burden that would ask for more time and effort.
Reflection: Group Interview or Individual Interview?
The discussion with AA was first held in the form of group interview. It turned out that:
(1) the time left for each participant was too limited to offer enough information.
(2) while the interview was conducted in a group, the participant felt more likely to be influenced by others. We ended up with fewer perspectives.
Session 2. Let's Discuss Mentoring and Rating!
For the 2nd session, we were ready to introduce the participants our main product ideas: mentoring and rating. Before getting into the "meat", we spent some time reviewing the survey results and asked follow-up questions.
Objectives
1. Understand the participants' opinions towards our product ideas: (1) peer-to-peer mentoring; (2) rating mechanism.
2. Deep dive in the reasons (personal and social) that shaped their opinions.
Examples
The attached was the plan for Costco User Session 2, which shows a revised and improved design compared to AA's as previously mentioned.
Costco User Session 2
March 21, 2016
I. Review Survey Results and Follow-Up (20 min)
As stated before, we included in the survey the questions regarding the participants' current training , social and cultural environment for mentoring and rating. We started the second session with survey results that we found interesting and worth a deep dive and asked follow-up questions about them.
II. Present Concept: Mentoring (20 min)
III. Present Rating Mechanism (20 min)
To understand (1) how well does the participant understand the idea; (2) how does the participant like the idea.
Method Applied to II & III:
1. Present product idea of Mentoring/Rating Mechanism in text and ask the participant to read through.
2. Ask follow-up questions.
-
What do you think made of the mentoring process/rating mechanic?
-
What are the goodness and badness of the process?
-
What would you suggest to improve the process?
IV. CLOSING (5 min)
Any last comments/questions, thank you, and wrap-up.
Note: why present the idea in the text?
We had an argument on how to present our ideas: whether to show the prototype or not? The decision was "No" and describing it in text instead for the following reasons:
1. The concept of mentorship was what mattered here. The prototype we had at that moment only showed one of the alternatives that could possibly digitalize and visualize the process. We'd like to maximize the creativity.
2. Usually, while presented with a finished prototype, the participants would end up with criticizing the usability issues, which wasn't what we were looking for at that moment.
Session 3. Let's Discuss Problem Solving and Learning Community!
For the 3rd session, we went back to the research question of learning/mentoring flow again, but at this time the flow was asked and examined in the context of problem-solving, which we assumed to initiate the learning/mentoring practice.
Objectives
1. Know about the participants' learning/mentoring workflow in a broader context of the problem-solving process, including the triggers (causes), the actions, and the effects.
2. Understand the participants' opinions towards the proposed community mechanism.
3. Test the proposed problem-solving flow by using paper prototype.
Examples
The attached was the plan for Costco User Session 3, which shows a revised and improved design compared to AA's as previously mentioned.
Costco User Session 3
April 4, 2016
I. Problem Solving (20 min)
To know the participants' regular problem-solving process by doing interviews. The information we were looking for: what would be the triggers to a learning initiative, what would be the most common actions they take, and how the actions turned out.
Method: User Interview
1. Could you share a recent story that made you strongly feel that you need to improve your certain soft skill?
2. What did you do to improve that skill, e.g., taking an action? Why?
3. How did it work?
II. Introducing Community Mechanism (20 min)
To understand (1) how well does the participant understand the idea; (2) how does the participant like the idea.
Method:
1. Present product idea of Community Mechanism in text and ask the participant to read through.
2. Ask follow-up questions.
-
What do you think made of the community mechanic?
-
What are the goodness and badness of the mechanism?
-
What would you suggest to improve the mechanism?
III. Test Proposed Pre-mentoring Flow (5 min)
Method: Paper Prototype
Show the proposed flow with paper prototypes
New Challenge: Habit-centered design put us back to where we started.
At the time when we were designing session 3, the founder brought a new methodology to the product design process, habit-centered design, which made a big influence on the pre-mentoring flow. Pre-mentoring flow demonstrates the process of a user gets triggered to open our application and start a mentorship with another user.
This big change in the methodology posed a new challenge for us to understand the participants' trigger-action-reward cycle. And that's why we designed the session 3 as it is now.
Session 4. Test The Process: Problem, Interact, and LearnTogether.
For the last session, we proposed a typical user journey based on the feedback from all previous user sessions. The journey was tested with all the participants by using paper prototype.
Objectives
1. Test new UIs for the pre-mentoring process.
2. Understand the participants' opinions towards a typical LearnTogether journey.
3. Get answers for how the participant would like to interact with people in what circumstance.
Examples
The attached was the plan for Costco User Session 4, which shows a revised and improved design compared to AA's as previously mentioned.
Costco User Session 4
April 18, 2016
I. Pre-mentoring UI (20 min)
Bring the new UIs for pre-mentoring flow and ask for feedback.
Method: Paper Prototype
Print the design on paper and show to the participants.
II. Alternatives of Solutions (25 min)
To understand what solutions the user would choose to solve the problem in what circumstance and how they would react to our 4 proposed solutions.
Method: Paper Prototype
Show the paper prototype of 4 solutions to engage in the community and solve a problem.
III. Closing Discussion (10 min)
Any final comments or thoughts? And thanks!
Note
Apart from designing the user sessions, revising and improving the existing user flow/user interface based on what we heard from the participants was also part of my responsibilities.
Every user sessions reflected a mini cycle of define - prototype - test process.
III. PRACTICE
In this 3-month long user study, I involved in the whole process of design, prepare, and practice for all the user sessions. I've been learning a lot in the process about coordinating, communicating, and presenting.
Alaska Airline Session 1
Feb 12, 2016
To Group 1: Participant A, B
Hosted the Section IV Part B: How do people improve their soft skill? (10 min)
To Group 2: Participant C, D
Hosted the Section IV Part B: How do people improve their soft skill? (10 min)
Reflection: the 1st session and lesson ever!
1. Smile, which I did pretty well :)
2. Be confident. My voice turned out to be too low and I tended to fade myself into the background.
3. Conclude with Rephrasing what you heard. "To make sure I understand correctly, you were saying that...", "Correct me if I'm wrong, you just said..."
Reflection:
1.Prepare for unexpected situations. In this session, an additional participant showed up without a heads-up and we didn't prepare enough materials.
2. Do not let just one of the participants do all the talking.
Alaska Airline Session 2
Feb 26, 2016
To Group 1: Participant A, B
Hosted the Section III: Presenting Rating Mechanism (20 min)
To Group 2: Participant C, D, E
Hosted the Section III: Presenting Rating Mechanism (20 min)
Costco Session 1
March 7, 2016
To Participant A, B
Hosted the Section IV Part A: How do people usually get trained in the company? (10 min)
Hosted the Section IV Part C: Suppose your company had the best soft skill development in the industry. (15 min)
Spoiler Alert!
You would be amazed at the exact opposite opinions that came from people from different age groups and company cultures.
Alaska Airline Session 3
March 11, 2016
To Group 1: Participant A
As Bob hosted the session, I facilitated the participant to walk through the prototype and proposed follow-up questions.
To Group 2: Participant C
As Bob hosted the session, I facilitated the participant to walk through the prototype and proposed follow-up questions.
Costco Session 2
March 21, 2016
To Participant A, B, C, D, E
Hosted the Section I: Survey Results Review and Follow-up (20 min)
Hosted the Section III: Presenting Concept: Mentoring. (20 min)
Spoiler Alert!
Really got inspired by how participants from Costco understood mentorship. This was the first time we got a super straight-forward but challenging feedback on the mentoring process.
Alaska Airline Session 4
March 25, 2016
To Group 1: Participant A
Independently facilitated the participant to walk through the prototype and proposed follow-up questions.
To Group 2: Participant C
Independently facilitated the participant to walk through the prototype and proposed follow-up questions.
Costco Session 3
March 21, 2016
To Participant A, B
Hosted the entire one-hour session:
I. Problems Solving
II. Present Community Mechanism
III. Test Proposed Pre-mentoring Flow
Reflection:
1. Beware of the time and get a helper! Given the first time that I had the full control of a session without a helper around, the session took 30 minutes longer than expected due to my loss of control :(
2. Pay attention to all the participants and do not let one do all the talking.
Let's just Hooray!!!
What a relief to close such a long-term user study I had ever done!
Costco Session 4
March 21, 2016
To Participant A
Hosted the Section I & II: Pre-mentoring UIs & Alternatives of Solutions. (50 min)
IV. RESULT
Affinity Diagram
The method we took to analyze such a great amount of data points was affinity diagram with the online tool Realtimeboard.
Affinity Diagram Process
Step 1 Log Data Points
Added all the data points to the board with sticky notes.
Step 2 Clustering
Divided the data points into groups and subgroups.
Step 3 Labeling
Labeled each group with an appropriate name.
Step 4 Mapping
Mapped the clusters based on the underlying logical relationships.
Findings
After the affinity diagram, we were about to present these findings to the founder and the rest of the team. The result was presented in the way of Q&A, which was trying to depict the research subject posed at the very beginning "soft skill learning in social (peer-to-peer) contexts regarding the problems, practices, constraints, and potentials" and the "social, cultural and technical fit of our ideas to their lives and model our UI and process with the participants."
Examples
There were quite a number of findings in this list of Q&A. Here I just put one of them as an example below.
Finding 4: What are the factors that stop people from self-learning and social-learning?
I. Unaware of the Problem/Weakness
People are unaware of their problems and thus would not take an initiative to improve. Sometimes it's because limited feedback is provided, and sometimes it's just one's personality to be satisfied with the current situation and not willing to improve.
"Channels to give and gain feedback are currently limited." - Participant A
“I’ve worked in this company for over ten years. Some people just don’t have such self-awareness.” - Participant B
II. Lack of Motivations/Incentives
III. Asking Someone for Help Could be Very Difficult which requires a healthy culture, a trustful relationship, and time commitment.
Asking others for help in a corporate environment could be hard, and it varies by companies and workgroups. People are unwilling to ask for help in a working environment due the following reasons:
-
Lack of an atmosphere or a culture that encourages (social) learning in the corporate or workgroup.
-
Lack of trust: they're worried that revealing weaknesses would result in bad consequences, e.g., being punished or affecting the career.
-
It's Hard to find the right connection and the right person with the corresponding strengths.
-
Problems with soft skill are always personal.
-
Learning/mentoring requires time commitment, which would better fit into their busy schedule.
"Because bonus is tight to how successful you are, people don’t want to admit that they are wrong." - Participant E
"There are concerns that reaching out for help would imply their incompetency and thus affect their careers." - Participant A
"Feeling unsafe to request help on soft skill improvement while the boss is being around." - Participant D
Reflection
Powerful idea of "self-awareness". This would never occur to my mind if I didn't participate in the user study. Participants from both Costco and Alaska Airlines made an emphasis on the importance of self-awareness and initiative.
It put us to think:
What are the triggers for them to use the product?
How can the product help the user to identify the problem? Or would the process of identifying the problem be the part of the learning experience itself?
First Degree Attention
The word "trust" could be the most repeated word in this user study. It takes trust to reveal your "vulnerability and incompetence" to others, let alone a mentorship.
They really really really care this! So it's an issue we must think about:
how do we help build trust?
Or can we just let the user start their journey with an existing trustful relationship?
Read Learn Together Part Two: Product Design to see how this study affected our design decisions in the process of developing LearnTogether Mobile App.