Applying Iterative Design

Principles to a Live Product

Name: Tibor Zahorecz

Date: 11th Dec 2020
 Step 1
Select KPIs
&
Evaluate Previous
Multivariate
Experiment Results
Select KPIs for Flyber
Analyses
● In MVP phase I have chosen the MAU/DAU, NPS, MRR

● For the data available, which KPI(s) best match Flyber’s

business model? (using supplement materials: link)
‚KPIs are quantifiable measurements or data points used to gauge your
company’s performance relative to some goal. For instance, a KPI could be related to
your goal of increasing sales, improving the return on investment of your marketing
efforts, or improving customer service.’

We have data logs available which help Flyber in the growth

stage:

KPI 1: Acquisition: the number of new visitors in a day

KPI 2: Activation: the number of visitors who sign up for a drive

KPI 3: Retention: % churn in searh phase

Select KPIs for Flyber
Analyses
● How would you calculate these KPI(s) using the available
event data logs?

i., Acquisition: the number of new visitors in a day = nr of new

user_id / event time (day)

ii., Activation: the number of visitors who sign up for a drive = nr of

user_id with event type (#_of_users)

● List other KPIs that might be important to Flyber but are not
calculable based on available data

Activation: The time users spend browsing the Flyber app on the
first visit

Retention: The number of users who renew their trip with Flyber

Revenue: Customer Acquisition Cost (CAC), Customer lifetime

value (LTV)
Describe the First
Multivariate Experiment
Describe the elements tested during the multivariate
experiment. You can use the image below when referencing the
tests
Describe the First
Multivariate Experiment
Describe the elements tested during the multivariate
experiment. You can use the image below when referencing
the tests
There are 2 variables in this multivariate experiment. The first
one is the text field “Tip Included”, the second one is the the
text on the CTA button (Fly Now or Book Flight).

● Control: CTA (Book Flight) + Tip included

● Experiment 1: vs Control CTA is changed to Fly Now

● Experiment 2: vs Control without Tip included

● Experiment 3: vs Control witthout Tip included and CTA is

changed to Fly Now
Review Multivariate Test
Results: Visualization
Provide a visual representation of the impact of the
experiment on the conversion rate of users booking
a flight (out of all users opening the app)

Step 1: comparing user_uuid vs records ➔ same

Review Multivariate Test
Results: Visualization
Provide a visual representation of the impact of the
experiment on the conversion rate of users booking
a flight (out of all users opening the app)

Step 2: conversion rate

Review Multivariate Test
Results: Visualization
Provide a visual representation of the impact of the
experiment on the conversion rate of users booking
a flight (out of all users opening the app)

Step 3: Visual representation

Review Multivariate Test
Results: Visualization

Step 4: Analyzing by convesion stage: Begin_Ride

Review Multivariate Test
Results: Significance Test
Determine if there was a significant difference
between the experiments and control states.

Explain how you would perform a t-test to determine

if the experimental results had a greater impact on
the booking conversion rate than the control state
In order to perform the t-test, we will first make hypothesis

➔Null hypothesis: Users do not convert better with the test

state as compared to control state

➔Alternative hypothesis: Users will convert differently with

the test state

➔Confidence interval:- 95% confidence

➔p- value < 0.025, difference is statistically significant, the null is
rejected.
Review Multivariate Test
Results: Significance Test
Determine if there was a significant difference
between the experiments and control states.
List the test results (p value) for each experiment
compared to the control

Data:
Review Multivariate Test
Results: Significance Test
Determine if there was a significant difference
between the experiments and control states.
List the test results (p value) for each experiment
compared to the control

Results: (using https://www.surveymonkey.com/mp/ab-

testing-significance-calculator/)

Control vs Experiment 1: p value: 0.1591

Review Multivariate Test
Results: Significance Test
Control vs Experiment 2: p value: 0.0843

Control vs Experiment 3: p value: 0.1848

Review Multivariate Test
Results: Significance Test
Determine if there was a significant difference
between the experiments and control states.
● Using the statistical significance calculator of your
choice, determine which experiments, if any, had a
significant result at the 95% level. Include your
calculations as part of your explanation:
* see onthe above slides the calculator
screenshots

● Based on your statistical significance calculations,

recommend if any of the experiments should be
expanded
* Result not significant for all three cases
based on p value at 95% confidence level
 Step 2
Funnel & Cohort
Analyses
User Funnel
Identifying the different stages the user funnel

Based on the event types in the data provided, list

the 3 or more steps a user can take from opening the
app to final booking of a ride

Event_type: open, #_of_users, search, begin_ride

User Funnel
Identifying the different stages the user funnel
Provide a graph showing the funnel from step to
step, including drop off rates.

Using data set

User Funnel
Identifying the different stages the user funnel
Provide a graph showing the funnel from step to
step, including drop off rates.
User Funnel
Identifying the different stages the user funnel
Provide a graph showing the funnel from step to
step, including drop off rates.
User Segments

● Identify 2 demographic attributes present in the

data that allow for segment analysis
'Demographic segments group users that share the same personal
characteristics’

Product managers use demographic segmentation to understand

if personal characteristics result in different product experiences.
When significant differences among different groups occur,
product managers can form a hypothesis for the cause of these
results and then investigate further. Such investigations often lead
to discoveries about the unmet needs of the underperforming
segment. These unmet needs to become opportunities to boost
KPIs with targeted product changes.

Demographic attributes: age, user_neighborhood

User Segments

For each demographic attribute, provide the number

of users in each segment group
User Segments

For each demographic attribute, provide the number

of users in each segment group
User Segments

For each demographic attribute, identify the segment

group with the largest number of users
User Segments

For each demographic attribute, identify the segment

group with the largest number of users
Segment Analysis of Funnel

Identify Opportunities for Improvement

● Perform a funnel analysis by segment for all

identified demographic attributes and describe
the results

Age analysis I.
Segment Analysis of Funnel

Identify Opportunities for Improvement

Perform a funnel analysis by segment for all

identified demographic attributes and describe the
results

Age analysis II.

Segment Analysis of Funnel

Identify Opportunities for Improvement

Perform a funnel analysis by segment for all

identified demographic attributes and describe the
results

Age analysis III.

Segment Analysis of Funnel

Identify Opportunities for Improvement

Perform a funnel analysis by segment for all

identified demographic attributes and describe the
results

Age analysis summary

In the 50+ age group, there is a large mass of interest (open) and
finally a very low use of the service (begin_ride) can be observed.
What is the reason of this age group has a high drop off rate in
the conversion funnel?

While the 30-39 and 18-29 ages have a much healthier funnel, but
they have a lower initial opening rate. The analysis shows that an
experiment should be conducted for these two age segments.
Why is the opening rate so low and how can be improved.
Segment Analysis of Funnel

Identify Opportunities for Improvement

Neighborhood Analysis I.
Segment Analysis of Funnel

Identify Opportunities for Improvement

Neighborhood Analysis I.
Segment Analysis of Funnel

Identify Opportunities for Improvement

Neighborhood Analysis I.
Segment Analysis of Funnel

Identify Opportunities for Improvement

Neighborhood Analysis II.

Segment Analysis of Funnel

Identify Opportunities for Improvement

Neighborhood Analysis summary

Manhattan has the largest nr of usage of the service, Brooklyn

come after it.
Drop off calculation: State of Island has the lowest #_nr_user
drop off rate (57.80%), in the search pahe Manhattan has the
lowest nr (79.83%) and Bronx has the best begin_ride rate
99.66%.
Analysing by experiements: by the largest nr of usage Manhattan
experiment_2 has the best conversion rate and Bronx
experiment_1
Segment Analysis of Funnel

Identify Opportunities for Improvement

Neighborhood & Age funnel drop analysis

Segment Analysis of Funnel

Identify Opportunities for Improvement

Neighborhood & Age funnel drop analysis

Segment Analysis of Funnel

Identify Opportunities for Improvement

Neighborhood & Age Analysis summary

Younger generations have got lower drop off rate in search

phase, and here we are loosing 50+ genreration.

In Bronx at the age goup of 30-39, 40-49 we have the highest

begin_ride %.
 Step 3
Hypothesis & Next
Steps
Review Qualitative Data

In this stage I use learnings from Udacity lecture:

Integrate Data into Iterative Design (video)
First, analyzing usage data, conversion funnels, and other quantitative
data collected can reveal users or areas that are under- and over-
performing.
Next, upon seeing areas for improvement via quantitative analysis,
product managers then turn to user interviews or other qualitative data
techniques to understand why such trends might be occurring.
Review Qualitative Data

Read user interviews to understand “why” any

funnel under-performance seen in Step 2 might
occur

In the user interview log file we can see 50+ age group over
represented but it is in line with quantitative data logs (open
rate).

Recognized behaviour patterns:

● Calling a service (taxi)
● Use assistant to drive their car
● Tell a phone (voice command) to call a service

Hypothesis behind the patterns:

● poor eyesight in older age | bad UX of the app
● entrenched habits like calling a dispatcher
● uses the phone to make calls and not booking service
Review Qualitative Data
List your hypothesis for what customer need is
being under-served
In this stage I use learnings from Udacity lecture: Jobs to Be Done
Framework

The Jobs to Be Done Framework pushes product managers to consider what job
people are hiring their product to do. This job goes beyond the simple actions a user
takes and instead examines the motivations and goals a customer is trying to
achieve.

Core functional job: this is the direct result that job executor is trying to achieve by
using a product. Note: it should be motivations and goals a customer is trying to
achieve, regardless of what tools they use. ➔ Being from A to B as fast as possible
fast and safe (detailed in the next slides)

Related jobs: these are jobs done either before, during, or following the execution of
the core functional job. By understanding related jobs, you can find opportunities to
add additional features to the product that helps customers.

Emotional job: this is how the job executor wants to feel or be perceived after
finishing the core functional job. ➔ super cool to fly and arrive to destination, gives
credits in society

video
Review Qualitative Data
List your hypothesis for what customer need is being
under-served

Step 1: Define: defines goals and the resources needed for achieving goals.
In this step, product managers can improve their product by making planning
simpler for the user.
➔ flyber user realizes she needs to be in Manhattan in 25 minutes for the
next meeting. She wants to avoid traffic jam.
➔ feature makes planning simpler: syncronize user calendar with flyber
app and give signals when to start trip based on traffic. Save user addresses and
favourite places make planning easier.

Step 2: Locate: gathers everything needed to accomplish a job, whether physical

tools or information.
A product manager can innovate in this step by making it easier for a user to gather
everything they need in one place.
➔ user checks the traffic situation, calls his cab service for estimated time
to arrive, check google maps
➔ feature makes easier gathering information: comparing ETA and costs
based on taxi, uber, own car and flyber

Step 3: Prepare: sets up the environment to do a job.

To make improvements, product managers can reduce required set up or provide
guidance that allows users to prepare more quickly.
➔ user prepares his mobile for booking, his car for driving
➔ feature helps to prepare more quickly: voice command for route
planning, virtual assistant for complete execution
Review Qualitative Data
List your hypothesis for what customer need is being
under-served

Step 4: Confirm: a final check to make sure the job is ready to do.
Product manager can innovate by giving users easy ways to confirm their readiness
for a job.
➔ user checks things before starting her trip: meeting notes, laptops, her
children present, clothes…
➔ feature helps this phase: Based on the user’s calendar entry, the app
analyzes the type of appointment and gives a forecast of what tools are needed for
the appointment. Eg. if she has a business meeting the app will remind the user to
bring her laptop and notes.

Step 5: Execute: begins to carry out a job.

A product manager improves this step by removing blockers and ensuring their
product is intuitive to use.
➔ user call the service or book the trip
➔ features: voice command, ‚press one button’, clear instruction (visible)

Step 6: Monitor: checks whether the job is being successfully carried out.
For this step, product managers can innovate by giving users more information
about progress towards their goal.
➔ user monitor the progress: in time?, no danger?, traffic changed?..
➔ features: show the ETA, progress on map vs cab & uber, show traffic
changes
Review Qualitative Data
List your hypothesis for what customer need is being
under-served

Step 7: Modify: make changes or adjustments to improve the outcomes of the job.
To improve a product in this step, a product manager should reduce the number of
modifications a user has to make when executing a job.
➔ user can change the destination or stop picking up her partner
➔ feature: options if destination changed (next flyber drop off station
with continuing uber pick up)

Step 8: Conclude: finishes executing the job.

To improve a product in this step, product managers can decrease the work needed
to conclude a job.
➔ user arrives her destination in good mood, fresh with pride
➔ features: payment without delay (not transactional one)

Emotional Needs Served

A flyber user has emotional need for feeling safer and winner. Showing others that
they are rich, tech-driven people.

Case Studies: Customer Empathy with Jobs to Be Done: The UBER Case, Build Products That Solve Real Problems With
This Lightweight JTBD Framework

Tony Ulwick – Put Jobs-To-Be-Done Theory Into Practice With Outcome-Driven Innovation
Review Qualitative Data
List your hypothesis for what customer need is being
under-served
Tony Ulwick – Put Jobs-To-Be-Done Theory Into Practice With Outcome-Driven
Innovation
 Review Qualitative Data
List your hypothesis for what customer need is being
under-served
Simple planning: syncronize user calendar with flyber app and give signals when to
start trip based on traffic. Save user addresses and favourite places make planning
easier.

Makes easier gathering information: comparing ETA and costs based on taxi,
uber, own car and flyber

Prepare the trip more quickly: voice command for route planning, virtual
assistant for complete execution

Booking : voice command, ‚press one button’, clear instruction (visible)

Prioritization
RICE framework
R (Reach): the number of people or customers a feature is expected to be used by,
in a set time period. (Qualitative log data 50+ age / location)
I (Impact): measures the effect a new feature is expected to have on KPIs such as
conversion rate or customer satisfaction. (data log conversion funnal drop off)
C (Confidence): describes how sure you are that your estimates are correct. The
more data you have about reach, impact, and effort, the more confident you would
be.
E (Effort): the amount of time it will take to make a feature go live, from design to
engineering to QA and launch. The effort can be measured in person-months,
which means the work one person can do in a number of months.
How to do it: link

RICE Score = (R * I * C) / E
Review Qualitative Data

Provide 3 or more quotes as evidence for this

hypothesis
'Honestly, I thought about using Flyber to suprise my grandson or granddaughter with a
visit to one of their sporting games. Luckily my daughter was around to help me book the
ride. I usually just use Uber because it remembers my addresses and has all my favorite
places saved, so I guess I always just open that up since it is so convenient and saves
me time. Though now that I say that, I really should use Flyber again since it would save
more time when it comes to fighting traffic!’

‚If the timing isn't different, I'll take a taxi or uber to save money. ‚

‚I just hail a taxi or tell my phone to call a cab to go to a certain address (I'm always on
the phone, so I just use voice commands with my phone most of the time)’

‚I have a personal car service on call. My assistant books Flyber whenever I'd be travelling
during peak NYC traffic hours. Time is money and Flyber saves me time! But I let my
assistant actually book the Flyber because the first few times I tried booking, the
instructions were too small.’
 Suggested Features &
Experimentation Plan

Share your hypothesis using the following format:

We believe [observed quantitative effect] Because

[hypothesized user “why”] And that by [general
change/opportunity for Flyber to improve] for
[targeted cohort] we will see [expected effect ]

We believe the age 50+ user group high drop off rate
in booking (search) phase because difficult for them to
use our app’s booking feature and by voice control
(voice to text and auto-fill feature) for this age group
we will see the same drop off rate as younger age
groups.
 Suggested Features &
Experimentation Plan

Suggest 2 or more features that would match your

hypothesis and determine a plan for multivariate
testing, including describing the control and
experimental conditions

• Voice control (voice to text)

• Auto-fill function
 Suggested Features &
Experimentation Plan

Determine who should be exposed to the

experimental changes

‚on the next slides’

List any additional metrics that would be helpful to

collect from your suggested features

‚on the next slides’

 Suggested Features &
Experimentation Plan
Instrumentation

Product managers do not always run experiments on all users,

but may focus on a particular type of users. This means you need
to instrument and collect information on the right type of users
and ensure that they are part of the experiment.

• It is vitally important to track which test state a user is

assigned. This enables you to analyze the difference in results
among experiment groups and to ensure the same test state
is shown to the same user until the experiment is over.

• The final thing to track during an experiment is the conversion

rate. It is the percentage of users who successfully performed
the desired action in each experiment group. The key to
choosing the correct conversion rate is to choose one that
relates to the important, long term impact you wish your
experiment to drive. This may often be a product or feature
KPI.
 Suggested Features &
Experimentation Plan

Track User Actions During Testing

• We need to track the number of unique visitors to the page

where they will be testing the new design.

• We will need to track whether a user sees the control or the

experimental design on the page.

• We will also likely want to track the device and browser that
the visitor is using in case the designs perform differently
across device types.

• Finally, to know which page is more effective, they will need

to track whether the visitor takes the desired action of using
the social linking feature
 Suggested Features &
Experimentation Plan

Ensure Unbiased Control & Test Groups

• When running experiments, you must make sure that users in the control
and the test/experiment group share the same characteristics. That way,
you can ensure the differences among experiment groups is caused by
the change of the product, not by different user characteristics. ➔ in our
expirement we should be careful with demographic (age 50+) and
locations characteristics.

• First, you need to determine what type of user you are testing and make
sure these target users are put into both the control and experiment
groups. ➔ age 50+

• In addition to everyone being a target user, you need to make sure other
user characteristics such as age, race, income are balanced between the
control and experiment groups.

• A great way to get a balance of user characteristics among the groups is

to randomize who is in the control and who is in the experiment group.
 Suggested Features &
Experimentation Plan
Calculate & Respond to Test Results

The results experiments can be evaluated by a T-Test. A T-test compares the means of two groups to
see whether the difference in the means is large enough that it is probably not due to random chance.
In the case of an A/B or multivariate test, the T-test helps us understand whether any differences
observed in the conversion rate are significant and are likely due to the change we made.

• Step 1: state the null hypothesis. The null hypothesis represents the idea that the control state is
not different from a test state, which is what we want to reject. A generic null hypothesis would be:
customers convert the same with the test state compared to the control state. ➔ In our case in this
will be the control state. No changes will be added here that is the current product state.

• Step 2: state the alternative hypothesis. The alternative hypothesis is the one you want to accept. A
generic alternative hypothesis would be: customers will convert differently with this change. ➔
Here we will add the two new features with several experiments

• Step 3: set a confidence threshold. Product managers almost universally user a 95% confidence
level, which means a 95% chance that the result appears statistically valid and a 5% chance that
the result is an error.

• Step 4: run the A/B or multivariate experiment.

• Step 5: perform a T-test. After you get the result from your experiment, you need to perform a T-
test to see if the differences you see in the experiment are significant. To perform a T-test, you will
need to know:

• The number of users in the control group and in each test group

• The number of conversions in each group

• The confidence threshold from step 3

• Decide if the T-test is a one-tailed or two-tailed test. A two-tailed test means you assume
the experiment conversion rate is different from the control, maybe worse OR better.
Appendix
Raw Data
Additional Info

You could include supporting or additional

information that can support your previous slides
but isn’t necessary for every person to see that
looks at your slides.