Towards Understanding Analytics in Software Startups

Usman Rafiq
Faculty of Computer Science, Free University of Bozen-Bolzano,
Bolzano, Italy
urafiq@unibz.it

ABSTRACT 1 INTRODUCTION
Analytics plays a crucial role in the data-informed decision-making In recent years, there has been an increasing interest in utilizing
processes of modern businesses. Unlike established software com- analytics to produce value in the software businesses [8, 14]. Often-
panies, software startups are not seen utilizing the potential of times, the power of analytics is used to analyze significant informa-
analytics even though a startup process should be primarily data- tion about software projects [8]. This vast amount of information
arXiv:2204.05182v1 [cs.SE] 11 Apr 2022

driven. There has been little understanding in the literature about leads to actions and decisions, transforming ways to revisit business
analytics for software startups. This study set out to address the methods and practices [16]. Particularly, software businesses avail
knowledge gap by exploring how analytics is understood in the this information to understand and make decisions while the project
context of software startups. To this end, we collected the qual- evolves [8]. The use of analytics to facilitate the decision-making
itative data of three analytics platforms that are mostly used by process has begun to proliferate since its inception [14]. Mainly,
startups from multiple sources. We covered platform documenta- established software businesses and companies are harnessing the
tion as well as experience reports of the software startups using full potential of analytics, possibly because of the access to a large
these platforms. The data was analyzed using content analysis tech- amount of information and abundant resources.
niques. Four high-level concepts were identified that encapsulate Startup companies are distinguished by their focus on inno-
the real understanding of software startups on analytics, including vation, under extreme uncertainty [3, 18]. These companies con-
instrumentation of analytics, experimentation, diagnostic analysis, stantly look for a sustainable and scalable business model with
and getting insights. The first concept describes how startups set up speed and focus. Software startups are types of startups, aiming
analytics and the latter three illustrate the usage scenarios of ana- to build software-intensive products or services [19]. Similar to
lytics. This study is the first step toward understanding analytics in general startup companies, while aiming at innovation, software
the software startup context. The identified concepts can guide fur- startups also need to move and act fast. Nevertheless, they are
ther investigation of analytics in this context. It also provides some confronted with a few other challenges, related to the software
insights for software startups to set up analytics for data-informed engineering field. Software startups use lean startup methodol-
decisions. Given the limitation of the data used in the study, the ogy to validate the product-market fit [4] and agile development
immediate next step is to ground as well as validate the acquired methodologies to develop the product [15]. This product and busi-
understanding using the primary data, by directly interacting with ness development cycle certainly brings a change in direction. It is
software startups referred to as pivoting in software startups [18] and brings a change
in the product as well as the business model. We, therefore, may
CCS CONCEPTS obtain a pertinent illustration that startups adjust themselves based
on the information they gain. The decisions regarding directions
• software and its engineering → Software creation and man-
could be diverse, like for example, changing the target customer
agement.
segments, adding or removing product features, improving user
experience, modifying customer acquisition strategies, and so forth.
KEYWORDS This is where software startups could utilize analytics, to better
analytics, metrics, abstraction, software startups understand and evaluate their actions.
The research regarding software startups, to date, has primarily
ACM Reference Format:
tended to focus on software development methodologies [19]. There
Usman Rafiq. 2022. Towards Understanding Analytics in Software Startups.
In 5th International Workshop on Software-intensive Business: Towards Sus-
is still uncertainty, however, how software startups understand and
tainable Software Business (IWSiB’22 ), May 18, 2022, Pittsburgh, PA, USA. apply analytics throughout the product as well as the business de-
ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3524614.3528632 velopment. The role of analytics in startups is unknown despite the
widespread use of analytics in other businesses [2, 17]. Therefore,
the current study seeks to address this gap. It will eventually help
Permission to make digital or hard copies of all or part of this work for personal or to understand how startups can use the power of analytics to make
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
data-informed decisions and illuminate their ways towards success.
on the first page. Copyrights for components of this work owned by others than ACM The following Research Question (RQ) is guiding our research: RQ:
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, How is analytics understood in the software startup context?
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org. The RQ turns our study exploratory. We utilized secondary data,
IWSiB’22 , May 18, 2022, Pittsburgh, PA, USA in the form of text documents, from analytics platforms for star-
© 2022 Association for Computing Machinery. tups, to answer the RQ. This study contributes to several important
ACM ISBN 978-1-4503-9302-7/22/05. . . $15.00
https://doi.org/10.1145/3524614.3528632 areas. It is the first step towards the understanding of analytics by
IWSiB’22 , May 18, 2022, Pittsburgh, PA, USA Usman Rafiq

software startups and eventually explains how do they leverage

their decision-making using analytics. Moreover, the findings iden-
tify important areas of product and business development where
analytics can be applied.

2 RELATED WORK
There are relatively few studies on analytics in the software startup
context. In recent work, Rafiq et. al [17] studied how software
startups deal with the information to make decisions from an ana-
lytics perspective. They reported ten types of recurrent analytics
mistakes that might derail software startups. The information deal-
ing mistakes are further grouped into four categories: information
collection, information analysis, information communication, and
information usage.
In another related study, while investigating the role of data
analytics in startup companies, Berg et al. [2] presented challenges
and barriers faced by startups. The study claims that startups are
aware of the benefits of applying analytics, however, they are also
facing challenges in implementing it. The reported challenges are
scarcity of resources, lack of training/skills, time management is-
sues, privacy issues in dealing with the data, and lack of access to
the data. It is further argued that the amount of data that startups
collect, in the early stages, is insufficient to apply analytics. We
find that this study analyzed hardware startups only and big data
analytics was brought into the question. This is done possibly be-
cause the term big data is considered as one of the key aspects of
business analytics, however, it differs from traditional data in terms
of velocity, volume, and variety [6].
On the other hand, we find some writers reporting on metrics
and measurements in startups. One such example is the multiple
case study of Kamulegeya et al. [11], where authors report 28 met- Figure 1: Data Collection and Analysis Procedure
rics that startups are utilizing or wish to utilize. The study [11]
classifies metrics list according to the measurement practices in
researcher with some other objective. Likewise, to analyze the data
established software companies and present five categories: busi-
set, we used content analysis, a qualitative research method [9]. A
ness metrics, product metrics, organizational performance metrics,
major advantage of using content analysis is that it is considered
project metrics, and design metrics.
an appropriate method in the absence of existing theory or when
Later, Kemell et al. [12], in their multi-vocal literature review,
the knowledge on the phenomenon is limited. In particular, we
report more than 100 metrics that startups can use. However, we
employed a conventional content analysis approach to build an
observed that most of these metrics are appropriate for the late
understanding of analytics in the startup context. Fig. 1 depicts the
stages in the life cycle of startups. Moreover, the study does not
complete process of data collection and analysis. The following
reflect on the information needed and the process to collect/track
subsections describe how we collected, filtered, and analyzed the
these metrics.
data.
Taken together, the studies presented, thus far, provide evidence
that there is a lack of understanding on how startups understand
the term analytics and apply analytics in the startup context. The
3.1 Data Collection
current study seeks to address this gap. Based on the research question, we planned to study analytics
platforms that are particularly intended for software startups. As a
preliminary step on getting information for platforms that startups
3 RESEARCH METHOD
are using., we consulted forums on Hacker News 1 . Hacker News
Our research is exploratory in nature as the knowledge on the is the most trusted, reliable, and prevalent source of news for the
understanding of software startups regarding analytics is limited. entrepreneurship and computer science community. It facilitates
Therefore, to develop an initial understanding of what software finding the latest information on startups which is hard to find
startups think about analytics, we decided to utilize secondary data elsewhere.
i.e. text data from analytics platforms. Nowadays, the widespread We applied search terms like "analytics", "metrics" and "startups"
usage of secondary data in research makes it a feasible approach to drive the search on the forum. These terms are originated from
to meet limited-time and resources [10]. Vartanian [20] describes
that secondary data is the data collected by someone other than the 1 https://news.ycombinator.com/
Towards Understanding Analytics in Software Startups IWSiB’22 , May 18, 2022, Pittsburgh, PA, USA

the research question of this research. Consequently, the manual We checked startup information from Crunchbase 4 . (5) Platform’s
analysis of the resulting discussions on forums revealed several ana- documentation contained data wherein at least one or multiple
lytics platforms. These platforms were brought up into a discussion stories were shared. These stories communicate experiences of
of the startup community, illustrating its features, usage scenarios, software startups while implementing analytics. We assessed this
and a short comparison. These platforms were ranged from open information from the collected documentation of the platform and
source to proprietary solutions and covered the following: later on checked individual startups on Crunchbase. In total, the
• Simple Analytics (https://simpleanalytics.com/) data contained 9 startup stories focusing on analytics, with varying
• Mixpanel (https://mixpanel.com/) text lengths.
• Google Analytics (https://analytics.google.com/)
• Matomo(https://matomo.org/) 3.2 Data Analysis
• Fathom (https://usefathom.com/) We used the qualitative analysis method, named, content analysis
• Plausible (https://plausible.io) to analyze the data. The use of content analysis in the software
• Open Web Analytics (https://www.openwebanalytics.com/) engineering context is not new to software engineering research.
• Kissmetrics (https://www.kissmetrics.io/) Recent studies, such as [7] and [13], advocate a rigorous process
• Amplitude (https://amplitude.com/) for its use in software engineering.
• Umami (https://umami.is/) Content analysis can be applied to the text data other than in-
• Goatcounter (https://www.goatcounter.com/) terview data and coding categories are directly derived from the
• Snowplow (https://snowplowanalytics.com/) data [9]. This is what makes it different from thematic analysis.
Thereafter, we visited individual platform websites, manually ex- Therefore, in the content analysis research method, text data can be
tracted the information, and stored it in text files. The informa- obtained from interviews, focus groups, narrative responses, survey
tion stemmed from several sources, like for example, associated questions, books, articles, or even manuals [9]. In particular, we
documentation, configuration information, developer information applied conventional content analysis inductively. This approach
in case of an open-source tool, user interface, testimonials of cus- is recommended when there is a lack of research knowledge on
tomers, and blog posts. We also found blogs of some of the platforms the topic [9]. While performing analysis, we read the whole text
where the vendors expressed their opinions about analytics and repeatedly to get a complete sense of the data. We then started
shared their experiences in instrumenting analytics at other startup coding phrases wherein the term analytics was indicated or the
companies. We read each of the available documentation and as- impression of analytics was arising. We highlighted the text that
sessed whether the data can be utilized to address the research appeared to contain the relevant terms and described the concepts.
question. It is noteworthy to state that we utilized every possible Subsequently, we utilized the author’s words to code this segment
way to extract the fragmented documentation associated with these of the text. Alongside, we ignored the summative indication of
platforms with an intent not to miss some useful information. To analytics in the text whereby the text was mainly referring to the
achieve this, we searched platform-specific documentation on the names of analytics platforms. Meanwhile, we avoided creating new
platform website, Hacker News forum and Github 2 , a repository codes as the analysis process continued. It further depicts that we
hosting service of open source applications. In this regard, we used only added new codes when we assessed that the existing codes
the platform’s hosting documentation on Github, known as a wiki are not fit to label the new text under consideration.
to extract some more information. Likewise, soon after finishing the development of codes, we then
To obtain another triangulation in the data collection and sam- moved the focus of our analysis to the coded data and examined
pling phase, we explored the website of G2 3 . G2 is a legit source each code. We eventually examined the text within the particular
to check the reviews of business software by real users. We found code and then merged, renamed, and eliminated codes while the
some of the platforms available on G2, however, few platforms were analysis process iterated. Based on the similarity in the codes and
not listed on it. This has helped us in screening analytics platforms the relationship between the codes, we created the clusters, even-
for the final data set. Lastly, we decided to utilize the information tually organizing clusters into a hierarchical structure. As a result,
from the following three platforms for our data set: we obtained 16 themes that were further clustered into 4 high-level
(1) Amplitude (https://amplitude.com/) themes/top-level categories.
(2) Mixpanel (https://mixpanel.com/) We used NVIVO 5 , a qualitative data analysis tool, to support
(3) Plausible (https://plausible.io) the conventional content analysis.
We considered the following inclusion/exclusion criteria while
screening the platform information for our final data set: (1) Plat- 4 FINDINGS
form is mature i.e. used by a huge startup community. We subjec- In this section, we present our research results. We identified 14
tively assessed these criteria from Hacker News forum and objec- themes, in our data set, that directly express or found more rele-
tively from stargazer on Github website (2) Platform is available on vant to the analytics process inside software startups. We catego-
G2 website, (3) Platform is having extensive documentation, from rized these themes into 4 high-level themes. The high-level themes
multiple sources like, for instance, website, blog posts and informa- include instrumenting, experimentation, diagnostic analysis, and
tion from forums. (4) Platform’s vendor is/was a software startup.
2 https://github.com/ 4 https://www.crunchbase.com/discover/organization.companies
3 https://www.g2.com/ 5 https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home/
IWSiB’22 , May 18, 2022, Pittsburgh, PA, USA Usman Rafiq

getting insights. The clustering of themes and labeling into high- often have a multitude of questions about their customers and how
level is purely inductive and thus based on two factors. The first their product is performing“.
one focuses on how to set up analytics in startups and we name Based on the data set, we encountered the following set of ques-
it instrumenting analytics. What we mean by instrumentation of tions that can be asked while instrumenting analytics:What exactly
analytics is the intuition and directions to set up analytics in star- works and what does not?, What do we want to be measuring?,
tups. On the flip side, three themes coexist together and focus what What’s the useful metric?, Is there an event we should be publish-
are the possible scenarios of applying analytics in startups. Here, ing?, Which features are popular?, Which users retain best?, which
we include experimentation, diagnostic analysis and finally getting types of users stick around, and for how long Who are my most
insights. These themes, when taken together, illustrate how the valuable customers?, What actions do those valuable customers
term analytics is understood by software startups. Fig. 2 shows how take?, Who are the customers who churn?, How many power users
software startups handle analytics. you have, What actions do churn users take?, What are the charac-
teristics of highly engaged users?, What are my users’ pain points?,
4.1 Instrumenting Analytics: What’s happening on your site live in that moment, What are peo-
4.1.1 Defining Goals: We figured out that all of the startups, em- ple doing when they visit the website? and What’s happening and
ploying analytics, have the sense of defining one or multiple goals how things are going?
to achieve, through analytics. Often analytics platform vendors also
found promoting culture to establish the goals to meet the business 4.1.3 Metrics As a Culture: We found several interesting examples
objectives. As an example, in the report of Amplitude platform (1), in the data where one startup repeatedly emphasized embedding
the concept is explained like this: “ Too often, companies develop metrics in the overall culture of the product development process.
feature after feature without thinking about how those features meet It means that no product feature should be left without publishing
overall business objectives.“. Therefore, these goals might be tied different metrics. It is aimed at serving the purpose of measuring
directly to the success of the startup. This further illustrates the and working towards large business goals. The same startup, said
thinking of measuring what matters the most and avoiding get- on one occasion, in the report of Mixpanel Platform (2): “Publishing
ting indulged in the noise. It is evident from one of the following the right business metrics must be integrated into your development
excerpts, of Amplitude Platform (1): “Measuring anything and ev- process“. The speaker, continued: “Baking this into the culture and
erything leads to unmanageable data. It shifts the burden to your the expectation is probably the most important and most impactful
team to try and make sense of the results.“. In another instance, the thing that you can do“. He further advised startups: “build it into the
company advised in a similar context:“keep your end goals in mind“. culture and build it into the development process so that it becomes
Going in the same vein, one of the startups, wanted their clients a given“. While establishing this culture, it is necessary that all
to hit on the platform. The goal was to increase customer retention. product platforms i.e. web and mobile, should be aligned and the
The possible ways they found against the goal were to reduce same vocabulary should be put in place across the development
churn rate and increase customer conversion. Churn rate is the life cycle. This alludes to the following notion: “Ensure that all your
rate at which customers discontinue using a service or product and stakeholders–both in web and mobile–are aligned on how to name
conversion means how new customers take a desired action [5]. In those events and properties. “. A similar conclusion is highlighted
the data of Mixpanel Platform (2), a startup described the situation in the report of Amplitude Platform (1), where the company said:
in the following words:“For each market, they have targets they want “They align on target product outcomes, define an event taxonomy to
partners to hit, so they know they are reducing churn and increasing measure those outcomes, and instrument tracking code“.
conversion“. A similar conclusion is echoed in another report of
Plausible Analytics Platform (3), where the company expressed its 4.1.4 Tracking Goals: We observed that another theme in the data
thoughts about setting goals: “Most web analytics allow site owners is usually concerned with tracking the goals. This theme is con-
to set goals and events to track those visitor actions that matter the cerned with tracking established goals. For example, a startup set
most to them.“. It is worthwhile to report that these established up a goal and wanted partners to grow. In contrast to this goal,
goals generate several metrics to monitor. The startup reflected the startup asked many questions and then tracked metrics. This is
this opinion in the following excerpt: “goals and events can be tied shown in the following excerpt of a startup, in Mixpanel Platform
directly to the monetary success of an enterprise which makes them (2) report: “... track how well their restaurant partners are doing on
essential metrics to follow to understand the state of the business“. the platform“. “and being able to look at these metrics is what’s going
to enable us to do that“, the startup reported. Another startup, in
4.1.2 Asking Questions: The data analysis, on the whole, suggests the same report, had a similar goal and wanted to know the most
that asking questions is a good way to establish the key focus of performing customer acquisition channels and increase customer
analytics, revealing what one wants to achieve through it. And, cer- conversion. The startup reflects its tracking of this goal in the fol-
tainly, this has a connection with setting goals while instrumenting lowing words: “...been able to measure and optimize high-performing
analytics in the startup. In the report of Mixpanel Platform (2), one acquisition channels, and also improve purchase funnels, making it
startup emphasized it in the following way: “We need to be asking easy for consumers to convert into loyal policyholders“. A similar
these questions“. The same report embarked again: “We want people indication about tracking goals about customer conversion is found
to engage with data, ask questions, and find the answers in data to in the report of Plausible Analytics Platform (3), where the startup
make the right decisions“. Likewise, Amplitude Platform (1) data, indicated to track several metrics against goals: “Track events and
reported a similar conclusion in the following excerpt: “Businesses goals to identify the number of unique converted visitors, the total
Towards Understanding Analytics in Software Startups IWSiB’22 , May 18, 2022, Pittsburgh, PA, USA

Figure 2: How Software Startups Understand Analytics

number of conversions, the conversion rate, and the referral sites that Finally, make certain that everyone is bought in on maintaining best
send the traffic that converts the best.“. practices“.

4.1.5 Unifying Analytics: Unifying Analytics is one of the chal-

lenges that startups must tackle while instrumenting analytics. 4.2 Experimentation
Based on the findings, we highlight two types of unification, a 4.2.1 Testing Hypotheses: A recurrent theme in the data was the
startup can achieve. The first type incorporates unifying analyt- utilization of analytics in testing hypotheses. Startups need to
ics on all of the fragmented products of the startup. Oftentimes, a quickly test several hypotheses. Various startup development ap-
startup offers multiple platforms, such as a combination of web and proaches like, for example, lean startup and customer development
mobile app. This is echoed by one of the startups, in the report of methodology are amplifying the use of Hypotheses, particularly at
Mixpanel Platform (2): “the company needed to unify its fragmented early stages [4, 18]. We found startups using analytics to test the
analytics ecosystem“. hypotheses. Simultaneously, analytics platforms are providing sup-
On the flip side, the other aspect of unification discussed in the porting features to achieve this goal. For example, the Amplitude
data is the shared understanding and goal alignment among differ- Platform (1) report indicated such features. Identifying the Winning
ent startup teams, like, for instance, between the engineering team features, customer acquisition campaigns, referral sources, and hy-
and product team. The shared understanding and alignment might potheses regarding customer behavior are prominent in the data.
include target product outcomes, taxonomy to measure outcomes, At one instance, the report of Amplitude Platform (1), indicated:“Its
and instrumenting coding accordingly. The report of Amplitude growth engine generates hypotheses data by observing customer be-
Platform (1) shows: “Implementing great product analytics requires havior while also amplifying winning features and campaign ideas“.
product and engineering teams to work together. They align on tar- We encountered many instances in the data where this feature
get product outcomes, define an event taxonomy to measure those was highlighted. It shows that the concerns regarding hypotheses
outcomes, and instrument tracking code“. This enables the engineer- testing through analytics are widespread among startups. This is
ing team to track only what is required by the product team. The evident from the following excerpt of a startup in the report of Mix-
company, Mixpanel Platform (2), agrees with such alignment in panel Platform (2): “We wanted to test this hypothesis, so we quickly
the following words: “Ensure that all your stakeholders–both in web threw events“. The startup continued alluding about the outcome
and mobile–are aligned on how to name those events and properties. of this testing: “ we were able to test and disprove our hypothesis of
IWSiB’22 , May 18, 2022, Pittsburgh, PA, USA Usman Rafiq

it being related to payday”“. The testing of the hypotheses saved realized a massive increase in the customer activation rates. A deep
the startup from investing more effort and time. The startup com- dig down through the analytics platform revealed pop-up removal
mented:“This finding was helpful... because it showed them that, since as the main reason for this. In the report of Amplitude Platform
restaurants were already using the feature, perhaps they didn’t need (1), the startup explained this event in the following words: “We
to invest more in it“. talk about growth and retention as “experiment-driven” processes so
much that we sometimes lose sight of the fact that many discover-
4.2.2 Conducting A/B Testing: In the data, at various places, star- ies occur by chance“. The startup regarded customer growth as an
tups expressed the use of analytics in conducting A/B testing. They experiment-driven process, as they also planned to test it in the
illustrated the benefits, this testing has brought to them and saved same way. Now, in a retrospect, they sense that although it was an
them in several ways. We found two examples where startups mea- accident, however, without the analytics the root cause cannot be
sured and followed the results of A/B testing through the use of exactly known. The startup explains this in the following excerpt:
analytics. Mainly, A/B tests were conducted to test the focus of the “They can only happen, however, if you’ve been tracking the data you
product’s features. At the same time, one startup instrumented the need to work back to the root cause later“. They continued sharing
process of analytics in their environment because they needed to their pleasant accident:“Without data to look back on, this whole
clarify and learn from A/B tests. The startup reports this finding in affair would have been regarded as a massive error “.
the report of Mixpanel Platform (2) using the following words: “The
company required a tool that could quickly clarify and streamline the
learnings from A/B testing“. The same startup continued expressing 4.3 Diagnostic Analysis
its interests in A/B testing measurements through analytics and 4.3.1 Uncovering Cause: The data analysis revealed that software
said that this helped them to improve account creation using A/B startups are utilizing analytics to diagnose a strange or unexpected
testing. It is expressed in the following excerpt:“monitor an A/B cause of a phenomenon. The strange situations may include sudden
test to see if it could increase another important metric: New Signer spikes in application traffic at particular times or a sudden decrease
Accounts“. in customer conversion rate. Customer conversion rate is the per-
While, on the other hand, in the Amplitude Platform (1) report, centage of users who take the desired action on the product[5]. We
a startup argued that analytics helps in A/B testing regardless of found 3 instances in the reports of two analytics platforms, where
the exclamation points the test generates. The startup commented: software startups reported their experience in finding the cause
“Analytics are for more complex than A/B testing whether two ex- of a strange phenomenon just because they were already into the
clamation points or three generate more opens“. The startup further analytics. For instance, we found that one of the startup customers
commented: “Analytics helped ... his team shift their entire messaging of Amplitude Platform (1) , noticed an unusual yet recurring trend
strategy toward what their users valued“. of a spike in traffic. They were new to applying analytics but they
4.2.3 Finding Product-Market Fit: A common view expressed in managed to sort out the reason eventually. And they assessed that
all the reports is the use of analytics to truly understand the target it is the start of the weekend and the traffic is spiking every Monday
customers and move towards the development of the right product or on Tuesday. The startup figured out, with the help of analytics,
for them. This means that startups can assess analytically whether that their partners were looking at their feedback, sales of last week,
their business model is flawed or has the potential to grow fast. In changes to opening hours or menus, at the beginning of every week.
several instances, in the data of Amplitude Platform (1), we encoun- The startup explained the diagnosis:“The company continued to col-
tered the company standing out in explaining the similar benefits. lect a month’s worth of data and realized that DAU was spiking every
For instance, at one place, the company reported:“analytics is the Monday—or, in the case of a long weekend—Tuesday. It’s the start of
epicenter of how digital-first companies figure out customer needs the work week and of course our partners are logging in“. Here, DAU
and measure the impact of their products“. It continued expressing is the abbreviation for daily active users.
significant use of analytics in the product-market fit for startups: A similar situation is reported by another startup at Mixpanel
“ To improve your odds of finding product-market fit, you need an platform (2). The startup experienced a sudden drop in the conver-
analytics platform that can go past reporting vanity metrics and basic sion rate and later it was found that a minor change on the home
conversion funnels to understanding the experience customers are page has caused this drop. The startup also utilized analytics in
having in your product“. In the same way, another company, Plau- finding the cause. According to the Mixpanel (2), “If you weren’t pay-
sible Analytics (3) also reported building an understanding of the ing close attention to your analytics, you would just wind up sitting
product benefits. The company alluded to the view in the following around and wondering why your conversion rate had dropped“.
words: “Understanding the interest around your product’s benefits..“.

4.2.4 Growing Customer Base: We find in our analysis that, a grow- 4.4 Getting Insights:
ing customer base is regarded as the experiment-driven process, 4.4.1 Assessing Efforts: In this theme, startups obtained insights
whereby different options are built and tested. We report an in- on the efforts that they put in terms of introducing new product
teresting event when a startup aimed to remove pop-ups from its features, making changes in the product layout, and hiring variant
application to increase the customer activation rates. After three customer acquisition as well as marketing channels. They want
months of onboarding, they accidentally removed the pop-ups even to observe which of their efforts are more result-oriented so that
when they were not ready to embrace this change. The accident they can shift all their focus to the identified direction. This new
went unnoticed for two weeks by the startup and suddenly they direction leads the future strategy of the startup and brings better
Towards Understanding Analytics in Software Startups IWSiB’22 , May 18, 2022, Pittsburgh, PA, USA

understanding the customer needs. As, one startup, Plausible An- 4.4.4 Understanding User Behaviour and Engagement: One possi-
alytics (3) indicated:“You may be putting a lot of time, effort, and ble use of analytics, our data analysis showed, is understanding
resources into different marketing campaigns and by looking at refer- user behavior and interaction with the product. It is used to clear
ral sources of your website traffic you can better understand which of the assumptions regarding the customer’s behavior. At the same
those campaigns are more worth than others“. The startup continued time, it brings to the surface, trending content, winning product
arguing that one needs to redesign its future strategy if the existing features, and highlights the customer journey from one platform to
efforts are misleading. “If you’re spending a lot of time and effort on another. Several analytics goals can be achieved through this, like,
a community but that effort doesn’t result in any benefits to your site for example, improving retention by monitoring and understanding
or business, then you need to reconsider things“, the startup remarked user behavior. The report of Amplitude Platform (1), highlights
on the situation. While talking about the future strategy, the startup this finding in the following words: “Identifying which customers
asserted to analyze efforts to optimize the future. It is reported in are most engaged and using this information to improve retention “.
the following phrase: “an addition layer and more depth to analyze The company continued expressing similar and concrete benefits at
your efforts which can then help you optimize your strategy for the another place in the following excerpt: “analytics helps you create
future“. that digital experience without any guesswork. It provides you with
A similar experience is shared in the report of Amplitude Plat- concrete information to optimize conversions, grow retention, and
form (1), where the company highlighted the need for analytics to maximize revenue. “.
deeply understand the customer needs and assess the outcomes Another interesting finding, we noticed, is finding reasons of
of development efforts. The statement reflects this finding: “The a user behaviour. “Good product teams don’t just ask, what are my
one thing we found missing was a powerful product platform to truly users doing? They ask, why? This requires context“, the report of
understand what users wanted and the impact our development efforts Amplitude Platform (1), revealed.
were having on their user experience“.

4.4.2 Measuring Progress: Savvy startups are often concerned with 5 DISCUSSION
measuring their progress. However, measuring the progress without Our study results confirm that analytics in the startup context
any yardstick is barely possible. This is what we pointed out while reflects a somewhat different meaning in contrast to what is de-
analyzing the data. We found that startups are using analytics to scribed in the software engineering literature. Therefore, when we
measure their progress using insights from different perspectives. talk about startups, analytics is not based on the big data as it is
The startup, Plausible Analytics (3), told in their report: “we use web indicated in [2, 14], nor it is characterized by what is given in terms
analytics to measure our startup’s progress and make better decisions“. of software analytics [8]. However, we find that while there exist
Generalizing this need, the startup claimed: “majority of website and many types of analytics in the literature, e.g. big data analytics, web
business owners want to see some level of stats that tells them what’s analytics, software analytics, and social media analytics, startup
going on “. The company further highlighted the key advantages of analytics is closer to the characteristics of web analytics. Along-
using analytics in measuring progress, in following excerpt: “some side, we need further investigation to ground the understanding of
of the main areas where web analytics can help website owners get a analytics in the startup context.
better idea of what’s happening and how things are going “. The study [17] also reported a few other analytics-related mis-
takes, e.g. “avoiding to collect further information“, “poor team com-
4.4.3 Making Decisions: The data analysis depicts that, startup munication“ and “mishandling information“. Our data shows that
decisions are based on the insights that the analytics generates for the first two mistakes can be mitigated by instrumenting analyt-
them. These insights oftentimes arise by measuring the progress of ics in startups. For instance, unifying analytics will reduce poor
startup, by assessing the startup efforts in making things done, or communication in the startup team. Similarly, for the last one, the
during the experimentation, however, these are considered holis- findings suggest looking at information from multiple perspectives
tically while making decisions. For instance, Plausible Analytics and also waiting for trends to appear in the information.
(3) claimed that they used analytics in making decisions. These Surprisingly, one of our findings emphasizes developing a cul-
decisions further lay down the foundation of future strategy. It is ture of metrics and embedding it into the development of product
apparent from the following excerpt: “... more depth to analyze your features. This invites the concept of measuring everything, espe-
efforts which can then help you optimize your strategy for the future cially in product development. A possible explanation for this result
“. In the same way, a startup, in the report of Mixpanel Platform might be that startups could need to produce hindsight. Therefore,
(2), shared that they decided not to put more effort into design- in a retrospect, they might need to do this. However, we find no
ing features when they observed that their partners are already clue on this finding from literature, and our study also remains
engaged with the product. The startup commented: “This finding unable to explain it to a further extent.
was helpful ... because it showed them that, since restaurants were In contrast to the earlier findings of [2], however, we confirm
already using the feature, perhaps they didn’t need to invest more that software startups are using analytics. This difference may
in it.“. The company Amplitude Platform (1) sheds more light on be explained by our chosen sample. We studied purely software
making decisions employing analytics in these words: “Your product startups while Berg et al. [2] studied startups with a hardware part
analytics can provide you with the exact data needed to drive your as well. Another important difference, we would like to illustrate
decision-making and continue to make improvements that delight is that our study focused on the understanding of startups about
your customers. “. analytics. We find that, when taken together, this understanding is
IWSiB’22 , May 18, 2022, Pittsburgh, PA, USA Usman Rafiq

pretty different from what general software engineering literature startups with the primary data. Moreover, the current findings have
states about analytics. The literature [8, 14, 16], presents analytics left many questions in need of further investigation. For example,
in terms of big data analytics, software analytics, or simply data further work is required to map the relationship of analytics with
analytics. It is interesting to reveal that our findings provide a the term metrics. As both these terms are used interchangeably.
diverse understanding of analytics from a startup’s perspective. Lastly, the relationship between stages of startup across the life-
However, it would be interesting to relate our findings with general cycle and use of analytics might be worth investigating.
software engineering literature. This would remain an important
issue for future avenues. REFERENCES
Lastly, the findings strongly show a connection of analytics with [1] Florian Auer, Chhong Shing Lee, and Michael Felderer. 2020. Continuous experi-
ment definition characteristics. In 2020 46th Euromicro Conference on Software
metrics. For example, goals and questions during analytics instru- Engineering and Advanced Applications (SEAA). IEEE, 186–190.
mentation generate metrics. However, our findings are not self- [2] Vebjørn Berg, Jørgen Birkeland, Ilias O Pappas, and Letizia Jaccheri. 2018. The
explanatory in explaining this connection and further assessment Role of Data Analytics in Startup Companies: Exploring Challenges and Barriers.
In Conference on e-Business, e-Services and e-Society. Springer, 205–216.
of the relationship between analytics and metrics. [3] Steve Blank. 2020. The four steps to the epiphany: successful strategies for products
that win. John Wiley & Sons.
[4] Jan Bosch, Helena Holmström Olsson, Jens Björk, and Jens Ljungblad. 2013. The
5.1 Threats to Validity early stage software startup development model: a framework for operational-
izing lean principles in software startups. In International Conference on Lean
One of the threats to validity, particularly, internal validity, is the Enterprise Software and Systems. Springer, 1–15.
use of secondary data. This type of data provides a lack of control [5] Alistair Croll and Benjamin Yoskovitz. 2013. Lean analytics: Use data to build a
on data volume and quality. To mitigate this threat, to a certain better startup faster. " O’Reilly Media, Inc.".
[6] Charles K Davis. 2014. Beyond data and analysis. Commun. ACM 57, 6 (2014),
extent, we triangulated data using multiple sources and by applying 39–41.
inclusion/exclusion criteria. This type of data has already been used [7] Joanna F DeFranco and Phillip A Laplante. 2017. A content analysis process for
by other studies, like for example, by [1]. qualitative software engineering research. Innovations in Systems and Software
Engineering 13, 2 (2017), 129–141.
Regarding threats to external validity, one of the threats lies [8] Latifa Guerrouj, Olga Baysal, David Lo, and Foutse Khomh. 2016. Software
in our data collection strategy. We selected 3 platforms out of 12 analytics: challenges and opportunities. In 2016 IEEE/ACM 38th International
Conference on Software Engineering Companion (ICSE-C). IEEE, 902–903.
for further investigation by applying several inclusion/exclusion [9] Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three approaches to qualitative
criteria. Later, we collected analytics platform documentation from content analysis. Qualitative health research 15, 9 (2005), 1277–1288.
every possible source e.g. platform website, blogs, and forums. The [10] Melissa P Johnston. 2017. Secondary data analysis: A method of which the time
has come. Qualitative and quantitative methods in libraries 3, 3 (2017), 619–626.
data, from the three platforms, was different in length and scope. [11] Grace Kamulegeya, Raymond Mugwanya, and Regina Hebig. 2018. Measurements
Besides that, we might have missed collecting some associated in the early stage software start-ups: a multiple case study in a nascent ecosystem.
data. However, the data set was containing startups from differ- Foundations of Computing and Decision Sciences 43, 4 (2018), 251–280.
[12] Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen-Duc, Jason Grendus, Tuure
ent geographical regions, market segments, and different product Tuunanen, and Pekka Abrahamsson. 2019. 100+ Metrics for Software Startups-A
platforms. It enhanced the ability to generalize the results of this Multi-Vocal Literature Review. arXiv preprint arXiv:1901.04819 (2019).
[13] Zijad Kurtanović and Walid Maalej. 2018. On user rationale in software engi-
ongoing research effort. neering. Requirements Engineering 23, 3 (2018), 357–379.
Regarding the reliability of this study, one possible threat is [14] Patrick Mikalef, Ilias O Pappas, John Krogstie, and Michail Giannakos. 2018. Big
concerned with the researcher’s bias in the coding process. The data analytics capabilities: a systematic literature review and research agenda.
Information Systems and e-Business Management 16, 3 (2018), 547–578.
coding process was done by one researcher solely, however, the [15] Jevgenija Pantiuchina, Marco Mondini, Dron Khanna, Xiaofeng Wang, and Pekka
early codes and themes were discussed with the other researcher Abrahamsson. 2017. Are software startups applying agile practices? The state of
before reporting. the practice from a large survey. In International Conference on Agile Software
Development. Springer, Cham, 167–183.
Lastly, we take the privilege to admit that the reported under- [16] Ilias O Pappas, Patrick Mikalef, Michail N Giannakos, John Krogstie, and George
standing of analytics, in the startup context, still needs further Lekakos. 2018. Big data and business analytics ecosystems: paving the way
towards digital transformation and sustainable societies. Information Systems
investigation, possibly with the primary data. It further means that and e-Business Management 16, 3 (2018), 479–491.
we need to continue grounding and validating our understanding [17] Usman Rafiq, Jorge Melegati, Dron Khanna, Eduardo Guerra, and Xiaofeng Wang.
of analytics in the startup context based on the primary data. 2021. Analytics Mistakes that Derail Software Startups. In Evaluation and
Assessment in Software Engineering. 60–69.
[18] Eric Ries. 2011. The lean startup: How today’s entrepreneurs use continuous inno-
6 CONCLUSIONS AND FUTURE WORK vation to create radically successful businesses. Currency.
[19] Michael Unterkalmsteiner, Pekka Abrahamsson, Xiaofeng Wang, Anh Nguyen-
Startups are confronted with several challenges while raising the Duc, Syed Shah, Sohaib Shahid Bajwa, Guido H Baltes, Kieran Conboy, Eoin
odds of success. Uncertainty, scarcity of resources, engineering Cullina, Denis Dennehy, et al. 2016. Software startups–a research agenda. e-
Informatica Software Engineering Journal 10, 1 (2016).
challenges, speed, and right focus are among few to mention. On [20] Thomas P Vartanian. 2010. Secondary data analysis. Oxford University Press.
their way, startups have to take plenty of decisions and adjust
directions accordingly. This is where analytics can serve them with
the right information. This is the first study that attempts to explain
how analytics is understood in the startup context. In this ongoing
research, we report an initial understanding of analytics in the
startups covering its instrumentation and the context, in which it
can be utilized.
Several questions still need to be answered in the future. One
significant direction is to validate this understanding of analytics in