Degree Project in the Main Field of Technology Embedded Systems and the Main

Field of Study Computer Science and Engineering

Second cycle, 30 credits

Continuous Integration for

Embedded Software with Modular
Firmware Architecture
FABIAN SEGATZ

Stockholm, Sweden 2023

Continuous Integration for
Embedded Software with
Modular Firmware
Architecture

FABIAN SEGATZ

Master’s Programme, Embedded Systems, 120 credits

Date: November 7, 2023

Supervisors: Han Fu, Gunnar Elgcrona

Examiner: Cyrille Artho
School of Electrical Engineering and Computer Science
Host company: Cobolt AB
Swedish title: Kontinuerlig Integration för Inbäddad Programvara
med Modulär Firmware-Arkitektur
© 2023 Fabian Segatz
 Abstract | i

Abstract
Continuous Integration (CI) techniques are widely adopted in web and ap-
plication development but have received limited attention in the embedded
software domain. This thesis investigates the application of CI techniques
in embedded software development through a case study at Cobolt AB, a
company specializing in optoelectronics. The study aims to identify suitable
CI techniques, assess implementation efforts, and evaluate the impact of CI
adoption in this domain.
A CI service is implemented using Jenkins as the automation server,
following an iterative development and deployment process. The service
incorporates multi-target compilation, automated unit testing, test reporting,
visual CI feedback, and trunk-based development. These techniques prove ef-
fective for embedded software with a modular firmware architecture. However,
automated system testing encounters limitations due to the need for manual
interaction with hardware targets.
Challenges encountered during implementation, such as learning CI tools,
managing build tool dependencies, and addressing manual input requirements
for system testing, are overcome through iterative implementation, distributed
build architecture, and selective test automation. Developers’ resistance to CI
adoption diminishes as they experience the positive impact of the CI service.
CI adoption in the embedded domain brings benefits such as fast bug
detection, increased developer motivation, higher confidence in code quality,
and encouragement for standardization n. No noticeable negative impacts are
observed.
Future research should focus on integrating hardware-in-the-loop simula-
tion systems for comprehensive automated system testing, exploring validation
on multiple hardware targets, and studying the vertical scaling capabilities of
distributed build architectures with Jenkins.

Keywords
Continuous integration, Embedded software development, Modular firmware
architecture, Jenkins, Software quality
ii | Abstract
 Sammanfattning | iii

Sammanfattning
Kontinuerlig integration (CI) tekniker används i stor utsträckning inom webb-
och applikationsutveckling, men har fått begränsad uppmärksamhet inom
inbyggd programvarudomän. Denna avhandling undersöker tillämpningen av
CI-tekniker inom inbyggd programvaruutveckling genom en fallstudie vid
Cobolt AB, ett företag specialiserat på optoelektronik. Studien syftar till
att identifiera lämpliga CI-tekniker, bedöma implementeringsinsatser och
utvärdera effekten av CI-användning inom detta område.
En CI-tjänst implementeras med Jenkins som automatiseringsserver, ef-
ter en iterativ utvecklings- och distribueringsprocess. Tjänsten inkluderar
kompilering för flera målenheter, automatiserad enhetstestning, testrapporte-
ring, visuell CI-återkoppling och utveckling baserad på huvudgrenen. Dessa
tekniker visar sig vara effektiva för inbyggd programvara med en modulär
firmware-arkitektur. Dock begränsas automatiserad systemtestning av behovet
av manuell interaktion med hårdvarumål.
Utmaningar som uppstår under implementeringen, såsom att lära sig CI-
verktyg, hantera byggverktygsberoenden och hantera manuella indatakrav
för systemtestning, övervinner genom iterativ implementering, distribuerade
byggarkitekturer och selektiv testautomatisering. Utvecklarnas motstånd mot
CI-användning minskar när de upplever de positiva effekterna av CI-tjänsten.
CI-användning inom inbyggd programvaruutveckling medför fördelar som
snabb upptäckt av fel, ökad utvecklar motivation, högre förtroende för kod-
kvalitet och främjande av standardisering. Inga märkbara negativa effekter
observeras.
Framtida forskning bör fokusera på att integrera hårdvaru-i-loop simule-
ring för omfattande automatiserad systemtestning, utforska validering på flera
hårdvarumål och studera de vertikala skalningsmöjligheterna hos distribuera-
de byggarkitekturer med Jenkins.

Nyckelord
Kontinuerlig integration, Inbyggd programvaruutveckling, Modulär fastvara
arkitektur, Jenkins, Programvarukvalitet
iv | Sammanfattning
 Acknowledgments | v

Acknowledgments
I want to express my gratitude to the following individuals whose support and
contributions have been invaluable in the completion of my thesis:
First and foremost, I would like to thank Cyrille, my examiner, for
providing exceptional feedback and asking the right questions during the
project planing phase in December. Your expertise and insightful guidance
have significantly improved the quality of my research. I am grateful for the
opportunity you gave me to explore this subject matter and for your belief in
my abilities.
I am immensely thankful to Han, my academic supervisor, for our regular
meetings. Your dedication, time, and extensive ideas about the direction of
my research have been instrumental in shaping this thesis. Your guidance
and support have been invaluable throughout the entire process. Additionally,
your effective communication has fostered a productive working relationship,
enabling me to navigate through challenges and make significant progress.
To Lukas and Gunnar, my company supervisors, I extend my sincerest
appreciation for trusting me with the project idea and providing support
during its execution. It has been uplifting to feel that my project is genuinely
appreciated. Your guidance and encouragement have motivated me to push
beyond my limits, and I am grateful for the opportunity to work with such
supportive mentors.
I would like to acknowledge the contributions of Per, my colleague. Your
assistance and insightful discussions regarding the build environment have
been immensely helpful. Our lunch talks have provided me with a broader
perspective and helped me overcome various obstacles. Your support and
guidance have made a significant difference, and I am thankful for your
presence throughout this journey.
To my mom and dad, I cannot thank you enough for everything you have
done. Your unwavering love, support, and encouragement have been my
constant source of strength. Your belief in my abilities has been a driving
force behind my achievements, and I am forever grateful for your presence in
my life.
Finally, I also want to thank my Stockholm family, including Marta,
Cate, Lucas, Anders, and Laurène. I am grateful for our thought-provoking
discussions and Friday co-working sessions.

Stockholm, November 2023

Fabian Segatz
vi | Acknowledgments
 CONTENTS | vii

Contents

1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Research Methodology . . . . . . . . . . . . . . . . . . . . . 4
1.6 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.6.1 Focus on Test Execution . . . . . . . . . . . . . . . . 4
1.6.2 Short Term Case Study on CI Service Impact . . . . . 5
1.7 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . 5

2 Background 7
2.1 Embedded Software Development . . . . . . . . . . . . . . . 7
2.1.1 Characteristics of Embedded Software . . . . . . . . . 8
2.1.2 Modular Firmware Architecture . . . . . . . . . . . . 8
2.1.3 Source Code Management . . . . . . . . . . . . . . . 9
2.1.4 Build Scripts . . . . . . . . . . . . . . . . . . . . . . 11
2.1.5 Build Automation . . . . . . . . . . . . . . . . . . . . 12
2.1.6 Continuous Integration . . . . . . . . . . . . . . . . . 12
2.1.7 Automated Software Testing . . . . . . . . . . . . . . 13
2.2 Cobolt’s Development Environment . . . . . . . . . . . . . . 14
2.2.1 Firmware Architecture — Lysa . . . . . . . . . . . . . 15
2.2.2 Hardware Target — Laser . . . . . . . . . . . . . . . 16
2.2.3 Build System Generator — CMake . . . . . . . . . . 16
2.2.4 Compiler — IAR C/C++ Compiler for ARM . . . . . 16
2.2.5 IDE — IAR Embedded Workbench . . . . . . . . . . 17
2.2.6 Version Control System — Git . . . . . . . . . . . . . 17
2.2.7 Unit Testing Tool — CxxTest . . . . . . . . . . . . . . 17
2.2.8 System Testing Tool — LATE . . . . . . . . . . . . . 18
viii | CONTENTS

2.3 Additional Tools . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3.1 Automation Server — Jenkins . . . . . . . . . . . . . 19
2.3.2 Build System — IAR Build Tool . . . . . . . . . . . . 20
2.3.3 Firmware Download — J-Link Commander . . . . . . 21
2.4 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.1 CI Services in Embedded Software Development . . . 21
2.4.2 Challenges for CI Adaption in the Embedded Domain . 22
2.4.3 Practices for CI Adoption in the Embedded Domain . . 23
2.4.4 Experiencing Benefits of CI in Industry . . . . . . . . 24

3 Methodology 25
3.1 Agile Research Process . . . . . . . . . . . . . . . . . . . . . 26
3.2 Experiment Design . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.1 Single-case Study . . . . . . . . . . . . . . . . . . . . 28
3.2.2 Iterative Implementation Plan . . . . . . . . . . . . . 28
3.2.3 Hardware to be used . . . . . . . . . . . . . . . . . . 29
3.3 Interpretive Research Paradigm . . . . . . . . . . . . . . . . . 30
3.4 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.4.1 Implementation Notebook . . . . . . . . . . . . . . . 31
3.4.2 User Feedback . . . . . . . . . . . . . . . . . . . . . 31
3.4.3 Final Survey . . . . . . . . . . . . . . . . . . . . . . 32
3.4.4 CI Metrics . . . . . . . . . . . . . . . . . . . . . . . 32
3.5 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.5.1 Stage Status . . . . . . . . . . . . . . . . . . . . . . . 34
3.5.2 Stage Duration . . . . . . . . . . . . . . . . . . . . . 35
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4 Implementation 37
4.1 Local Jenkins Setup and First Test . . . . . . . . . . . . . . . 37
4.2 Declarative Pipeline for Parallel Firmware Compilation . . . . 38
4.3 Add Unit Testing Stage . . . . . . . . . . . . . . . . . . . . . 38
4.4 Deploying CI Service . . . . . . . . . . . . . . . . . . . . . . 39
4.5 Firmware Download to Target . . . . . . . . . . . . . . . . . 41
4.6 Adding System Testing Stage . . . . . . . . . . . . . . . . . . 42
4.7 Final CI Service . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
 CONTENTS | ix

5 Results and Analysis 47

5.1 Implementation Notebook . . . . . . . . . . . . . . . . . . . 47
5.2 User Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.3 Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.3.1 Part A: General Questions . . . . . . . . . . . . . . . 50
5.3.2 Part B: Feedback on the CI Service . . . . . . . . . . 51
5.3.3 Part C: Quantitative Evaluation of the CI Service . . . 52
5.4 CI Server Metrics . . . . . . . . . . . . . . . . . . . . . . . . 53
5.4.1 Data Preperation . . . . . . . . . . . . . . . . . . . . 53
5.4.2 Stage Status . . . . . . . . . . . . . . . . . . . . . . . 57
5.4.3 Stage Duration . . . . . . . . . . . . . . . . . . . . . 61
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

6 Discussion 69
6.1 CI Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6.1.1 Multi-target Compilation . . . . . . . . . . . . . . . . 69
6.1.2 Automated Unit Testing . . . . . . . . . . . . . . . . 70
6.1.3 Iterative CI Implementation . . . . . . . . . . . . . . 70
6.1.4 Automated System Testing . . . . . . . . . . . . . . . 71
6.1.5 Distributed Build Architecture . . . . . . . . . . . . . 72
6.1.6 Visual CI Service Feedback . . . . . . . . . . . . . . 72
6.1.7 Test Report in xUnit XML Format . . . . . . . . . . . 72
6.1.8 Trunk-based Development . . . . . . . . . . . . . . . 72
6.2 Efforts for CI Adoption . . . . . . . . . . . . . . . . . . . . . 73
6.2.1 Learn Using the CI Tool . . . . . . . . . . . . . . . . 73
6.2.2 Adress Build Tool Dependencies . . . . . . . . . . . . 73
6.2.3 Adress Manual Input for System Testing . . . . . . . . 74
6.2.4 Improve Visibility of CI Service . . . . . . . . . . . . 74
6.3 Impacts of CI Adoption . . . . . . . . . . . . . . . . . . . . . 74
6.3.1 Fast Bug Detection . . . . . . . . . . . . . . . . . . . 75
6.3.2 Increased Motivation . . . . . . . . . . . . . . . . . . 75
6.3.3 Confidence in Code Quality . . . . . . . . . . . . . . 75
6.3.4 Encourage Standardisation . . . . . . . . . . . . . . . 76

7 Conclusions and Future work 77

7.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7.4 Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
x | Contents

References 81

A Questionnaire 89

B Questionnaire Results 96
 LIST OF FIGURES | xi

List of Figures

2.1 Trunk-based development . . . . . . . . . . . . . . . . . . . . 10

2.2 Branching with Git flow . . . . . . . . . . . . . . . . . . . . 11
2.3 Simplified UML class diagram of Lysa firmware . . . . . . . . 15
2.4 Schematic of Jenkins’s infrastructure . . . . . . . . . . . . . . 19

3.1 UML state diagram of the agile research network process . . . 26

4.1 Parallel execution of the build stages . . . . . . . . . . . . . . 38

4.2 Parallel execution of the unit testing and build system genera-
tion stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3 CI pipeline with firmware downloading stage . . . . . . . . . 41
4.4 CI pipeline with system testing stage . . . . . . . . . . . . . . 42
4.5 CI infrastructure with principal-agent architecture . . . . . . . 43
4.6 CI integration flow model . . . . . . . . . . . . . . . . . . . . 44

5.1 Radar chart for CI service evaluation . . . . . . . . . . . . . . 52

5.2 Logarithmic box plot for each stage duration of Slice 2 . . . . 62
5.3 Logarithmic box plot for each stage duration of Slice 3 . . . . 63
5.4 Logarithmic box plot for each stage duration of Slice 5 . . . . 65
xii | LIST OF FIGURES
 LIST OF TABLES | xiii

List of Tables

3.1 Iteration levels of CI prototype implementation . . . . . . . . 28

3.2 Specification for the build machine . . . . . . . . . . . . . . . 30
3.3 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4 Pipeline stage metrics . . . . . . . . . . . . . . . . . . . . . . 35

4.1 Deployment specification of the CI service . . . . . . . . . . . 40

5.1 Allocation table for stage ids and names . . . . . . . . . . . . 53

5.2 Slices for further data analysis . . . . . . . . . . . . . . . . . 55
5.3 Status of all stages in Slice 2 . . . . . . . . . . . . . . . . . . 56
5.4 Status of all stages in Slice 3 . . . . . . . . . . . . . . . . . . 57
5.5 Status count for each stage of Slice 2 . . . . . . . . . . . . . . 58
5.6 Status count for each stage of Slice 3 . . . . . . . . . . . . . . 59
5.7 Status count for each stage of Slice 5 . . . . . . . . . . . . . . 60
5.8 Analysis of the stage durations of Slice 2 . . . . . . . . . . . . 61
5.9 Analysis of the stage durations of Slice 3 . . . . . . . . . . . . 63
5.10 Analysis of the stage durations of Slice 5 . . . . . . . . . . . . 66
xiv | LIST OF TABLES
 List of acronyms and abbreviations | xv

List of acronyms and abbreviations

CD continuous deployment

CDE continuous delivery

CI continuous integration

CLI command line interface

DevOps development operations

HILS hardware-in-the-loop simulation

IDE integrated development environment

JTAG joint test action group

LATE laser tester

RDP remote desktop protocol

RTOS real time operating system

SCM source code management

SILS software-in-the-loop simulation

SUT software under test

SWD serial wire debug

TDD test-driven development

UML unified modeling language

xvi | List of acronyms and abbreviations
 Introduction | 1

Chapter 1

Introduction

Software projects are typically conducted by multiple developers at the same

time. The process of synchronizing the work of individuals to a complete
system is called integration. With the upcoming of extreme programming in
1999 [1], the idea of applying continuous integration (CI) arose. New code
should be integrated into a system in no more than a few hours, to keep the
differences in the code between developers small and make the integration
phase simpler. Today, CI is an important part of development operations
(DevOps) — the most widely used practice for software development [2] and
there are many tools and literature existing that help adopt CI services to
software projects. A systematic literature review has shown that until 2016
only a small percentage of the attention was given to CI for embedded software
development [3]. It can be observed that the interest in this area for CI is rising,
but yet there is not enough reference literature existing. Especially solution
proposals for CI adoption in the embedded domain are sparse.

1.1 Background
Cobolt AB∗ is at the very forefront of the industry in the development and man-
ufacturing of high-performance continuous and pulsed lasers. Cobolt AB was
founded in 2000 and is a spin-off of a research project at the laser physics group
from KTH. Today Cobolt AB belongs to the HÜBNER Photonics Group† and
the research department consists of 15 engineers working in a cross-functional
team within CAD, laser-, software- and electronic engineering.
∗
https://www.linkedin.com/company/cobolt-ab (Accessed at
†
09/12/2022) https://www.hubner-photonics.com/ (Accessed at 13/12/2022)
2 | Introduction

The company Cobolt AB wants to improve the workflow of its firmware

development team by applying DevOps methods. Cobolt sells a wide range of
different lasers, some of them customised only for specific customers. All new
lasers rely on the same code base that is complemented by a set of firmware-
specific components. When an adjustment to the firmware core is made, the
correct functionality of all firmware targets is verified through manual testing.
In the scope of this degree project, we build a prototype for a CI pipeline
that suits modular firmware architectures. The implementation efforts are
documented and the benefits that are obtained from the CI service are analysed.

1.2 Problem
Firmware projects with modular code architecture are used to allow code
reusability for multiple firmware targets. While this potentially helps to reduce
development time and implement bug fixes and features for many products at
the same time, it can create issues when changes in the code base create bugs
in just a subset of the compilation targets. When not properly tested, it may
be that regression bugs stay undetected. At a later point, when the bug gets
found it might be harder to resolve it compared to when it would have been
done right away.
CI is the state of the art for agile software practices [4]. Especially
with the rising complexity of software systems in the present day, where
many developers collaborate on one product, there must be strategies for
streamlining the integration of code changes. This mostly contains the
execution of automatic tests with every code integration. When working with
modular firmware architectures, CI can be used to automate tests and get
feedback on multiple compilation targets at the same time. Given, that there is
sufficient test coverage, this can help identify regression bugs fast. To the best
of my knowledge, there is no scientific literature available that analyses which
CI techniques are suitable for modular firmware architectures in the embedded
domain.
We would like to answer the question if CI can benefit a project that
involves a modular firmware architecture for embedded systems in a company
like Cobolt. There are no metrics available to evaluate the quality of Cobolt’s
current processes, which makes it complicated to quantitatively measure
improvement that may be achieved with a CI service. Further, it is not clear
which CI techniques are suitable for the development of embedded software
with modular firmware architectures.
 Introduction | 3

1.3 Purpose
This degree project, first of all, benefits the company Cobolt AB, which gets
a tailored CI service, that suits their development environment and workflow.
The findings are generalised so that other agents are able to use the thesis report
as a fundament for their own CI implementation.
To comply with ethical challenges that are inherent in research, this project
is aligned with the individual goals for researchers that are described in the
European Code of Conduct for Research Integrity [5]. To take state-of-the-art
research into account a semi-systematic literature study is conducted, that is
carefully analysed and documented. The intellectual contribution of others
are be reported in an appropriate form. This research does not interfere
with personal rights. It does not harm humans and animals nor is it doing
severe damage to the environment. The increased power consumption due to
additional hardware is neglectably small. The intelligence of Cobolt lies in
their laser physics knowledge and not their software development. Thus, there
is no interest conflict in publishing the research outcome to its full extent.
This research addresses goal 9 of the 17 sustainability goals that the United
Nations adopted in their 2030 agenda for sustainable development. Goal 9
is about prospering technological progress. The results of this project are
published on the Diva platform so that anyone can use it to support their
development of CI pipelines and thus help to thrive innovation.

1.4 Goals
This project entails a thorough study of the current state-of-the-art practices for
establishing and examining CI systems in the embedded domain. We explore
how an automatic CI system can be implemented for a modular firmware
architecture used for compiling multiple firmware targets. To achieve this, we
implement a CI service prototype at the company Cobolt. The implementation
effort is reported and analysed. Once the CI service is deployed, we focus on
examining its impact on the current development process. Based on these
considerations, the research questions can be summarised as follows:

R1: What CI techniques suit modular firmware architectures for embedded

systems?

R2: What are the efforts to build up such CI infrastructure?

R2.a: What challenges occur during the implementations of a CI service?

4 | Introduction

R2.b: What countermeasures help to overcome those challenges?

R3: What are the benefits of building up such CI infrastructure?

1.5 Research Methodology

This study is a single case study with one contributing company. It consists
of an engineering project (implementation of a CI service) and an analysis of
the challenges and benefits of the proposed solution. Qualitative evaluation
data is obtained from personal experience, user feedback, and quantifiable
data (such as speed, number of detected software bugs, and usage data) from
the implemented CI service. The data is evaluated using an interpretive
research paradigm, which considers knowledge as an individual understanding
of a subject within the context in which it is observed. The research work
is conducted as an agile process to allow data collection in parallel to the
implementation.
A long-term study of the implemented system would not be feasible
in the scope of this thesis work. The presented results are specific to
small software companies using modular firmware architectures with multiple
firmware targets. However, this is a typical scenario for embedded software
development, and other organizations benefit from the results of this thesis,
which is given in a generalised manner.

1.6 Delimitations
CI services can include a broad variety of different tools and activities. While
it is part of this thesis to identify relevant parts for a CI service for modular
firmware architecture of customizable embedded products, there are some
activities that must be excluded from the beginning because of the time
limitation of this project.

1.6.1 Focus on Test Execution

Software testing can be divided into three subtopics. Test design, execution,
and evaluation [6]. The company Cobolt has both a unit testing framework
and a system testing framework in place, so this thesis purely focuses on
automating the test execution as part of the implemented CI service.
 Introduction | 5

1.6.2 Short Term Case Study on CI Service Impact

It would be interesting to collect quantifiable data about the benefits of
adapting CI in this use case for a longer time period. However, this is infeasible
in the time window of this thesis work. Instead, conclusions about the
impact of the implemented CI service are drawn from immediately available
user feedback, personal experience and short-term quantifiable data that is
available, because of an iterative deployment process.

1.7 Structure of the Thesis

Chapter 2 presents relevant background information about embedded software
development using modern methods of software engineering as well as intro-
ducing the tools that are used for the CI service solution proposal. Chapter 3
reflects on the usage of an agile research process with an interpretative
paradigm to answer the posed research questions. Chapter 4 reports on the
implementation process. Chapter 5 presents and analyses the results of the
planned data collection and in Chapter 6 the reported data is discussed to
answer the research questions. Ultimately, the whole report is concluded in
Chapter 7 and an outlook on future work is given.
6 | Introduction
 Background | 7

Chapter 2

Background

DevOps is the most widely used practice for software development [2].
Despite the fact that using DevOps methods such as CI and continuous deploy-
ment (CD), automated testing, and monitoring would result in less error-prone
development and improved maintenance of embedded software, those modern
methods of software engineering are still rarely used in embedded software
development. This is because of the unique characteristics and requirements
that the embedded domain poses. Section 2.1 presents an overview of
techniques that can be used in embedded software development. After first
inspecting the characteristics and requirements of embedded software, this
Section explains modular firmware architecture, source code management
(SCM), build scripts, build automation, CI, and automated testing in the
context of embedded software. As in this thesis, implementing CI services is
often done to improve an existing development process. Thus it is important
to understand the existing environment to be able to include and reuse
its components for the CI implementation. Section 2.2 shows the current
development process at Cobolt. All additional tooling that was introduced
to implement the CI service is specified in Section 2.3. Section 2.4 reflects on
related work to this paper.

2.1 Embedded Software Development

As already stated, modern software engineering methods are still not com-
monly used in embedded software development. This results from a set of
limitations that arise from working in the embedded domain. For instance, the
concurrent development of the target hardware platform often poses obstacles
to software development [7]. Shanin et al. [3] did a systematic literature review
8 | Background

about approaches, tools, challenges, and practices of CI, CD and continuous

delivery (CDE) between 2010 and 2016. They have shown that only 5 of
their 69 selected papers were applied to the application domain of embedded
systems.
However, many recommended practices that work in web or app develop-
ment may also be used in the embedded domain. In their book “Patterns in the
Machine: A Software Engineering Guide to Embedded Development”, Taylor
and Taylor [8] describe concepts that enable a modern way of developing
embedded software. They focus on prominent topics in software development.
These include SCM, build scripts, automation, testing, and integration. The
goal is to meet the requirements specific to embedded software characteristics.
Those topics are summarised and presented in the following sections.

2.1.1 Characteristics of Embedded Software

Embedded software is software that runs on hardware that is embedded
in a specific context. This context is usually a runtime environment that
requires the hardware to comply with special working conditions (i.e. safety
regulations, real-time performance demands, lack of space, and low-energy
consumption). Often, the word firmware is used as a synonym for software
that is device-specific. Embedded software developers are obliged to focus on
the underlying hardware when writing code.
In contrast to classic software development, embedded software devel-
opment is often not performed directly on the target platform. Embedded
systems lack a keyboard and display interface, do not run an operating system
supporting integrated development environment (IDE) installation, and have
limited processing power for compilation. Sometimes there is no actual
compiler. Instead, embedded developers use a development workstation
running Windows, Linux, or Mac OS and cross-compile the firmware for the
hardware target as a binary image. This image is then downloaded/flashed to
the respective target device using either a bootloader that is preinstalled on the
microcontroller or via a debugger.

2.1.2 Modular Firmware Architecture

In 2000, IEEE released a recommendation for specifying the architectural
description of software-intensive systems, which uses the following definition
for software architecture:

“The fundamental organization of a system embodied in its com-

 Background | 9

ponents, their relationships to each other, and to the environment,

and the principles guiding its design and evolution.” [9]

The purpose of software architectures is to satisfy a set of quality attribute

requirements [10]. The design choices to achieve this goal are called archi-
tectural tactics. Márquez et al. [11] have done a systematic mapping study
of architectural tactics and concluded on twelve quality attributes that are
addressed in publications about software architectures: adaptability, depend-
ability, reliability, modifiability, interoperability, deployability, scalability,
performance, safety, availability, fault tolerance, and security.
Modular software/firmware architecture is not a fixed term. However,
it often refers to siloing functional components into modules with a clear
interface specification. This approach targets multiple quality attributes.
Firmware projects that follow a modular design can be modified easier because
new modules can be integrated just by matching the interface. Deployability
is improved, especially for teams with many developers, because the risk of
merge conflicts is lowered as long as they are not working on the same module.
Modular architectures often introduce an abstraction of the application to the
hardware layer, which improves scalability as it is simpler to port a firmware
product to a different hardware architecture. Martin [12] presents code
reusability and its resulting developer team productivity and quality increase
as an additional motivation for applying modular firmware architectures.
When modules are well isolated and, therefore loosely coupled, they can
be easily integrated into other firmware projects. This is especially relevant
for companies that have many firmware targets with similar hardware or
functionality.

2.1.3 Source Code Management

SCM is used to organise code repositories. Most importantly, it tracks source
code modifications over time, helps to preserve specific states (snapshots) of
the repository, and supports independently-working developers to integrate
their work into the shared code base.
Git is a popular, open-source SCM tool. It organises repositories around
branches. A branch is a history of code changes (commits). There is usually
one main branch where the latest commit reflects the current status of the
repository. It is possible to create a new branch from any commit in the
history of another. This new branch then shares the history up to the respective
commit. Developers use this, for example, to create a release branch from the
main branch. The release branch is then used to fix bugs discovered during
10 | Background

the operation while isolated from newly implemented features. Branches can
be merged to integrate changes from one branch into another. Depending on
the selected merging strategy, this modifies the history of a branch in different
ways.
There is a wide range of different git-enabled workflows that follow
different branching strategies. Taylor and Taylor [8], name two, that they
identify as typical for embedded software development.

Trunk-Based Development. Trunk-based development is a branching strat-

egy, where all developers commit to one shared branch, the so-called trunk or
mainline. There must not be any other long-lived branch. Release branches
are created, but commits on them are only added by cherry-picking them
from the mainline. This ensures that bugs are fixed on the shared trunk and
prevents integration issues from merging the release branch back. While trunk-
based development is originally designed for small teams with experienced
developers, there is also an adaption for scaling it for bigger teams. The
so-called scaled trunk-based development allows short-lived feature branches
that one person works on for a couple of days, which are then merged to the
mainline locally or via a pull request. However, it is important that they are
only used temporarily [13].

release
cherry-picked

trunk

(a) Classic trunk-based development

release
cherry-picked

trunk
featureB
featureA

(b) Scaled trunk-based development

Figure 2.1: Trunk-based development (inspired by Hammant [13])

 Background | 11

Git Flow Development. Git Flow development utilises multiple primary

branches. These branches include the main branch, where each commit
represents a production-ready state, and the development branch, which keeps
track of ongoing work. Alongside these, feature, hotfix, and release branches
can also be employed. Feature branches are used to implement new program
functionality and are always merged into the development branch. Release
branches are used to prepare a release. Once a stable version is reached, it is
merged both in the development and main branch. Hotfixes must be created
when bugs are found for the release under operation. They are also later
merged into the development and master branch [14].

main

hotfix

release

develop

feature

Figure 2.2: Branching with Git flow (adapted from Driessen [14]

Comparison. Taylor and Taylor suggest using the Git flow strategy for
embedded software projects because it adds an extra layer of security to the
mainline. Especially less-experienced developers do not need to worry about
breaking the trunk branch. Above that, it prevents pollution of the history of
the mainline [8]. However, Driessen, the inventor of the Git flow method,
sees his strategy as dogmatic nowadays. Thus he would not recommend it any
more for projects that do not need to maintain multiple release versions. He
recommends using a simpler strategy, such as the GitHub flow method, which
is, in fact, very similar to the scaled trunk-based approach. Both strategies
have only one long-lived branch; however, GitHub flow is not that explicit
about how long a secondary branch may exist.

2.1.4 Build Scripts

Software development is often based around an IDE, with all necessary tools
such as a code editor, build tool and an integrated debugger. This is handy
12 | Background

when trying to get something working quickly. However, it also hinders

developers from changing their development environment. IDEs usually store
all configuration parameters in an IDE-specific project file [8]. To assure
flexibility, it is advisable to use build scripts. Those can be Shell, Batch,
or Python scripts that automatically execute all necessary steps for the build
process. Above that, as build scripts are executed through the command line,
they can be automated without additional effort.

2.1.5 Build Automation

A build process can include various steps, among others, the compilation of a
program, linking its libraries, compiling unit tests, executing them, packaging
the compiled software, and creating an installer. Those steps are repetitive and
can be easily automated.
A set of tools is commonly used to archive different levels of automa-
tion. Build systems such as GNU Make∗ or Ninja† are command-line tools
that work similarly to build scripts; however, they usually have some more
abstraction, making them more suitable for larger software projects. Build
system generation tools such as CMake‡ abstract even further. They are used
to make the build automation independent from the used build system, as they
might be platform dependent. Finally, build automation tools, often called CI
orchestrators, can be used to automate anything. Typically, they continuously
monitor a SCM tool for repository changes and trigger the execution of a build
pipeline (a set of dependent jobs). Taylor and Taylor [8], name the usage of a
build automation tool as a prerequisite for implementing CI services.

2.1.6 Continuous Integration

An early definition of continuous integration (CI) was provided by Beck [1] in
his article from 1999 about extreme programming:

“New code is integrated with the current system after no more than
a few hours. When integrating, the system is built from scratch
and all tests must pass or the changes are discarded.” [1]

In more recent definitions of CI it is further specified that there must be an

automated system. When an agent checks in their revised code, this system
∗
https://www.gnu.org/software/make/ (Accessed at 24/03/2023) † https://ninja-build.org/
‡
(Accessed at 31/01/2023) https://cmake.org/ (Accessed at 31/01/2023)
 Background | 13

picks up the change and runs a set of commands to verify the correctness
automatically [15, 16]. This enables faster feedback and less error-prone
testing because a machine will not forget about executing every defined step.
Duvall et al.[17] introduced essential practices for CI, later distilled into
seven cornerstones by Mårtenson et al. [18, Tab. 1]. These include local
code validation before commits, daily commits, automated integration builds
with thorough testing, immediate fixing of broken builds, and ongoing process
improvement through developer engagement.
Taylor and Taylor [8, Fig. 8-1] present a schematic for a typical CI
setup A developer works on a local workstation and has the possibility to
compile, test and debug because all necessary tools and packages are locally
installed. To track the code history and enable collaboration, developers use
a SCM repository. A build machine fetches code changes from the SCM and
automatically executes a set of jobs using some automation tool. For CI, it is
important that the build machine has the same build and test tools installed
as the developer. Further, it is supposed to provide feedback to the developer
about the functional correctness of the code base.

2.1.7 Automated Software Testing

The IEEE Computer Society defines software testing as follows:
“Software testing consists of the dynamic verification that a
program provides expected behaviours on a finite set of test cases,
suitably selected from the usually infinite execution domain.” [19]
Testing is supposed to be not only executed after the coding phase but
instead continuously during the whole development and maintenance life cycle
[19]. However, depending on the number of test cases, which one can assume
is increasing if the software is actively maintained, this task can get very time-
consuming or even unfeasible. Above that, manual testing is error-prone [4].
Thus, it is recommended to automate this repetitive task.
Further, testing can be performed on three different test levels. Unit tests
target a single module or component while integration testing focuses on a
group of modules. As the name suggests, system testing tests functionalities
involving the complete system [19].

Unit Testing
To conduct unit testing, a small chunk of code, the so-called software under test
(SUT), is compiled and executed as part of a test program. The test program
14 | Background

asserts that an expected condition results from the test and thus verifies the
correct functionality of the SUT [20].
In embedded systems, it is important that the SUT is isolated from its
hardware dependencies to avoid testing hardware and software at the same
time [21]. This can be achieved by writing mock functions to satisfy the
build dependencies, tracing the hardware function calls, or even simulating
hardware response. It is recommended to execute the unit tests off-target [20].
Unit tests are typically written when new functionality is developed,
sometimes even prior to the actual functionality, as in test-driven development
(TDD). After their initial usage, they remain valuable when run as regression
tests [20].

System Testing
System testing — or end-to-end testing as it is called in some literature —
is used to verify the correct system behaviour when a set of conditions is
applied. Its main goal is to test non-functional system requirements (i.e. speed,
security, accuracy, and reliability) and the interaction with different operating
environments [19]. Unlike unit tests, embedded system tests should be
executed on the actual firmware target or at least in a target-like simulation
environment as done in software-in-the-loop simulation (SILS).
System tests on real embedded systems have to be selected carefully. In
safety-critical systems such as fighter jets, a non-acceptable danger for the
environment and humans can result from system failure [22]. But even with
systems that are not safety-critical, system tests can result in self-destruction
[22]. This is an unwanted cost factor, especially for automated system tests, as
the same hardware is supposed to be used many times without being manually
replaced.

2.2 Cobolt’s Development Environment

This thesis is conducted together with the company Cobolt AB. Cobolt’s
research and development department consists of 15 engineers working on
improving the palette of high-performance continuous and pulsed lasers. The
firmware development team at Cobolt currently consists of four developers.
Each of them works on fixing bugs, resolving issues, or developing features
for the Lysa project. They do this by checking out the relevant branch and
committing their changes to a new development branch. Before merging it
back, they build and download the system to the target hardware to verify
 Background | 15

its correct behaviour and may run some unit tests. The following sections
describe relevant parts of Cobolt’s development environment, that is used in
the scope of this thesis.

2.2.1 Firmware Architecture — Lysa

The project Lysa was Cobolt’s approach to make their firmware development
more streamlined. Lysa follows a modular firmware architecture, that can be
compiled to currently three different firmware targets with the names Alpaca,
Bobcat, and Capybara, which correspond to three different laser driver boards.
Lysa tries to abstract the business logic from the low-level hardware
drivers. The business logic follows the object-oriented programming paradigm
and is written in C++. The communication to the low-level drivers is done
through an interface, so the real time operating system (RTOS) and standard
peripheral drivers can be replaced easily.

Core Statemachine<_State> Module CalibrationModule

TecModule

LaserModule

Property<T> can get connected CommModule

to module class
member variable

configures class members

Firmware Component expands
of modules as properties

<Firmware name> Board

Instantiate Hardware Interface Drivers Bridge

Project configuration Software representation Libraries

Firmware Specific
class; instantiates of all hardware features
Core Extentions and Drivers
core; connects it to of electronic circuit
hardware; board Standard
RTOS
Peripheral Drivers

Figure 2.3: Simplified unified modeling language (UML) class diagram of

Lysa firmware (inspired by Kalinski [23])
16 | Background

Lysa’s folder structure is aligned with its module structure. Figure 2.3
presents a simplified overview of the modules and their interconnections.
<Firmware name> is a placeholder for the name of the respective firmware,
i.e. Capybara. This class specifies the core component, among others by
configuring Property<T> objects. These properties specify a Set/Get
interface for member variables of the different core modules, i.e. Laser-
Module. Properties are the main element for inter-module and module-
hardware communication. The hardware is thereby described in software by
the board class, which specifies and configures the drivers. This structure
allows abstracting the low-level hardware access from the high-level business
logic.

2.2.2 Hardware Target — Laser

As previously mentioned, project Lysa is currently compiling three different
firmware targets that are intended to be executed on 3 different laser platforms.
Alpaca is running on lasers of the Cobolt Tor Series [24], while Bobcat is
running on the DPL version of the 06-01 series lasers, and Capybara will
replace the firmware 06-01 series laser MLP version [25].

2.2.3 Build System Generator — CMake

CMake is an open-source build system generation tool. It enables platform-
independent building by generating configuration files for different build
systems. Cobolt uses CMake to generate build files for the Ninja build system.
Ninja orchestrates the compilation and linking process. Cobolt uses the
output of Ninja check for compilation errors and allows an IDE independent
development process.
Additionally, Cobolt created a custom CMake command, that generates an
IAR project file from the relevant build information, i.e. source files, including
directories and libraries.

2.2.4 Compiler — IAR C/C++ Compiler for ARM

Cobolt uses the IAR ANSI C/C++ Compiler V7.70.2.11706/W32 for ARM
with the C++03 standard and the Extended Embedded C++ enabled. This
compiler is proprietary, bound to the Windows operating system, and requires
a USB license key. A short test has shown, that Lysa could theoretically
also be compiled with the open-source ARM GNU Toolchain. However, the
 Background | 17

RTOS Embos that Lysa is currently tied to, can only be compiled with the IAR
compiler.

2.2.5 IDE — IAR Embedded Workbench

Together with their ARM build tools, IAR distributes an IDE which is named
“IAR Embedded Workbench for ARM”. Cobolt uses version 7.5 because of
historical reasons. The IAR project file that is generated with CMake is used to
build release versions of the firmware targets: Alpaca, Bobcat, and Capybara.
Further, hardware debugging is currently only possible using this IDE.

2.2.6 Version Control System — Git

Cobolt currently uses a Git SCM to host the Lysa project. The chosen
branching strategy is mostly classic trunk-based (see Section 2.1.3) with one
clear main branch that is used to synchronise the work of each developer fast.
However, there are additional long-lived branches for side projects. These
branches may never be merged into the mainline but continue to be developed.
This aligns with the typical approach of the git flow branching strategy (see
Section 2.1.3). Cobolt’s developers have communicated, that they are open
to discussing the branching strategy in the future, which may be helpful for
applying CI.

2.2.7 Unit Testing Tool — CxxTest

Lysa’s core module implements its fundamental functionality. It is made
sure that this code is platform-independent, which makes it suitable for unit
testing. A lot of the functionality is implemented in a test-driven development
fashion, where unit tests are written prior to the implementation of the actual
functionality. There are no metrics on test coverage.
All tests are written using the CxxTest unit testing framework∗ . The
tests are kept separate from the software under test. Cxxtest compiles into
a command line program and thus shows the results as command line output
as well.
∗
https://cxxtest.com/ (Accessed at 03/03/2023)
18 | Background

2.2.8 System Testing Tool — LATE

Laser tester (LATE) is software for testing lasers (or in fact any serial devices)
using serial communication with plain text commands. Each test is written in
a text file which is then processed by LATE.
A LATE test file consists of instructions that are a combination of a
command and a value. The most important commands are SEND, WAIT and
EXPECT. The value specifies what to send to a connected serial device, how
long the program should wait before executing the next instruction or what
return value to expect from the tested device. Instructions are separated by
line breaks.
LATE allows executing runtime tests, that involve multiple consecutive
inputs. It is currently used to document sequences of inputs that lead to
bugs and was originally implemented as a convenience tool for the bug-fixing
procedure. However, there are a couple of tests existing now, that can be
executed as part of a CI pipeline to assure, that those bugs are not recreated.
It is important to mention, that LATE is not used to automatically test the
physical qualities of a laser, as there are other processes implemented by the
production department that takes care of this.
Yet, LATE can be classified as a system testing tool, specifically for user
interaction through the serial interface of the laser.
The LATE framework includes some unit tests to validate the correct
functionality of itself. The unit testing utilises Pytest ∗

2.3 Additional Tools

Setting up a CI service largely involves the automation of pre-existing pro-
cesses and tools. Section 2.3.1 introduces Jenkins, a popular automation server
that is often used for this task. The grand challenge lies in automating those
tools that require input through a graphical interface such as IDEs. Luckily,
many graphical programs use command line interface (CLI) tools under the
hood, that can be used stand-alone. In Section 2.3.2, the IAR Build Tool is
presented, a build system that is used to compile multi-file software projects.
Section 2.3.3 introduces the J-Link commander, a tool to interface Segger J-
Link debugging probes and download firmware to microcontrollers.
∗
https://docs.pytest.org/en/7.3.x/ (Accessed at 15/06/2023)
 Background | 19

2.3.1 Automation Server — Jenkins

Jenkins∗ is an open-source, platform-independent and self-hosted server ap-
plication, that can be used to automate tasks. In the literature, Jenkins is often
categorised as CI server, CI orchestrator, or CI coordinator and represents one
of the most common tools in this sector. The systematic literature review of
continuous activities and tooling by Shahin et al. has shown that 11 of 25
papers that were using a CI server were using Jenkins [3]. Rostami et al. have
identified Jenkins as the most used, self-hosted CI tool [26]. Jenkins has a
sophisticated handbook [27].

Server Workstation

Jenkins Server Application Jenkins Agent Application

Coordinator

allocate tasks

Node Node

Executor Executor Executor Executor

Figure 2.4: Schematic of Jenkins’s infrastructure

Jenkins infrastructure reflects a principal-agent model, as shown in Fig-

ure 2.4. The principal server application runs a coordinator that allocates tasks
to nodes. By default, there is a node running as part of the server application
internally. However, there is the option to connect external agents to the
principal, which themselves act as interfaces to external nodes. Each node
specifies a finite number of executors that can run tasks in parallel. Thus, this
number most often reflects the number of cores of the respective workstation.

Pipelines. Tasks are usually organised in so-called pipelines. There are

declarative and scripted pipelines. Scripted pipelines are more expressive and
flexible since they lack the stricter structure that declarative pipelines have (see
Chapter Pipeline→Pipeline Syntax of Jenkins Handbook [27]). Declarative
∗
https://www.jenkins.io/ (Accessed at 09/12/2022)
20 | Background

pipelines, however, are easier to write, read and therefore maintain. They
consist of stages that can get executed sequentially or in parallel on multiple
executors. A stage consists of steps (or tasks) that are executed sequentially.
The execution of a pipeline is called a job and is referenced with a job number,
which allows for keeping a history of previous executions. If a stage in a
pipeline fails, all downstream stages are aborted and considered failing as
well. This is because Jenkins generally considers stages to be dependent on
the output of their predecessors.

Trigger. Jobs can be initiated manually, but typically they are supposed to be
run automatically. Either scheduled or event-based with a trigger, i.e. through
a webhook that signalises a code change in the SCM. Setting up webhooks
usually requires administration rights for the SCM. When those are not given,
Jenkins users have the option to set up a schedule that checks an SCM for code
changes. The pipeline is only running if there are indeed code changes.

Blue Ocean UX. Jenkins is an open-source project, that has many active
developers contributing. This results in Jenkins having a rich ecosystem with
1800+ community plugins. A popular plugin is Blue Ocean∗ , which offers a
modern pipeline visualisation.

JUnit Test Report. Another popular plugin is called JUnit† . It serves as

a valuable tool for processing and displaying test outcomes using the widely
adopted xUnit XML format. This format has emerged as a de facto standard
for reporting test results and enjoys extensive support from various unit testing
frameworks like JUnit, CxxTest, and Pytest.

2.3.2 Build System — IAR Build Tool

Cobolt uses the IAR Embedded Workbench for ARM v7.5 for compiling the
release version of their firmware targets. This final firmware image is the
one, that shall also be built by a CI service. However, IDEs typically can
not be automated themselves as they require input through a graphical user
interface. Fortunately, IAR makes the build system, that their IDE is using
also accessible through a CLI program–the IAR Build Tool [28]. It requires
only the IAR project file as input, which is generated by CMake.
∗
https://plugins.jenkins.io/blueocean (Accessed at
†
14/04/2023) https://plugins.jenkins.io/junit/ (Accessed at 16/06/2023)
 Background | 21

2.3.3 Firmware Download — J-Link Commander

Embedded software is cross-compiled and then downloaded to the firmware
target. Typically this is done with a debugger, i.e. a Segger J-Link debugger,
which can manipulate the memory of a microcontroller through a joint
test action group (JTAG) or serial wire debug (SWD) interface. J-Link
Commander is a CLI tool that interfaces J-Link debugging probes from the
workstation side. In the scope of the thesis, this program was used to download
firmware images to hardware devices.

2.4 Related Work

Our goal is to build a CI service, identify challenges and conclude on its
impact. Section 2.4.1 gives an overview of existing solution proposals for CI
adoption in the embedded domain. Section 2.4.2 reports on what challenges
with CI adoption in the embedded domain were found in various case studies.
Section 2.4.3 presents practices that were identified as key enablers for CI
adoption in the embedded domain. Section 2.4.4 summarises how other
researchers approached identifying benefits of using CI in industry.

2.4.1 CI Services in Embedded Software Development

Shahin et al. [3] classified existing literature about continuous activities before
2016 according to their research type. They only identified 5 of 69 papers
as solution proposals of which only one was associated with the domain of
embedded systems. Even after 2016 existing literature in this area seems to be
sparse.
One main function of CI systems is automating a variety of tasks. Common
stages of CI services are build automation, unit testing, integration testing,
system testing, code coverage analysis, security analysis, packaging, checking
non-code artefacts, dependency management and feedback on communication
channels [26]. It seems that CI systems in the embedded domain often
implement just a subset of those activities. Bernhardt [29] implemented build
automation, packaging and system testing. Seth and Khare [30] include only
a build and testing stage and stay vague regarding the level of their testing
activities. Kozwan and Pietrzak [31] report also just on their build and test
stage. However, their pipeline is not fully automated as some of the testing
must be done manually. Apart from that, they also include a reviewing process
for their CI service.
22 | Background

In this thesis, it is also not the goal to implement a CI system that includes
all of the possible CI activities. We agree with Rostami et al. [26] who state
that it is very use-case specific which stages are relevant to implement. Our use
case is embedded systems with modular firmware architectures that compile
multiple firmware products. Previous solution proposals do not cover this topic
to the best of our knowledge.
The complexity of CI services varies largely [32]. Some services in the
embedded domain consist of a single pipeline [30]. Others use a two-pipeline
approach [29] to cope with two different compile products that are triggered
independently, but are dependent on each other. Complex services with
manual activities and multiple triggers may require multiple interconnected
pipelines [31]. This thesis presents a fundamental CI service that in fact builds
multiple software products. Yet one pipeline is sufficient since their build is
triggered by the same event. Further, we aim to implement a system with a
small complexity as the cooperating company, Cobolt hasn’t used CI before.

2.4.2 Challenges for CI Adaption in the Embedded

Domain
There is a wide range of publications available studying the challenges that
developers encounter when implementing and practising CI in the embedded
domain. The results are manifold.
First and foremost, many researchers identify hardware dependencies,
which is a central difference to non-embedded software engineering, as a major
barrier for adapting CI. Hardware has long development cycles and slows
down the software cycle-time [33, 34]. It increases maintenance effort [33]
and testing effort [18] since it is not uncommon that there must be multiple
supported hardware configurations. The test effort is further affected because
system tests on hardware must be well selected to not affect or even destroy
the hardware itself [22].
An additional issue is that hardware is a limited, expensive resource that
affects the testing capabilities [18]. So do limited human resources [34] with
limited domain-expertise [34]. Not every developer has the same capability of
automating tasks and designing tests, since their confidence is varying. Even
tooling for adapting CI strategies in the embedded domain is insufficient [34,
33].
A fast time-to-feedback of CI services is an important factor for the
adoption of CI in general. The speed of building and testing the SUT is
substantially affecting this metric. Embedded systems often represent tightly
 Background | 23

coupled systems [18]. As a consequence, the whole system must be rebuilt,

when just one component was changed. Legacy code is often reused to reduce
the time-to-market. Unfortunately, it limits the possibility to use newer build
tools [33], and creates more complicated and time-intensive build procedures.
Finally, it is relevant to find a good trade-off between test coverage and testing
time [34], to have a well functioning CI service.

2.4.3 Practices for CI Adoption in the Embedded Do-

main
Aside from the identification of challenges, there are many papers that identify
practices for CI adoption. It is relevant to observe how CI is applied by
experienced developers in real cases — for this thesis specifically in the scope
of embedded software.
A thorough activity planning seems to be a key enabler for CI. Especially
using a strategy for work breakdown [35]. This enables developers to work
in parallel on small tasks, with lower integration effort. This is easier when
the software system follows a modular architecture with loosely coupled
components [35, 36, 3]. This even improves the build speed, as components
can be built in parallel. It is important to have a fast CI service, as it is
relevant that feedback is provided as fast as possible [36]. This is somewhat
antagonizing to the practice of improving testing activities. More tests and
thus greater coverage can improve the confidence in the CI service, making
developers integrate their code more often [3, 36]. However, more tests result
also in longer testing times, which reduces the feedback speed.
Another key enabler is to rethink the role of developers. On one side
this means dedicating time and human resources to the maintenance and
improvement of the implemented CI system [3]. Mårtenson et al. [36] have
further proposed to rethink the role of the software architect. Instead of
creating an architecture and enforcing it, they should help developers to make
decisions, by creating respective tools and promoting discussions about design
choices.
Mårtenson et al. also suggest that continuous integration is not black or
white [35]. One has to distinguish between inter-component, component-
system and system-system integration and only implement the levels that
are relevant to the respective project. Rostami et al. [26] have performed a
multi-case study on CI/CD tool usage and migration. They have shown, that
companies initially start implementing certain phases (build automation, unit
testing, integration testing, system testing, code analysis) and progressively
24 | Background

added new functionality. It is very use-case specific which features are

relevant, i.e. automatically labelling and closing pull requests or integrating
communication channels for feedback is not useful for every organisation.

2.4.4 Experiencing Benefits of CI in Industry

Research question R3 of this thesis is about identifying the benefits of an
implemented CI service (see Section 1.4). The research community has been
quite active in this field and explored different ways of identifying the effects
of CI implementations.
Many authors performed multi-case studies with semi-structured inter-
views to identify similarities in various cases and/or answer their hypothesis
[37, 32, 38, 35, 18, 36, 34, 33]. Ståhl and Bosch [37] describe the reason for
this method well. According to them, it is favourable to design questions, and
study effects, discovered in previous literature, but encourage discussion and
elaboration to get a more contextual understanding.
Elazhary et al. further argue that “CI and its benefits can only be understood
using qualitative methods” [38]. We disagree with this general statement, as
other researchers have already shown use cases in which quantitative data was
successfully used to explore the effects of CI. Ståhl and Bosch asked their
interview partners to rate the perceived effects of CI on a scale [37]. In a later
study, Ståhl and Bosch again used a quantifiable questionnaire to identify the
impact of their respective software integration flows [32]. Miller [39] used
quantifiable data for the time overhead of implementing and maintaining a CI
service to compare it to an alternative software integration process without CI
to conclude, that CI can reduce the development cost.
Further, researchers used data-driven analysis on CI artefacts to identify
CI practices. This is needed in order to be able to design relevant questions
that can identify benefits that are linked to specific practices [38, 32].
We came to the conclusion, that we want to use both quantifiable and
qualitative data in this thesis work, to maximise the knowledge acquisition.
Additionally, to a qualitative questionnaire with open questions to understand
the context, we want to use qualitative questions to evaluate the implemented
CI service using the guidelines that Ståhl and Bosch have concluded in their
study [32]. The six evaluation categories are comprehensiveness, effective
communication, immediacy, appropriate dimensioning, accuracy and lucidity.
This is further discussed when explaining the means of data collection in
Section 3.4.
 Methodology | 25

Chapter 3

Methodology

In an experimental single-case study, we implement a prototype for a CI

service using Jenkins at Cobolt, a company operating in the embedded systems
domain. The study aims to identify suitable CI techniques, analyse the
implementation efforts of the CI service and assess its impact on Cobolt’s
development process. From our aims, we formulate the following research
questions:

R1: What CI techniques suit modular firmware architectures for

embedded systems?

R2: What are the efforts to build up such CI infrastructure?

R2.a: What challenges occur during the implementations of a CI

service?
R2.b: What countermeasures help to overcome those challenges?

R3: What are the benefits to build up such CI infrastructure?

The research process follows an agile approach (as detailed in Section 3.1),
allowing for continuous stakeholder feedback throughout the study. This
approach enables timely data collection in parallel with the implementation
process, making it suitable within the 20-week timeframe of the study.
The agile research process is roughly planned in four iteration stages that
each produce a deployable CI service. Section 3.2 reflects on those stages as
well as the hardware that is required for the CI service prototype.
We answer the research questions according to the interpretative research
paradigm (see Section 3.3).
26 | Methodology

Section 3.4 presents the data collection for this thesis. Qualitative data is
gathered from the experience of the researcher during the implementation, as
well as informal and formal user feedback. The qualitative data is supported
with quantitative data from various CI metrics.
The planned data analysis of the CI metrics is specified in Section 3.5

3.1 Agile Research Process

This research project is conducted in collaboration with the company Cobolt
AB and the TCS/EECS Department at KTH Stockholm. Both organizations
act as two individual stakeholders. Cobolt is majorly interested in a tailored
solution for a CI service specific to their context, while KTH requires academic
output in the form of generalised results, that can be reused by other agents.
Barroca et al. identify those alternating goals of industry and academia as a
typical scenario and propose the agile research network approach to overcome
this challenge [40]. Figure 3.1 visualises their process schematically. Note that
the original specification of the agile research network process comes without
the looping option back to the investigation phase (dashed lines). In this thesis,
feedback and learnings from the implementation and evaluation phase are used
to identify relevant topics that need further investigation.

Collaboration
Investigation Implementation Evaluation
kick-off

Figure 3.1: UML state diagram of the iterative research process

Collaboration Kick-Off. The kick-off aims to identify the goals of the

stakeholders. The initial project descriptions, prioritization, and focus areas
are discussed [40]. The kick-off phase gives the project a general direction.
Concrete goals are defined in the investigation phase.
In the first kick-off meeting with Gunnar Elgcrona from Cobolt at the
beginning of December 2022, different thesis topics were discussed. Cobolt
communicated that they want to improve the rigour of their software develop-
ment. CI was identified as a possible solution.
The project proposal was formulated and concluded during the first meet-
ing with Han Fu from the TCS/EECS department. The discussions led to the
proposal hand-in in mid-December.
 Methodology | 27

The kick-off phase was finalised with the approval of the preliminary time
plan by Han Fu and the signing of the contract with Cobolt. The start date for
the investigation phase was set to be the last week of January 2023.

Initial Investigation. The initial investigation needs to be highlighted, as it

differs from the continuous investigation, which is part of the iterative process.
Its purpose is to refine the focus area through interviews and work practice
observations. Criteria for the project result are defined, and the initial literature
research is done to survey and summarise existing solutions [40].
For this thesis work, the current development process and environment of
the software development team must be understood. We learned how to set up
the build tools and simulated a development task. We asked questions to the
developers at Cobolt and identified challenges for a CI service.
In addition, we investigated existing CI solutions — Specifically in the
embedded domain.

Iterative Cycle. Similar to software engineering projects, it is possible that

the goals and requirements of research projects change over time. Iterative,
agile methods allow responding flexibly to changing demands to the cost of an
increased communication time effort [41].
Cobolt preferred to have intermediate, functioning deliveries that imple-
ment parts of the final CI service and that can be operated immediately. This
allows getting feedback right away and additionally shows the capability and
effort for adding new functions. This goes at the cost of more time being
spent on redesigning or refactoring, compared to a more thoroughly planned
software system.
During the initial investigation phase, we defined four preliminary iteration
stages that specify the tollgates of each iteration cycle. Those are presented
later in Section 3.2.2.

Final Evaluation. This thesis project is conducted in a fixed period of 20

weeks, the current state of the research ultimately needs to be assessed in a
final evaluation phase.

3.2 Experiment Design

We want to propose a CI service for embedded software with modular firmware
architectures. The solution proposal is produced in an experimental single-
28 | Methodology

case study at the company Cobolt. In Section 3.2.1, relevant background to

the company Cobolt AB is given. Section 3.2.2 explains the iteration stages of
the experiment. Section 3.2.3 elaborates on the required hardware to replicate
the proposed CI service infrastructure.

3.2.1 Single-case Study

In an experimental study, we implemented a CI service to support the firmware
development division of Cobolt, which currently consists of three full-time
developers that work or have worked with the modular firmware project Lysa.
All results need to be understood in the scope of a small embedded firmware
development team developing firmware for a high-tech embedded system.

3.2.2 Iterative Implementation Plan

Our implementation process follows an iterative approach. We begin by
analyzing various proposals for CI solutions specific to the embedded domain
(see Section 2.4.1). Additionally, we assess the development environment at
Cobolt (refer to Section 2.2). Based on these evaluations, we have identified a
series of iteration stages, outlined in Table 3.1.

Table 3.1: Iteration levels of CI prototype implementation

Level Name Output Description

1 Firmware tar- Compile Automate compilation of the 3 firmware
get compila- warnings & targets Alpaca, Bobcat, and Capybara
tion errors and give feedback on correct compilation,
warnings, and errors.
2 Unit test Business Execute the existing CxxTest unit tests
logic faults that are integrated into Lysa and give
feedback.
3 Download Booting fail- Download firmware target to test board
and run ure and verify the correct bootup: (a) by
firmware on checking the correct firmware info via the
hardware serial interface.
4 System test System test Automate execution of the LATE runtime
results test software and return results.
 Methodology | 29

3.2.3 Hardware to be used

In Section 2.1.6, we have specified that a CI service requires additional
hardware that runs the automation server.
Research conducted by Seth et al. [30] highlighted the benefits of utilizing
a distributed build architecture, consisting of a central server for automation
coordination and additional agent build machines, for CI in the embedded
domain.
Apart from the advantages in the development process, this architecture
also proves to be advantageous from a deployment perspective. By employing
this approach, the IT department is relieved of the burden of managing special
user rights on the principal server, mitigating potential security concerns.
Instead, developers only require full access to the build machine, which can
be treated as a standard workstation.

Principal Server
The principal server is installed inside a virtualisation container on the existing
development server at Cobolt. Since it only acts as a controller and not as
a build agent, it does not require sophisticated hardware. The containerised
environment helps to prevent malicious Jenkins scripts to interfere with the
server itself [42]. From the allocated 900 MB of memory, there were never
more than 250 MB used.

Build Machine
There were two incentives for the selection of the build machine. Firstly, a
machine setup must be selected that allows the execution of all required build
tools (see Section 2.2). Secondly, it is preferred to have a large amount of avail-
able USB slots to attach target hardware test units and debuggers. In general,
selecting hardware is always a trade-off between cost and performance [8].
For this thesis work, an already existing workstation was selected, to not have
extra costs for new hardware. The specs can be obtained from Table 3.2.

J-Link Plus Compact Debugger

For downloading firmware to the hardware targets, we decided to use a J-
Link debugging probe, which allows interfacing the microcontroller memory
content of Cobolt lasers via their JTAG/SWD debugging port.
30 | Methodology

Table 3.2: Specification for the build machine

Component Description
Operating System Windows 10
Processor Intel I5-10500 @3.10 GHz.
Cores 6
Memory 8 GB
Storage SSD Samsung MZVLQ256HAJD-000H1
USB Slots 9

3.3 Interpretive Research Paradigm

We chose to follow an interpretive research paradigm which understands
reality as a subjective view that must be considered in the context that it
is observed in [43]. Knowledge is obtained as the interpretation of the
researcher of qualitative data such as individual experience, formal discussion
and interviews [43]. The research result is not considered to be universal laws
as it is biased by the assumptions and beliefs of the researcher [43]. Yet it is
seen as a valuable contribution to science and is preferably used, when it is
complicated to obtain quantifiable data, i.e. when the research focus lies on
perceptions and new understandings as a contribution [43].
Most of the related literature to this study (see Section 2.4) uses the
interpretative paradigm. They represent case studies using qualitative data
gathered from interviews and obtain their results through interpretation.

3.4 Data Collection

The research questions R1 “What CI techniques are suitable for modular
firmware architectures in embedded systems?” and R2 “What are the efforts
involved in building such a CI infrastructure?” are best addressed by reflecting
on personal experiences gained during the investigation and implementation
process.
Research question R3 “What are the benefits of building such a CI infras-
tructure?” can be answered by comparing quantifiable data before and after
the implementation of the CI service. Another option is to evaluate individual
developer experiences. Interviews and reflections on formal discussions can
be used to describe the situation with and without a CI service qualitatively
and then utilized to understand the benefits of adopting CI.
 Methodology | 31

This section focuses on the various methods used for data collection.
Section 3.4.1 describes how qualitative data is obtained through an under-
standing of the implementation process, allowing evaluation of the efforts
involved in setting up a CI service. In Section 3.4.2, the possibility of
gathering informal user feedback is introduced to comprehend the impact of
the CI service. Section 3.4.3 outlines the design of a questionnaire to be
administered after the final implementation of the CI service, aiming to collect
both qualitative and quantifiable data to provide feedback on the CI service.
Finally, in Section 3.4.4, our plan to utilize quantifiable data obtained from the
automation server is presented to evaluate the effectiveness of the implemented
CI techniques.

3.4.1 Implementation Notebook

To address research question R1, we implemented a range of CI techniques
and conducted an evaluation to assess their applicability in the context of
embedded systems with modular firmware architecture. Furthermore, we
provide our own interpretation and analysis to determine the suitability of these
techniques for such systems.
We wrote a protocol to document the implementation steps and our
thoughts throughout the process. This protocol helps to identify challenges
and solutions, which are used to answer research question R2 concerning the
effort required to build the CI infrastructure. The protocol is summarised in
Chapter 4, where the implementation details are described.

3.4.2 User Feedback

The mindset of the team towards a CI service is an important factor for its
success [3]. Developers need to have confidence in it, to use it appropriately.
User feedback is a valuable tool to measure the acceptance of the implemented
CI service and allow steering the project in the right direction.
Feedback is further relevant to give insights into observations and interpre-
tations by the stakeholders, i.e. about usability and the impact of the system.
Users can identify benefits or downsides which is relevant to answering
research question R3. In the scope of this thesis, this feedback is gathered
in informal discussions with Cobolt’s developers.
32 | Methodology

3.4.3 Final Survey

Additional to the informal user feedback, we also conducted a survey after the
final CI service deployment for more formal feedback. A questionnaire was
filled out by the 3 firmware developers at Cobolt to understand their perception
of the implemented CI service and thereby the used CI techniques. Further,
we observed the opinion of the developers on the impact of CI on Cobolt’s
development process. The following variables led to the questionnaire design:

1. Interests and thoughts on CI.

2. Wishes and hopes on CI service.

3. Feedback on the implemented CI service.

4. Thoughts on improvements and additional features.

The questionnaire is split into three parts. Part A focuses on the general
thoughts of the developers on CI. Part B targets their perception of the
implemented CI service. Part C lets the developers evaluate the CI service
according to the 6 guidelines that were proposed by Ståhl and Bosch [32].
As suggested in Saris and Gallhofer’s [44] book about designing ques-
tionnaires for survey research, an introduction to a set of survey questions
was given. This helps to clarify what the subject is supposed to focus on
when answering the question. The complete questionnaire can be found in
Appendix A.
The survey was conducted using a Google Forms∗ questionnaire. The
questionnaire allowed participants to provide their responses and make revi-
sions within a 48-hour timeframe. Each of the three developers (subjects)
completed the questionnaire individually. To ensure privacy, the results
were reported in an anonymized manner, respecting the confidentiality of the
subjects.

3.4.4 CI Metrics
Qualitative data can be used to support the analysis of qualitative data. There
are two sources for quantifiable data in the CI service: the SCM tool and the
automation server.
∗
https://www.google.com/forms/about/ (Accessed at 22/05/2023)
 Methodology | 33

Table 3.3: Questionnaire

No. Question
Part A
QA.1 What does CI mean to you?
QA.2 In what way does CI change the way of developing?
QA.3 What experiences do you have developing without a CI Service in place?
QA.4 What kind of improvements do you expect from using CI?
QA.5 What downsides do you see in using CI?
Part B
QB.1 In what way did you already use the implemented CI service?
QB.2 What impact does the implemented CI service have on your work?
QB.3 What other feature do you desire to be implemented into the CI Service?
QB.4 Do you have any other feedback on the CI service prototype?
Part C
QC.1 Comprehensive Activities: Are the artefacts produced by the CI service
of value to you (i.e. unit test results)?
QC.2 Effective Communication: Is the output of the CI service easy to access
and understand?
QC.3 Immediacy: Is the CI service output available quickly enough?
QC.4 Appropriate Dimensioning: Does the CI service have a good balance
between completeness and lucidity of the output? In other words: Is a
reasonable amount of activities implemented?
QC.5 Accuracy: Are you confident that the CI service always processes the
recent source code and not an old version?
QC.6 Lucidity: Do you understand all activities included in the CI service and
the chronological order of their execution?

SCM Data. The SCM tool is responsible for managing various metrics
related to the version control system. It keeps track of metrics such as the
number of commits made per branch, per developer, and per specific time
period. Additionally, it provides detailed information about each commit,
including the number of files and lines of code that have been changed, added,
or deleted. This data provides valuable insights into the development activity
and the effects of code changes on the codebase over time. Consequently, it can
be utilised to assess the impact of a CI service on the development process.
However, in the case of Cobolt, we lack a comprehensive understanding of
their development practices, which would be necessary to accurately interpret
this data from before the adoption of the CI service. Therefore, this particular
method cannot be employed to explore the influence of the CI service on the
34 | Methodology

development activity at Cobolt.

Automation Server Data. The automation server stores comprehensive

data on each pipeline execution, including detailed information about the
stages involved. This information encompasses the number of executions, the
duration of each execution, the status of each stage, and a list of the new
commits associated with the pipeline. By analyzing this data, it becomes
possible to evaluate the effectiveness of various stages and, consequently,
assess the impact of different CI techniques that have been implemented within
the CI service.

Python Library for Data Collection. Python-Jenkins∗ offers a convenient

means to access the automation server data by utilising the Jenkins Remote
Access API† . The Jenkins Remote Access API provides information on the
number of pipeline executions within our implementation and allows us to
retrieve detailed information about each stage of the pipeline executions in the
form of a JSON file.
By leveraging Python-Jenkins and parsing the obtained data, we can
generate a list of builds. Each build contains a list of stages, and each stage is
represented by a dictionary with key-value pairs that describe the attributes of
that particular stage. These key-value pairs are displayed in Table 3.4.

3.5 Data Analysis

The process of analyzing the quantifiable data collected for CI metrics must be
conducted in an appropriate manner. This section provides an explanation of
how the two key-value pairs, namely status and durationMillis, are processed.

3.5.1 Stage Status

Analyzing the status key provides valuable insights into the frequency and
occurrence of stage failures. By leveraging the number of stage failures as
an indicator of bugs introduced through tested commits, we can evaluate the
effectiveness of individual stages and, consequently, the effectiveness of the
corresponding CI techniques.
∗
https://pypi.org/project/python-jenkins/ (Accessed at 01/06/2023)
†
https://www.jenkins.io/doc/book/using/remote-access-api/ (Ac-
cessed at 01/06/2023)
 Methodology | 35

Table 3.4: Pipeline stage metrics

Key Description
id Unique identifier for the stage.
name Stage name, that was specified in the stage definition.
execNode Name of the executing node.
status Denotes the execution status of a stage. It can have
one of the following values: [S:SUCCESS, F:FAILED,
A:ABORTED, U:UNSTABLE, NE:NOT_EXECUTED,
IP:IN_PROGRESS, PPI:PAUSED_PENDING_INPUT,
PPA:PAUSED_PENDING_APPROVAL,
U:UNKNOWN].
startTimeMillis Timestamp indicating the start time of the stage execution
in milliseconds.
durationMillis Duration of the stage execution in milliseconds.
pauseDurationMillis Duration of any pauses or delays encountered during the
stage execution in milliseconds.
error Error message or details, if any, associated with the stage
execution.

We employ a Python script that counts the occurrence of each status for
every pipeline stage.

3.5.2 Stage Duration

Our literature research indicates that achieving a fast time-to-feedback is
a significant challenge in adopting CI practices (see Section 2.4.2). The
durationMillis attribute provides information on the time each stage takes
in the pipeline. By analyzing the stage durations, we can evaluate the
effectiveness of various CI techniques in meeting the objective of quick
turnaround times.
To visualize the distribution of stage durations, we plan to utilize a boxplot.
This boxplot is created using the boxplot function provided by the Python
library Matplotlib∗ .
Additionally, we calculate characteristic values such as the mean, standard
deviation, and median using the Python library Numpy† . These values serve
as representative measures for each stage’s duration.
∗
https://matplotlib.org/ (Accessed on 16/06/2023)† https://numpy.org/
(Accessed on 16/06/2023)
36 | Methodology

3.6 Summary
This chapter presents the research methodology for our study on CI for
modular firmware architectures in the embedded systems domain. The
primary objectives of the study include identifying suitable CI techniques,
understanding the efforts and challenges involved in building a CI infras-
tructure and exploring the benefits of implementing such an infrastructure.
The research focuses on implementing and analyzing a CI service prototype
at Cobolt, a company in the embedded systems domain. The emphasis is
on studying the implementation efforts and examining the impact of the CI
service on the development process. The research follows an agile approach,
divided into four iterative stages. Each stage results in a deployable CI
service increment, enabling continuous stakeholder feedback and parallel data
collection. To gather data for the study, a combination of qualitative data
from the researcher’s experience and user feedback, along with quantitative
data from CI metrics, is collected and analyzed. An interpretive research
paradigm is employed in this study, involving the interpretation of qualitative
data within the specific context of Cobolt. This approach allows for a
deeper understanding of the implications and outcomes of implementing
CI techniques in the company. The chapter also discusses the hardware
requirements for implementing the CI service.
 Implementation | 37

Chapter 4

Implementation

This chapter presents the development effort, challenges, and countermeasures

encountered during the implementation of the planned CI service. Each
section is organized to represent a relevant implementation step and should
be read in the specified order. The information presented in these sections is
derived from the implementation notebook as described in Section 3.4.1.

4.1 Local Jenkins Setup and First Test

Same as with the selection of their SCM tool, Cobolt wishes to use a CI server
that is self-hosted. Jenkins is assumably one of the most common, self-hosted
CI tools as shown in [3, 26]. All developers at Cobolt were familiar with it and
thus they were confident, that it would be a good candidate for their CI server.
We set up a first prototype locally on a workstation to validate if Jenkins
would be a good choice. Two days and 7 hours were spent in total, to set up a
Jenkins Freestyle job — a type of job that consists just of one pipeline stage.
This job compiles the Lysa firmware Capybara. One job execution at this stage
only takes 40 seconds.
The most effort went into getting familiar with Jenkins. The implemen-
tation of the freestyle pipeline was intuitive with some scripting knowledge,
however, the tight coupling between the build script and the installation
location of the build tools seems to not be ideal for scaling reasons.
38 | Implementation

4.2 Declarative Pipeline for Parallel Firmware

Compilation
It is possible to create many Freestyle jobs and chain them together by using
triggers. However, for sophisticated pipelines the Jenkins handbook [27]
suggests using declarative pipelines, which are described in Section 2.3.1.
Building the three different firmware targets (Alpaca, Bobcat, and Capy-
bara) is independent of each other. Yet, they share the same initialisation steps,
such as checking out the source code with Git and generating the build system
with CMake. Thus, it makes sense to execute the building stages in parallel
with the initialisation as upstream stages as shown in Figure 4.1.

Checkout Generate
Build Alpaca
SCM: Lysa Build System

Build Bobcat

Build
Capybara

Figure 4.1: Parallel execution of the build stages

This pipeline architecture can be easily adjusted to accommodate ad-

ditional firmware targets. The number of parallel executions is directly
determined by the available executors, which can be increased by adding more
build machines or by utilizing the processor cores of existing machines. It is
worth noting that when employing multiple build machines, the build artefacts
from upstream stages would need to be mirrored to each machine. However,
this was not a concern in our case as we had a build machine capable of
supporting six parallel executions.

4.3 Add Unit Testing Stage

In the next step, we automated the unit testing with CxxTest. The unit testing is
independent of the firmware build process and can thus be executed completely
in parallel.
Sadly, Jenkins’s declarative pipeline syntax does not allow multiple levels
of parallelism without using a workaround with generative functions that
 Implementation | 39

prevent displaying the pipeline correctly in the BlueOcean user interface [45].
We decided to instead execute only the build system generation and unit testing
in parallel as shown in Figure 4.2 and thus are still able to visualise the pipeline
entirely.

Checkout Generate
Build Alpaca
SCM: Lysa Build System

Build Lysa Execute Lysa

Build Bobcat
Unit Test Unit Test

Build
Capybara

Figure 4.2: Parallel execution of the unit testing and build system generation
stage

The unit testing framework is executed on the build machine itself and thus
must be compiled with a C/C++ compiler for x86/x64 architectures.
After adding the unit testing pipeline stage, it did not work right away. It
appeared, that the script that generates the parameters for the unit test suite
generator cxxtestgen is dynamic and dependent on the project location. The
long Jenkins path for the workspace (C:/Jenkins/workspace/Lysa) resulted in
exceeding the Windows command-line string limitation boundary of 8191
chars [46]. The analysis and resolution of this issue took one day. The
final solution was to rewrite the script to generate parameters that use relative
instead of absolute paths, to make the script execution independent from the
source code location.
Finally, we had to decide how the unit test results should be displayed.
The Jenkins handbook [27] suggests that there are plugins for displaying any
widely-used test report format. Since CxxTest can output test results in an
XML format that conforms to the XML standard used by other xUnit tools, we
selected the common JUnit plugin [47] to display the unit test results. Using
the BlueOcean UX plugin. When test cases fail, the Jenkins build is marked
as UNSTABLE. However, when the test suite compilation fails, the Jenkins
build is marked as FAILED.

4.4 Deploying CI Service

At this stage, it was time for the first status report with the stakeholders at
Cobolt. The feedback was positive and it was decided to deploy the CI service
40 | Implementation

in its current status. The proposal to have a distributed service architecture

with a principal server that just acts as a controller and external build nodes
was agreed on immediately. The principal server was installed inside a Docker
container on the same server as the version control system and maintained by
the IT department. The access is hardly restricted to preempt security con-
cerns. Build machines are physically accessible so that additional hardware
can be easily installed. Further, they can be dynamically administered by the
developers through remote desktop protocol (RDP) access with administration
rights. Table 4.1 summarises the deployment specification.

Table 4.1: Deployment specification of the CI service

Principal Server Build Machine

Location Docker container on Workstation that is physically accessi-
a company server. ble to the developers.
Developer Administration of Physical access to attach hardware and
access Jenkins through the administration through RDP to install
web interface. build tools.
Design Security: Limit ac- Flexibility: Maximise accessibility to
incentive cess and activity on reduce integration effort and deploy-
the principal server. ment time.

After physically installing the build machine, connecting it to the intranet

of the company, and installing the build dependencies (see Section 3.2.3), we
registered it as a build node on the principal server. The Jenkins handbook
[27] gives an introduction to how to manage nodes. The only things left to do
was:

1. Create login credentials for each developer and distribute them.

2. Store credentials for the Bitbucket SCM server to get access to the source
code.

3. Copy previously described pipeline job, its build trigger, and SCM
repository connection.

In just one week the CI service detected two feature integrations that broke
the unit testing stage. The respective developers did not know until they
were manually notified. Each time the issue was resolved after less than 1h.
Since this is only a CI service prototype, we opted against implementing email
notifications for unsuccessful builds. This choice aims to avoid any potential
 Implementation | 41

intrusion. Our preference was to encourage developers to engage with the self-
sufficient web frontend for updates instead.

4.5 Firmware Download to Target

For the next iteration level, the process of downloading and running the
firmware on its respective hardware target had to be automated. The build
stage of the automation pipeline produces the required artefact (firmware) for
this stage and thus must be upstream to the downloading stage.

Checkout Generate
Build Alpaca
SCM: Lysa Build System

Build Lysa Execute Lysa Download

Build Bobcat
Unit Test Unit Test Bobcat

Build
Capybara

Figure 4.3: CI pipeline with firmware downloading stage

Figure 4.3 shows only the download of Bobcat. Due to limited hardware
availability, we decided to only use a 06-01 DPL laser for this test setup and
integrate the other Lysa hardware targets later.
Usually, Cobolt developers download firmware to test hardware using the
IAR Embedded Workbench IDE. However, in a CI setup, this needs to be
done using a command line program instead. For this, we used the command
line tool J-Link Commander, which is used in version 6.00h by the IAR
Embedded Workbench IDE. The biggest challenge was thereby, that the J-
Link Commander, does not allow to specify the firmware image as a program
parameter and instead requires to select the file in a command line dialogue.
Luckily, there is the option to automate tasks using the J-Link script language
[48]. The workaround was thus, to dynamically create a J-Link Command File
in the Jenkins pipeline that would then be given as an argument to the J-Link
Commander.
After initially implementing the downloading stage, we discovered that the
new firmware was not booting correctly. It became clear that the Ninja build
system that was generated by CMake did not compile an identical firmware
image to the one compiled by the IAR Embedded Workbench IDE. It was not
clear why the images differed, but a workaround was found after 3 days. The
IAR Embedded Workbench IDE uses the command line tool iarbuild to build
IAR project targets. After changing the Jenkins pipeline to use the iarbuild
42 | Implementation

build system instead of Ninja to compile the firmware image, everything

worked as expected.
The purpose of downloading the firmware to a hardware target was to
detect booting failures, as specified in Section 3.2.2. To achieve this a serial
connection would have to be set up to the laser, to confirm, that the firmware
is running, by testing the communication channel. We realised that this is
redundant, as this was going to be done in iteration level 4 “system testing”.

4.6 Adding System Testing Stage

At the final iteration level, we implemented system testing with Cobolt’s LATE
framework. Just like the downloading stage, we only added system testing
for the Bobcat firmware for now, as only a 06-01 DPL laser was available.
Nevertheless, the general strategy can easily be duplicated for testing the other
firmware targets. Figure 4.4 presents the resulting CI pipeline.

Checkout Generate
Build Alpaca
SCM: Lysa Build System

Build Lysa Execute Lysa Build Download System Test

Unit Test Unit Test Bobcat Bobcat Bobcat

Checkout Execute Late Build

SCM: Late Unit Test Capybara

Figure 4.4: CI pipeline with system testing stage

Notice, that there are two new, additional stages. Apart from the actual
system test stage, there is also a stage for unit testing the LATE test program
with the program pytest, which is executed in parallel to the Lysa unit testing
stage. Pytest offers printing test results in xUnit XML format and thus can
easily be parsed to the general test viewer using the Jenkins JUnit plugin [46]
The LATE testing framework included a large number of tests, that
required manual user input, to the command line and the laser hardware. Those
tests were excluded from the initial CI service implementation, as they would
require designing a hardware-in-the-loop simulation (HILS) setup to simulate
the input. Since LATE is a rather new software product at Cobolt, it is not so
feature-rich and we had to implement a parser for xUnit XML output first.

4.7 Final CI Service

Finally, a CI service infrastructure was created as visualised in Figure 4.5.
 Implementation | 43

CI Controller installed on
Webinterface
(Jenkins)

installed on installed on
Principal Server

monitors for
code changes CI Controller
SCM (Jenkins)
monitors for (Bitbucket)
CI feedback

initiate and control scalable

job execution

get/store Lysa and get Lysa and

Late source code Late source code
CI Agent
(Jenkins)
works on

Developer
installed on
Compiler toolchain
can manipulate
and build tools

connected
installed on installed on
connected to to

Laser hardware Local Workstation Build Server Laser hardware

Figure 4.5: CI infrastructure with principal-agent architecture

Each developer works on their personal workstation as they did before the
CI service implementation. The workstation has all the necessary tooling
installed and is connected to laser hardware, so the developer can test and
debug their code. A developer usually works on code changes that are specific
to one specific laser, that the code is tested on.
When a feature or a bug fix is ready, it is pushed to the SCM system and
code changes are merged. Jenkins is installed on the same server as the SCM
system and monitors it for code changes. When new code is available, it
triggers a pipeline execution on an external build server, that has the same
tooling installed as the workstations of the developers. The build server is
supposed to test on all available hardware targets, that are compiled from the
Lysa firmware project. However, for this prototype only one Cobolt 06-01 DPL
laser [25] is connected, thus only one firmware is tested with the system testing
framework LATE. More build machines can be integrated easily to scale up
the number of test runners.
44 | Implementation

The results of the pipeline execution are reported back from the Jenkins
Agent to the Jenkins controller on the principal server and displayed in a web
interface. Developers monitor the web interface to receive feedback on the
pipeline execution. When tests or stages of the pipeline fail, they are supposed
to react instantly by hot-fixing the issue.
After completing the setup of the CI service and infrastructure deployment,
the next step was to discuss its usage with the Cobolt developers. In their work,
Ståhl et al. [32] propose an integration flow model that effectively visualises
CI processes. Figure 4.6 shows the model that was presented to the Cobolt
developers.

manual

triggers

triggers Jenkins Pipeline

SCM: Lysa source SCM: Late
Execution
(mainline) source (jenkins_dev)

Figure 4.6: CI integration flow model

In the figure, a CI activity is represented by a rectangle. Triangles

symbolise SCM repositories, which can serve as triggers and sources for a
CI service. Circles represent external triggers that are not SCM repositories.
Solid lines indicate trigger relationships, while dashed lines represent source
relationships.
For our CI service, we utilise the mainline branch of the SCM repository
for project Lysa as the primary source and trigger. Whenever a new code
change is detected on this branch, the Jenkins pipeline depicted in Figure 4.4
is executed. Within the pipeline, the Late framework is checked out from the
"jenkins_dev" branch of the corresponding SCM repository. Alternatively, de-
velopers can manually trigger the Jenkins pipeline through the web interface.
In a final deployment notification, we reminded the Cobolt developers how
to access the CI service feedback and encouraged them to refer to the Blue
Ocean pipeline view after each code change. Furthermore, we emphasised the
importance of promptly addressing any bugs or issues that result in unstable
or failing stages.
 Implementation | 45

4.8 Summary
This chapter details the step-by-step CI service implementation and addresses
challenges with proactive solutions.
We established a local Jenkins server for initial testing and created a
Freestyle job for compiling Lysa firmware Capybara, before introducing a
declarative pipeline for parallel firmware compilation. Three distinct firmware
targets were built independently, sharing common initialization steps.
We added a unit testing stage using CxxTest. The build system generation
and unit testing were executed concurrently. The script for generating the
unit testing suite faced issues with long file paths, necessitating rewriting for
relative paths. The JUnit plugin displayed unit test results in Jenkins’ web
view, marking builds as successful, unstable, or failed based on tests.
We addressed deploying the CI service next, proposing a distributed
architecture with a principal server as a controller and external build nodes
for scalability. The principal server was Docker-installed, while the physically
accessible build machine facilitated hardware installation and administration.
Then, we added a download stage to the automation pipeline, utilizing J-
Link Commander CLI for the firmware download. Challenges arose spec-
ifying the firmware image as a program parameter, requiring J-Link script
language.
We introduced a system testing stage using Cobolt’s LATE framework.
The pipeline included parallel unit testing of the LATE framework alongside
the unit testing of Lysa. Parsing xUnit XML from LATE testing required the
implementation of a custom parser.
The chapter concludes with CI service infrastructure setup. Developers
continued working on workstations, linked to laser hardware. Pushing code to
the Git repository triggered the CI service to build and test, offering feedback
on status and results.
46 | Implementation
 Results and Analysis | 47

Chapter 5

Results and Analysis

In the subsequent sections, we present the findings obtained from our study. To
begin with, Section 5.1 analyses the results derived from the implementation
notebook, which is summarised in Chapter 4. Then, Section 5.2 reflects on
the informal user feedback that was gathered throughout the study. Moving
on, Section 5.3 presents and summarizes the knowledge acquired from the
final survey, offering additional perspectives on the perceived impact of
the implemented CI service. This section includes an overview of the
survey results and the insights they provide. Lastly, Section 5.4 presents
the quantifiable data collected using the Jenkins API. We explain the nec-
essary preparations undertaken to analyse the data according to the methods
described in Section 3.5.

5.1 Implementation Notebook

In Chapter 4, we documented the implementation efforts. Now, in this section,
we aim to analyse and reflect on our findings. Based on our implementation
notebook, the following refined lessons can be derived:
1. Learning to use a CI tool like Jenkins effectively requires a significant
initial effort. Providing training resources can be beneficial in facilitat-
ing the learning process.
2. Jenkins is well-suited for implementing CI services iteratively in an
agile manner. Its pipeline syntax allows for easy addition of stages and
adjustment of their execution order.
3. The utilisation of proprietary components, such as the RTOS EmbOS,
introduces dependencies on specific build tools like the IAR toolchain.
48 | Results and Analysis

This dependency can be a limiting factor when attempting to incorporate

modern build tools such as Ninja.

4. CI services in the embedded domain necessitate build machines that are

equivalent to developers’ workstations. These machines must run the
same tooling and be connected to hardware when tasks like firmware
downloading and system testing need to be performed.

5. Jenkins simplifies the setup of distributed build infrastructure. It

enables the use of build machines, as described above, that can be
located alongside developers, allowing physical access. Additionally,
Jenkins facilitates vertical scaling and is therefore ideal for concurrently
validating multiple firmware targets.

6. System testing poses challenge es, particularly when certain tests require
manual user input, which cannot be automated without a HILS system.

7. Jenkins JUnit plugin makes it easy to visualise test results. However,

they must be given in the xUnit XML format. Many unit testing tools
can print their results natively within this format (CxxTest, Pytest), and
implementing a parser for adding it to the system testing framework
LATE was fast.

8. Developers create build scripts with the objective of maximizing ef-

ficiency, as most of the development time should be dedicated to
delivering results through feature development. For CI purposes, build
scripts should be designed to be independent of the source code’s file
path to avoid issues similar to those encountered with the execution of
cxxtestgen (as described in Section 4.3).

5.2 User Feedback

In this section, we present the informal user feedback we received during the
course of this thesis project.
During the first status report meeting on March 22nd, we showcased the
CI service, including the unit testing stage, to the developers. The response
was overwhelmingly positive, and the developers expressed their interest in
utilising the service. They specifically mentioned their appreciation of the
intuitive Jenkins Blue Ocean UX to view the CI feedback. As a result, we
made the decision to deploy the system.
 Results and Analysis | 49

However, following the deployment, it became apparent that the developers

were not adequately prepared to use the CI service. Code changes were being
made without subsequent checks on the CI feedback. Initially, two users
failed to notice when they broke the unit testing stage. Upon notification, they
promptly resolved the respective bugs and began independently reviewing the
CI feedback.
On one occasion, a user questioned the functionality of the CI service when
the unit testing stage failed after their code change, despite being confident that
their code was correct. We believe it is natural for users to require time to gain
confidence in a newly deployed tool.
Until the implementation of the system testing stage, we received minimal
feedback, and users seemed to be using the service as expected. One user
reported that he likes the clear representation of successful and failing stages
in the pipeline view of the Jenkins plugin Blue Ocean UX.
After the introduction of the system testing stage, discussions arose re-
garding the necessity of passing all LATE tests. One developer argued that it
slowed down the development process, but eventually conceded that it might
be a necessary trade-off to maintain a correct code base.
It further seems that developers lack confidence in the effectiveness of
the LATE system testing framework. This lack of confidence stems from
the newness of the framework and its ongoing development, which involves
numerous design choices. Consequently, developers often prioritise other
tasks over addressing issues with the LATE tests, especially considering that
it is currently utilised solely for a CI service prototype.
One user provided two feature requests for the CI service. Firstly, they
expressed a desire to automate the release process, specifically by implement-
ing a mechanism to store the firmware binary in a file server. This would
ensure that engineers from other departments always have access to the latest
firmware for testing purposes.
Secondly, the user requested the ability to track firmware metrics, such
as the memory allocated to the static heap and stack on the microcontroller.
This information would enable developers to monitor and optimise memory
usage in the firmware, leading to more efficient code and potentially avoiding
memory-related issues.

5.3 Survey
The survey was sent out on Monday, 2023-05-23 at 11:30 to the three
developers at Cobolt. All replies were received within 24 hours. In this
50 | Results and Analysis

Section, we summarise the relevant results achieved from Parts A, B and C

of the form. See Appendix A for a table with all answers. The respondents are
anonymised and are named D1, D2 and D3.

5.3.1 Part A: General Questions

This section summarises and discusses the results given for Part A of the
questionnaire, which is considered to give an overview of the respondents’
general understanding of CI. We want to use the given answers to reflect on
the developers’ interest and thoughts on CI as well as their wishes and hopes
in CI adoption at Cobolt. Each paragraph shows the results of one question.

QA.1: What does CI mean to you? All three respondents answered that
CI means automated testing to them. D2 mentioned “safeguarding the main
branch” from introducing bugs as a purpose of the automated testing. D3
thinks that fast development and delivering cycles is a motivation for using
CI.

QA.2: In what way does CI change the way of developing? CI impacts the
way of developing by removing test effort from developers (D1, D2, D3). In
software projects with multiple compilation targets, this is wanted because of
the higher regression testing effort (D3). According to D3, Lysa’s modular
firmware architecture shares 90-95% of the source code between multiple
targets, which means that code changes most likely affect all of them. D1
mentions that the usage of CI for testing removes the “works on my machine”
factor, as the build machine represents a non-changing build environment,
unlike the developers’ workstations. All three developers agree, that CI helps
to improve the confidence that checked-in code is correct, which D3 sees as
a key enabler for more frequent releasing. Ultimately, D3 finds it pleasant
that CI allows automating processes with many manual steps (i.e. the release
process that consists of 15 manual steps at Cobolt).

QA.3: What experiences do you have developing without a CI Service

in place? D1 mentions that without a CI service developers have to be
more cautious when integrating new code. D2 says, that “confusion and/or
annoyance” can occur because small flaws (i.e. forgotten validation steps) can
result in big issues. Those are sometimes only detected after some time. We
assume that D2, mentions this because it is then harder to trace problems to
their origin.
 Results and Analysis | 51

QA.4: What kind of improvements do you expect from using CI? D1

sees faster feedback about bugs as an improvement of using CI. It was further
posed that CI can be used to automate repetitive tasks (D2) and that it helps
developers to make better use of their time (D2, D3).

QA.5: What downsides do you see in using CI? The downsides of CI are
linked to the initial effort that goes into the setup (D1) and its maintenance
overhead (D2, D3). D3 sees it as an issue, that somebody needs to be
responsible for the CI and D1 argues, that the CI server might get overloaded
which results in unnecessary wait time.

5.3.2 Part B: Feedback on the CI Service

Part B of the questionnaire was designed to receive direct feedback on the
implemented CI service and its impact on the development process at Cobolt.
Further, we want to understand what additional features are desired and if
anything can be improved.

QB.1: In what way did you already use the implemented CI service? In
the scope of this thesis, D1 was only assigned to projects that were not using
the Lysa project and thus didn’t use CI service so far. The other two developers
used the pipeline view of the job execution to verify their code integrations.
D3 looked closer into the CI activities to plan the release process automation.

QB.2: What impact does the implemented CI service have on your work?
Despite not having used the CI service so far, D1 assumes that when they need
to work with Project Lysa again, it can be considered that the main branch is
in a stable state. D3 comes to the same conclusion and has already used the
CI service. D2 and D3 report that they are more motivated. D2 argues that
implemented CI makes the work-life “less cumbersome” which we believe
is due to the automation of otherwise manual tasks. Using CI apparently
eliminates the uncertainty of making mistakes (D3). We assume that D3
talks about mistakes that happen when validating code changes during their
integration. Overall the implemented CI gives confidence in Project Lysa’s
code quality (D3).

QB.3: What other feature do you desire to be implemented into the CI

Service? The most wanted feature seems to be the automating of the release
process (D1, D2, D3) which consists of 15 manual steps as reported in D3’s
52 | Results and Analysis

answer to question QA.2. D2 further mentions the wish to have elaborate

hardware tests in an environment that can simulate physical input to the laser
hardware. We assume he means a HILS system that can automate the physical
input.

QB.4: Do you have any other feedback on the CI service prototype? D1

didn’t give an answer, while D2 and D3 use this question to express their
positive perception of the implemented CI service. D3 further gives another
answer to the impact of the implemented CI service (QB.3), which is an
increased working speed.

5.3.3 Part C: Quantitative Evaluation of the CI Service

The quantitative evaluation of the implemented CI service aligns well with
the positive feedback given in QB.4. Figure 5.1 presents the values given
by the three contributors in a radar chart. The larger the area enclosed by a
graph, the stronger the endorsement of the evaluation criteria. On average, all
categories are evaluated with a score between the two highest values (5 and
6). Respondent D1, who didn’t use the CI service so far gives the lowest score
in all categories.

Comprehensive Activities D1
6 D2
5 D3
Average
4
Lucidity 3 Effective Communication

Accuracy Immediacy

Appropriate Dimensioning

Figure 5.1: Radar chart for CI service evaluation

 Results and Analysis | 53

5.4 CI Server Metrics

Using the method that we described in Section 3.4.4 we obtained a dataset of
121 pipeline executions. Each pipeline execution includes information about
all the stages that were executed as part of it. The following section provides
a summary of the results obtained from this dataset.

5.4.1 Data Preperation

When initially processing the data we noticed that unlike originally antici-
pated, the key id does not actually represent a unique stage identifier. It seems
that one stage can have multiple different ids and even one id can be reassigned
to another stage. Table 5.1 presents all existing stage ids and the stage names
that were allocated with them throughout the whole dataset. Note that the stage
names do not reflect the names that were used in Chapter 4 and instead are the
names used for the Jenkins pipeline description.

Table 5.1: Allocation table for stage ids and names

id name
0 1 2
6 Declarative: Checkout SCM
13 Unit testing Pre Firmware Build Declarative: Post Actions
17 CMake: Generate Build System
18 CMake: Generate Build System
19 Unit testing
21 CMake: Generate Build System / IAR Projects Pre System Testing
22 Build Firmware Targets
23 Unit Testing Build Firmware Targets Build Test Suite
25 CMake: Generate Build System / IAR Projects
27 Clone LATE framework
29 Build Test Suite
30 Clone LATE framework Alpaca
31 Alpaca
32 Build Test Suite Bobcat
33 Bobcat
34 Execute Test Suite Capybara
35 Capybara
38 Execute Late unit tests
40 Execute Test Suite
46 Build Firmware Targets
47 Execute Late unit tests
48 Declarative: Post Actions
49 Declarative: Post Actions
50 Execute Test Suite
51 Declarative: Post Actions
54 Alpaca
55 Build Firmware Targets
56 Bobcat
54 | Results and Analysis

58 Execute Test Suite Capybara

62 Build Firmware Targets
63 build Alpaca
65 build Bobcat
67 Capybara
70 Build Firmware Targets Alpaca
71 upload
72 build Bobcat
73 upload
74 build Capybara
75 upload
76 upload
78 Alpaca
79 build
80 upload Bobcat
81 build
82 Capybara
83 download
87 build
89 build
90 download test
94 test
95 download
96 download
98 download
99 test
100 download
102 test
105 test

From Table 5.1, it can be observed, that some stage ids are allocated
with 2 and sometimes even 3 stages. Our first guess was, that this reflects
a changing name of one specific stage. However, we never changed the
stage with the name “Alpaca” throughout the whole implementation process.
Yet, the allocated name for id 30 changed from “Alpaca” to “Clone LATE
framework”. A more plausible explanation is, that when changing the pipeline
architecture, some stages get a new identifier, which allows reallocating the
now obsolete ones, even though they have been used before.
Unfortunately, we did not know about this prior to the implementation
process. Otherwise, we would have given every stage unique consistent
names so we could later sort the data with them. Now some stages such as
“upload” (later renamed to “download”) were used to describe both the stage
of downloading the Bobcat as well as the Capybara firmware. Yet, most stages
have unique names and can be further processed.
 Results and Analysis | 55

Data Slicing
To capture different iteration states of the CI service, we divided the dataset
into meaningful slices, each representing a subset of pipeline executions.
These slices were carefully selected to ensure they reflect specific iteration
levels of the implementation process. Importantly, there are no duplicate
allocations of any id across these slices. The table below, Table 5.2, provides
an overview of the slicing, along with a description for each slice.

Table 5.2: Slices for further data analysis

Nr. Slice Description

1 1–52 Setup process
2 53–76 Stable unit testing
3 77–79 Stable firmware downloading
4 80 Setup for LATE
5 81–121 Stable system testing

We can disregard slices 1 and 4 as they do not reflect a stable state of

the CI pipeline, and therefore, they do not offer valuable feedback on the
corresponding CI techniques.
The remaining three slices represent stable states of the pipeline. Slice
2 encompasses pipeline executions that occurred after the unit testing stage
was deployed. Slice 3 represents pipeline executions after including firmware
downloading to the CI service. Finally, Slice 5 reflects the state of the CI
service after implementing system testing, thereby representing the final state.

Ignoring Alternating Stage Ids after Failing Upstream Stages

From Table 5.1, it is also observable that some stage names are allocated
to multiple stage ids. As previously mentioned, some stages were given the
same name, i.e. the stage for downloading the firmware to the Bobcat and the
Capybara target were both called “upload” (was later renamed to “download”).
However, “Declarative: Post Actions” was the name for only one stage. Yet,
there are multiple ids for it.
During our analysis, we observed that when upstream stages fail, down-
stream stages are assigned alternate ids before they themselves fail. Table 5.3
presents the status of all stages in all pipeline executions of Slice 2. Table
3.4 lists all possible status values and their abbreviations. It is evident
that whenever the stage “CMake: Generate Build System” fails, which
56 | Results and Analysis

Table 5.3: Status of all stages in Slice 2 (S: SUCCESS; F: FAILED; NE:
NOT_EXECUTED)

id name pipeline execution

53 54–56 57–64 65–70 71–72 73 74 75–76
6 Declarative: Checkout SCM S S S S S S S S
13 Unit testing S F S F S S F S
18 CMake: Generate Build System S S S S S S S S
22 Build Firmware Targets S S S
23 Build Firmware Targets S S S S S
30 Alpaca F F F
31 Alpaca NE NE NE NE NE
32 Bobcat F F F
33 Bobcat S S S F S
34 Capybara F F F
35 Capybara S S S S S
48 Declarative: Post Actions S S S
51 Declarative: Post Actions S S S S S

occurs before the stages “Alpaca,” “Bobcat,” and “Capybara” responsible for
compiling different firmware targets, as well as the “Declarative: Post Actions”
stage used for parsing unit test results to the JUnit plugin, these downstream
stages receive alternate stage ids. We excluded those ids from further analysis,
as they do not represent properly executed stages.

Ignore Stage Blocks

We also noticed that Jenkins records stage blocks — stages that consist of a set
of other stages but do not have another function — in the same way as regular
stages. Table 5.4 presents the status of all stages in all pipeline executions of
Slice 3. There are two types of stage blocks. Stage id 13 “Pre Firmware Build”
describes a parallel block stage that contains the unit testing of project Lysa and
the build system generation with CMake. Stage id 58 “Unit testing” describes
a sequential block stage that contains stage id 23 and 34, the building and
execution of the unit test suite. We removed all stage blocks from the analysis.

Merge Identical Stages

We mergeed the results for multiple stage ids when they refer to the same
activity. Table 5.4 has multiple ids for the stages “upload [Bobcat]” and “up-
load [Capybara]” which presumably resulted from an architecture adjustment
throughout the pipeline executions.
 Results and Analysis | 57

Table 5.4: Status of all stages in Slice 3 (The stage names “build” and
“upload” were manually appended with a descriptor; S: SUCCESS; NE:
NOT_EXECUTED)

id name pipeline execution

77 78 79
6 Declarative: Checkout SCM S S S
13 Pre Firmware Build S S S
19 Unit testing S S S
21 CMake: Generate Build System / IAR Projects S S S
23 Build Test Suite S S S
34 Execute Test Suite S S S
46 Build Firmware Targets S S S
54 Alpaca NE NE NE
56 Bobcat S S S
58 Capybara S S S
63 build [Bobcat] S S S
65 build [Capybara] S S S
71 upload [Capybara] NE
73 upload [Bobcat] S
75 upload [Capybara] NE
76 upload [Capybara] S
80 upload [Bobcat] S

5.4.2 Stage Status

We wrote a Python script to analyse the status data of the relevant slices 2,
3, and 5. Tables 5.5, 5.6 and 5.7 display how often each status was found for
each stage identifier. The value NaN represents hereby, that there was no data
available for the respective stage for a pipeline execution.

Slice 2 — Unit Testing

Slice 2 consists of 7 individual stages. The stage “Declarative: Checkout
SCM” is checking out the source code for project Lysa. “Unit testing” executes
the building and execution of the unit testing of Lysa. The stages with the
names “Alpaca”, “Bobcat” and “Capybara” refer to the build stage of the
respective firmware targets. “Declarative: Post Actions” is the name for a
pipeline post stage that parses the unit test results to the Jenkins JUnit plugin.
From Table 5.5 it can be seen, that the unit testing stage failed 10 times and
the Bobcat building stage failed once. The stage “Alpaca” was never executed
58 | Results and Analysis

since its development is halted and it is not in a functioning state.

Table 5.5: Status count for each stage of Slice 2 (S: SUCCESS; F: FAILED;
NE: NOT_EXECUTED; NaN: No value)

nr name id status
S F NE NaN
1 Declarative: Checkout SCM 6 24 0 0 0
2 Unit testing 13 14 10 0 0
3 CMake: Generate Build System 18 24 0 0 0
4 Alpaca 31 0 0 14 10
5 Bobcat 33 13 1 0 10
6 Capybara 35 14 0 0 10
7 Declarative: Post Actions 51 14 0 0 10

Based on the observations from Table 5.3, it is apparent that pipeline

executions with failing unit tests occur in clusters. For example, after
encountering a failure in the unit testing stage in pipeline execution 54, it took
the respective developer three iterations before resolving the issue in execution
57. It is important to note that a failing unit testing stage does not necessarily
indicate failed unit tests. The status “UNSTABLE” in the stage “Declarative:
Post Actions” would be used to report failed unit tests. Instead, a failing unit
testing stage suggests that the unit test suite could not be compiled successfully.
In pipeline execution 73, it was observed that the firmware target Bobcat
failed to build, while Capybara was successfully built. Before fixing the bug
in execution 75, the unit testing stage failed once in execution 74.

Slice 3 — Firmware Download

Slice 3 consists of only 3 pipeline executions where no stage failed (see
Table 5.6). The unit testing stage was split up into two sequential stages to
differentiate between failing the build or execution of the unit test suite. The
stage “Declarative: Post Actions” used to parse the unit test results, is now
included in the “Execute Test Suite” stage. The stage build system generation
stage was renamed after it was adjusted to produce the IAR projects for
building the firmware with the IAR Build Tool, as reported in Section 4.5.
Finally the stages “Bobcat” and “Capybara” were split up into a building and
downloading stage. Note that the downloading stage was named “upload” at
this point. This mistake was corrected in Slice 5.
In pipeline execution 77 the Capybara firmware was downloaded to the
Cobolt 06-01 DPL laser, which is the hardware target for the Bobcat firmware.
 Results and Analysis | 59

This was fixed with pipeline execution 78. This architectural modification
of the pipeline resulted in the downloading stages being assigned to multiple
stage ids. We merged their results.

Table 5.6: Status count for each stage of Slice 3 (The stage names “build”
and “upload” were manually appended with a descriptor; S: SUCCESS; F:
FAILED; NE: NOT_EXECUTED; NaN: No value)

nr name id status
S F NE NaN
1 Declarative: Checkout SCM 6 3 0 0 0
2 CMake: Generate Build Sys- 21 3 0 0 0
tem / IAR Projects
3 Build Test Suite 23 3 0 0 0
4 Execute Test Suite 34 3 0 0 0
5 Alpaca 54 0 0 3 0
6 build [Bobcat] 63 3 0 0 0
7 build [Capybara] 65 3 0 0 0
8 upload [Capybara] 71, 73, 76 1 0 2 0
9 upload [Bobcat] 75, 80 2 0 0 1

Slice 5 — System Testing

Slice 5 represents the state of the CI service after the completion of the
pipeline. As of the data collection date, a total of 41 pipeline executions
were recorded in this slice. Two additional stages, namely “Clone LATE
framework” and “Execute Late unit tests,” were introduced in this slice.
These stages are responsible for checking out the source code for the LATE
framework from the SCM system and executing its unit tests to validate the
stability of the LATE framework. It is important to note that, unlike the
Lysa unit test suite, the LATE test suite does not require building as it is
a Python program. Finally, the stage “test [Bobcat]” reports on the system
testing activity.
We merge the stages with the ids 83, 95, 96, 98 and 100, as we identified
them to represent the same stage.
Table 5.7 reveals that the new system testing stage “test [Bobcat]” expe-
rienced two failures that happen in execution 98 and 99. These failures were
caused by a connection error with the laser. Further investigation revealed
that the issue originated from a bug in the serial interface driver. This was
60 | Results and Analysis

Table 5.7: Status count for each stage of Slice 5 (S: SUCCESS; F: FAILED;
U: UNSTABLE; NE: NOT_EXECUTED; A: ABORTED; NaN: No value)

nr name id status
S F U NE A NaN
1 Declarative: Checkout SCM 6 35 1 0 0 5 0
2 CMake: Generate Build Sys- 25 34 0 0 1 0 6
tem / IAR Projects
3 Clone LATE framework 27 34 0 0 0 0 7
4 Build Test Suite 29 34 0 0 0 0 7
5 Execute Late unit tests 47 34 0 0 0 0 7
6 Execute Test Suite 58 31 3 0 0 0 7
7 Alpaca 78 0 3 0 31 0 7
8 download [Bobcat] 83, 95, 96, 27 7 0 0 1 6
98, 100
9 build [Bobcat] 87 27 7 0 0 0 7
10 build [Capybara] 89 29 5 0 0 0 7
11 test [Bobcat] 105 2 2 23 0 0 14

resolved in execution 101, leading to successful subsequent executions of the

system testing stage.
The stage “test [Bobcat]” reported the status “SUCCESS” only two times
in execution 102 and 103. For most of the other pipeline runs it was either
“UNSTABLE” or not even executed when the pipeline failed already before.
The status “UNSTABLE” indicates that the outcomes of one or more test cases
diverge from the anticipated values. Developers frequently encountered issues
with failing LATE tests after doing interface adjustments and were uncertain
regarding the process of reworking the tests. The test scripts were stored in a
separate SCM repository from the Lysa source code. Additionally, they were
isolated within an individual development branch. We believe, that this led to
the developers neglecting to fix the “UNSTABLE” status.
In execution 100, an incident occurred where a developer accidentally
initiated the execution from the parallel block stage “Build Firmware Targets”,
instead of starting it from the beginning. Thus, the stage “CMake: Gen-
erate Build System/IAR Projects” had the status “NOT_EXECUTED”. The
respective developer subsequently aborted the same run at stage “download
[Bobcat]”.
From pipeline executions 105 to 109, it was observed that the stage
“Declarative: Checkout SCM” was not functioning properly. The pipeline
appeared to be stuck at this stage, prompting a developer to abort it. Despite
 Results and Analysis | 61

efforts, the exact cause of this issue could not be determined and it was decided
to restart the build machine. This caused a failed status in execution 110 but
apparently solved the issue and the stage started functioning correctly again in
execution 111.
The unit test execution stage “Execute Test Suite” failed 3 times due to
faulty code changes in execution 111, 112 and 116. Unlike in most of the
other cases that did not result in a changing stage id for most downstream
stages. Thus, the stages “Alpaca”, “build [Bobcat]”, “build [Capybara]” and
“download [Bobcat]” report 3 extra failing states, even when they were actually
skipped.
In execution 96, it occurred that the build stage for Bobcat failed, while
the Capybara firmware was still building correctly. This was resolved by the
respective developer in two iterations. In execution 118 and 119, both Bobcat
and Capybara could not be built. This was resolved at execution 120. The four
failures of “build [Bobcat]” resulted in 4 failure reports for stage “download
[Bobcat]” when it actually was skipped.

5.4.3 Stage Duration

We created a Python script to analyze the duration data for slices 2, 3, and 5.
The script calculates the mean, median, and standard deviation to understand
the duration of each pipeline stage. It also generates a box plot to visually
represent the data.

Slice 2 — Unit Testing

Figure 5.2 shows the distribution of data points for the duration of each stage
recorded in Slice 2. Table 5.8 shows the mean, median and standard deviation
values for the duration.

Table 5.8: Analysis of the stage durations of Slice 2

nr name id mean [ms] stdev [ms] median [ms]

1 Declarative: Checkout SCM 6 3584 2354 2536
2 Unit testing 13 42401 1912 41948
3 CMake: Generate Build System 18 940 871 1220
4 Alpaca 31 189 59 174
5 Bobcat 33 31425 7231 34029
6 Capybara 35 31693 5294 33643
7 Declarative: Post Actions 51 230 213 151
62 | Results and Analysis

104
duration [ms]

103

102

1 2 3 4 5 6 7
stage nr

Figure 5.2: Logarithmic box plot for each stage duration of Slice 2

The unit testing stage (nr 13) takes the longest with 42401 ± 1912 ms.
The second longest takes the building of the firmware. Bobcat takes 31425 ±
7231ms and Capybara 31693±5294ms. The large standard deviation for those
firmware build stages comes from one outlier stemming from the pipeline
execution 73. The build of Bobcat failed after 8135 ms and Capybara builds in
just 16426 ms. Both firmware targets are built in parallel and we assume, that
because of the fast failure of the Bobcat build, more hardware resources were
distributed to the Capybara build executer.
The firmware target Alpaca was marked to be skipped in the pipeline
description file and its execution time is measured to be 189 ± 59 ms.

Slice 3 — Firmware Download

Slice 3 consists of only 3 data points and every stage has only 3 or lesser
reported durations (see Table 5.4). In consequence, outliers have a stronger
influence on the mean and median value of the stage duration.
The duration values of Slice 3 for stage id 6 that are reported in the box
plot (see Figure 5.3) lie in the same area as for Slice 2. Yet, the characteristic
 Results and Analysis | 63

105

104
duration [ms]

103

1 2 3 4 5 6 7 8 9
stage nr

Figure 5.3: Logarithmic box plot for each stage duration of Slice 3

Table 5.9: Analysis of the stage durations of Slice 3

nr name id mean [ms] stdev [ms] median [ms]

1 Declarative: Checkout 6 10875 5858 13009
SCM
2 CMake: Generate Build 21 6112 1286 6840
System / IAR Projects
3 Build Test Suite 23 46206 758 46707
4 Execute Test Suite 34 1014 245 864
5 Alpaca 54 166 17 155
6 build [Bobcat] 63 78787 1554 78534
7 build [Capybara] 65 78756 1281 78488
8 upload [Capybara] 76 18567 0 18567
9 upload [Bobcat] 75, 80 18617 25 18617

values differ quite a lot. While this stage takes 3584 ± 2354 ms in Slice 3, it
takes 10875 ± 5858 ms in Slice 3.
In execution 77, stage id 6 took 2875 ms, while in execution 78, it took
16741 ms. The command line logs of Jenkins revealed that in execution 78,
64 | Results and Analysis

the git repository had to be initialised instead of just being synchronised with
the SCM. It is assumed that the workspace folder was manually deleted before
execution 78.
To verify this assumption, we manually deleted the workspace and trig-
gered a new pipeline execution. Jenkins reported an execution time of
19533 ms, supporting our conclusion that deleting the workspace led to the
need for git repository initialization.
Another noticeable observation is the increased duration for the build
system generation and firmware building stages from Slice 2 to Slice 3. The
build system generation now takes an average of 6112 ± 1286 ms. The build
of Bobcat takes an average of 78787 ± 1554 ms, and Capybara requires an
average of 78756 ± 1554 ms to build. We attribute this increase to the switch
from using the faster Ninja build tool to the slower IAR build tool.
The stage “upload [Capybara]” has only one valid record point (18567 ms)
from execution 77, before being skipped in the other two executions of Slice
3. Downloading the firmware to the Bobcat target takes 18617 ± 25 ms.
The unit testing, which was split up into a build and an execution phase
takes 46206 ± 758 ms for the building and 1014 ± 245ms which is slightly
higher than what was reported for Slice 2. We attribute this increase to the
additional step introduced in execution 77, where the build folder is always
deleted before the building process begins. This extra step adds some overhead
to the overall execution time.

Slice 5 — System Testing

Slice 5 is the largest slice with between 27 and 41 data points for each stage.
This section presents the duration for each stage that are part of the final CI
service. Figure 5.4 shows the distribution of the durations for each stage and
Table 5.10 presents the characteristic values for Slice 5.
After careful consideration, we manually removed the reported values
for the following stages:“Alpaca”, ”build [Bobcat]”, “build [Capybara]” and
“download [Bobcat]” when they were skipped due to a failing upstream stage
but still reported duration values.
One striking observation is the unusually long duration of 3105077 ±
13388395 ms for stage id 6. The median value is still in the expected range
with a value of 3482 ms. Looking at the box plot, we notice the presence of
8 outliers. Among these outliers, the two smallest values were recorded in
execution 87 (6008 ms) and execution 111 (7122 ms), for which we currently
lack a satisfactory explanation.
 Results and Analysis | 65

108

107

106
duration [ms]

105

104

103

102
1 2 3 4 5 6 7 8 9 10 11
stage nr

Figure 5.4: Logarithmic box plot for each stage duration of Slice 5

The remaining six outliers occurred between execution 105 and 110. As
previously discussed in Section 5.4.2, these outliers correspond to instances
where the checkout stage became stuck and had to be manually aborted.
The stage for generating the build system with CMake is slightly lower in
duration with an average of 4927 ± 1085 ms compared to Slice 3, but it falls
within a similar range. However, we would have expected a slight increase
in duration due to the extra stages for cloning (id 27) and unit testing of the
LATE framework that are executed in parallel to the Lysa unit testing and the
build system generation. Especially because this behaviour is noticeable in the
Lysa unit testing stages. The stage “Build Test Suite” takes 51645 ± 5554 ms
and stage “Execute Test Suite” requires 2006 ± 6641 ms. The relatively large
standard deviation for the latter can be explained by the existence of two large
outliers reported in execution 111 (39562 ms) and 112 (2781 ms), which were
caused by an error in the unit test suite execution due to a faulty code change.
The cloning of the late framework always initialises its git repository as
the respective folder gets deleted before checking out the source code. This
explains the relatively long duration of 23252 ± 1411 ms. The unit tests of
LATE take an average of 2006 ± 1177 ms. While the duration of the cloning
66 | Results and Analysis

Table 5.10: Analysis of the stage durations of Slice 5

nr name id mean [ms] stdev [ms] median [ms]

1 Declarative: Checkout SCM 6 3108077 13388395 3482
2 CMake: Generate Build 25 4927 1085 4990
System / IAR Projects
3 Clone LATE framework 27 23252 1411 23166
4 Build Test Suite 29 51645 5554 49266
5 Execute Late unit tests 47 2208 1177 1893
6 Execute Test Suite 58 2006 6641 804
7 Alpaca 78 178 41 165
8 download [Bobcat] 83, 14068 7277 18613
95,
96,
98,
100
9 build [Bobcat] 87 76897 4275 78456
10 build [Capybara] 89 77493 3454 78249
11 test [Bobcat] 105 6854 1417 7666
Crit. path 158287

process could certainly be improved, it is not necessary as the bottleneck of

the parallel stage block it is part of is the Lysa unit testing, which takes more
than double the amount of time.
The execution of the system testing stage “test [Bobcat]” requires an
average of 6854 ± 1417 ms. In executions 98 and 99, there are two outliers
(3501 ms and 3505 ms) which were caused by a bug that made the serial
interface of the laser unavailable, resulting in the termination of the test
program.
The build stage for the firmware targets Bobcat and Capybara is roughly
the same as it was in Slice 3.
The downloading stage “download [Bobcat]” has a similar median value of
18613ms. However, the average and the standard deviation differ substantially
with 14068 ± 7277 ms. When looking at command line logs of downloading
stages that reported different times, we noticed that the J-Link Commander
compares the memory of the target hardware and only downloads sections
of the firmware that changed. I.e. when the pipeline was executed without
firmware changes in execution 84 and 104 the downloading stage took only
around 1000 ms because no firmware had to be loaded.
The total execution time is limited by the critical path of the pipeline. We
 Results and Analysis | 67

marked the median values of the critical path in bold in Table 5.10 and summed
them to get the median value for the duration of the critical path which is
158287ms or approximately 2min38s.

5.5 Summary
We presented the findings from our experimental single-case study. We first
discuss the learned lessons from the implementation process, such as the initial
effort to learn Jenkins, its suitability for iterative development, challenges
due to proprietary components, and the benefits of using distributed build
infrastructures.
User feedback revealed positive interest initially due to Jenkins’ intuitive
interface, yet developers lacked preparation, leading to code changes without
subsequent CI checks. Over time, user confidence grew, though concerns
arose about passing system tests and testing framework effectiveness.
A survey among developers highlighted CI’s association with automated
testing, reducing test efforts, and improving code quality. Expectations
included faster bug feedback, task automation, and better time utilization.
Initial setup and maintenance, along with potential CI server overload, were
noted as downsides.
The survey also gauged the impact of the implemented CI service on
Cobolt’s development, showing increased code quality confidence and mo-
tivation. Quantitative evaluation endorsed the CI service.
Metrics from the CI server, especially pipeline executions and stages,
were analyzed. Consistent stage identifiers were stressed for comprehensive
analysis. Findings supported the success of CI techniques like automated unit
testing and multi-target compilation.
68 | Results and Analysis
 Discussion | 69

Chapter 6

Discussion

In this chapter, we utilize the findings presented in Chapter 5 to address the

research questions formulated in Chapter 3. The upcoming sections, namely
6.1, 6.2, and 6.3, systematically examine the research questions R1, R2, and
R3, respectively.

6.1 CI Techniques
We identified 8 CI techniques throughout our experimental study and per-
formed an iterative CI implementation using Jenkins. The service incorporates
multi-target compilation, automated unit testing, test reporting, visual CI
feedback, and trunk-based development. These techniques prove effective
for embedded software with a modular firmware architecture. However,
automated system testing encounters limitations due to the need for manual
interaction with hardware targets. In this section, we discuss each of the
identified techniques.

6.1.1 Multi-target Compilation

Modular firmware architectures are to have a shared codebase for multiple
firmware targets. In this way, bugs can be fixed for various products si-
multaneously, reducing the development effort. However, the overall testing
effort increases, because the code changes need to be tested for each relevant
hardware.
Our study demonstrates the effectiveness of automatically compiling the
firmware for multiple targets as part of a CI service. We observed a scenario
(see Section 5.4.2) where a developer introduced code changes with the
70 | Discussion

intention of adding a new feature to a specific firmware target. However, these

modifications inadvertently caused another firmware target to fail. Without the
aid of a CI service that compiles all firmware targets, the developer may not
have immediately realised the impact of their code changes on the other target.
Our research highlights the crucial role played by the CI service in promptly
identifying such issues, enabling developers to rectify them efficiently.

6.1.2 Automated Unit Testing

Among the stages of the CI service, the unit testing phase stands out as the
most successful in detecting bugs, with a total of 13 failures. It is worth
noting that while developers have the option to execute unit tests on their own
workstations, this practice is not consistently followed before committing new
code changes. This observation aligns with the statement of Gloor et al. that
"with manual testing, human errors can become quite common" [4].
Our final survey reveals that developers at Cobolt appreciate the contribu-
tions of the CI service, particularly in eliminating the uncertainty associated
with making mistakes. They also mention that their work life has become more
convenient due to the automation of manual tasks, which can be attributed,
among other factors, to the automated unit testing stage of the CI service.
In our experimental study, we discovered that the average execution
time of the unit test framework compilation is the second-longest, taking
approximately 51645 ± 5554 ms. Currently, the unit testing framework is built
entirely from scratch. However, we believe the time required for this process
can be reduced by only compiling the portion of the test suite that has changed.
In our particular case, we did not deem it necessary to implement this
optimization since the compilation time is still under a minute. Furthermore,
we can execute it in parallel with other pre-firmware build stages, making it
satisfactory for our specific use case.

6.1.3 Iterative CI Implementation

Rostami et al. [26] have shown, that many companies progressively added new
functionality to their CI service. Our study also implemented a CI service
iteratively in four cycles. Initially, we introduced multi-target compilation,
followed by the unit testing stage. We deployed the service at this stage
and subsequently added firmware downloading and the system testing stage.
Throughout the iterative implementation process, we did not encounter any
significant challenges. However, restructuring a Jenkins pipeline poses diffi-
 Discussion | 71

culties for later analysis of CI metrics, as it hinders the clear tracking of stage
behaviours over time.
This iterative implementation approach offers the advantage of faster
availability of CI activities. Even after the initial deployment, which included
only the multi-target compilation and unit testing stage, we were able to
identify specific bugs, as mentioned in Section 5.4.2.
Based on our experiences with implementing CI service iteratively, we
conclude it to be a valuable technique for CI in the embedded domain.

6.1.4 Automated System Testing

Cobolt has developed the LATE framework to automate their system testing
process. This involves automating command sequences sent to the hardware
target via its serial interface. During our implementation of the system testing,
we encountered challenges in achieving the intended level of automation.
Many existing tests require physical interaction with the laser hardware, which
can not be automated due to the absence of a HILS system.
The subset of tests, that does not require manual interaction, was not
considered sufficiently relevant to ensure consistent passing results. Con-
sequently, out of 41 executions of the CI service, 23 instances reported an
“UNSTABLE” status due to test failures. Additionally, 2 instances reported
a “FAILED” testing stage, which was not a bug with the system-under-test
itself but rather an interface change that was not implemented in the LATE
framework.
The preparation of the system test framework can be executed in parallel
with other pre-firmware build steps, incurring no additional time overhead to
the CI service, given the relatively long duration of the unit testing stages.
However, the system test execution must be performed after downloading
the latest firmware to the hardware target, thereby increasing the overall
pipeline duration. In our case, the system test execution duration averages
6854 ± 1417 ms, although this duration is subject to variation depending on
the size of the implemented system testing tool.
Based on informal feedback received from the developers, we believe that
there is a lack of confidence in the relatively new system testing framework
LATE.
In conclusion, automating system testing for embedded systems is a
complex task that requires an established framework designed to function
without manual user inputs. Additionally, developers must have sufficient
confidence in the system testing process, which we believe comes with the
72 | Discussion

maturity and reliability of the framework being used.

6.1.5 Distributed Build Architecture

A distributed build architecture is in our opinion a key enabler for using CI
services in the embedded domain. Specifically, for initial CI setups it is useful
that build machines can be used that are a direct replicate of the workstations
of a developer. This setup allows setting up the CI pipeline with just shallow
knowledge of the existing development process and tools, as they can run on
the build machine in the same way as they do on the workstation of developers.
Additionally, it completely averts the build tool dependency challenge that
is often present in adapting CI in embedded software development, which is
discussed later in Section 6.2.2.

6.1.6 Visual CI Service Feedback

We opted for the Jenkins plugin Blue Ocean UX to provide a visual represen-
tation of the CI service results. Our survey reveals that developers currently
involved in project Lysa utilise the pipeline view to monitor feedback on their
code modifications. One developer specifically expressed appreciation for
the ability to easily identify failed (red) and successful (green) stages of the
pipeline.
Given the absence of any negative feedback, we conclude that this ap-
proach meets the requirements for reporting CI service results in our specific
use case.

6.1.7 Test Report in xUnit XML Format

We discover that testing tools that provide feedback in the xUnit XML format
can be easily parsed and visualised using the Jenkins JUnit plugin.
Both unit testing frameworks (CxxTest and PyTest) natively support this
format and we were fast (just two days) to implement it to the LATE system
testing framework.

6.1.8 Trunk-based Development

Our CI service makes use of the trunk-based branching strategy, that the
firmware development team of Cobolt uses. This strategy suggests, that the
mainline branch needs to always be in a production-ready state. Our CI service
observes the mainline branch and continuously verifies it. This gives the
 Discussion | 73

developers confidence that the mainline is always correct as one developer

of Cobolt reported in the questionnaire.
Furthermore, trunk-based development revolves around a single, long-
lived branch, resulting in a single automatic trigger point for the CI service.
We contend that this contributes to the high lucidity score of 5.67 given by the
Cobolt developers, as reported in Section 5.3.3.

6.2 Efforts for CI Adoption

We identify four challenges that we encountered when implementing our CI
service: learning CI tools, managing build tool dependencies, and addressing
manual input requirements for system testing. The following sections reflect
on each of them and the countermeasures that we used to overcome them.

6.2.1 Learn Using the CI Tool

One downside associated with CI that developers at Cobolt have identified is
the initial effort required to set up the CI service. We found that a significant
portion of this effort stems from learning to use the specific CI tool.
Nevertheless, our iterative implementation approach allowed us to consult
the Jenkins handbook in a problem-solving manner rather than delving into
extensive theoretical study. In Chapter 4, we have documented this approach,
which involved substantial refactoring, particularly with regard to the overall
pipeline architecture.
It is challenging to determine whether this approach is superior to investing
time in a comprehensive training course for mastering the CI tool before using
it. However, it is evident that we successfully achieved our project goals within
a reasonable timeframe, and we can consider the challenge of learning the CI
tool as overcome.

6.2.2 Adress Build Tool Dependencies

Already Lwaktare et al. [33] identified build tool dependencies as a challenge
for adopting CI for embedded systems. When implementing our service at
Cobolt, we experienced build tool dependencies that tied us also in the choice
of the build machine.
However, using the same approach as Seth et al. [30] and setting up a
distributed build architecture allowed us to overcome this challenge. As previ-
74 | Discussion

ously described in Section 6.1.5, we can use a build machine, that is identical
to developer workstations when utilising a distributed build architecture.

6.2.3 Adress Manual Input for System Testing

As specified in Section 2.1.7, testing embedded software on a system level
necessitates running the firmware on the target hardware or a target-like
environment.
During our efforts to automate the system testing framework LATE at
Cobolt, we encountered complications due to the need for manual physical
input to the hardware target in many of the included tests. Unfortunately,
without a currently unavailable HILS system, automating these tests is not
feasible.
However, certain system tests that focus on verifying the correct function-
ality of the serial interface can still be executed. Ultimately, to overcome this
challenge, the development of a HILS system is required.

6.2.4 Improve Visibility of CI Service

Even though the developers at Cobolt were informed about the initial deploy-
ment of the CI service, they initially ignored its feedback on code changes.
We believe that this was primarily due to a lack of understanding of how to
effectively utilise the feedback.
We managed to encourage developers to utilise the system by personally
informing them about failing stages and providing them with a link to the
pipeline view of the respective pipeline execution.
Over time, we observed that developers started independently consulting
the CI feedback and resolving broken builds. In the final questionnaire, both
developers currently working on project Lysa reported using the pipeline view
of the job execution to verify their code changes.

6.3 Impacts of CI Adoption

Developers at Cobolt expressed expectations of faster bug feedback, automa-
tion of repetitive tasks, and better utilization of their time through CI. They
also see downsides including the initial setup and maintenance effort, as well
as potential overload on the CI server resulting in wait time.
We identify faster bug detection, increased motivation, confidence in code
quality, and encouragement for standardization as benefits to our CI service
 Discussion | 75

and discuss them in the following.

6.3.1 Fast Bug Detection

We have demonstrated the effectiveness of our CI system in detecting bugs.
Through the automation of unit tests and compilation of different firmware
targets, faulty code changes were identified on multiple occasions. The
feedback from the CI system is available in less than 3 minutes, eliminating
any significant wait time for the developers at Cobolt.
It is important to note that code changes often involve adding a feature or
fixing a bug specific to one hardware target. However, our CI service detected
two instances where these code changes resulted in a failing build stage for
another hardware target. Without our CI service, these faults would have
remained undiscovered for longer than 3 minutes, and it would have been less
clear which developer and code changes introduced the bug.

6.3.2 Increased Motivation

Developers at Cobolt have reported an increase in motivation as a result of
the implemented CI service. One developer mentioned that the continuous
verification of code changes motivates them to address issues promptly in
order to enable more regular releases. Another developer noted that the work
environment has become more exciting, presumably due to the automation of
“boring, time-consuming, and repetitive tasks”, as another developer antici-
pated when discussing the expected impact of the CI service.
While it is important to conduct further research to evaluate this increased
motivation, i.e. by studying the impact of CI on developers’ commit frequency.
We did not pursue this investigation in the case of Cobolt due to a lack of
contextual information on the development process before the implementation
of CI.
However, we believe that the perceived increase in motivation among
developers is already a benefit in itself, as it indicates a greater level of
confidence and engagement.

6.3.3 Confidence in Code Quality

Another notable impact of the CI service is the increased confidence in the
code quality of project Lysa. Developers have expressed trust in the stability
of the mainline source code, as it undergoes continuous validation through the
76 | Discussion

CI service. Additionally, developers have reported that the CI service enhances

their confidence in their own code changes, as it alerts them when a commit
is faulty. We believe that this improvement in confidence positively influences
the work mentality and overall code quality.

6.3.4 Encourage Standardisation

Our CI service has served as an impetus for developers at Cobolt to develop
and standardize processes that can be automated. Two specific processes that
have been discussed and initiated after the initial CI deployment are the release
process and the collaboration process with Git.
Regarding the release process, developers have started planning its au-
tomation by formalizing the steps involved. During the questionnaire, the
release process consisted of 15 steps, but it has already undergone two
revisions since then. This focus on automation aims to streamline the release
process and reduce manual effort.
In addition, discussions have taken place regarding the collaboration
process with Git. The intention is to transition from a classic trunk-based
development approach to a scaled trunk-based development model with pull
requests. This transition would enable having a code-reviewing process to
enhance code quality and facilitate better inter-team communication.
We believe that establishing standardized and streamlined development
processes is important, and management may eventually enforce these prac-
tices when the cost-benefit tradeoff is justified. However, it is noteworthy that
developers themselves are motivated to drive these changes as it results in
automating otherwise laborious processes.
 Conclusions and Future work | 77

Chapter 7

Conclusions and Future work

In this final chapter we present our conclusions (Section 7.1), present our
limitation (Section 7.2), discuss future work (Section 7.3), and elaborate on
our reflection (Section 7.4).

7.1 Conclusions
CI is a software development activity that supports the synchronisation of
multiple developers coding activities. The idea is to encourage developers to
commit their code changes faster by having a service in place, that automati-
cally validates the correctness of the code changes. While CI is widely adopted
in web and application development, there is a lack of solution proposals
for applying it to the embedded domain. Embedded software — also called
firmware — is software that runs on hardware that is embedded in a specific
context. The hardware and firmware are often developed concurrently and the
product needs to comply with special working conditions. Embedded software
development is often cross-compiled on an external workstation. The firmware
must then be downloaded to the memory of the hardware target as a binary file.
In this thesis, we investigated the application of CI techniques in the
context of embedded software development. Our study identifies suitable
CI techniques for the embedded domain, assesses the implementation efforts
involved, and evaluates the impact of CI adoption.
Through an iterative development and deployment process, we success-
fully implemented a CI service using the automation server Jenkins at Cobolt
AB, a company specializing in optoelectronics. The CI service incorporated
iterative CI implementation, multi-target compilation, automated unit testing,
test reporting in xUnit XML format, visual CI feedback, and trunk-based
78 | Conclusions and Future work

development. These techniques proved to be effective for embedded software

with a modular firmware architecture.
Additionally, we explored automated system testing but found limited
success in our case. System testing in the embedded domain often requires
manual interaction with the hardware target, which cannot be automated
without a HILS system in place. The subset of system tests that could be
automated without manual input was deemed less critical and not prioritised
for immediate resolution in case of failure.
Throughout the implementation process, we encountered noteworthy chal-
lenges. Learning to use CI tools like Jenkins required dedicated effort, but
we found that iterative implementation of CI services facilitated efficient
learning. By continuously consulting the Jenkins handbook, we achieved our
project goals without extensive initial training. In the final evaluation, Cobolt
developers expressed approval of our CI service across six categories: compre-
hensiveness, effective communication, immediacy, appropriate dimensioning,
accuracy, and lucidity.
We also addressed the challenge of build tool dependencies, which can
limit the build environment. We demonstrated that a distributed build archi-
tecture, with a separation of the CI coordinator from the build environment,
allows replication of developers’ workstations as build machines. This ap-
proach ensures a consistent build environment compatible with all necessary
build tools while minimizing implementation effort.
Moreover, we recognised the challenge of manual input required for many
system tests. To mitigate this issue, we selected a subset of tests that could be
executed without manual input. Over time, developers themselves realised the
positive impact of the CI service after being reminded of its existence at the
beginning, eliminating the initial resistance to adoption.
The adoption of CI in the embedded domain resulted in several benefits.
Notably, it enabled fast bug detection, increased motivation among developers,
improved confidence in code quality, and encouraged standardization. We did
not observe any noticeable negative impacts directly linked to the CI service
itself.
In conclusion, this thesis contributes to the understanding of CI adoption
in the embedded domain and provides insights into the challenges, techniques,
and impacts associated with implementing CI in embedded software develop-
ment. The identified CI techniques tailored to the modular firmware architec-
ture of Cobolt AB demonstrate the potential for successful CI integration in
similar contexts.
 Conclusions and Future work | 79

7.2 Limitations
The Cobolts lasers are not readily available in large quantities to the de-
velopment team. Consequently, the team must prioritize the distribution
of hardware resources. Due to this limitation, we were constrained to
downloading and system-testing the firmware exclusively on the DPL version
of the 06-01 series laser. We focused on developing an extendable CI service,
that we believes allows easy integration of additional hardware targets at a later
stage. However, it is important to note that our thesis work does not encompass
the challenges that may arise from the parallel usage of multiple debuggers and
hardware targets.
Part B of the questionnaire gauges the developers’ perception of the
implemented CI service. However, due to the ongoing research on the
implemented CI techniques, we refrained from seeking concrete feedback
on them. Instead, we requested general feedback, which restricted our
understanding of the developers’ perception of specific CI techniques.

7.3 Future work

We encountered challenges with automating the system testing at Cobolt as
initially planned. To address this, we propose exploring the integration of
HILS systems. This approach can enhance automated system testing in the
embedded domain, enabling more comprehensive testing.
In addition, we identified another area for future exploration, which
involves incorporating parallel automated testing on multiple hardware targets.
Given the prevalence of embedded software with numerous hardware targets,
this topic holds significant relevance.
The Jenkins handbook suggests that distributed build architectures are
advantageous for scaling CI services [42]. While we concur with this
statement, we were unable to find literature supporting it. Therefore, we
propose conducting an experimental study using our solution proposal to
establish a distributed build architecture with multiple build machines. Such a
study would provide valuable insights into the effectiveness of this approach.

7.4 Reflections
The purpose of our research, as outlined in Section 1.3, is to contribute to the
United Nations’ sustainable development goal 9, “Promoting Technological
80 | Conclusions and Future work

Progress.” Specifically, our discussion focuses on relevant CI techniques and

the adoption efforts within the embedded domain.
In our research, we presented eight techniques that can be utilized by other
stakeholders when implementing a CI service in the embedded domain. Addi-
tionally, we addressed four notable challenges that are commonly encountered
throughout the implementation process. By identifying and discussing these
challenges, we aim to facilitate better communication and understanding in
this field.
Furthermore, we highlighted four benefits that arise from the adoption
of CI, providing additional insights and justification for its implementation
in the embedded domain. These benefits further support the promotion of
technological progress in line with sustainable development goals.
 REFERENCES | 81

References

[1] K. Beck, “Embracing change with extreme programming,” Computer,

vol. 32, no. 10, p. 70–77, Oct 1999. doi: 10.1109/2.796139. [Online].
Available: https://doi.org/10.1109/2.796139 (Accessed at 01/02/2023).

[2] S. Vadapalli, DevOps: Continuous Delivery, Integration, and

Deployment with DevOps. Packt Publishing Ltd., Mar 2018. ISBN
978-1-78913-299-1. [Online]. Available: https://learning.oreilly.com/
library/view/devops-continuous-delivery/9781789132991/ (Accessed
at 10/12/2022).

[3] M. Shahin, M. Ali Babar, and L. Zhu, “Continuous integration,

delivery and deployment: A systematic review on approaches, tools,
challenges and practices,” IEEE Access, vol. 5, p. 3909–3943, 2017.
doi: 10.1109/ACCESS.2017.2685629. [Online]. Available: https:
//doi.org/10.1109/ACCESS.2017.2685629 (Accessed at 01/02/2023).

[4] M. Gloor, M. Mazalin, and A. Zimmermann, “Automated payment

terminal testing: How to achieve continuous integration for systems that
are almost impossible to virtualize,” IEEE Software, vol. 38, no. 6, p.
17–23, Nov 2021. doi: 10.1109/MS.2021.3094955. [Online]. Available:
https://doi.org/10.1109/MS.2021.3094955 (Accessed at 10/03/2023).

[5] ALLEA, The European Code of Conduct for Research

Integrity Revised Edition. ALLEA - All European
Academies, 2017. ISBN 978-3-00-055767-5. [Online].
Available: https://www.allea.org/wp-content/uploads/2017/05/
ALLEA-European-Code-of-Conduct-for-Research-Integrity-2017.pdf
(Accessed at 01/02/2023).

[6] V. Garousi, M. Felderer, C. M. Karapıçak, and U. Yılmaz,

“Testing embedded software: A survey of the literature,”
Information and Software Technology, vol. 104, p. 14–45,
82 | REFERENCES

Dec 2018. doi: 10.1016/j.infsof.2018.06.016. [Online]. Available:

https://doi.org/10.1016/j.infsof.2018.06.016 (Accessed at 13/04/2023).

[7] M. Shen, W. Yang, G. Rong, and D. Shao, “Applying agile methods to

embedded software development: A systematic review,” in 2012 Second
International Workshop on Software Engineering for Embedded Systems
(SEES), 2012. doi: 10.1109/SEES.2012.6225488 pp. 30–36. [Online].
Available: https://doi.org/10.1109/SEES.2012.6225488 (Accessed at
10/12/2022).

[8] J. T. Taylor and W. T. Taylor, Patterns in the Machine: A Software

Engineering Guide to Embedded Development. Apress, Jan 2021.
ISBN 978-1-4842-6440-9. [Online]. Available: http://doi.org/10.1007/
978-1-4842-6440-9 (Accessed at 16/03/2023).

[9] IEEE Computer Society, “IEEE recommended practice for architectural

description for software-intensive systems,” IEEE Std 1471-2000,
p. 1–30, Oct 2000. doi: 10.1109/IEEESTD.2000.91944 Conference
Name: IEEE Std 1471-2000. [Online]. Available: http://doi.org/10.
1109/IEEESTD.2000.91944 (Accessed at 29/03/2023).

[10] L. Bass, P. Clement, and R. Kazman, Software Architecture in

Practice, Third Edition, 3rd ed., ser. SEI series in software
engineering. Addison-Wesley, 2012. ISBN 978-0-13-294279-
9. [Online]. Available: https://learning.oreilly.com/library/view/
software-architecture-in/9780132942799/ (Accessed at 05/04/2023).

[11] G. Márquez, H. Astudillo, and R. Kazman, “Architectural tactics

in software architecture: A systematic mapping study,” Journal
of Systems and Software, vol. 197, p. 111558, 2023. doi:
10.1016/j.jss.2022.111558. [Online]. Available: http://doi.org/10.1016/
j.jss.2022.111558 (Accessed at 05/04/2023).

[12] T. Martin, Chapter 15 - MCU Software Architecture, 3rd ed. Newnes,

2023, p. 555–582. ISBN 978-0-323-85494-8. [Online]. Available:
https://doi.org/10.1016/B978-0-323-85494-8.00017-6 (Accessed at
05/04/2023).

[13] P. Hammant, Trunk-Based Development And Branch By

Abstraction. Leanpub, May 2020. [Online]. Available: https:
//trunkbaseddevelopment.com/ (Accessed at 22/03/2023).
 REFERENCES | 83

[14] V. Driessen, “A successful Git branching model,” Jan 2010. [Online].

Available: http://nvie.com/posts/a-successful-git-branching-model/
(Accessed at 22/03/2023).
[15] M. Meyer, “Continuous integration and its tools,” IEEE Software,
vol. 31, no. 3, p. 14–16, May 2014. doi: 10.1109/MS.2014.58.
[Online]. Available: https://doi.org/10.1109/MS.2014.58 (Accessed at
02/02/2023).
[16] M. Leppänen, S. Mäkinen, M. Pagels, V.-P. Eloranta, J. Itkonen,
M. V. Mäntylä, and T. Männistö, “The highways and country roads
to continuous deployment,” IEEE Software, vol. 32, no. 2, p.
64–72, Mar 2015. doi: 10.1109/MS.2015.50. [Online]. Available:
https://doi.org/10.1109/MS.2015.50 (Accessed at 02/02/2023).
[17] P. M. Duvall, S. Matyas, P. Duvall, and A. Glover,
Continuous Integration: Improving Software Quality and
Reducing Risk. Addison-Wesley, 2007. ISBN 978-0-321-
33638-5. [Online]. Available: https://learning.oreilly.com/library/
view/continuous-integration-improving/9780321336385/ (Accessed at
18/04/2023).
[18] T. Mårtensson, D. Ståhl, and J. Bosch, “Continuous integration
applied to software-intensive embedded systems – problems and
experiences,” in Product-Focused Software Process Improvement, ser.
Lecture Notes in Computer Science, P. Abrahamsson, A. Jedlitschka,
A. Nguyen Duc, M. Felderer, S. Amasaki, and T. Mikkonen, Eds.
Cham: Springer International Publishing, 2016. doi: 10.1007/978-
3-319-49094-6_30. ISBN 978-3-319-49094-6 p. 448–457. [Online].
Available: https://doi.org/10.1007/978-3-319-49094-6_30 (Accessed at
18/04/2023).
[19] P. Bourque and R. Fairley, Guide to the Software
Engineering Body of Knowledge - SWEBOK V3.0, 3rd ed.
Washington, DC, USA: IEEE Computer Society Press,
Jan 2014. ISBN 978-0-7695-5166-1. [Online]. Available:
https://www.researchgate.net/publication/342452008_Guide_to_
the_Software_Engineering_Body_of_Knowledge_-_SWEBOK_V30
(Accessed at 6/04/2023).
[20] S. Branam, “Unit testing for embedded software
development,” Aug 2022. [Online]. Available: https://dojofive.
84 | REFERENCES

com/blog/unit-testing-for-embedded-software-development/ (Accessed
at 11/04/2023).

[21] N. Van Schooenderwoert and R. Morsicato, “Taming the embedded

tiger - agile test techniques for embedded software,” in Agile
Development Conference, Jun 2004. doi: 10.1109/ADEVC.2004.21 p.
120–126. [Online]. Available: https://doi.org/10.1109/ADEVC.2004.21
(Accessed at 11/04/2023).

[22] M. Z. Iqbal, A. Arcuri, and L. Briand, Automated system testing

of real-time embedded systems based on environment models,
ser. Technical Report 2011-19. Simula Research Laboratory,
2011. [Online]. Available: https://www.simula.no/sites/default/files/
publications/Simula.simula.840.pdf (Accessed at 13/04/2023).

[23] L. Kalinski, “Lysa layers overview 1.0,” Nov 2022, unpublished.

[24] Cobolt AB, “Cobolt Tor series datasheet,” 2023. [Online].

Available: https://hubner-photonics.com/wp-content/uploads/2014/11/
D0764-B-Datasheet-Tor-Series.pdf (Accessed at 03/03/2023).

[25] Cobolt AB, “Cobolt 06-01 series datasheet,” 2023.

[Online]. Available: https://hubner-photonics.com/wp-content/uploads/
2014/11/D0352-O-Datasheet-Cobolt-06-01-Series.pdf (Accessed at
03/03/2023).

[26] P. Rostami Mazrae, T. Mens, M. Golzadeh, and A. Decan, “On

the usage, co-usage and migration of CI/CD tools: A qualitative
analysis,” Empirical Software Engineering, vol. 28, no. 2, p. 52,
Mar 2023. doi: 10.1007/s10664-022-10285-5. [Online]. Available:
https://doi.org/10.1007/s10664-022-10285-5 (Accessed at 20/04/2023).

[27] Jenkins, Jenkins Handbook, Apr 2023. [Online]. Available: https:

//www.jenkins.io/doc/book/ (Accessed at 14/04/2023).

[28] IAR Systems AB, IAR IDE Project Management and Building Guide
(UIDEARM-9), 2015, this guide applies to version 7.4x of IAR
Embedded Workbench® for Advanced RISC Machines Ltd’s ARM core
family.

[29] A. J. Bernhardt, “CI/CD pipeline from android to embedded devices

with end-to-end testing based on containers,” Master’s thesis, KTH,
 REFERENCES | 85

School of Electrical Engineering and Computer Science (EECS),

2021. [Online]. Available: https://urn.kb.se/resolve?urn=urn:nbn:se:
kth:diva-305911 (Accessed at 09/12/2022).
[30] N. Seth and R. Khare, “ACI (automated continuous integration)
using Jenkins: Key for successful embedded software development,”
in 2015 2nd International Conference on Recent Advances in
Engineering & Computational Sciences (RAECS), Dec 2015. doi:
10.1109/RAECS.2015.7453279 p. 1–6. [Online]. Available: https:
//doi.org/10.1109/RAECS.2015.7453279 (Accessed at 17/04/2023).
[31] M. Kowzan and P. Pietrzak, “Continuous integration in validation of
modern, complex, embedded systems,” in 2019 IEEE/ACM International
Conference on Software and System Processes (ICSSP), May 2019.
doi: 10.1109/ICSSP.2019.00029 pp. 160–164. [Online]. Available:
https://doi.org/10.1109/ICSSP.2019.00029 (Accessed at 09/12/2022).
[32] D. Ståhl and J. Bosch, “Automated software integration flows in
industry: a multiple-case study,” in Companion Proceedings of the
36th International Conference on Software Engineering, ser. ICSE
Companion 2014. New York, NY, USA: Association for Computing
Machinery, May 2014. doi: 10.1145/2591062.2591186. ISBN 978-1-
4503-2768-8 p. 54–63. [Online]. Available: https://doi.org/10.1145/
2591062.2591186 (Accessed at 19/05/2023).
[33] L. E. Lwakatare, T. Karvonen, T. Sauvola, P. Kuvaja, H. H.
Olsson, J. Bosch, and M. Oivo, “Towards DevOps in the embedded
systems domain: Why is it so hard?” in 2016 49th Hawaii
International Conference on System Sciences (HICSS), Jan 2016.
doi: 10.1109/HICSS.2016.671. ISSN 1530-1605 p. 5437–5446.
[Online]. Available: http://doi.org/10.1109/HICSS.2016.671 (Accessed
at 27/02/2023).
[34] F. Zampetti, D. A. Tamburri, S. Panichella, A. Panichella,
G. Canfora, and M. D. Penta, “Continuous integration and delivery
practices for cyber-physical systems: An interview-based study,”
ACM Transactions on Software Engineering and Methodology, Nov
2022. doi: 10.1145/3571854 Just Accepted. [Online]. Available:
https://doi.org/10.1145/3571854 (Accessed at 18/04/2023).
[35] T. Mårtensson, D. Ståhl, and J. Bosch, “Continuous integration
impediments in large-scale industry projects,” in 2017 IEEE
86 | REFERENCES

International Conference on Software Architecture (ICSA), Apr

2017. doi: 10.1109/ICSA.2017.11 p. 169–178. [Online]. Available:
http://doi.org/10.1109/ICSA.2017.11 (Accessed at 16/03/2023).

[36] T. Mårtensson, D. Ståhl, A. Martini, and J. Bosch, “Continuous

architecture: Towards the Goldilocks zone and away from vicious
circles,” in 2019 IEEE International Conference on Software
Architecture (ICSA), Mar 2019. doi: 10.1109/ICSA.2019.00022
p. 131–140. [Online]. Available: https://doi.org/10.1109/ICSA.2019.
00022 (Accessed at 20/04/2023).

[37] D. Ståhl and J. Bosch, “Experienced benefits of continuous integration

in industry software product development: A case study,” IASTED
Multiconferences - Proceedings of the IASTED International Conference
on Software Engineering, SE 2013, p. 736–743, Mar 2013. doi:
10.2316/P.2013.796-012. [Online]. Available: https://doi.org/10.2316/
P.2013.796-012 (Accessed at 19/05/2023).

[38] O. Elazhary, C. Werner, Z. S. Li, D. Lowlind, N. A. Ernst,

and M.-A. Storey, “Uncovering the benefits and challenges of
continuous integration practices,” IEEE Transactions on Software
Engineering, vol. 48, no. 7, p. 2570–2583, Jul 2022. doi:
10.1109/TSE.2021.3064953 Conference Name: IEEE Transactions
on Software Engineering. [Online]. Available: https://doi.org/10.1109/
TSE.2021.3064953 (Accessed at 19/05/2023).

[39] A. Miller, “A hundred days of continuous integration,” in Proceedings

of the Agile 2008, ser. AGILE ’08. USA: IEEE Computer Society, Aug
2008. doi: 10.1109/Agile.2008.8. ISBN 978-0-7695-3321-6 p. 289–293.
[Online]. Available: https://doi.org/10.1109/Agile.2008.8 (Accessed at
19/05/2023).

[40] L. Barroca, H. Sharp, D. Salah, K. Taylor, and P. Gregory,

“Bridging the gap between research and agile practice: an evolutionary
model,” International Journal of System Assurance Engineering
and Management, vol. 9, no. 2, p. 323–334, Apr 2018. doi:
10.1007/s13198-015-0355-5. [Online]. Available: https://doi.org/10.
1007/s13198-015-0355-5 (Accessed at 24/04/2023).

[41] T. Rane, “Agile research — getting beyond the buzzword,”

International Journal of Science and Research (IJSR), vol. 11,
 REFERENCES | 87

p. 362–367, Aug 2022. doi: 10.21275/SR22805100538. [Online].

Available: https://www.researchgate.net/publication/362578850_
Agile_Research_-Getting_beyond_the_Buzzword (Accessed at
24/04/2023).

[42] Jenkins, “Architecting for scale,” May 2023. [Online]. Available: https:
//www.jenkins.io/doc/book/scaling/architecting-for-scale/ (Accessed at
02/05/2023).

[43] H. Alharahsheh and A. Pius, “A review of key paradigms: positivism

vs interpretivism,” Global Academic Journal of Humanities and
Social Sciences, vol. 1, no. 2(3), p. 39–43, Dec 2019. [Online].
Available: https://www.researchgate.net/publication/338244145_A_
Review_of_key_paradigms_positivism_VS_interpretivism (Accessed
at 25/04/2023).

[44] W. E. Saris and I. N. Gallhofer, Design, Evaluation, and Analysis

of Questionnaires for Survey Research, 2nd ed., ser. Wiley Series
in Survey Methodology. Hoboken, New Jersey: John Wiley &
Sons, Ltd, Mar 2014. ISBN 978-1-118-63461-5. [Online]. Available:
https://doi.org/10.1002/9781118634646 (Accessed at 15/05/2023).

[45] KymikoLoco, “Answer to "how to build a complex parallel

Jenkins pipeline?"",” Apr 2021. [Online]. Available: https:
//devops.stackexchange.com/a/13759 (Accessed at 05/05/2023).

[46] Deland-Han, “Command prompt (cmd.exe) command-

line string limitation,” Sep 2021. [Online]. Avail-
able: https://learn.microsoft.com/en-us/troubleshoot/windows-client/
shell-experience/command-line-string-limitation (Accessed at
08/05/2023).

[47] A. Bayer, A. Rodriquez, J. Glick, K. Kawaguchi, O. Gondža, S. Wolf,

D. Nusbaum, M. Baptiste, E. Gutierrez, F. Fernández, R. Leon, rsandell,
and T. Jacomb, “Jenkins JUnit plugin,” Apr 2023. [Online]. Available:
https://plugins.jenkins.io/junit (Accessed at 08/05/2023).

[48] S. M. G. . C. KG, J-Link / J-Trace User Guide (V6.00), Jun 2016, this
guide applies to version 7.70.x of IAR Embedded Workbench® for AR.
88 | REFERENCES
 Appendix A: Questionnaire | 89

Appendix A

Questionnaire

Following are the screenshots displaying the questionnaire we’ve put together
using the Google Forms platform.
 Questionnaire: Continuous Integration for Firmware
Development Division at Cobolt
As part of the master thesis
"Continuous Integration for Embedded Software with Modular Firmware Architecture" we want to understand:

1. Which Continuous Integration (CI) techniques are suitable for embedded software with
modular firmware architecture?
2. What are the benefits from using a CI service in the embedded domain?

Please enter your email, so you have the possibility to revise your answers.
All data will be presented anonymized in the thesis report.

* Indicates required question

1. Email *

PART A: General

An early definition of Continuous Integration (CI) was provided by Beck in his 1999’s article about extreme programming:

"New code is integrated with the current system after no more than a few hours. When integrating, the system is built from scratch
and all tests must pass or the changes are discarded."

In more recent definitions of CI, especially in the context of DevOps, it is further specified that there must be an automated system.
When an agent checks in their revised code, this system picks up the change and runs a set of commands to verify the correctness
automatically.

Please answer the following questions based on your current understanding and opinion on CI.

2. What does CI mean to you?

3. In what way does CI change the way of developing?

4. What experiences do you have developing without a CI Service in place?

5. What kind of improvements do you expect from using CI?

6. What downsides do you see in using CI?

PART B: Impact of the implemented CI service

A CI service
prototype was implemented in Cobolt. Its infrastructure can be viewed in Figure 1. Figure 2 shows
the CI pipeline that is initiated on the CI Agent by the CI Controller. This
part of the questionnaire is targeted on getting your feedback regarding the
implementation.

Figure 1: CI service infrastructure

Figure 2: CI pipeline executed on CI agent

7. In what way did you already use the implemented CI service?

8. What impact does the implemented CI service have on your work?

9. What other feature do you desire to be implemented into the CI Service?

10. Do you have any other feedback on the CI service prototype?

PART C: Evaluation of CI service

Ståhl and Bosch (Link) have proposed 6 guidelines for the implementation of CI services, that are used to evaluate the CI service that
was implemented at Cobolt.

Please rate the following criteria on a scale from 0 to 6. if you want to discuss what any of the categories mean, please get in touch
with me.
11. Comprehensive Activities *
Are the artefacts produced by the CI service of value to you (i. e. unit test results)?

Mark only one oval.

not at all

very much

12. Effective Communication *

Is the output of the CI service easy to access and understand?

Mark only one oval.

not at all

very much
13. Immediacy *
Is the CI service output available quick enough?

Mark only one oval.

not at all

very much

14. Appropriate Dimensioning *

Does the CI service have a good balance between completeness and lucidity of the output?
In other words: Is a reasonable amount of activities implemented?

Mark only one oval.

not at all

very much
15. Accuracy *
Are you confident that the CI service always processes the recent source code and not an old version?

Mark only one oval.

not at all

very much

16. Lucidity *
Do you understand all activities included in the CI service and the chronological order of their execution?

Mark only one oval.

not at all

very much

This content is neither created nor endorsed by Google.

Forms
 96 | Appendix B: Questionnaire Results

Appendix B

Questionnaire Results
Respondant/Question D1 D2 D3 Summary
What does CI mean to you? Automated tests and building on a server Automatic testing safeguarding the main branch from A relatively fast cycle of developing new code, * Automated testing (D1, D2, D3)
implementing new problems after fixing another merging it with other developers' code, running tests * Safeguarding the main branch (D2)
and delivering a binary ready to be tested by non-dev * Fast development and delivering cycles (D3)
people.
In what way does CI change Automated tests takes away a lot of the effort from Added confidence when checking in code. Otherwise The overhead of verifying that all projects compile (as * Remove testing effort from developers (D1, D2, D3)
the way of developing? testing, which makes it possible to make progress lots of doubt whether one has thought of everything, 90-95% of our code is shared between projects) and * Regression testing for multiple project targets (90-
faster. Building on a server takes a way the "works on and that problems may occur down the road. works after every merge into master is gone. The 95% of Lysa code is shared between multiple
my machine" factor. overhead of doing formal releases for internal testing projects) (D3)
(currently about 15 manual steps) is dramatically * Takes away the "works on my machine" factor (D1)
reduced. This obviously makes the process much * Increases confidence that checked in code is correct
more pleasant and the otherwise inherent reluctance (D1, D2, D3)
to do frequent releases (or replace broken releases) * -> more freqient releases (D3)
is no more. * Pleasant to automate release process (15 manual
steps) (D3)

What experiences do you You have to me more careful Both doubt and problems down the road, as That's how we have been working for 5 years now, so * Being more careful (D1)
have developing without a CI mentioned. This sometimes leads to confusion and or quite a lot. * Small flaws (forgotten safety steps) result in big
Service in place? annoyance. A small thing one forgets or don't think problems (D2)
about when having implemented something, may lead * Confusion for the team, as problems cannot be
to a bigger thing for either the same person after traced to a specific change (D2)
some time or another person.

What kind of improvements Notice quicker if something doesn't work right. More of actual problem solving and improvements, Developers can focus on developing while a lot of * Faster feedback about bugs (D1)
do you expect from using CI? less silly made up problems boring, time consuming and repetitive tasks are no * Better use of time (less problem analysis) (D2, D3)
longer their (or any one else's) responsibility. * Automation of repetitive tasks (D3)

What downsides do you see Initial effort to set it up. Sometimes servers get over Overhead in general. Added moment of inertia if e.g Someone has to be responsible for maintaining the CI * Iniial effort to set up (D1)
in using CI? loaded with work and you have to wait tests get obsolete after changing some requirements environment (but given the positive results I believe * Wait time because of server load (D1)
that's totally worth it). * Maintainance overhead (i.e. requirements change ->
tests needs to be changed) (D2, D3)
* New responsibility (D3)
In what way did you already Haven't used it yet See green/red boxes clearly in web interface in case I As a receipt that what I push to master is actually * One has not used it (assigned to other project) (D1)
use the implemented CI forgot something. Sometime exposed and been working. I also see that there's just little work left * Pipeline view to verify merge (D2, D3)
service? reminded from colleague that something did not pass before most of the release procedure can be * Waits for release process to be automated (D3)
automated.
What impact does the If I would check out master to do work on Lysa, it Makes worklife more exciting and less cumbersome Knowing that the quality of the firmware is being * Main line is always tested (D1, D3)
implemented CI service have would already be tested. continuously verified significantly increases * Increased motivation (D2, D3)
on your work? motivation to fix things fast and release them * Elimitates uncertanty of doing something wrong (D3)
regularly. Before this, releasing felt a bit * Confidence that product is of good quality (D3)
overwhelming, always having this worry that
something fundamental that I fail to detect is going to
be released to the users, leading to their irritation and
lack of confidence in the produced code.

What other feature do you create dfu file (if it isn't already) Release procedure. Also would be nice with more Automated release procedure (this is very close now). * Automate release process (D1, D2, D3)
desire to be implemented into hardware tests, including simulated external user * More hardware tests incl. simulated input (D2)
the CI Service? inputs
Do you have any other Seems good Very grateful for the work done. I clearly see how my * Positive perception (D2, D3)
feedback on the CI service working speed has increased. * Increased working speed (D2)
prototype?
Comprehensive Activities 6 6 6 6
Effective Communication 4 6 6 5,333333333
Immediacy 4 6 5 5
Appropriate Dimensioning 4 6 6 5,333333333
Accuracy 6 6 6 6
Lucidity 5 6 6 5,666666667
TRITA-EECS-EX-2023:847

www.kth.se
For DIVA
{
"Author1": {
"Last name": "Segatz",
"First name": "Fabian",
"Local User Id": "u1sdcy2v",
"E-mail": "segatz@kth.se",
"ORCiD": "0000-0002-1802-0849",
"organisation": {"L1": "School of Electrical Engineering and Computer Science ",
}
},
"Degree": {"Educational program": "Master’s Programme, Embedded Systems, 120 credits"},
"Title": {
"Main title": "Continuous Integration for Embedded Software with Modular Firmware Architecture",
"Language": "eng" },
"Alternative title": {
"Main title": "Kontinuerlig Integration för Inbäddad Programvara med Modulär Firmware-Arkitektur",
"Language": "swe"
},
"Supervisor1": {
"Last name": "Fu",
"First name": "Han",
"Local User Id": "u1d7tcbu",
"E-mail": "hfu@kth.se",
"organisation": {"L1": "School of Electrical Engineering and Computer Science ",
"L2": "CS" }
},
"Supervisor2": {
"Last name": "Elgcrona",
"First name": "Gunnar",
"E-mail": "gunnar.elgcrona@cobolt.se",
"Other organisation": "Cobolt AB"}
},
"Examiner1": {
"Last name": "Artho",
"First name": "Cyrille",
"Local User Id": "u1tesytk",
"E-mail": "artho@kth.se",
"organisation": {"L1": "School of Electrical Engineering and Computer Science ",
"L2": "CS" }
},
"Cooperation": { "Partner_name": "Cobolt AB"},
"Other information": {
"Year": "2023", "Number of pages": "xv,97"}
}