Case Study Method and Research Design for the Dynamic Multilevel Component Selection Problem

Abstract

The architecture of a system changes after the deployment phase due to new requirements of the stakeholders. The software architect must make decisions about the selection of the right software components out of a range of choices to satisfy a set of requirements. This work deals with the component selection problem with a multilevel system view in a dynamic environment.

To validate our approach we have used the case study method. Three different case studies were performed but only one is presented in the current paper. The research design was conducted using a research question, propositions and for interpreting the study’s findings we have use the Wilcoxon signed ranks statistical test. The tests performed show the potential of evolutionary algorithms for the dynamic multilevel component selection problem.

1 Introduction

The problems of identification and selection the right software components out of a range of choices to satisfy a set of requirements have received considerable attention in the field of component-based software engineering during the last two decades [2, 8].

Identification of a software architecture for a given system may be achieved in two ways: (1) Component Identification and (2) Component Selection. Component Identification has the scope to partition functionalities of a given system into non-intersecting logical components to provide the starting points for designing the architecture. The aim of Component Selection methods is to find suitable components from repository to satisfy a set of requirements under various constraints/criteria (i.e. cost, number of used components, etc.). This paper has focused on the component selection process, the goal being to provide the suitable existing components matching software requirements.

After the deployment phase, the maintenance phase requires more attention and the software architects need assistance in the decisions of the frequent changes of the software system. Thus, software maintenance phase is dedicated not only to the removal of possible remaining bugs and/or release of new software versions but also needs to consider adding new requirements (i.e., the stakeholder need some new requirements to be added) or removing some of the requirements (i.e., the development team found out that some requirements were not correctly specified first or even not needed). So, the other perspective concerning component configurations refers to updating, adding, removing one or many requirements from an already constructed system. This represents the reconfiguration problem [13], transforming the structural view of a component system, changing the system’s functionality. Our previous research regarding this perspective was proposed in [11].

The contribution to this paper is the use of the case study method and the research design from the book of Yin [14] to validate our research proposal for the Dynamic Multilevel Component Selection Problem. A research question and propositions are used to conduct research design. For interpreting the study’s findings we used the Wilcoxon signed ranks statistical test.

Our approach combines the multilevel configuration [10] of the component selection problem with the dynamical changing requirements, i.e. updating, adding, removing requirements (or components) from an already constructed system [11].

Our previous studies analyzed the current state of art regarding the component selection problem and also analyzed the difference compared with our approach. For more details see [9]. A similar approach considering evolution of software architecture was proposed by [3].

The paper is organized as follows: Sect. 2 contains configuration and reconfiguration description problems, the description of the optimization process, and the proposed evolutionary-based algorithm approach. Section 3 presents the reasons for using case study method and the research design, the criteria used to interpret the findings of the results, i.e. the evaluation are presented in Sect. 4. In Sect. 5 we apply the approach to one example to validate the proposed approach. Some experiments are performed considering two dynamics: requirements changes over time and component repository varies over time. We conclude our paper in Sect. 6.

2 Dynamic Multilevel Component Selection Problem

2.1 Component Systems, Configurations and Reconfigurations

A component [4] is an independent software package that provides functionality via defined interfaces. The interface may be an export interface through which a component provides functionality to other components or an import interface through which a component gains services from other components.

A configuration [13] of a component system is described as the structural relationship between components, indicated by the layout of components and connectors. A reconfiguration is to modify the structure of a component system in terms of additions, deletion, and replacement of components and/or connectors. While the reconfiguration transforms the structural view of a component system, it changes the system’s functionality and service specifications. From another aspect, a reconfiguration may consist of several individual updates or changes to components and/or connectors.

There are two type of components: simple component - is specified by the inports (the set of input variables/parameters), outports (the set of output variables/parameters) and a function (the computation function of the component) and compound component - is a group of connected components in which the output of a component is used as input by another component from the group. For details about the component model please refer to [10].

2.2 Dynamic Multilevel Component Selection Problem Formulation

An informal specification of the configuration problem is described in the following. It is needed to construct a final system specified by input data and output data. We can see the final system as a compound component and thus the input data becomes the required interfaces of the component and the output data becomes the provided interfaces, and in this context we have the required interfaces as provided and we need to provide the internal structure of the final compound component by offering the provided interfaces.

A formal definition of the configuration problem [10] (seen as a compound component) is as follows. Consider SR the set of final system requirements (the provided functionalities of the final compound component) as $SR=\{r_{1}, r_{2}, . . . , r_{n}\}$ and SC the set of components (repository) available for selection as $SC=\{c_{1}, c_{2}, . . . , c_{m}\}.$ Each component $c_{i}$ can satisfy a subset of the requirements from SR (the provided functionalities) denoted $SP_{c_{i}}=\{ p_{i_{1}}, p_{i_{2}}, . . ., p_{i_{k}}\}$ and has a set of requirements denoted $SR_{c_{i}}=\{ r_{i_{1}}, r_{i_{2}}, . . ., r_{i_{h}}\}$. The goal is to find a set of components Sol in such a way that every requirement $r_{j}$ ($j=\overline{1,n}$) from the set SR can be assigned a component $c_{i}$ from Sol where $r_{j}$ is in $SP_{c_{i}}$ ($i=\overline{1,m}$), while minimizing the number of used components and the total cost of assembly. All the requirements of the selected components must be satisfied by the components in the solution. If a selected component is a compound component, the internal structure is also provided. All the levels of the system are constructed.

The reconfiguration problem [13] is define similar to the configuration problem but considering the dynamical changes of either requirements or component. Regarding the reconfiguration problem [11], the dynamics of the component selection problem can be viewed in two ways: the system requirements or the repository containing the components varies over time.

2.3 Dynamic Multilevel Component Selection Optimisation Process

Our approach starts by considering a set of components (repository) available for selection and the specification of a final system (input and output). The optimisation process begins with the Dynamic Multilevel Component Selection Problem Formulation (see Fig. 1 for details). The result of this step is the transformation of the final system specification as the set of required interfaces (and the set of provided interfaces). In the second step, the construction of the multilevel configurations is done by applying the evolutionary optimisation algorithm (from the fourth step, see Fig. 1) for each time steps (from the Dynamic Changing Requirements or Dynamic Changing Components step). The evolutionary optimisation algorithm is applied for each time steps (i.e. if there are still changing requirements or components) and for each compound component from each level. The solution with best fitness value is selected at each level. The fifth step presents the results.

Fig. 1.

Dynamic multilevel component selection optimisation process.

2.4 Evolutionary Optimisation

The approach presented in this paper uses principles of evolutionary computation and multiobjective optimization [6]. First, the problem is formulated as a multiple objective optimization problem having 5 objectives: the number of used components, the number of new requirements, the number of provided interfaces, the number of the initial requirements that are not in solution, and the cost of a component (group of components). All objectives are to be minimized. The percentage importance of each objective to the fitness functions are: $30\,\%$ number of distinct used components, $30\,\%$ number of new requirements, $5\,\%$ number of provided interfaces, $5\,\%$ number of initial requirements that are not in solution, and $30\,\%$ cost value. We have selected these percentages because of their impact in finding the final solution (number of new requirements will finally be 0 for final solutions (but is needed to be $30\,\%$ during the population “improvement” process). The number of provided interfaces does not play an important role in finding the final solution but the number of distinct components does. Also, the cost value plays an important role in selecting the final cheapest solution among many obtained.

There are several ways to deal with a multiobjective optimization problem. In this paper the Pareto dominance [6] principle is used.

Definition 1

Pareto dominance. Consider a maximization problem. Let x, y be two decision vectors (solutions) from the definition domain. Solution x dominate y (also written as x $\succ$ y) if and only if the following conditions are fulfilled: $f_{i}(x) = f_{i}(y)$, $\forall$ $i = 1,n$; $\exists j \in \{1, 2, ... n\}: f_{j}(x) > f_{j}(y)$. That is, a feasible vector x is Pareto optimal if no feasible vector y can increase some criterion without causing a simultaneous decrease in at least one other criterion.

Solution Representation. The current solution representation was used in [11].

A solution (chromosome) is represented as a 5-tuple $(lstProv,\ lstComp,$ $\ lstInitReq,\ lstNewReq,\ cost)$ with the following information: list of provided interfaces (lstProv); list of components (lstComp); list of initial requirements (lstInitReq); list of new requirements (lstNewReq); cost (sum of the cost of each component in the chromosome). The value of $i-th$ component represents the component satisfying the $i-th$ provided interface from the list of provided interfaces. A chromosome is initialized with the list of provided interfaces by the list of requirements of the final required system and with the list of initial requirements with the list of the requirements of the final required system (these will be provided as implicit, being input data of the problem/system). An example is given in what follows.

A valid chromosome may be structured as follows:

$Crom_{0}=\ ((3,\ 4),\ (12,\ 24),\ (1,\ 2),\ (5,\ 7,\ 8,\ 11,\ 33,\ 30),(67)).$ This chromosome does not represent a solution, it is only an initialized chromosome without any applied genetic operator. The provided interfaces $(3,\ 4)$ are offered by the components $(12,\ 24)$. The set of initial requirements are: $(1,\ 2).$ By using a component we need to provide it’s requirements: component 12 requires the $(5,\ 7,\ 8,\ 11)$ new requirements and component 24 requires the $(33,\ 30)$ new requirements.

Genetic Operator. Because the current paper uses the same genetic algorithm as in [11] paper, the mutation operator keeps the computation method. There are two types of mutations that ca be applied to a chromosome, depending of the chromosome “status”.

If the chromosome still has new requirements to satisfy then the following steps will be applied:

• randomly select a requirement from the list of new requirements;

• add the associated provided interface of the new requirement in the list of provided interfaces;

• add the component that satisfies the added provided interface (a component is randomly selected from the set of components that offer it);

• remove the selected requirement from the list of new requirements, and add to the list of new requirements the requirements of the added component (if there exist).

If the chromosome representations does not have any other new requirements to be satisfied then the following steps will be applied:

• randomly select a provided from the list of providers;

• search if exists another component in the chromosome that could satisfy this provided interface.

  • If exist then add this component. No new requirements are needed to be added.;

  • If it does not exist another component to satisfy the provided interface then choose another component. All the dependencies of the previous used component are deleted and new requirements (of the new added component) are added (if exists).

The above chromosome after applying mutation operator has the internal structure:

$Crom_{0}=\ ((3,\ 4,\ 8),\ (12,\ 24,\ 2),\ (1,\ 2),\ (5,\ 7,\ 11,\ 33,\ 30,\ 9,\ 10),\ (122)).$

In order to provide the $8^{th}$ new requirement we have selected component 2. New requirements are added for the component 8. Each time the mutation operator is applied to this new chromosome a new requirement is satisfied by a randomly selected component (that can provide the selected requirement) and new requirements may be added due to component selection and dependencies. A final chromosome may be as the following:

$Crom_{0}=\ ((3,\ 4,\ 8,\ 9,\ 5,\ 11,\ 7,\ 10,\ 6),$ $(12,\ 12,\ 2,\ 21,\ 6,\ 21,\ 2,\ 21,\ 6),\ (1,\ 2),$ $(),\ (153)).$ All the above mutation cases were applied to the chromosome described above. The selection of the component 2 for satisfying the provided interface 8 corresponds to the first type of mutation but the modification of the 24 component (to satisfy the 4 provided) into the component 12 corresponds to the second type of mutation. The dependencies of the 24 component were deleted from the chromosome (33 and 30).

3 Case Study Method and Research Design

The book of Yin [14] provided us with a strategy of identifying the method for our research project, showing when to choose the case study method and how to do research design. Defining the Research Questions is the most important step to be taken in a research study. In general, case studies are the preferred strategy when “how” or “why” questions are being posed.

Our Research Question: How and Why do Search-based Algorithms (in our case a Genetic Algorithm and a Random Search Algorithm) provide different results for the Dynamic Multilevel Component Selection Problem?

Another component of the research design are the Propositions that direct attention to something that should be examined within the scope of study. These propositions begin to tell you where to look for relevant evidence.

Our Proposition: The Search-based Algorithms (in our case a Genetic Algorithm and a Random Search Algorithm) provide different results for the Dynamic Multilevel Component Selection Problem because in the case of Genetic Algorithm the fitness of a solution is reevaluated.

Criteria for interpreting a study’s findings represents the third component of the research design of a case study according to [14]. Statistical analysis offer some explicit criteria for such interpretation.

Our Criteria for interpreting the study is based on the Wilcoxon signed ranks statistical test that aims to detect significant differences between two sample means, that is, the behavior of the two algorithms. For more information regarding the Wilcoxon statistical test see Sect. 4.

After covering these components of research designs, the construction of a theory related to our topic of study will follow. Our Theory development: The case study will show why the Genetic Algorithm performs better than the Random Search Algorithm.

An issue related to case studies is referring to the generalization from a case study to theory. According to [14] statistical generalization is the common way when doing surveys, but in doing case studies the analytic generalization should be used. Multiple cases resemble multiple experiments and under these circumstances the mode of generalization is analytic. If two or more cases are shown to support the same theory, replication [14] may be claimed. The replication logic is analogous to that used in multiple experiments. Some of the replications might attempt to duplicate the exact conditions of the original experiment. Other replications might alter one or two experimental conditions considered unimportant to the original findings, to see whether the findings could still be duplicated. Only with such replications would the original finding be considered robust.

Our Replication strategy: The time steps for the dynamic changing requirements (Level 1) and for the dynamic changing components (Level 1, 2, and 3) were modified for Case study 2 and for Case Study 3.

Thus, we have selected the case study method and conducted three case studies, the first one is a real case study for constructing a Reservation System (reported in [9]) and the last two of them are constructed using artificial data (the current paper reports the second experiment and the third experiment is going to be published). For each case study we specify the component selection problem. After that, the experimental studies are followed for each considered case study: the two perspectives, changing requirements and changing component repository. Following the replication approach to multiple-case studies [14], each individual case study will be finalized by an individual case report that will be next considered to be part of a summary report, i.e. a cross-case conclusions. Thus, in our case the results obtained are reported and conclusions about the potential of evolutionary algorithms for the dynamic multiobjective multilevel component selection problem are drawn.

4 Interpreting the Study

When comparing [7] two algorithms, the best fitness values obtained by the searches concerned are an obvious indicator to how well the optimisation process performed. Inferential statistics may be applied to discern whether one set of experiments are significantly different in some aspect from another. Usually we wish to be in a position to make a claim that we have evidence that suggests that Algorithm A (Genetic Algorithm) is better than Algorithm B (Random Search). The Wilcoxon signed ranks test [5] is used for answering the following question: do two samples represent two different populations? It is a nonparametric procedure employed in hypothesis testing situations, involving a design with two samples. It is a pairwise test that aims to detect significant differences between two sample means, that is, the behavior of two algorithms. The best fitness value (from the entire population) was used for comparing the two algorithms.

The Wilcoxon signed ranks test has two hypothesis:

• 1. Null hypothesis $H_{0}$: The median difference is zero versus.

• 2. Research hypothesis $H_{1}$: The median difference is not zero, $\alpha =0.05$.

Steps of the Wilcoxon signed ranks test: Compute $W_{-}$ and $W_{+}$, Check if $W_{-}+W_{+}$ = n(n+1)/2, Select the test statistic (for the two tailed test the test statistic is the smaller of $W_{-}$ and $W_{+}$), We must determine whether the observed test statistic $W_{t}$ supports the $H_{0}$ or $H_{1}$, i.e. we determine a critical value of $W_{c}$ such that if the observed value of $W_{t}$ is less or equal to critical value $W_{c}$, we reject $H_{0}$ in favor to $H_{1}$.

Due to stochastic nature of optimisation algorithms, searches must be repeated several times in order to mitigate against the effect of random variation. How many runs do we need when we analyze and compare algorithms? In many fields of science (i.e. medicine and behaviour science) a common rule of thumb [1] is to use at least $n=30$ observations. We have also used in our evaluation 30 executions for each algorithm.

5 Experimental Results

In this section, the proposed approach is evaluated and the results are reported. According to the book of Yin [14] and as stated in Sect. 3 we have conducted three different experiments. The first one was reported in [9]. The current paper reports the second experiment. The third experiment is going to be published.

5.1 Component Selection Problem Formulation

Having specified two input data (customerData, calendarData) and two output data (doneReservation, requestConfirmation) needed to be computed, and having a set of 150 available components, the goal is to find a subset of the given components such that all the requirements are satisfied considering the optimisation criteria specified above. The set of requirements $SR=\{ r_{6}, r_{8} \}$ (view as provided interfaces $\{ p_{6}, p_{8} \}$) and the set of components $SC=\{ c_{0}, c_{1}, c_{2}, c_{3}, c_{4}, c_{5}, c_{6},..., c_{150} \}$ are given. The final system has as input data (transformed in required interfaces) the set $\{ r_{3}, r_{4}\}$.

Remark. Due to lack of space the component repository is not described in this paper but may be found at [12].

Experimental Studies - Case 1: Dynamic Changing Requirements. As in the case of the first case study, we consider two types of dynamics and, consequently two experiments corresponding to each of them: the requirements of the problem change over time, and the components available at a certain time step change.

Five different time steps are built using artificially generated data and the dynamics at each of these steps are: T $=$ 1 - The initial requirements, T $=$ 2- Add one new requirement, T $=$ 3- Add one new requirement, T $=$ 4 - Remove one requirement, and T $=$ 5 - Add one new requirement.

Performed tests. The role of the performed test was to see if the number of iterations and the population size play a role in finding the Pareto solutions. The conclusions about the findings of this type of experiments are given in Sect. 5.2.

Multilevel configurations. The compound components are next constructed by applying the same algorithm but with different requirements and input data. For the Second Level of the system the set of required interfaces is $\{ r_{1},\ r_{2}\}$ and the set of provided interfaces is $\{ p_{18},\ p_{19}\}$. For the Third Level of the system the set of required interfaces is $\{ r_{27},\ r_{31}\}$ and the set of provided interfaces is $\{ p_{28},\ p_{34}\}$. The conclusions about the findings of this type of experiments are given in Sect. 5.2.

Remark. We have not presented the charts regarding the influence of population size or iteration number in finding the Pareto solutions, because we have concentrated our findings in comparing the algorithm using the Wilcoxon statistical test.

Wilcoxon statistical test. In Sect. 4 we have described in details the Wilcoxon statistical test that we have use to compare our Genetic Algorithm with the Random Search Algorithm. In Table 1 we have the test results for the Case Study 1 - Dynamic Changing Requirements.

Table 1. Wilcoxon statistical test for Case Study 1 - changing requirements experiment.

The Wilcoxon statistical test (see Table 1) shows that we have statistically significant evidence at $\alpha =0.05$ to show that the median is positive, i.e. the $H_{0}$ Null-Hypothesis is rejected in favor of $H_{1}$ for all levels and for all time steps.

Experimental Studies - Case 2: Dynamic Changing Components. As in the first and second case study, the repository containing components changes over time. This modification of the available components may be seen as an update of the COTS market, new components being available or other being withdrawn from the market.

Four different time steps are built using artificially generated data and the dynamics at each of these steps are: T $=$ 1 - The initial components, and T $=$ 2 - Add two new components.

Performed tests. The aim of the performed tests is the same as in the first and second case study: to see if the number of iterations and the population size play a role in finding the Pareto solutions. The conclusions about the findings of this type of experiments are given in Sect. 5.2.

Multilevel configurations. The compound components are next constructed by applying the same algorithm but with different requirements and input data. For the second level of the system the set of required interfaces is $\{ r_{1},\ r_{2}\}$ and the set of provided interfaces is $\{ p_{18},\ p_{19}\}$. For the second level we have four time steps: T $=$ 1 - No modifications of the component repository, T $=$ 2 - Adding three new components and removing three old components, T $=$ 3 - Adding two new components, and T $=$ 4 - Adding one new component and removing one old component.

For the third Level of the system the set of required interfaces is $\{ r_{27},\ r_{31}\}$ and the set of provided interfaces is $\{ p_{28},\ p_{34}\}$. The conclusions about the findings of this type of experiments are given in Sect. 5.2. For the third level we have five time steps: T $=$ 1 - No modifications of the component repository, T $=$ 2 - Adding two new components and removing two old components, T $=$ 3 - Adding one new component and removing one old component, T $=$ 4 - Adding one new component and removing one old component, and T $=$ 5 - Adding one new component and removing two old components.

Remark. We have not presented the charts regarding the influence of population size or iteration number in finding the Pareto solutions, because we have concentrated our findings in comparing the algorithm using the Wilcoxon statistical test.

Wilcoxon statistical test. In Sect. 4 we have described in details the Wilcoxon statistical test that we have use to compare our Genetic Algorithm with the Random Search Algorithm. In Table 2 we have the test results for the Case Study 3 - Dynamic Changing Components.

Table 2. Wilcoxon statistical test for Case Study 3 - changing components experiment.

The Wilcoxon statistical test (see Table 2) shows that we have statistically significant evidence at $\alpha =0.05$ to show that the median is positive, i.e. the $H_{0}$ Null-Hypothesis is rejected in favor of $H_{1}$ for all levels and for all time steps.

5.2 Summary Report

For each individual case (also for those in [9]), the report indicated that the proposition from Sect. 5 was demonstrated, i.e. “The Search-based Algorithms (in our case a Genetic Algorithm and a Random Search Algorithm) provided different results for the Dynamic Multilevel Component Selection Problem because in the case of Genetic Algorithm the fitness of a solution is reevaluated.”

Regarding the research question from Sect. 5, i.e. “How and Why do Search-based Algorithms (in our case a Genetic Algorithm and a Random Search Algorithm) provide different results for the Dynamic Multilevel Component Selection Problem?”, the conclusions sustained by the conducted case studies is that the Genetic Algorithm provides better results than the Random Search Algorithm for the Dynamic Multilevel Component Selection Problem and that we have statistically significant evidence at $\alpha =0.05$.

6 Conclusion

The current work investigated the potential of evolutionary algorithms in a particular case of multiobjective dynamic system: multilevel component selection problem. The case study method was used to validate our approach and the research design was conducted. Referring to the generalization from a case study to theory, three different case studies were performed but only one is presented in the current paper.

Our Criteria for interpreting the study is based on the Wilcoxon signed ranks statistical test that was used to compare our Genetic Algorithm approach with a Random Search Algorithm. The tests performed show the potential of evolutionary algorithms for this particular problem and for other similar ones.