Spectrum analysis on quality requirements consideration in software design documents

In the same way as functional requirements, quality requirements, such as security, usability, reliability, and efficiency should be defined at the requirements definition stage because these requirements are the dominant factor in development costs and efforts. How to define quality requirements completely and correctly is thus well studied (Kaiya et al. 2010a; Firesmith 2005; Zhang et al. 2008). However, we have no standard ways to confirm such quality requirements are adapted in upcoming software artifacts such as design documents or test cases. For functional requirements, traditional stepwise refinement techniques can be used for this purpose. At the last stage of software development, we can confirm whether quality requirements are adapted and implemented through testing. It is, however, too late to find incorrect or missing quality requirements in software implementation at this stage because the design and/or codes would need to be revised. We thus have to develop a method for validating quality requirements considerations in intermediate software artifacts, such as design documents.

In this paper, we propose and evaluate this method. We use a technique called “spectrum analysis for quality requirements” (Kaiya et al. 2008) to construct this method. The contributions of the method are as follows. First, software engineers can become aware of missing and/or incorrect quality requirements considerations in a design document even if such considerations are scattered over the document. Second, the software engineers can narrow down the parts of the design document containing the missing and/or incorrect quality requirements. Third, the method can be applied almost automatically because domain knowledge can be reused for spectrum analysis. Fourth, the method can be applied to most requirements and design documents because it does not require specific notations such as mathematical equations but that the documents be written in natural language. Fifth, the method can be applied to a wide range of development styles such as waterfall or spiral styles because it does not give any constraints on the ordering of software development. A CASE tool was developed for supporting analysts in performing this method.

The rest of this paper is organized as follows. In the section “Spectrum analysis”, we briefly introduce the “Spectrum analysis for software quality requirements” technique used to analyze software quality requirements in a document. We then propose the method for validating quality requirements considerations in a design document in the section “Method”. By using our method, we assume we can apply our spectrum analysis from the section “Spectrum analysis” to a design document. To apply this analysis to a design document efficiently, a supporting tool is crucial because the size of such a document is not so small. In the section “Supporting tool”, we explain the supporting tool, and how to use it about a design document. We then confirm the assumption that the spectrum analysis can be applied to a design document in the section “Preliminary evaluation” by comparing the result of an analysis performed by an expert in commercial systems with the result obtained by the method. In the section “Evaluation”, we show an evaluation to confirm that the method works well. We finally review related works, summarize our current results, and show future problems.

Spectrum analysis for quality requirements is one measurement technique for summarizing quality features scattered over a requirements document. The technique enables us to visualize quality characteristics in the same way as sound waves. We explain how to perform this analysis by using the example in Figure 1. In the figure, there is a list of requirements and a list of quality characteristics. These two lists are the inputs of the technique. An analyst decides the quality characteristics related to each requirement. For example, requirement #3 is related to both resource efficiency and changeability. After that, the analyst counts the number of requirements related to each quality characteristic. For example, resource efficiency is related to three out of five requirements. The counted number of each quality characteristic lets us know how often each quality characteristic is mentioned in a requirements document. We confirmed that the importance of each quality characteristic was reflected in the number (Kaiya et al. 2012). We call the number the “power” of a quality characteristic. To normalize the power, it is divided by the total number of requirements. For example, in this figure, resource efficiency is more important than the others because it is frequently mentioned in the list of requirements. We may give some weights to each requirement on the basis of the priority among the requirements. We may also reformat a requirements document (Ncube et al. 2007) if the document is highly structured like i* or KAOS goal models.

We can maintain the traceability links between the power of a quality requirement in a spectrum and requirements because the power is derived from the requirements. For example in Figure 1, the power of resource efficiency is derived from requirements #1, #2 and #3. When we want to know the reasons for the differences between two spectra, we simply trace the links mentioned above. The spectrum analysis for quality requirements only focuses on the type of quality requirements. It cannot handle the value and/or level of quality requirements. For example, it can detect whether a quality requirement related to resource efficiency is considered in a document. An example of such a quality requirement is “the number of users.” However, it cannot decide whether the number itself (e.g., 10, 100 or 10000 people) is correctly considered in a document. For this kind of decision on the value/level of quality requirements, specific methods are required for each type of quality requirements. This is one of the limitations of our spectrum analysis.

Establishing the relationships between requirements and quality characteristics in Figure 1 is not an easy task. To help, a term-characteristics map (TCM) was proposed, and its effectiveness was validated (Kaiya et al. 2010a). The TCM is a matrix between terms and quality characteristics defined in each problem domain. By looking up the TCM, the relationships in Figure 1 can be easily established because an analyst simply needs to check the occurrences of terms in each requirement. Developing a TCM is not easy task as well. We develop a TCM in a domain by gathering the results of a spectrum analysis. If sentences containing the term “xxx” are frequently related to a quality characteristic such as “security,” we develop a TCM containing the data that “xxx” is related to “security”. We empirically validated that the confidence of the TCM in each domain increases when the number of the results of the spectrum analysis increase (Kaiya et al. 2011).

In this section, we introduce the method for confirming quality requirements considerations in a design document by using spectrum analysis.

Overview

Figure 2 shows the overview of the method. In the method, spectrum analysis is first applied to the requirements and design respectively. We then compare one spectrum with another to identify quality characteristics defined incorrectly and/or incompletely. In Figure 2, “Qbility” seems to be defined incompletely in design because the power of “Qbility” in the design spectrum is smaller than that in the requirements spectrum. “Sbility” seems to be defined incorrectly in the design because the power of “Sbility” in the design spectrum is larger than the power in the requirements. Because we can find methods (functions in desgin) related to each characteristic on the basis of the relationships between characteristics and methods, we can investigate the causes of such incompleteness and/or incorrectness. We explain each step in this method below.

Spectrum analysis for design

There are several kinds of software design documents because there are several kinds of people who read them, such as project managers, programmers, architects, and testers. We focus on a design document for programmers in this paper. We thus regard the explanations for each method or function as a design document because programmers have to know the functionalities and roles of each method or function.

The spectrum analysis for design is also performed in accordance with the technique in the section “Spectrum analysis”. Instead of a sentence(s) for each requirement, a sentence(s) used for explaining each method is used for the spectrum analysis. We thus normalize the powers in a spectrum on the basis of the number of methods.

In addition, we have to revise the spectrum in accordance with external libraries such as application programmable interfaces (API) and/or frameworks. Both external organizations and the developer him/herself will provide such external libraries. Such a revision is performed as follows.

1. 1.

   In accordance with the design description, external libraries to be used are selected.

    
2. 2.

   For each library, quality characteristics supported by the library are identified. We call such characteristics “library-characteristics”. The documents for each library help us to identify them.

    
3. 3.

   For each method, the libraries used in the method are identified. We then establish the relationships between the method and the library-characteristics of the libraries.

    

Aspect-oriented design (AOD) is currently out of the scope of this method. In the case of using AOD, the revision steps above cannot be applied because main routines do not know their required libraries (aspects), but the libraries know the routines that call the libraries. All aspects should be woven together before applying our method if AOD is used.

We show an example of a design document for programmers in Figures 3 and 4 written in Javadoc (Oracle 2004). In this example, a supporting system for a drugstore is shown, and the system will be written in Java. In this design, 13 classes are defined except test classes. In Java, test classes are normally ended with “Test” because that is the common practice when using the unit test tool called “JUnit”. In Figure 3, methods in the class “Medicine” are listed with brief explanations. However, programmers cannot implement each method with only such brief explanations because such explanations contain only the functionality of the method. Designers thus have to give more detailed explanations for each method, as exemplified in Figure 4. In the figure, the method “sell” in the class “Medicine” is explained in detail. By reading this detailed explanation, programmers can correctly implement the method. In this detailed examination, programmers can know that the sell method does not require checking the amount of stock. These kinds of explanations avoid useless double checks for parameters that are required by the policy of “design by contract” (Meyer 1997). The spectrum analysis for the design is applied to this kind of detailed explanations. Terms and words that are used in the spectrum analysis depend on the contents of the TCM. In our evaluation, the same TCM is used in both the requirements and design documents.

Comparison

We can simply visualize the spectra of both the requirements and design and identify the differences in order to find the causes of inconsistent and/or missing requirement considerations. When an engineer finds such causes, he or she modifies the design documents to remove them. We explain how to find such causes by using the example in Figure 2. As mentioned in the first sub-section of the section “Method”, we can find that “Qbility” seems to be defined incompletely in the design because its power in the design is smaller than that in the requirements. By examining the relationships at the top of the figure, methods except for methods 2 and 3 are suspected to cause the incompleteness because they are not related to “Qbility” in the figure. To remove such causes, an engineer has to establish more relationships between the quality characteristics and design than exist now. For “Sbility”, methods 3 and 10 are suspected to cause the incorrectness in the design because “Sbility” is not defined in the requirements but is defined in the design as shown in Figure 2. To remove such causes, the engineer has to establish less or no relationships between the quality characteristics and design than exist now.

Because a quality spectrum is a kind of vector, we use cosine similarity (cossim) to decide whether two spectra are similar to each other. Cosine similarity can be a metric for identifying the inconsistent and incomplete quality requirements considerations in design. The definition of cosine similarity between $\overrightarrow{a}$ and $\overrightarrow{b}$ is as follows.

$\mathit{\text{cossim}}(\overrightarrow{a},\overrightarrow{b})=\frac{a_1\ast b_1+\cdots +a_n\ast b_n}{\sqrt{a_1^2+\cdots +a_n^2}\ast \sqrt{b_1^2+\cdots +b_n^2}}$

When two vectors are completely the same, the value is 1. Because a quality spectrum never has a negative value in its vector, cosine similarity between two quality spectra varies from 0 to 1. Therefore, we may regard two quality spectra as similar if their cosine similarity is close to 1. In addition, cosine similarity does not take the length of the vector into account. Figure 5 shows an intuitive meaning of the cosine similarity with two-dimensional vectors. As show in the figure, two vectors D and C are almost the same as each other. The cosine similarity among them thus takes almost one (0.99). In comparison, vectors A and E make a right angle. As mentioned above, all values in a vector for our spectrum analysis take non-negative values. The direction of vector A is thus completely different from that of vector E. The cosine similarity between these vectors takes 0 by the definition above.

As mentioned in the section “Spectrum analysis”, the idea of the spectrum analysis is simple, but it is too dull for humans to perform the analysis manually. We thus developed a supporting tool to perform the spectrum analysis. Because the tool is designed to analyze natural language sentences, it can be applied to both requirements documents and design documents, such as Java docs.

Software quality requirements are widely focused on in the field of software, and we can find a special magazine issue of IEEE Software published in 2008 (Blaine and Cleland-Huang 2008). In the issue, the importance and challenges of software quality requirements were summarized, and one of the challenges is measurement and traceability for software quality requirements. In this section, we briefly review research on this measurement and traceability to clarify the importance of software quality requirements. We use non-functional requirements (NFRs) as a synonym for quality requirements even though NFR contains more things than do quality requirements (Blaine and Cleland-Huang 2008; Glinz 2008).

First, we focus on traceability links among different kinds of software engineering artifacts. Managing explicit links among different artifacts is a normal idea. For example, links between a section in a requirements document and classes and packages in design documents help us to find the impacts of changes on design.

However, maintaining such links takes a lot of effort in general. To mitigate such effort, several kinds of ideas are proposed. Lucia et al. used information retrieval techniques to manage such links (Lucia et al. 2009). Mandelin et al. used a probability model to do that (Mandelin et al. 2005). Ratanotayanon et al. used the differences between different versions of artifacts to manage traceability links efficiently (Ratanotayanon et al. 2009). Lopez et al. used the techniques of natural language processing (NLP) and machine learning (ML) to trace quality requirements to architecture (Gokyer et al. 2008). Model-driven approaches can enable us maintain traceability automatically. Alebrahim et al. proposed a method to derive software architectures from quality requirements (Alebrahim et al. 2011). In the method, problem frames (Jackson 2000) and its UML profile was used for specifying functional and quality requirements. They also used patterns for deriving architecture. Although the method seems to work well for maintaining traceability, we have to use specific and normally complex notations.

Second, we focus on a central model shared by software engineering artifacts. By using this model, we can easily trace an artifact to another via the model. Jane et al. proposed a method called “Goal Centric Traceability” for quality requirements (Cleland-Huang 2005; Cleland-Huang et al. 2008). In the method, a goal model plays the role of a central model. The thesaurus and ontology are popular notations for a central model. Daneva et al. proposed an ontology for NFRs (2009; Kassab et al. 2008). However, how to make links between ontology and software artifacts was not mentioned. Saeki et al. used a domain ontology for traceability between documents and source codes (Yoshikawa et al. 2009).

Kaiya et al proposed another idea for traceability called “projection traceability” (Kaiya et al. 2010b ), and the method proposed in this paper is based on this idea. In that paper (Kaiya et al. 2010b ), the traceability between requirements and codes is analyzed, but the execution tests can be used instead of this analysis because the codes can be executable. In comparison, the method in this paper is about the traceability between requirements and design, and it helps the software developers find missing and/or incorrect quality requirements considerations earlier than with the analysis in (Kaiya et al. 2010b).

Finally, we briefly review research on measuring quality requirements. One famous catalog for software quality requirements is the ISO9126 standard (International Standard ISO/IEC 9126-1 2001 ), which contains about 20 subcharacteristics such as accuracy, and reliability. Washizaki et al. provided measurement methods that use the usual metrics on source codes and design diagrams such as lines of codes (LOC) and cyclomatic complexity (CC) for each subcharacteristic (Washizaki et al. 2008 ). Jane et al. proposed a method for detecting and categorizing NFRs contained in a document by using information retrieval (IR) and NLP techniques. (Cleland-Huang et al. 2006 ). To count and normalize the number of NFRs in a document, we can visualize the distribution of NFRs. Kaiya et al. proposed a technique for summarizing such a distribution and visualizing it on the basis of a metaphor of spectrum analysis in optics (Kaiya et al. 2009 ). They used the technique to identify domain specific commonality by directly comparing one spectrum of a system to another. Whether a requirement is related to a quality requirement is decided on the basis of the occurrences of terms, which characterize the quality requirement in the study. Measuring quality requirements is used for prioritizing requirements (Otero et al. 2010 ), but how to establish relationships between quality requirements (a quality feature in the study) is not explained in the study. Because quality requirements qualify functional requirements, measuring and predicting quality requirements on the basis of the content of each functional requirement is a natural idea. Kaiya et al. developed such an idea by using semi-formal functional requirement notation (Kaiya and Ohnishi 2011 ). They also used a machine learning technique to automate the prediction and measurement (Tanaka et al. 2012). This kind of research is useful for tracing quality requirements indirectly.

In this paper, we proposed the method for validating quality requirements considerations in a design document. We also proposed a supporting tool for the method. Because the method uses spectrum analysis for quality requirements, it does not give any constraints on design notations and activities. Through an evaluation on commercial software systems, we confirmed the method works well.

Currently, we have no integrated CASE tools for supporting design activities and design analysis based on this method. Our current tool analyzes only documents on the basis of our spectrum analysis, so analysts have to manually modify these documents if problems are found. One important goal in the future is to develop an integrated CASE tool for supporting both design descriptions and analyses for software development.

^a Line separators depend on the kinds of operating system. For example, a line separator in UNIX is normally ∖n, while one in Windows is ∖r∖n.