Table of Contents

Cover

Title

Dedication

Preface

List of Figures

List of Tables

CHAPTER 1: BASIC CONCEPTS AND PRELIMINARIES

1.1 QUALITY REVOLUTION
1.2 SOFTWARE QUALITY
1.3 ROLE OF TESTING
1.4 VERIFICATION AND VALIDATION
1.5 FAILURE, ERROR, FAULT, AND DEFECT
1.6 NOTION OF SOFTWARE RELIABILITY
1.7 OBJECTIVES OF TESTING
1.8 WHAT IS A TEST CASE?
1.9 EXPECTED OUTCOME
1.10 CONCEPT OF COMPLETE TESTING
1.11 CENTRAL ISSUE IN TESTING
1.12 TESTING ACTIVITIES
1.13 TEST LEVELS
1.14 SOURCES OF INFORMATION FOR TEST CASE SELECTION
1.15 WHITE-BOX AND BLACK-BOX TESTING
1.16 TEST PLANNING AND DESIGN
1.17 MONITORING AND MEASURING TEST EXECUTION
1.18 TEST TOOLS AND AUTOMATION
1.19 TEST TEAM ORGANIZATION AND MANAGEMENT
1.20 OUTLINE OF BOOK
REFERENCES

CHAPTER 2: THEORY OF PROGRAM TESTING

2.1 BASIC CONCEPTS IN TESTING THEORY
2.2 THEORY OF GOODENOUGH AND GERHART
2.3 THEORY OF WEYUKER AND OSTRAND
2.4 THEORY OF GOURLAY
2.5 ADEQUACY OF TESTING
2.6 LIMITATIONS OF TESTING
2.7 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 3: UNIT TESTING

3.1 CONCEPT OF UNIT TESTING
3.2 STATIC UNIT TESTING
3.3 DEFECT PREVENTION
3.4 DYNAMIC UNIT TESTING
3.5 MUTATION TESTING
3.6 DEBUGGING
3.7 UNIT TESTING IN EXTREME PROGRAMMING
3.8 JUNIT: FRAMEWORK FOR UNIT TESTING
3.9 TOOLS FOR UNIT TESTING
3.10 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 4: CONTROL FLOW TESTING

4.1 BASIC IDEA
4.2 OUTLINE OF CONTROL FLOW TESTING
4.3 CONTROL FLOW GRAPH
4.4 PATHS IN A CONTROL FLOW GRAPH
4.5 PATH SELECTION CRITERIA
4.6 GENERATING TEST INPUT
4.7 EXAMPLES OF TEST DATA SELECTION
4.8 CONTAINING INFEASIBLE PATHS
4.9 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 5: DATA FLOW TESTING

5.1 GENERAL IDEA
5.2 DATA FLOW ANOMALY
5.3 OVERVIEW OF DYNAMIC DATA FLOW TESTING
5.4 DATA FLOW GRAPH
5.5 DATA FLOW TERMS
5.6 DATA FLOW TESTING CRITERIA
5.7 COMPARISON OF DATA FLOW TEST SELECTION CRITERIA
5.8 FEASIBLE PATHS AND TEST SELECTION CRITERIA
5.9 COMPARISON OF TESTING TECHNIQUES
5.10 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 6: DOMAIN TESTING

6.1 DOMAIN ERROR
6.2 TESTING FOR DOMAIN ERRORS
6.3 SOURCES OF DOMAINS
6.4 TYPES OF DOMAIN ERRORS
6.5 ON AND OFF POINTS
6.6 TEST SELECTION CRITERION
6.7 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 7: SYSTEM INTEGRATION TESTING

7.1 CONCEPT OF INTEGRATION TESTING
7.2 DIFFERENT TYPES OF INTERFACES AND INTERFACE ERRORS
7.3 GRANULARITY OF SYSTEM INTEGRATION TESTING
7.4 SYSTEM INTEGRATION TECHNIQUES
7.5 SOFTWARE AND HARDWARE INTEGRATION
7.6 TEST PLAN FOR SYSTEM INTEGRATION
7.7 OFF-THE-SHELF COMPONENT INTEGRATION
7.8 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 8: SYSTEM TEST CATEGORIES

8.1 TAXONOMY OF SYSTEM TESTS
8.2 BASIC TESTS
8.3 FUNCTIONALITY TESTS
8.4 ROBUSTNESS TESTS
8.5 INTEROPERABILITY TESTS
8.6 PERFORMANCE TESTS
8.7 SCALABILITY TESTS
8.8 STRESS TESTS
8.9 LOAD AND STABILITY TESTS
8.10 RELIABILITY TESTS
8.11 REGRESSION TESTS
8.12 DOCUMENTATION TESTS
8.13 REGULATORY TESTS
8.14 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 9: FUNCTIONAL TESTING

9.1 FUNCTIONAL TESTING CONCEPTS OF HOWDEN
9.2 COMPLEXITY OF APPLYING FUNCTIONAL TESTING
9.3 PAIRWISE TESTING
9.4 EQUIVALENCE CLASS PARTITIONING
9.5 BOUNDARY VALUE ANALYSIS
9.6 DECISION TABLES
9.7 RANDOM TESTING
9.8 ERROR GUESSING
9.9 CATEGORY PARTITION
9.10 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 10: TEST GENERATION FROM FSM MODELS

10.1 STATE-ORIENTED MODEL
10.2 POINTS OF CONTROL AND OBSERVATION
10.3 FINITE-STATE MACHINE
10.4 TEST GENERATION FROM AN FSM
10.5 TRANSITION TOUR METHOD
10.6 TESTING WITH STATE VERIFICATION
10.7 UNIQUE INPUT–OUTPUT SEQUENCE
10.8 DISTINGUISHING SEQUENCE
10.9 CHARACTERIZING SEQUENCE
10.10 TEST ARCHITECTURES
10.11 TESTING AND TEST CONTROL NOTATION VERSION 3 (TTCN-3)
10.12 EXTENDED FSMS
10.13 TEST GENERATION FROM EFSM MODELS
10.14 ADDITIONAL COVERAGE CRITERIA FOR SYSTEM TESTING
10.15 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 11: SYSTEM TEST DESIGN

11.1 TEST DESIGN FACTORS
11.2 REQUIREMENT IDENTIFICATION
11.3 CHARACTERISTICS OF TESTABLE REQUIREMENTS
11.4 TEST OBJECTIVE IDENTIFICATION
11.5 EXAMPLE
11.6 MODELING A TEST DESIGN PROCESS
11.7 MODELING TEST RESULTS
11.8 TEST DESIGN PREPAREDNESS METRICS
11.9 TEST CASE DESIGN EFFECTIVENESS
11.10 SUMMARY
11.11 LITERATURE REVIEW
REFERENCES

CHAPTER 12: SYSTEM TEST PLANNING AND AUTOMATION

12.1 STRUCTURE OF A SYSTEM TEST PLAN
12.2 INTRODUCTION AND FEATURE DESCRIPTION
12.3 ASSUMPTIONS
12.4 TEST APPROACH
12.5 TEST SUITE STRUCTURE
12.6 TEST ENVIRONMENT
12.7 TEST EXECUTION STRATEGY
12.8 TEST EFFORT ESTIMATION
12.9 SCHEDULING AND TEST MILESTONES
12.10 SYSTEM TEST AUTOMATION
12.11 EVALUATION AND SELECTION OF TEST AUTOMATION TOOLS
12.12 TEST SELECTION GUIDELINES FOR AUTOMATION
12.13 CHARACTERISTICS OF AUTOMATED TEST CASES
12.14 STRUCTURE OF AN AUTOMATED TEST CASE
12.15 TEST AUTOMATION INFRASTRUCTURE
12.16 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 13: SYSTEM TEST EXECUTION

13.1 BASIC IDEAS
13.2 MODELING DEFECTS
13.3 PREPAREDNESS TO START SYSTEM TESTING
13.4 METRICS FOR TRACKING SYSTEM TEST
13.5 ORTHOGONAL DEFECT CLASSIFICATION
13.6 DEFECT CAUSAL ANALYSIS
13.7 BETA TESTING
13.8 FIRST CUSTOMER SHIPMENT
13.9 SYSTEM TEST REPORT
13.10 PRODUCT SUSTAINING
13.11 MEASURING TEST EFFECTIVENESS
13.12 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 14: ACCEPTANCE TESTING

14.1 TYPES OF ACCEPTANCE TESTING
14.2 ACCEPTANCE CRITERIA
14.3 SELECTION OF ACCEPTANCE CRITERIA
14.4 ACCEPTANCE TEST PLAN
14.5 ACCEPTANCE TEST EXECUTION
14.6 ACCEPTANCE TEST REPORT
14.7 ACCEPTANCE TESTING IN eXtreme PROGRAMMING
14.8 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 15: SOFTWARE RELIABILITY

15.1 WHAT IS RELIABILITY?
15.2 DEFINITIONS OF SOFTWARE RELIABILITY
15.3 FACTORS INFLUENCING SOFTWARE RELIABILITY
15.4 APPLICATIONS OF SOFTWARE RELIABILITY
15.5 OPERATIONAL PROFILES
15.6 RELIABILITY MODELS
15.7 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 16: TEST TEAM ORGANIZATION

16.1 TEST GROUPS
16.2 SOFTWARE QUALITY ASSURANCE GROUP
16.3 SYSTEM TEST TEAM HIERARCHY
16.4 EFFECTIVE STAFFING OF TEST ENGINEERS
16.5 RECRUITING TEST ENGINEERS
16.6 RETAINING TEST ENGINEERS
16.7 TEAM BUILDING
16.8 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 17: SOFTWARE QUALITY

17.1 FIVE VIEWS OF SOFTWARE QUALITY
17.2 MCCALL’S QUALITY FACTORS AND CRITERIA
17.3 ISO 9126 QUALITY CHARACTERISTICS
17.4 ISO 9000:2000 SOFTWARE QUALITY STANDARD
17.5 SUMMARY
LITERATURE REVIEW
REFERENCES

CHAPTER 18: MATURITY MODELS

18.1 BASIC IDEA IN SOFTWARE PROCESS
18.2 CAPABILITY MATURITY MODEL
18.3 TEST PROCESS IMPROVEMENT
18.4 TESTING MATURITY MODEL
18.5 SUMMARY
LITERATURE REVIEW
REFERENCES

GLOSSARY

INDEX

End User License Agreement

List of Tables

CHAPTER 3: UNIT TESTING

TABLE 3.1 Hierarchy of System Documents
TABLE 3.2 Code Review Checklist
TABLE 3.3 McCabe Complexity Measure

CHAPTER 4: CONTROL FLOW TESTING

TABLE 4.1 Examples of Path in CFG of Figure 4.7
TABLE 4.2 Input Domain of openfiles()
TABLE 4.3 Inputs and Paths in openfiles()
TABLE 4.4 Paths for Statement Coverage of CFG of Figure 4.7
TABLE 4.5 Paths for Branch Coverage of CFG of Figure 4.7
TABLE 4.6 Two Cases for Complete Statement and Branch Coverage of CFG of Figure 4.9a
TABLE 4.7 Interpretation of Path Predicate of Path in Figure 4.10.
TABLE 4.8 Interpretation of Path Predicate of Path in Figure 4.14.
TABLE 4.9 Test Data for Statement and Branch Coverage

CHAPTER 5: DATA FLOW TESTING

TABLE 5.1 Def() and c-use() Sets of Nodes in Figure 5.4
TABLE 5.2 Predicates and p-use() Set of Edges in Figure 5.4

CHAPTER 6: DOMAIN TESTING

TABLE 6.1 Two Interpretations of Second if() Statement in Figure 6.2
TABLE 6.2 Detection of Boundary Shift Resulting in Reduced Domain (Closed Inequality)
TABLE 6.3 Detection of Boundary Shift Resulting in Enlarged Domain (Closed Inequality)
TABLE 6.4 Detection of Boundary Tilt (Closed Inequality)
TABLE 6.5 Detection of Closure Error (Closed Inequality)
TABLE 6.6 Detection of Boundary Shift Resulting in Reduced Domain (Open Inequality)
TABLE 6.7 Detection of Boundary Shift Resulting in Enlarged Domain (Open Inequality)
TABLE 6.8 Detection of Boundary Tilt (Open Inequality)
TABLE 6.9 Detection of Closure Error (Open Inequality)

CHAPTER 7: SYSTEM INTEGRATION TESTING

TABLE 7.1 Check-in Request Form
TABLE 7.2 Example Software/Hardware Compatibility Matrix
TABLE 7.3 Framework for SIT Plan
TABLE 7.4 Framework for Entry Criteria to Start System Integration
TABLE 7.5 Framework for System Integration Exit Criteria

CHAPTER 8: SYSTEM TEST CATEGORIES

TABLE 8.1 EMS Functionalities
TABLE 8.2 Regulatory Approval Bodies of Different Countries

CHAPTER 9: FUNCTIONAL TESTING

TABLE 9.1 Number of Special Values of Inputs to FBS Module of Figure 9.1
TABLE 9.2 Input and Output Domains of Functions of P in Figure 9.6
TABLE 9.3 Pairwise Test Cases for System S
TABLE 9.4 L₄ (2³) Orthogonal Array
TABLE 9.5 Commonly Used Orthogonal Arrays
TABLE 9.6 Various Values That Need to Be Tested in Combinations
TABLE 9.7 L₉ (3⁴) Orthogonal Array
TABLE 9.8 L₉ (3⁴) Orthogonal Array after Mapping Factors
TABLE 9.9 Generated Test Cases after Mapping Left-Over Levels
TABLE 9.10 Generated Test Cases to Cover Each Equivalence Class
TABLE 9.11 Decision Table Comprising Set of Conditions and Effects
TABLE 9.12 Pay Calculation Decision Table with Values for Each Rule
TABLE 9.13 Pay Calculation Decision Table after Column Reduction

CHAPTER 10: TEST GENERATION FROM FSM MODELS

TABLE 10.1 PCOs for Testing Telephone PBX
TABLE 10.2 Set of States in FSM of Figure 10.8
TABLE 10.3 Input and Output Sets in FSM of Figure 10.8
TABLE 10.4 Transition Tours Covering All States in Figure 10.8
TABLE 10.5 State Transitions Not Covered by Transition Tours of Table 10.4
TABLE 10.6 Transition Tours Covering All State Transitions in Figure 10.8
TABLE 10.7 UIO Sequences of Minimal Lengths Obtained from Figure 10.14
TABLE 10.8 Examples of State Blocks
TABLE 10.9 Outputs of FSM G₂ in Response to Input Sequence 11 in Different States
TABLE 10.10 Output Sequences Generated by FSM of Figure 10.17 as Response to W₁
TABLE 10.11 Output Sequences Generated by FSM of Figure 10.17 as Response to W₂
TABLE 10.12 Test Sequences for State Transition (D, A, a /x) of FSM in Figure 10.17

CHAPTER 11: SYSTEM TEST DESIGN

TABLE 11.1 Coverage Matrix [A_ij]
TABLE 11.2 Requirement Schema Field Summary
TABLE 11.3 Engineering Change Document Information
TABLE 11.4 Characteristics of Testable Functional Specifications
TABLE 11.5 Mapping of FR QoS Parameters to ATM QoS Parameters
TABLE 11.6 Test Case Schema Summary
TABLE 11.7 Test Suite Schema Summary
TABLE 11.8 Test Result Schema Summary

CHAPTER 12: SYSTEM TEST PLANNING AND AUTOMATION

TABLE 12.1 Outline of System Test Plan
TABLE 12.2 Equipment Needed to be Procured
TABLE 12.3 Entry Criteria for First System Test Cycle
TABLE 12.4 Test Case Failure Counts to Initiate RCA in Test Cycle 1
TABLE 12.5 Test Case Failure Counts to Initiate RCA in Test Cycle 2
TABLE 12.6 Test Effort Estimation for FR–ATM PVC Service Interworking
TABLE 12.7 Form for Computing Unadjusted Function Point
TABLE 12.8 Factors Affecting Development Effort
TABLE 12.9 Empirical Relationship between Function Points and LOC
TABLE 12.10 Guidelines for Manual Test Case Creation Effort
TABLE 12.11 Guidelines for Manual Test Case Execution Effort
TABLE 12.12 Guidelines for Estimation of Effort to Manually Execute Regression Test Cases
TABLE 12.13 Benefits of Automated Testing

CHAPTER 13: SYSTEM TEST EXECUTION

TABLE 13.1 States of Defect Modeled in Figure 13.1
TABLE 13.2 Defect Schema Summary Fields
TABLE 13.3 State Transitions to Five Possible Next States from Open State
TABLE 13.4 Outline of Test Execution Working Document
TABLE 13.5 EST Metric in Week 4 of Bazooka Project
TABLE 13.6 EST Metric in Bazooka Monitored on Weekly Basis
TABLE 13.7 DAR Metric for Stinger Project
TABLE 13.8 Weekly DRR Status for Stinger Test Project
TABLE 13.9 Weekly OD on Priority Basis for Stinger Test Project
TABLE 13.10 Weekly CD Observed by Different Groups for Stinger Test Project
TABLE 13.11 ARD Metric for Bayonet
TABLE 13.12 Sample Test Data of Chainsaw Test Project
TABLE 13.13 Framework for Beta Release Criteria
TABLE 13.14 Structure of Final System Test Report
TABLE 13.15 Scale for Defect Age
TABLE 13.16 Defect Injection versus Discovery on Project Boomerang
TABLE 13.17 Number of Defects Weighted by Defect Age on Project Boomerang
TABLE 13.18 ARD Metric for Test Project
TABLE 13.19 Scale for PhAge

CHAPTER 14: ACCEPTANCE TESTING

TABLE 14.1 Outline of ATP
TABLE 14.2 ACC Document Information
TABLE 14.3 Structure of Acceptance Test Status Report
TABLE 14.4 Structure of Acceptance Test Summary Report

CHAPTER 15: SOFTWARE RELIABILITY

TABLE 15.1 Example of Operational Profile of Library Information System

CHAPTER 17: SOFTWARE QUALITY

TABLE 17.1 McCall’s Quality Factors
TABLE 17.2 Categorization of McCall’s Quality Factors
TABLE 17.3 McCall’s Quality Criteria

CHAPTER 18: MATURITY MODELS

TABLE 18.1 Requirements for Different Maturity Levels
TABLE 18.2 Test Maturity Matrix

List of Illustrations

CHAPTER 1: BASIC CONCEPTS AND PRELIMINARIES

Figure 1.1 Shewhart cycle.
Figure 1.2 Ishikawa diagram.
Figure 1.3 Examples of basic test cases.
Figure 1.4 Example of a test case with a sequence of < input, expected outcome >.
Figure 1.5 Subset of the input domain exercising a subset of the program behavior.
Figure 1.6 Different activities in program testing.
Figure 1.7 Development and testing phases in the V model.
Figure 1.8 Regression testing at different software testing levels. (From ref. 41. © 2005 John Wiley & Sons.)

CHAPTER 2: THEORY OF PROGRAM TESTING

Figure 2.1 Executing a program with a subset of the input domain.
Figure 2.2 Example of inappropriate path selection.
Figure 2.3 Different ways of comparing power of test methods: (a) produces all test cases produced by another method; (b) test sets have common elements.
Figure 2.4 Context of applying test adequacy.

CHAPTER 3: UNIT TESTING

Figure 3.1 Steps in the code review process.
Figure 3.2 Dynamic unit test environment.
Figure 3.3 Test-first process in XP. (From ref. 24. © 2005 IEEE.)
Figure 3.4 Sample pseudocode for performing unit testing.
Figure 3.5 The assertTrue() assertion throws an exception.
Figure 3.6 Example test suite.

CHAPTER 4: CONTROL FLOW TESTING

Figure 4.1 Process of generating test input data for control flow testing.
Figure 4.2 Symbols in a CFG.
Figure 4.3 Function to open three files.
Figure 4.4 High-level CFG representation of openfiles(). The three nodes are numbered 1, 2, and 3.
Figure 4.5 Detailed CFG representation of openfiles(). The numbers 1–21 are the nodes.
Figure 4.6 Function to compute average of selected integers in an array. This program is an adaptation of “Figure 2. A sample program” in ref. 10. (With permission from the Australian Computer Society.)
Figure 4.7 A CFG representation of ReturnAverage(). Numbers 1–13 are the nodes.
Figure 4.8 Dashed arrows represent the branches not covered by statement covering in Table 4.4.
Figure 4.9 Partial CFG with (a) OR operation and (b) AND operation.
Figure 4.10 Example of a path from Figure 4.7.
Figure 4.11 Path predicate for path in Figure 4.10.
Figure 4.12 Method in Java to explain symbolic substitution [11].
Figure 4.13 Path predicate expression for path in Figure 4.10.
Figure 4.14 Another example path from Figure 4.7.
Figure 4.15 Path predicate expression for path in Figure 4.14.
Figure 4.16 Input data satisfying constraints of Figure 4.13.
Figure 4.17 Binary search routine.

CHAPTER 5: DATA FLOW TESTING

Figure 5.1 Sequence of computations showing data flow anomaly.
Figure 5.2 State transition diagram of a program variable. (From ref. 2. © 1979 IEEE.)
Figure 5.3 Definition and uses of variables.
Figure 5.4 Data flow graph of ReturnAverage() example.
Figure 5.5 Relationship among DF (data flow) testing criteria. (From ref. 4. © 1988 IEEE.)
Figure 5.6 Relationship among FDF (feasible data flow) testing criteria. (From ref. 4. © 1988 IEEE.)
Figure 5.7 Limitation of different fault detection techniques.
Figure 5.8 Binary search routine.
Figure 5.9 Modified binary search routine.

CHAPTER 6: DOMAIN TESTING

Figure 6.1 Illustration of the concept of program domains.
Figure 6.2 A function to explain program domains.
Figure 6.3 Control flow graph representation of the function in Figure 6.2.
Figure 6.4 Domains obtained from interpreted predicates in Figure 6.3.
Figure 6.5 Predicates defining the TT domain in Figure 6.4.
Figure 6.6 ON and OFF points.
Figure 6.7 Boundary shift resulting in reduced domain (closed inequality).
Figure 6.8 Boundary shift resulting in enlarged domain (closed inequality).
Figure 6.9 Tilted boundary (closed inequality).
Figure 6.10 Closure error (closed inequality).
Figure 6.11 Boundary shift resulting in reduced domain (open inequality).
Figure 6.12 Boundary shift resulting in enlarged domain (open inequality).
Figure 6.13 Tilted boundary (open inequality).
Figure 6.14 Closure error (open inequality).
Figure 6.15 Equality border.
Figure 6.16 Domains D₁, D₂, and D₃.

CHAPTER 7: SYSTEM INTEGRATION TESTING

Figure 7.1 Module hierarchy with three levels and seven modules.
Figure 7.2 Top-down integration of modules A and B.
Figure 7.3 Top-down integration of modules A, B, and D.
Figure 7.4 Top-down integration of modules A, B, D, and C.
Figure 7.5 Top-down integration of modules A, B, C, D, and E.
Figure 7.6 Top-down integration of modules A, B, C, D, E, and F.
Figure 7.7 Top-down integration of modules A, B, C, D, E, F and G.
Figure 7.8 Bottom-up integration of modules E, F, and G.
Figure 7.9 Bottom-up integration of modules B, C, and D with E, F, and G.
Figure 7.10 Bottom-up integration of module A with all others.
Figure 7.11 Hardware ECO process.
Figure 7.12 Software ECO process.
Figure 7.13 Module hierarchy of software system.

CHAPTER 8: SYSTEM TEST CATEGORIES

Figure 8.1 Types of system tests.
Figure 8.2 Types of basic tests.
Figure 8.3 Types of functionality tests.
Figure 8.4 Types of robustness tests.
Figure 8.5 Typical 1xEV-DO radio access network. (Courtesy of Airvana, Inc.)

CHAPTER 9: FUNCTIONAL TESTING

Figure 9.1 Frequency selection box of Bluetooth specification.
Figure 9.2 Part of form ON479 of T1 general—2001, published by the CCRA.
Figure 9.3 Functionally related variables.
Figure 9.4 Function in context.
Figure 9.5 (a) Obtaining output values from an input vector and (b) obtaining an input vector from an output value in functional testing.
Figure 9.6 Functional testing in general.
Figure 9.7 System S with three input variables.
Figure 9.8 (a) Too many test inputs; (b) one input selected from each subdomain.
Figure 9.9 Gold standard oracle.
Figure 9.10 Parametric oracle.
Figure 9.11 Statistical oracle.

CHAPTER 10: TEST GENERATION FROM FSM MODELS

Figure 10.1 Spectrum of software systems.
Figure 10.2 Data-dominated systems.
Figure 10.3 Control-dominated systems.
Figure 10.4 FSM model of a dual-boot laptop computer.
Figure 10.5 Interactions between system and its environment modeled as FSM.
Figure 10.6 PCOs on a telephone.
Figure 10.7 FSM model of a PBX.
Figure 10.8 FSM model of PBX.
Figure 10.9 Interaction of test sequence with SUT.
Figure 10.10 Derived test case from transition tour.
Figure 10.11 Conceptual model of test case with state verification.
Figure 10.12 Finite-state machine G₁ (From ref. 5. © 1997 IEEE.)
Figure 10.13 1 UIO tree for G₁ in Figure 10.12. (From ref. 5. © 1997 IEEE.)
Figure 10.14 Identification of UIO sequences on UIO tree of Figure 10.13.
Figure 10.15 Finite-state machine G₂.
Figure 10.16 Distinguishing sequence tree for G₂ in Figure 10.15.
Figure 10.17 FSM that does not possess distinguishing sequence. (From ref. 11. © 1994 IEEE.)
Figure 10.18 DS tree for FSM (Figure 10.17).
Figure 10.19 Abstraction of N entity in OSI reference architecture.
Figure 10.20 Abstract local test architecture.
Figure 10.21 Abstract external test architecture.
Figure 10.22 Local architecture.
Figure 10.23 Distributed architecture.
Figure 10.24 Coordinated architecture.
Figure 10.25 Remote architecture.
Figure 10.26 Structure of module in TTCN-3.
Figure 10.27 Definitions of two subtypes.
Figure 10.28 Parameterized template for constructing message to be sent.
Figure 10.29 Parameterized template for constructing message to be received.
Figure 10.30 Testing (a) square-root function (SRF) calculator and (b) port between tester and SRF calculator.
Figure 10.31 Defining port type.
Figure 10.32 Associating port with component.
Figure 10.33 Test case for testing SRF calculator.
Figure 10.34 Executing test case.
Figure 10.35 Comparison of state transitions of FSM and EFSM.
Figure 10.36 Controlled access to a door.
Figure 10.37 SDL/GR door control system.
Figure 10.38 Door control behavior specification.
Figure 10.39 Door control behavior specification.
Figure 10.40 Transition tour from door control system of Figures 10.38 and 10.39.
Figure 10.41 Testing door control system.
Figure 10.42 Output and input behavior obtained from transition tour of Figure 10.40
Figure 10.43 Test behavior obtained by refining if part in Figure 10.42.
Figure 10.44 Test behavior that can receive unexpected events (derived from Figure 10.43).
Figure 10.45 Core behavior of test case for testing door control system (derived from Figure 10.44.)
Figure 10.46 User interface of ATM.
Figure 10.47 Binding of buttons with user options.
Figure 10.48 Binding of buttons with cash amount.
Figure 10.49 FSM G.
Figure 10.50 FSM H.
Figure 10.51 FSM K.
Figure 10.52 Nondeterministic FSM.

CHAPTER 11: SYSTEM TEST DESIGN

Figure 11.1 State transition diagram of requirement.
Figure 11.2 Test suite structure.
Figure 11.3 Service interworking between FR and ATM services.
Figure 11.4 Transformation of FR to ATM cell.
Figure 11.5 FrAtm test suite structure.
Figure 11.6 State transition diagram of a test case.
Figure 11.7 State transition diagram of test case result.

CHAPTER 12: SYSTEM TEST PLANNING AND AUTOMATION

Figure 12.1 Concept of cycle-based test execution strategy.
Figure 12.2 Gantt chart for FR–ATM service interworking test project.
Figure 12.3 Broad criteria of test automation tool evaluation.
Figure 12.4 Test selection guideline for automation.
Figure 12.5 Characteristics of automated test cases.
Figure 12.6 Six major steps in automated test case.
Figure 12.7 Components of automation infrastructure.

CHAPTER 13: SYSTEM TEST EXECUTION

Figure 13.1 State transition diagram representation of life cycle of defect.
Figure 13.2 Projected execution of test cases on weekly basis in cumulative chart form.
Figure 13.3 PAE metric of Bazooka (PE: projected execution; AE: actually executed) project.
Figure 13.4 Pareto diagram for defect distribution shown in Table 13.12.
Figure 13.5 Cause–effect diagram for DCA.

CHAPTER 15: SOFTWARE RELIABILITY

Figure 15.1 Relationship between MTTR, MTTF, and MTBF.
Figure 15.2 Graphical representation of operational profile of library information system.
Figure 15.3 Failure intensity λ as function of cumulative failure µ (λ₀ = 9 failures per unit time, ν₀ = 500 failures, θ = 0.0075).
Figure 15.4 Failure intensity λ as function of execution time τ (λ₀ = 9 failures per unit time, ν₀ = 500 failures, θ = 0.0075).
Figure 15.5 Cumulative failure µ as function of execution time τ (λ₀ = 9 failures per unit time, ν₀ = 500 failures, θ = 0.0075).

CHAPTER 16: TEST TEAM ORGANIZATION

Figure 16.1 Structure of test groups.
Figure 16.2 Structure of software quality assurance group.
Figure 16.3 System test team hierarchy.
Figure 16.4 Six phases of effective recruiting process.
Figure 16.5 System test organization as part of development.

CHAPTER 17: SOFTWARE QUALITY

Figure 17.1 Relation between quality factors and quality criteria [6].
Figure 17.2 ISO 9126 sample quality model refines standard’s features into subcharacteristics. (From ref. 4. Ó 1996 IEEE.)

CHAPTER 18: MATURITY MODELS

Figure 18.1 CMM structure. (From ref. 3. © 2005 John Wiley & Sons.)
Figure 18.2 SW-CMM maturity levels. (From ref. 3. © 2005 John Wiley & Sons.)
Figure 18.3 Five-level structure of TMM. (From ref. 5. © 2003 Springer.)

PREFACE

karmany eva dhikaras te; ma phalesu kadachana; ma karmaphalahetur bhur; ma te sango stv akarmani.

Your right is to work only; but never to the fruits thereof; may you not be motivated by the fruits of actions; nor let your attachment to be towards inaction.

— Bhagavad Gita

We have been witnessing tremendous growth in the software industry over the past 25 years. Software applications have proliferated from the original data processing and scientific computing domains into our daily lives in such a way that we do not realize that some kind of software executes when we do even something ordinary, such as making a phone call, starting a car, turning on a microwave oven, and making a debit card payment. The processes for producing software must meet two broad challenges. First, the processes must produce low-cost software in a short time so that corporations can stay competitive. Second, the processes must produce usable, dependable, and safe software; these attributes are commonly known as quality attributes. Software quality impacts a number of important factors in our daily lives, such as economy, personal and national security, health, and safety.

Twenty-five years ago, testing accounted for about 50% of the total time and more than 50% of the total money expended in a software development project—and, the same is still true today. Those days the software industry was a much smaller one, and academia offered a single, comprehensive course entitled Software Engineering to educate undergraduate students in the nuts and bolts of software development. Although software testing has been a part of the classical software engineering literature for decades, the subject is seldom incorporated into the mainstream undergraduate curriculum. A few universities have started offering an option in software engineering comprising three specialized courses, namely, Requirements Specification, Software Design, and Testing and Quality Assurance. In addition, some universities have introduced full undergraduate and graduate degree programs in software engineering.

Considering the impact of software quality, or the lack thereof, we observe that software testing education has not received its due place. Ideally, research should lead to the development of tools and methodologies to produce low-cost, high-quality software, and students should be educated in the testing fundamentals. In other words, software testing research should not be solely academic in nature but must strive to be practical for industry consumers. However, in practice, there is a large gap between the testing skills needed in the industry and what are taught and researched in the universities.

Our goal is to provide the students and the teachers with a set of well-rounded educational materials covering the fundamental developments in testing theory and common testing practices in the industry. We intend to provide the students with the “big picture” of testing and quality assurance, because software quality concepts are quite broad. There are different kinds of software systems with their own intricate characteristics. We have not tried to specifically address their testing challenges. Instead, we have presented testing theory and practice as broad stepping stones which will enable the students to understand and develop testing practices for more complex systems.

We decided to write this book based on our teaching and industrial experiences in software testing and quality assurance. For the past 15 years, Sagar has been teaching software engineering and software testing on a regular basis, whereas Piyu has been performing hands-on testing and managing test groups for testing routers, switches, wireless data networks, storage networks, and intrusion prevention appliances. Our experiences have helped us in selecting and structuring the contents of this book to make it suitable as a textbook.

Who Should Read This Book?

We have written this book to introduce students and software professionals to the fundamental ideas in testing theory, testing techniques, testing practices, and quality assurance. Undergraduate students in software engineering, computer science, and computer engineering with no prior experience in the software industry will be introduced to the subject matter in a step-by-step manner. Practitioners too will benefit from the structured presentation and comprehensive nature of the materials. Graduate students can use the book as a reference resource. After reading the whole book, the reader will have a thorough understanding of the following topics:

Fundamentals of testing theory and concepts
Practices that support the production of quality software
Software testing techniques
Life-cycle models of requirements, defects, test cases, and test results
Process models for unit, integration, system, and acceptance testing
Building test teams, including recruiting and retaining test engineers
Quality models, capability maturity model, testing maturity model, and test process improvement model

How Should This Book be Read?

The purpose of this book is to teach how to do software testing. We present some essential background material in Chapter 1 and save the enunciation of software quality questions to a later part of the book. It is difficult to intelligently discuss for beginners what software quality means until one has a firm sense of what software testing does. However, practitioners with much testing experience can jump to Chapter 17, entitled “Software Quality,” immediately after Chapter 1.

There are three different ways to read this book depending upon someone’s interest. First, those who are exclusively interested in software testing concepts and want to apply the ideas should read Chapter 1 (“Basic Concepts and Preliminaries”), Chapter 3 (“Unit Testing”), Chapter 7 (“System Integration Testing”), and Chapters 8–14, related to system-level testing. Second, test managers interested in improving the test effectiveness of their teams can read Chapters 1, 3, 7, 8–14, 16 (“Test Team Organization”), 17 (“Software Quality”), and 18 (“Maturity Models”). Third, beginners should read the book from cover to cover.

Notes for Instructors

The book can be used as a text in an introductory course in software testing and quality assurance. One of the authors used the contents of this book in an undergraduate course entitled Software Testing and Quality Assurance for several years at the University of Waterloo. An introductory course in software testing can cover selected sections from most of the chapters except Chapter 16. For a course with more emphasis on testing techniques than on processes, we recommend to choose Chapters 1 (“Basic Concepts and Preliminaries”) to 15 (“Software Reliability”). When used as a supplementary text in a software engineering course, selected portions from the following chapters can help students imbibe the essential concepts in software testing:

Chapter 1: Basic Concepts and Preliminaries
Chapter 3: Unit Testing
Chapter 7: System Integration Testing
Chapter 8: System Test Category
Chapter 14: Acceptance Testing

Supplementary materials for instructors are available at the following Wiley web-site: http:/www.wiley.com/sagar.

Acknowledgments

In preparing this book, we received much support from many people, including the publisher, our family members, and our friends and colleagues. The support has been in many different forms. First, we would like to thank our editors, namely, Anastasia Wasko, Val Moliere, Whitney A. Lesch, Paul Petralia, and Danielle Lacourciere who gave us much professional guidance and patiently answered our various queries. Our friend Dr. Alok Patnaik read the whole draft and made numerous suggestions to improve the presentation quality of the book; we thank him for all his effort and encouragement. The second author, Piyu Tripathy, would like to thank his former colleagues at Nortel Networks, Cisco Systems, and Airvana Inc., and present colleagues at NEC Laboratories America.

Finally, the support of our parents, parents-in-law, and partners deserve a special mention. I, Piyu Tripathy, would like to thank my dear wife Leena, who has taken many household and family duties off my hands to give me time that I needed to write this book. And I, Sagar Naik, would like to thank my loving wife Alaka for her invaluable support and for always being there for me. I would also like to thank my charming daughters, Monisha and Sameeksha, and exciting son, Siddharth, for their understanding while I am writing this book. I am grateful to my elder brother, Gajapati Naik, for all his support. We are very pleased that now we have more time for our families and friends.

Kshirasagar Naik

University of Waterloo

Waterloo

Priyadarshi Tripathy

NEC Laboratories America, Inc.

Princeton

LIST OF FIGURES

1.1 Shewhart cycle

1.2 Ishikawa diagram

1.3 Examples of basic test cases

1.4 Example of a test case with a sequence of < input, expected outcome >

1.5 Subset of the input domain exercising a subset of the program behavior

1.6 Different activities in program testing

1.7 Development and testing phases in the V model

2.1 Executing a program with a subset of the input domain

2.2 Example of inappropriate path selection

2.3 Different ways of comparing power of test methods: (a) produces all test cases produced by another method; (b) test sets have common elements.

2.4 Context of applying test adequacy

3.1 Steps in the code review process

3.2 Dynamic unit test environment

3.4 Sample pseudocode for performing unit testing

3.5 The assertTrue() assertion throws an exception

3.6 Example test suite

4.1 Process of generating test input data for control flow testing

4.2 Symbols in a CFG

4.3 Function to open three files

4.4 High-level CFG representation of openfiles(). The three nodes are numbered 1, 2, and 3.

4.5 Detailed CFG representation of openfiles(). The numbers 1–21 are the nodes

4.6 Function to compute average of selected integers in an array. This program is an adaptation of “Figure 2. A sample program” in ref. 10. (With permission from the Australian Computer Society.)

4.7 A CFG representation of ReturnAverage(). Numbers 1–13 are the nodes.

4.8 Dashed arrows represent the branches not covered by statement covering in Table 4.4

4.9 Partial CFG with (a) OR operation and (b) AND operations

4.10 Example of a path from Figure 4.7

4.11 Path predicate for path in Figure 4.10

4.12 Method in Java to explain symbolic substitution [11]

4.13 Path predicate expression for path in Figure 4.10

4.14 Another example of path from Figure 4.7

4.15 Path predicate expression for path shown in Figure 4.14

4.16 Input data satisfying constraints of Figure 4.13

4.17 Binary search routine

5.1 Sequence of computations showing data flow anomaly

5.3 Definition and uses of variables

5.4 Data flow graph of ReturnAverage() example

5.7 Limitation of different fault detection techniques

5.8 Binary search routine

5.9 Modified binary search routine

6.1 Illustration of the concept of program domains

6.2 A function to explain program domains

6.3 Control flow graph representation of the function in Figure 6.2

6.4 Domains obtained from interpreted predicates in Figure 6.3

6.5 Predicates defining the TT domain in Figure 6.4

6.6 ON and OFF points

6.7 Boundary shift resulting in reduced domain (closed inequality)

6.8 Boundary shift resulting in enlarged domain (closed inequality)

6.9 Tilted boundary (closed inequality)

6.10 Closure error (closed inequality)

6.11 Boundary shift resulting in reduced domain (open inequality)

6.12 Boundary shift resulting in enlarged domain (open inequality)

6.13 Tilted boundary (open inequality)

6.14 Closure error (open inequality)

6.15 Equality border

6.16 Domains D₁, D₂ and D₃

7.1 Module hierarchy with three levels and seven modules

7.2 Top-down integration of modules A and B

7.3 Top-down integration of modules A, B, and D

7.4 Top-down integration of modules A, B, D, and C

7.5 Top-down integration of modules A, B, C, D, and E

7.6 Top-down integration of modules A, B, C, D, E, and F

7.7 Top-down integration of modules A, B, C, D, E, F and G

7.8 Bottom-up integration of modules E, F, and G

7.9 Bottom-up integration of modules B, C, and D with E, F, and G

7.10 Bottom-up integration of module A with all others

7.11 Hardware ECO process

7.12 Software ECO process

7.13 Module hierarchy of software system

8.1 Types of system tests

8.2 Types of basic tests

8.3 Types of functionality tests

8.4 Types of robustness tests

8.5 Typical 1xEV-DO radio access network. (Courtesy of Airvana, Inc.)

9.1 Frequency selection box of Bluetooth specification

9.2 Part of form ON479 of T1 general—2001, published by the CCRA

9.3 Functionally related variables

9.4 Function in context

9.5 (a) Obtaining output values from an input vector and (b) obtaining an input vector from an output value in functional testing

9.6 Functional testing in general

9.7 System S with three input variables

9.8 (a) Too many test inputs; (b) one input selected from each subdomain

9.9 Gold standard oracle

9.10 Parametric oracle

9.11 Statistical oracle

10.1 Spectrum of software systems

10.2 Data-dominated systems

10.3 Control-dominated systems

10.4 FSM model of dual-boot laptop computer

10.5 Interactions between system and its environment modeled as FSM

10.6 PCOs on a telephone

10.7 FSM model of a PBX

10.8 FSM model of PBX

10.9 Interaction of test sequence with SUT

10.10 Derived test case from transition tour

10.11 Conceptual model of test case with state verification

10.14 Identification of UIO sequences on UIO tree of Figure 10.13

10.15 Finite-state machine G₂

10.16 Distinguishing sequence tree for G₂ in Figure 10.15

10.18 DS tree for FSM (Figure 10.17)

10.19 Abstraction of N-entity in OSI reference architecture

10.20 Abstract local test architecture

10.21 Abstract external test architecture

10.22 Local architecture

10.23 Distributed architecture

10.24 Coordinated architecture

10.25 Remote architecture

10.26 Structure of module in TTCN-3

10.27 Definitions of two subtypes

10.28 Parameterized template for constructing message to be sent

10.29 Parameterized template for constructing message to be received

10.30 Testing (a) square-root function (SRF) calculator and (b) port between tester and SRF calculator

10.31 Defining port type

10.32 Associating port with component

10.33 Test case for testing SRF calculator

10.34 Executing test case

10.35 Comparison of state transitions of FSM and EFSM

10.36 Controlled access to a door

10.37 SDL/GR door control system

10.38 Door control behavior specification

10.39 Door control behavior specification

10.40 Transition tour from door control system of Figures 10.38 and 10.39

10.41 Testing door control system

10.42 Output and input behavior obtained from transition tour of Figure 10.40

10.43 Test behavior obtained by refining if part in Figure 10.42

10.44 Test behavior that can receive unexpected events (derived from Figure 10.43)

10.45 Core behavior of test case for testing door control system (derived from Figure 10.44)

10.46 User interface of ATM

10.47 Binding of buttons with user options

10.48 Binding of buttons with cash amount

10.49 FSM G

10.50 FSM H

10.51 FSM K

10.52 Nondeterministic FSM

11.1 State transition diagram of requirement

11.2 Test suite structure

11.3 Service interworking between FR and ATM services

11.4 Transformation of FR to ATM cell

11.5 FrAtm test suite structure

11.6 State transition diagram of a test case

11.7 State transition diagram of test case result

12.1 Concept of cycle-based test execution strategy

12.2 Gantt chart for FR–ATM service interworking test project

12.3 Broad criteria of test automation tool evaluation

12.4 Test selection guideline for automation

12.5 Characteristics of automated test cases

12.6 Six major steps in automated test case

12.7 Components of a automation infrastructure

13.1 State transition diagram representation of life cycle of defect

13.2 Projected execution of test cases on weekly basis in cumulative chart form

13.3 PAE metric of Bazooka (PE: projected execution; AE: actually executed) project

13.4 Pareto diagram for defect distribution shown in Table 13.12

13.5 Cause–effect diagram for DCA

15.1 Relationship between MTTR, MTTF, and MTBF

15.2 Graphical representation of operational profile of library information system

15.3 Failure intensity λ as function of cumulative failure µ (λ₀ = 9 failures per unit time, ν₀ = 500 failures, θ = 0.0075)

15.4 Failure intensity λ as function of execution time τ (λ₀ = 9 failures per unit time, ν₀ = 500 failures, θ = 0.0075)

15.5 Cumulative failure µ as function of execution time τ (λ₀ = 9 failures per unit time, ν₀ =500 failures, θ = 0.0075)

16.1 Structure of test groups

16.2 Structure of software quality assurance group

16.3 System test team hierarchy

16.4 Six phases of effective recruiting process

16.5 System test organization as part of development

17.1 Relation between quality factors and quality criteria [6]

SOFTWARE TESTING AND QUALITY ASSURANCE

Theory and Practice

PREFACE

Who Should Read This Book?

How Should This Book be Read?

Notes for Instructors

Acknowledgments

LIST OF FIGURES