Software Construction & Testing

Construction

• Software construction fundamentals
• Management
• Practical Considerations
• Technologies
• Tools

Software Construction Fundamentals

• Minimizing complexity: emphasizing code creation that is simple and readable rather than clever
• Anticipating change: enhance a software product without disrupting the underlying structure
• Constructing for verification: faults can be readily found by the software engineers writing the software as well as by the testers and users
• Reuse: using existing assets in solving different problems
• Standards in construction: helps achieve a project's objectives for efficiency, quality, and cost.

Managing Construction

• Construction in Life Cycle Models
• Construction planning: affects the project team’s ability to reduce complexity, anticipate change, and construct for verification
• Construction measurement: code developed, code modified, code reused, code destroyed, code complexity, code inspection statistics, fault-fix and fault-find rates, effort, and scheduling.

Practical Considerations

• Design
• Languages
• Coding practices
• Testing: unit and integration testing
• Reuse: variability analysis and design
• Quality: testing, inspections, technical review, static analysis
• Integration: individually constructed routines, classes, components, and subsystems into a single system

Technologies & Tools

Technologies:

• API
• Parameterization and Generics
• Error Handling, Exception Handling, and Fault Tolerance
• Concurrency
• Distributed & Heterogeneous Systems
• Platform
• Test-Driven Development

Tools: - IDEs, Builders, Unit Testing Tools, Profiling, Performance Analyzers, etc.

Metrics

• Management
• Quality
• Requirements
• Design
• Maintenance and Operation

Management Metrics

• Size: Lines of Code (LOC*), Thousand Lines of Code (KLOC)
• Size: Function points, Feature Points
• Individual Effort: hours
• Task Completion Time: hours, days, weeks
• Project Effort: person-hours
• Project Duration: months
• Schedule: earned value
• Risk Projection: risk description, risk likelihood, risk impact

Software Quality Metrics

• Defect Density - Defects/KLOC (e.g., for system test)
• Defect Removal Rate - defect removed/hour (for review and test)
• Test Coverage
• Failure Rate

Software Requirements Metrics

• Change requests received, open, closed
• Change request frequency
• Effort required to implement a requirement change
• Status of requirements traceability
• User stories in the backlog

Software Design Metrics

• Cyclomatic Complexity
• Weighted Methods per Class
• Cohesion - Lack of Cohesion of Methods
• Coupling - Coupling Between Object Classes
• Inheritance - Depth of Inheritance Tree, Number of Children

Software Maintenance and Operation

• Mean Time Between Changes (MTBC)
• Mean Time To Change (MTTC)
• System Reliability
• System Availability
• Total Hours of Downtime

Testing

There are currently five standards:

• ISO/IEC 29119-1: Concepts & Definitions (published September 2013)
• ISO/IEC 29119-2: Test Processes (published September 2013)
• ISO/IEC 29119-3: Test Documentation (published September 2013)
• ISO/IEC 29119-4: Test Techniques
• ISO/IEC 29119-5: Keyword Driven Testing

The ISO/IEC/IEEE 29119 standards replace a number of existing software testing standards:

• IEEE 829 Test Documentation
• IEEE 1008 Unit Testing
• BS 7925-1 Vocabulary of Terms in Software Testing
• BS 7925-2 Software Component Testing Standard

Motivation for ISO/IEC/IEEE 29119

• Demand for existing 'standards'
• Conflicts in current definitions and processes
• Gaps in the current standards provision
• A Baseline for the Testing Discipline
• Current industry practice is lacking
• Buyers unclear on what is 'good test practice'

ISO/IEC/IEEE 29119 - Structure

structure

ISO/IEC/IEEE 29119-1: Concepts & Definitions

This part of ISO/IEC/IEEE 29119 specifies definitions and concepts in software testing.

• informative and no conformance
• contain three standards where conformance may be claimed:
  • test processes (ISO/IEC/IEEE 29119-2)
  • test documentation (ISO/IEC/IEEE 29119-3)
  • test techniques (ISO/IEC/IEEE 29119-4)

Software Testing

Software testing should focus on providing information about a software product and finding as many defects as possible, as early as possible in the development process, under given constraints of cost and schedule.

• Testing is a process.
• Testing should be planned, monitored and controlled.
• Testing addresses some of the verification and validation activities.

Software Testing (cont.)

Software testing is necessary because:

• Information on the quality characteristics of the test item(s) is required by decision makers
• The test item(s) being tested does not always do what it is expected to do
• The test item(s) being tested needs to be verified
• The test item(s) being tested needs to be validated
• Evaluation of the test item(s) needs to be conducted throughout the software and system development slife cycle.

Exhaustive Testing

Due to the complexity of systems and software, it is not possible to exhaustively test every single aspect of any given test item. - Recognize that exhaustive testing is not possible - Target to best fulfil the test objectives for a test item. - Risk-based testing is an approach that uses risk to direct test effort.

Testing as a Heuristic

A heuristic is an experience-based (trial and error) method that can be used as an aid to problem solving and design.

• Heuristics can be used to solve problems, but are fallible.

Much of systems and software testing is based on heuristics.

• E.g., creating models of the system to be tested
• May fail to fully model the system and so defects in the system may not be found even though the testing appears to be complete.

So, mitigate the risk of an ineffective test strategy by employing multiple test strategies.

Testing Processes

• Organizational Test Process
• Test Management Process
• Dynamic Test Process

ISO/IEC 29119-2: Test Processes

test-process

Instantiating Testing Processes

Test Management Processes

test-mng-process

Dynamic Test Processes

dyn-test-process

Test Planning

test-planning

Test Monitoring & Control

test-monitor

ISO/IEC/IEEE 29119-3: Test Documentation

• Organizational Test Process Documentation:
  • Test Policy
  • Organizational Test Strategy
• Test Management Process Documentation:
  • Test Plan (including a Test Strategy)
  • Test Status Report
  • Test Completion Report
• Dynamic Test Process Documentation:
  • Test Specification: Design, Case, Procedure
  • Test Data Requirements & Readiness Report
  • Test Environment Requirements &Readiness Report
  • Actual Results
  • Test Result, Execution Log, Incident Report

Hierarchy of Test Documentation

test-docs

Test Policy

• Test Policy defines the objectives and principles of software testing to be applied within the organization.
• It defines what should be accomplished by testing, but does not detail how testing is performed.
• It provides a framework for establishing, reviewing, and continually improving the organization’s Test Policy.

It includes:

• Objectives of testing
• Test process & organization structure
• Tester training
• Testing ethics
• Standards & relevant policies
• Measuring the value of testing
• Test process improvement

Example structure of the Organizational Test Strategy

test-strategy

Test Plan

• provides a test planning and test management document (single or multiple)
• describes the decisions made during the initial planning
• evolves as re-planning is performed as part of the control activity.

Test Plan (cont.)

Includes:

• Document specific information
• Introduction: information about the context and structure of the document
• Context of the testing
• Testing communication
• Risk register: identifies the risks considered by the testing covered by this plan
• Test strategy: describes the approach to testing for the specified test project or test sub-process
• Testing activities and estimates: identifies all necessary testing activities based on the test process to be used
• Test Status Report: the status of the testing that is performed in a specific reporting period
• Test Completion Report: summary of the testing that was performed

ISO/IEC/IEEE 29119-4: Test Techniques

• Specification-based techniques
• Structure-based techniques
• Experience-based techniques
• Other techniques
  • Model-based testing
  • Data-driven testing
  • Keyword-driven Testing

test-techs