13 March 2017 Tokyo, Japan


1st International Workshop on Testing Extra-Functional Properties and Quality Characteristics of Software Systems

Co-located with ICST 2017


« Click on the titles for more details »

9:05 - 10:00
Scalable Software Testing and Verification of Non-Functional Properties through Heuristic Search and Optimization (Keynote 1)

Prof. Lionel Briand University of Luxembourg

Testing and verification problems in the software industry come in many different forms, due to significant differences across domains and contexts. But one common challenge is scalability, the capacity to test and verify increasingly large, complex systems. Another concern relates to practicality. Can the inputs required by a given technique be realistically provided by engineers? This talk reports on 10 years of research tackling verification and testing as a search and optimization problem, often but not always relying on abstractions and models of the system under test. Our observation is that most of the problems we faced could be re-expressed so as to make use of appropriate search and optimization techniques to automate a specific testing or verification strategy.

* Slides Available *
10:00 - 10:30
A Process for Sound Conformance Testing of Cyber-Physical Systems

Hugo Araujo, Gustavo Carvalho, Augusto Sampaio, Mohammad Reza Mousavi, Masoumeh Taromirad

We present a process for sound conformance testing of cyber-physical systems. The process starts with a hybrid model of cyber-physical systems in which the correct behavior of the system (at its interface level) is specified. Such a model captures both discrete behavior and evolution of continuous dynamics of the system in time. Since conformance testing inherently involves comparing continuous dynamics, the key parameters of the process are (1) error margin (in the comparison of continuous signals), and (2) the conformance bounds defining when two signals are sufficiently close to each other. The final parameter of this process is (3) finding (and adjusting) the sampling rate of the dynamic behavior. In the specified process, we provide different alternatives for fixing the error margins of the conformance testing if the sampling rate is determined, establishing the sampling rate if the error margin is fixed and finding conformance bounds once the sampling rate and the error margins are fixed.

11:00 - 11:30
Testing Cache Side-channel Leakage Best Paper

Tiyash Basu, Sudipta Chattopadhyay

Cache timing attacks retrieve secret information (e.g. a secret key) about a program by analyzing the cache behaviour in program executions. It is, therefore, crucial to understand whether a program is vulnerable to cache timing attacks. But how can we test a program to discover its vulnerability against cache timing attacks? In this paper, we propose, design and evaluate a test generation methodology that systematically discovers the cache side-channel leakage of arbitrary software binaries. At the core of our test generation is a method that systematically explores the program input space and it adapts based on the observed cache performance in the executed tests. We have implemented our test generator and evaluated it with several open-source subject programs, including programs from OpenSSL and Linux GDK libraries. Our evaluation effectively reveals cache side-channel leakage in such real-world programs. We also empirically show that our test generator is more effective in revealing cache side-channel leakage than traditional fuzz testing tools Radamsa and AFL.

11:30 - 12:00
Simulation-Based Safety Testing Brake-By-Wire

Nils Mullner, Saifullah Khan, Md. Habibur Rahman, Wasif Afzal, Mehrdad Saadatmand

Mechanical systems in cars are being replaced by electronic equivalents. To be authorized for the road, validation is required that the replacements are at least as good as the old systems. For braking systems, this goodness translates to safety in terms of maintaining timing constraints. This paper presents work-in-progress in the light of discusses the state-of-the-art. The goal is to motivate the connection of a Simulink model of a brake-by-wire system in a traffic simulator for conducting the required validation. The validation focuses on non-functional safety impacted by timing and environmental influences. This can be exploited for instance to justify more expensive real world experimentation or to determine the boundaries to which sufficient safety can be guaranteed.

12:00 - 12:30
Targeted Mutation: Efficient Mutation Analysis for Testing Non-Functional Properties

Björn Lisper, Birgitta Lindström, Pasqualina Potena, Mehrdad Saadatmand, Markus Bohlin

Mutation analysis has proven to be a strong technique for software testing. Unfortunately, it is also computationally expensive and researchers have therefore proposed several different approaches to reduce the effort. None of these reduction techniques however, focuses on non-functional properties. Given that our goal is to create a strong test suite for testing a certain non-functional property, which mutants should be used? In this paper, we introduce the concept of targeted mutation, which focuses mutation effort to those parts of the code where a change can make a difference with respect to the targeted non-functional property. We show how targeted mutation can be applied to derive efficient test suites for estimating the Worst-Case Execution Time (WCET). We use program slicing to direct the mutations to the parts of the code that are likely to have the strongest influence on execution time. Finally, we outline an experimental procedure for how to evaluate the technique.

14:00 - 15:00
Performance-driven software model refactoring (Keynote 2)

Prof. Vittorio Cortellessa University of L'Aquila

Refactoring is a disciplined technique for supporting the extremely volatile software life cycle, by providing better ways to reduce and manage the growing complexity of software systems while improving developer's productivity. Developers can use it to clean up their software, to improve readability, understandability and quality. The focus of this talk is on refactoring of software models driven by changes triggered from performance issues. The problem of capturing performance issues is critical in the software design, mostly because the results of performance analysis (i.e. mean values, variances, and probability distributions) are difficult to be interpreted for providing feedback to software modellers. Support to the interpretation of performance analysis results that helps to fill the gap between numbers and design alternatives is still lacking. The aim of this talk is to present the work that has been done in the last few years on filling such gap. The work is centered on software performance antipatterns, that are recurring solutions to common mistakes (i.e. bad practices) affecting performance. Such antipatterns can play a key role in the refactoring of software models, since they can be used in the investigation of performance problems as well as in the identification of appropriate refactoring actions.

15:00 - 15:30
Automatic Test Generation for Energy Consumption of Embedded Systems Modeled in EAST-ADL

Raluca Marinescu, Eduard Paul Enoiu, Cristina Seceleanu, Daniel Sundmark

Testing using architectural design models is intended to determine if the realized system meets its specifications, and works as a whole in terms of computational components and interactions among such components. The growing complexity of embedded systems makes testing of extra-functional requirements, like energy usage of components and system, very necessary in order to obtain valid implementations. To achieve this, new techniques that are able to support testing extra-functional properties at high levels of abstraction such as the architectural level are needed. Some architectural languages such as EAST-ADL let designers to annotate the models with resource consumption. However, it is not very well studied how such annotations can be used to generate performance tests, such as energy consumption tests, in an efficient and effective manner. In this paper, we show how architectural models described in EAST-ADL can also be used for testing the energy consumption of embedded systems, after transforming them into networks of formal models called priced timed automata. Assuming an EAST-ADL model annotated with energy usage information, we show how to automatically generate energy-aware test cases based on statistical model-checking of the resulting network of priced timed automata. We automate the generation of executable test cases with UPPAAL SMC using a test strategy based on several random simulation runs of the system. By seeding the original formal model with a set of energy-consumption related faults, we are able to carry out fault detection analysis. We apply this technique on a Brake-by-Wire system from the automotive domain, and evaluate it in terms of efficiency and model fault detection.

16:00 - 16:30
Runtime Verification for Detecting Suspension Bugs in Multicore and Parallel Software

Sara Abbaspour Asadollah, Daniel Sundmark, Hans Hansson

Multicore hardware development increases the popularity of parallel and multicore software while testing and debugging these software becoming more difficult, frustrating and costly. Among all types of software bugs, concurrency bugs are also important and troublesome. This type of bugs is increasingly becoming an issue, particularly due to the growing prevalence of multicore hardware. Suspension-based-locking bugs are one type of concurrency bugs. This paper proposes a model based on runtime verification and reflection technique in the context of multicore and parallel software to monitor and detect suspension-based-locking bugs. This model is not only able to detect faults, but diagnose and even repair them. The model is composed of four layers: Logging, Monitoring, Suspension Bug Diagnosis and Mitigation. The logging layer will observe the events and save them into a file system. The monitoring layer will detect the presents of bugs in the software. The suspension bug diagnosis will identify the Suspension bugs by comparing the captured data with the suspension bug properties. Finally, the mitigation layer will reconfigure the software to mitigate the suspension bugs. A functional architecture of runtime verification tool is also proposed in this paper. This architecture is based on the proposed model and is comprised of different modules. Runtime verification and concurrency bugs detection can be an important step towards robust, self-organizing and self-healing tools for testing and debugging the multicore and parallel software.

16:30 - 17:00
Generating Controllably Invalid and Atypical Inputs for Robustness Testing

Simon Poulding, Robert Feldt

One form of robustness in a software system is its ability to handle, in an appropriate manner, inputs that are unexpected compared to those it would experience in normal operation. In this paper we investigate a generic approach to generating such unexpected test inputs by extending a framework that we have previously developed for the automated creation of complex and high-structured test data. The approach is applied to the generation of valid inputs that are atypical as well as inputs that are invalid. We demonstrate that our approach enables the parameterisation of the `degree' to which the test data is invalid or atypical, and show empirically that this parameterisation can alter the extent to which the robustness of a software system is exercised during testing.

2:20 PM - 4:00 PM
Opening a business anywhere

Michael Lambert Harvard M.D.

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Michael Lambert Harvard M.D.

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Michael Lambert Harvard M.D.

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Call For Papers

The workshop does not accept papers that focus purely on functional testing!

Download: Flyer as PDF ----- Call-for-Papers as text

As the presence and role of computer systems in our daily life increases, we rely more and more on the services that are provided by software. On one hand, more tasks and functions are delegated to software systems (e.g., in the automotive domain), and on the other hand, the expectations and demands on the variety of services provided by these systems are dramatically growing (e.g., in mobile phones). In this context, the success of a software product may not only be dependent on logical correctness of its functions, but also on their quality characteristics and how they perform. Such system characteristics, which are referred to and captured as Extra-Functional Properties (EFPs), or Non-Functional Properties, have determinant importance particularly in resource constrained systems. For instance, in real-time embedded domain there can be limitations on available memory, CPU and processing capability, power consumption, and so on, that need to be considered along with timing requirements of an application. Considering the rapid development towards increased integration of software with the social and physical world that we see today, quality aspects become more important in an increasing number of the systems and devices we use and depend on. These systems therefore, need to be tested with a special attention to EFPs such as safety, security, performance and robustness.

Testing a system with respect to its EFPs, however, poses specific challenges and traditional functional testing methods and approaches may not simply be applicable. Examples of such challenges are: fault localization, the need to have appropriate techniques for different types of EFPs, the role and impact of the environment in testing EFPs, observability and testability issues, coverage and test-stop criteria, modeling EFPs and generating meaningful test cases, mutation operators for EFPs, etc.

ITEQS provides a well-focused forum with the goal of bringing together researchers and practitioners to share ideas, identify challenges, propose solutions and techniques, and in general expand the state of the art in testing EFPs and quality characteristics of software systems and services.

Check the topics of interest for more details.

Towards the goal of the ITEQS, the topics of interest for the workshop include, but are not limited to, the following:

  • Model-based testing of EFPs; e.g., choice of modeling languages to capture EFPs and their role on testability, model-based test case generation, etc.
  • Mutation-based testing for EFPs; e.g., application of mutation techniques for testing of EFPs particularly introduction of EFP-specific mutation operators
  • Search-based testing techniques for EFPs
  • Testability, observability, and the role of the platform; e.g., how choosing an operating system can impact testability of EFPs, for instance, a real-time operating system, introducing testability mechanisms into a platform, designing middlewares for testing of EFPs
  • Empirical studies and experience reports; e.g., on the importance of testing EFPs, evaluation of testing methods, case-study and reports on project failures due to EFPs, comparison of methods and techniques
  • Quality assurance, standards, and their impact on testing EFPs
  • Requirements and testing EFPs; e.g., identification and generation of test oracles for EFPs from requirements, requirements for testability, traceability
  • Coverage criteria in testing EFPs
  • Processes and their role in testing EFPs; e.g., agile and TDD
  • Fault localization for EFPs and debugging
  • Formal methods, model-checking, and reasoning about EFPs
  • Parallelism, Concurrency, and Testing of multicore applications
  • Performance, Robustness, and Security Testing
  • Testing real-time, embedded, and cyber-physical systems, and their challenges
  • Testing quality characteristics of distributed, mobile, and cloud applications

The workshop promotes two types of contributions in IEEE double-column format:

  1. 1) position papers and also solid work-in-progress papers consisting of 4 pages. WiP papers should provide concrete (but not necessarily complete) solutions and results as well as detailed plans for future extensions
  2. 2) full research papers consisting of 6-10 pages.

Each submitted paper will receive at least three reviews. To be faithful to the scope and goals of the workshop, papers that focus only on functional aspects and fail to address any EFPs and quality attributes will not be accepted.

Papers must conform to the IEEE double-column format (http://www.ieee.org/conferences_events/conferences/publishing/templates.html). Authors can submit their papers through easychair at: https://easychair.org/conferences/?conf=iteqs2017.

Important Dates

Submission Deadline Dec 12 (extended) Dec 1 , 2016
Notification of Acceptance Jan 14, 2017
Camera-ready Deadline Jan 30, 2017
Workshop Date March 13, 2017

Organizing Committee

Contact us: iteqs2017 [at] easychair [dot] org or mehrdad [at] sics [dot] se

Mehrdad Saadatmand

General Chair

SICS Swedish ICT, Västerås, Sweden

Birgitta Lindström

Program co-chair

University of Skövde, Sweden

Markus Bohlin

Program co-chair

SICS Swedish ICT, Västerås, Sweden

Program Committee

Antonia Bertolino

CNR (National Research Council), Italy

Mark Harman

University College London, UK

Vahid Garousi

Hacettepe University, Turkey

Jeff Offutt

George Mason University, USA

James H. Hill

Indiana University-Purdue University Indianapolis, USA

Vittorio Cortellessa

University of L'Aquila, Italy

Mohammad Mousavi

Halmstad University, Sweden

Brian Nielsen

Aalborg University, Denmark

Robert Feldt

Blekinge Institute of Technology, Sweden

Shiva Nejati

University of Luxembourg, Luxembourg

Björn Lisper

Mälardalen University, Sweden

Hadi Hemmati

University of Manitoba, Canada

Wasif Afzal

Mälardalen University, Sweden

Tsong Yueh Chen

Swinburne University of Technology, Australia

Pasqualina Potena

SICS Swedish ICT, Sweden

Bestoun S. Ahmed

Dalle Molle Institute for Artificial Intelligence Research (IDSIA) USI-SUPSI, Switzerland

General sponsors

organizations that support us


  • Waseda University - Nishiwaseda Campus, Tokyo