* ITEQS 2021 is held as a virtual workshop *
12 April 2021Porto de Galinhas, Brazil


5th International Workshop on Testing Extra-Functional Properties and Quality Characteristics of Software Systems

Co-located with ICST 2021
Submission Deadline: January 20, 2021 (Extended)


« Click on the titles for more details »

« ITEQS 2021 is held as a virtual workshop. Contact iteqs2021[at]easychair.org for details»

09:15 - 10:15
Keynote: Testing Security and Resilience

Ana Rosa Cavalli

Testing techniques are used to check if a given system implementation satisfies its specification or some predefined properties. In the last years an important research activity has taken place concerning the definition of testing techniques for security. Resilience has also become a crucial issue to guarantee the security and robustness of systems. In this talk, we will present the main features of security testing and resilience, and propose some solutions. We will also present the project H2020 VeriDevOps to illustrate the application of our approach.

* Slides Available *
10:30 - 11:00
Software Defects Rules Discovery

Andreea Vescan, Camelia Serban, Gloria Cerasela Crisan

Developing highly qualitative software systems is important but at the same time difficult to be achieved due to a constant increase in size and complexity of the systems. Early detection of software defects becomes a must, various methods being researched. The Software Defects Rules Discovery (SDRD) approach aims to address the generation of rules for software defects. The model is based on code metrics values and uses the ant colony system algorithm to discover the best solution. Performed experiments considered one theoretical example and two open-source projects with various setups, in total 22 different configurations being investigated. The obtained results indicated that from the set of considered metrics the ones that provide the best rules are: CBO (Coupling Between Objects), LOC (Lines Of Code), and NPM (Number of Private Methods).

11:00 - 11:30
Online GANs for Automatic Performance Testing Best Paper

Ivan Porres, Hergys Rexha, Sebastien Lafond

We present a novel algorithm for automatic performance testing that uses an online variant of the Generative Adversarial Network (GAN) to optimize the test generation process. The objective is to generate a test suite for a given test budget with a high number of tests revealing performance defects. This is achieved using a GAN to generate the tests and predict their outcome. This GAN is trained online, while generating and executing the tests and it does not require a prior training set or model of the system under test. We consider that the presented algorithm serves as a proof of concept and we hope that it can spark a research discussion on the application of GANs to test generation.

11:45 - 12:15
Assuring Fairness of Algorithmic Decision Making

Marc P. Hauer, Rasmus Adler, Katharina Zweig

Assuring fairness of an algorithmic decision making (ADM) system is a challenging task involving different and possibly conflicting views on fairness as expressed by multiple fairness measures. We argue that a combination of the agile development framework Acceptance Test-Driven Development (ATDD) and the concept of Assurance Cases from safety engineering is a pragmatic way to assure fairness levels that are adequate for a predefined application. The approach supports examinations by regulating bodies or related auditing processes by providing a structured argument explaining the achieved level of fairness and its sufficiency for the application.

12:15 - 12:45
Enabling Fast Exploration and Validation of Thermal Dissipation Requirements for Heterogeneous CPU Platforms

Joel Öhrling, Dragos Truscan, Sebastien Lafond

The management of the energy consumption and thermal dissipation of multi-core heterogeneous platforms is becoming increasingly important as it can have direct impact on the platform performance. This paper discusses an approach that enables fast exploration and validation of heterogeneous system on chips (SoCs) platform configurations with respect to their thermal dissipation. Such platforms can be configured to find the optimal trade-off between performance and power consumption. The directly reflects in the head dissipation of the platform, which when increases over a given threshold will actually decrease the performance of the platform. Therefore, it is important to be able to quickly probe and explore different configurations and identify the most suitable one. However, this task is hindered by the large space of possible configurations of such platforms and by the time required to benchmark each configurations. As such, we propose an approach in which we construct a model of the thermal dissipation of a given platform using a system identification methods and then we use this model to explore and validate different configurations. The approach allows us to decrease the exploration time with several orders of magnitude. We exemplify the approach on an Odroid-XU4 board featuring an Exynos 5422 SoC.

Call For Papers

The workshop does not accept papers that focus purely on functional testing!

Download: Flyer as PDF ----- Call-for-Papers as text

Towards the goal of ITEQS, the topics of interest for the workshop include, but are not limited to, the following:

  • Model-based testing of EFPs; e.g., choice of modeling languages to capture EFPs and their role on testability, model-based test case generation, etc.
  • Testing of AI and Machine Learning based systems with respect to quality attributes; e.g., such as safety, security, and ethical concerns
  • Performance, Robustness, and Security Testing
  • Testing quality in software product lines and variant-intensive system
  • Testing quality characteristics of distributed, mobile, and cloud applications
  • Mutation-based testing for EFPs; e.g., application of mutation techniques for testing of EFPs particularly introduction of EFP-specific mutation operators
  • Search-based and AI techniques for testing EFPs
  • Testability, observability, and the role of the platform; e.g., how choosing an operating system can impact testability of EFPs, for instance, a real-time operating system, introducing testability mechanisms into a platform, designing middlewares for testing of EFPs
  • Empirical studies and experience reports; e.g., on the importance of testing EFPs, evaluation of testing methods, case-study and reports on project failures due to EFPs, comparison of methods and techniques
  • Quality assurance, standards, and their impact on testing EFPs
  • Requirements and testing EFPs; e.g., identification and generation of test oracles for EFPs from requirements, requirements for testability, traceability
  • Coverage criteria in testing EFPs
  • Processes and their role in testing EFPs; e.g., agile and TDD
  • Fault localization for EFPs and debugging
  • Formal methods, model-checking, and reasoning about EFPs
  • Parallelism, Concurrency, and Testing of multicore applications
  • Testing real-time, embedded, and cyber-physical systems, and their challenges

As the presence and role of computer systems in our daily life increases, we rely more and more on the services that are provided by software. On one hand, more tasks and functions are delegated to software systems (e.g., in the automotive domain), and on the other hand, the expectations and demands on the variety of services provided by these systems are dramatically growing (e.g., in mobile phones). In this context, the success of a software product may not only be dependent on logical correctness of its functions, but also on their quality characteristics and how they perform. Such system characteristics, which are referred to and captured as Extra-Functional Properties (EFPs), or Non-Functional Properties, have determinant importance particularly in resource constrained systems. For instance, in real-time embedded domain there can be limitations on available memory, CPU and processing capability, power consumption, and so on, that need to be considered along with timing requirements of an application. Considering the rapid development towards increased integration of software with the social and physical world that we see today, quality aspects become more important in an increasing number of the systems and devices we use and depend on. These systems therefore, need to be tested with a special attention to EFPs such as safety, security, performance and robustness.

Testing a system with respect to its EFPs, however, poses specific challenges and traditional functional testing methods and approaches may not simply be applicable. Examples of such challenges are: fault localization, the need to have appropriate techniques for different types of EFPs, the role and impact of the environment in testing EFPs, observability and testability issues, coverage and test-stop criteria, modeling EFPs and generating meaningful test cases, mutation operators for EFPs, etc.

ITEQS provides a well-focused forum with the goal of bringing together researchers and practitioners to share ideas, identify challenges, propose solutions and techniques, and in general expand the state of the art in testing EFPs and quality characteristics of software systems and services.

Check the topics of interest for more details.

The workshop promotes two types of contributions in IEEE double-column format:

  1. 1) Position papers and also solid Work-in-Progress papers consisting of 4 pages. WiP papers should provide concrete (but not necessarily complete) solutions and results as well as detailed plans for future extensions. Position papers should provide a clear position about research in testing EFPs arguing for a particular research direction in this area, discussing open problems and challenges, emerging topics and applications, and so on.
  2. 2) Full research papers consisting of 6-10 pages.

Each submitted paper will receive at least three reviews. To be faithful to the scope and goals of the workshop, papers that focus only on functional aspects and fail to address any EFPs and quality attributes will not be accepted.

Papers must conform to the IEEE double-column format (http://www.ieee.org/conferences_events/conferences/publishing/templates.html). Authors can submit their papers through easychair at: https://easychair.org/conferences/?conf=iteqs2021. Accepted and presented papers will be included in the IEEE CPS Proceedings.

Important Dates

Submission Deadline January 20, 2021 (Extended)
Notification of Acceptance February 09, 2021
Camera-ready Deadline February 20, 2021
Workshop Date April 12, 2021

Organizing Committee

Contact us: iteqs2021 [at] easychair [dot] org or mehrdad.saadatmand [at] ri [dot] se

Mehrdad Saadatmand

General chair

RISE SICS Västerås, Sweden

Dragos Truscan

Program chair

Åbo Akademi University, Finland

Eduard Enoiu

Program chair

Mälardalen University, Sweden

Program Committee

Antonia Bertolino

CNR (National Research Council), Italy

Brian Nielsen

Aalborg University, Denmark

Yvan Labiche

Carleton University, Canada

Ali Sedaghatbaf

RISE Research Institutes of Sweden, Sweden

Franz Wotawa

Graz University of Technology, Austria

Björn Lisper

Mälardalen University, Sweden

Zohaib Iqbal

National University of Computer and Emerging Sciences, Pakistan

Fabrizio Pastore

University of Luxembourg, Luxembourg

Bestoun S. Ahmed

Karlstad University, Sweden

Shaukat Ali

Simula Research Laboratory, Norway

Mahsa Varshosaz

IT University of Copenhagen, Denmark

Bernhard K. Aichernig

Graz University of Technology, Austria

General sponsors

organizations that support us


  • Porto de Galinhas, Brazil Virtual