9 April 2018 Västerås, Sweden


2nd International Workshop on Testing Extra-Functional Properties and Quality Characteristics of Software Systems

Co-located with ICST 2018
Aros Congress Center, Room 107


« Click on the titles for more details »

9:10 - 10:00
Some Experiences from Testing and Verifying Complex Concurrent and Distributed Systems (Keynote 1)

Prof. Konstantinos SagonasUppsala University

This talk will report on some recent experiences in applying two different stateless model checking (SMC) tools in two different case studies. The first of them applied Nidhugg, an SMC tool for C/Pthread programs, to the code of Tree RCU, the Hierarchical Read-Copy-Update synchronization mechanism for mutual exclusion used in the Linux kernel, a low-level and quite complex concurrent program. The second case study applied Concuerror, an SMC tool for Erlang programs, to test and verify, during their design phase by engineers at VMWare, chain repair methods for CORFU, a distributed shared log which aims to be scalable and reliable in the presence of failures and asynchrony. Besides the results from these two case studies, we will try to present some experiences and lessons learned about engaging in testing projects of that size and complexity.

* Slides Available *
10:00 - 10:30
A Testability Analysis Framework for Non-Functional Properties Best Paper

Michael Felderer, Bogdan Marculescu, Francisco Gomes, Robert Feldt, Richard Torkar

This position paper presents background, the basic steps and an example for a testability analysis framework for non-functional properties.

11:00 - 11:30
Model-Based Mutation Testing of Real-Time Systems via Model Checking

Florian Lorber, Kim Guldstrand Larsen, Brian Nielsen

Model-based mutation testing is a fault-based technique for generating effective test suites. Given a formal model of a system and a set of slightly altered variants of the model (mutants), the approach generates a test suite that is able to distinguish the original from all (non-equivalent) mutants. Generally, this is done via conformance checks between the mutants and the correct specification where witness traces of non-conformance are transformed into test cases. While this procedure generates high-quality test suites, the conformance checks are computationally expensive, resulting in a high test-case generation time. To ensure that the specification is correct, many safety-critical systems are additionally specified via properties, e.g., timed computation tree logic (TCTL) properties. These can be used to verify the specification model via model checking. After this check, the properties are often not used any further. In this paper, we propose to reuse the properties in a combination of model-based mutation testing and the model-checking of properties for real-time systems. That is, we generate a set of mutated models, and check whether they still satisfy the specified properties. In case of a violation, we receive a trace to this violation, which can be transformed into a test case in the same way as in the traditional approach. The advantages of the approach are manifold: the checking of individual small properties is often more efficient than a full conformance check; the resulting test suite is smaller in size and mainly focuses on the safety-critical parts that were specified by the properties; the distribution of the generated test cases gives a very good indication on the quality of the properties; the technique can also be applied to certain kinds of non-functional properties. We apply the technique to an industrial case study of an automated gear controller. It is modeled as a network of timed automata and specified via 46 TCTL properties. We use the tool UPPAAL and its model-checking capabilities to demonstrate the approach.

11:30 - 12:00
Identifying useful mutants to test time properties

Birgitta Lindström, Jeff Offutt, Loreto Gonzalez-Hernandez, Sten F. Andler

Real-time systems have to be verified and tested for timely behavior as well as functional behavior. Thus, time is an extra dimension that adds to the complexity of software testing. A timed automata model with a model-checker can be used to generate timed test traces. To properly test the timely behavior, the set of test traces should challenge the different time constraints in the model. This paper describes and adapts mutation operators that target such time constraints in timed automata models. Time mutation operators apply a delta to the time constraints to help testers design tests that exceed the time constraints. We suggest that the size of this delta determine how easy the mutant is to kill and that the optimal delta varies by the the program, mutation operator, and the individual mutant. To avoid trivial and equivalent time mutants, the delta should be set individually for each mutant. We discuss mutant subsumption and define the problem of finding dominator mutants in this new domain. In this position paper, we outline an iterative tuning process where a statistical model-checker, UPPAAL SMC, is used to: (i) create a tuned set of dominator time mutants, and (ii) generate test traces that kill the mutants.

12:00 - 12:30
Learning-Based Self-Adaptive Assurance of Timing Properties in a Real-Time Embedded System

Mahshid Helali Moghadam, Mehrdad Saadatmand, Markus Borg, Markus Bohlin, Björn Lisper

Providing an adaptive runtime monitoring and assurance mechanism to meet the performance requirements of a real-time system without the need for a precise model could be a challenge. Adaptive performance assurance based on monitoring the status of timing properties can bring a flexible performance control and more robustness to the underlying platform. At the same time, the results or the achieved policy of this quality assurance procedure could be used as feedback to update the initial model, and consequently for producing proper test cases. Reinforcement-learning has been considered as a promising adaptive technique for tuning and assuring the satisfaction of performance properties of software-intensive systems in recent years. In this work-in-progress paper, we propose an adaptive runtime timing assurance procedure based on reinforcement learning to satisfy the performance requirements in terms of response time. The timing control problem is formulated as a Markov Decision Process and details of all parts of the learning-based timing assurance technique are described.

14:00 - 15:00
Towards Compositional Testing of Real-Time Systems (Keynote 2)

Prof. Brian Nielsen Aalborg University

In this talk, we discuss how we may achieve a methodology for compositional testing of real-time systems. We revisit the notion of compositional testing in the setting of real-time systems. In particular, we introduce crucial notions of real-time conformance testing and compositional verification of real-time systems. We illustrate these notions on a small example, and show how the tools Uppaal Tron (online real-time testing), Uppaal Ecdar (refinement-checking) and Uppaal SMC (statistical model-checking for stochastic hybrid automata) provide strong support for a compositional testing methodology.

* Slides Available *
15:00 - 15:30
Scan Code Injection Flaws in HTML5-based Mobile Applications

Tuong Lau

HTML5-based mobile apps are becoming popular in the development of a cross-platform mobile. They are also built using web technologies, including HTML5, CSS, and JavaScript, so it may face with code injection attacks like web apps. However, code injection attacks are exploited in web apps and in mobile web apps are distinguished. The code injection attacks in web apps are often exploited by attackers throughout the cross-site scripting. In HTML5-based mobile apps, attackers can deploy attacks by various code injection channels such as inter-apps, inter-components, inter-devices communication, and local device resources such as WiFi, SMS, Contact. The plugin APIs are implemented for the code injection channels are defined as the sensitive plugin APIs. The previous approaches aimed at modeling known sensitive plugin APIs, and applying the data flow analysis to detect sensitive information flows from such modeled sensitive plugin APIs to vulnerable APIs. However, their method can miss code injection flaws caused by unknown sensitive plugin APIs. Besides, analyzing information flows in JavaScript is challenging. We found that the previous approaches are not able to analyze various contexts of callback functions. In this paper, we developed a static analysis tool called SCANCIF to scan code injection flaws. SCANCIF identifies the sensitive plugin APIs based on code injection tags, and analyzes flows of information based on modeling contexts of callback functions passed in function calls. We evaluated our approach on a data set of 3,204 HTML5-based mobile apps, as a result, SCANCIF scanned 220 vulnerable apps. We manually reviewed them and found 4 new code injection channels.

16:00 - 16:50
Panel Discussion: Challenges of Testing Autonomous Systems

Vahid Garousi, Cristina Seceleanu, Konstantinos Sagonas, Brian Nielsen, Bernhard Aichernig.

2:20 PM - 4:00 PM
Opening a business anywhere

Michael Lambert Harvard M.D.

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Michael Lambert Harvard M.D.

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Michael Lambert Harvard M.D.

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Call For Papers

The workshop does not accept papers that focus purely on functional testing!

Best papers from the workshop will be invited to submit an extended version of their work to
the Special Issue on "Testing Extra-Functional Properties" in the Software Testing, Verification and Reliability Journal!

Download: Flyer as PDF ----- Call-for-Papers as text

As the presence and role of computer systems in our daily life increases, we rely more and more on the services that are provided by software. On one hand, more tasks and functions are delegated to software systems (e.g., in the automotive domain), and on the other hand, the expectations and demands on the variety of services provided by these systems are dramatically growing (e.g., in mobile phones). In this context, the success of a software product may not only be dependent on logical correctness of its functions, but also on their quality characteristics and how they perform. Such system characteristics, which are referred to and captured as Extra-Functional Properties (EFPs), or Non-Functional Properties, have determinant importance particularly in resource constrained systems. For instance, in real-time embedded domain there can be limitations on available memory, CPU and processing capability, power consumption, and so on, that need to be considered along with timing requirements of an application. Considering the rapid development towards increased integration of software with the social and physical world that we see today, quality aspects become more important in an increasing number of the systems and devices we use and depend on. These systems therefore, need to be tested with a special attention to EFPs such as safety, security, performance and robustness.

Testing a system with respect to its EFPs, however, poses specific challenges and traditional functional testing methods and approaches may not simply be applicable. Examples of such challenges are: fault localization, the need to have appropriate techniques for different types of EFPs, the role and impact of the environment in testing EFPs, observability and testability issues, coverage and test-stop criteria, modeling EFPs and generating meaningful test cases, mutation operators for EFPs, etc.

ITEQS provides a well-focused forum with the goal of bringing together researchers and practitioners to share ideas, identify challenges, propose solutions and techniques, and in general expand the state of the art in testing EFPs and quality characteristics of software systems and services.

Check the topics of interest for more details.

Towards the goal of ITEQS, the topics of interest for the workshop include, but are not limited to, the following:

  • Model-based testing of EFPs; e.g., choice of modeling languages to capture EFPs and their role on testability, model-based test case generation, etc.
  • Mutation-based testing for EFPs; e.g., application of mutation techniques for testing of EFPs particularly introduction of EFP-specific mutation operators
  • Search-based testing techniques for EFPs
  • Testability, observability, and the role of the platform; e.g., how choosing an operating system can impact testability of EFPs, for instance, a real-time operating system, introducing testability mechanisms into a platform, designing middlewares for testing of EFPs
  • Empirical studies and experience reports; e.g., on the importance of testing EFPs, evaluation of testing methods, case-study and reports on project failures due to EFPs, comparison of methods and techniques
  • Quality assurance, standards, and their impact on testing EFPs
  • Requirements and testing EFPs; e.g., identification and generation of test oracles for EFPs from requirements, requirements for testability, traceability
  • Coverage criteria in testing EFPs
  • Processes and their role in testing EFPs; e.g., agile and TDD
  • Fault localization for EFPs and debugging
  • Formal methods, model-checking, and reasoning about EFPs
  • Parallelism, Concurrency, and Testing of multicore applications
  • Performance, Robustness, and Security Testing
  • Testing real-time, embedded, and cyber-physical systems, and their challenges
  • Testing quality characteristics of distributed, mobile, and cloud applications

The workshop promotes two types of contributions in IEEE double-column format:

  1. 1) Position papers and also solid Work-in-Progress papers consisting of 4 pages. WiP papers should provide concrete (but not necessarily complete) solutions and results as well as detailed plans for future extensions. Position papers should provide a clear position about research in testing EFPs arguing for a particular research direction in this area, discussing open problems and challenges, emerging topics and applications, and so on.
  2. 2) Full research papers consisting of 6-10 pages.

Each submitted paper will receive at least three reviews. To be faithful to the scope and goals of the workshop, papers that focus only on functional aspects and fail to address any EFPs and quality attributes will not be accepted.

Papers must conform to the IEEE double-column format (http://www.ieee.org/conferences_events/conferences/publishing/templates.html). Authors can submit their papers through easychair at: https://easychair.org/conferences/?conf=iteqs2018. Accepted and presented papers will be included in the IEEE CPS Proceedings.

Important Dates

Submission Deadline Jan 21 (extended)Jan 12, 2018
Notification of Acceptance Feb 21, 2018
Camera-ready Deadline Feb 28, 2018
Workshop Date April 9, 2018

Organizing Committee

Contact us: iteqs2018 [at] easychair [dot] org or mehrdad.saadatmand [at] ri [dot] se

Mehrdad Saadatmand

Program chair

RISE SICS Västerås, Sweden

Birgitta Lindström

Program chair

University of Skövde, Sweden

Bernhard K. Aichernig

Program chair

Graz University of Technology, Austria

Program Committee

Antonia Bertolino

CNR (National Research Council), Italy

Brian Nielsen

Aalborg University, Denmark

Lionel Briand

University of Luxembourg, Luxembourg

Mohammad Mousavi

University of Leicester, UK

Vahid Garousi

Wageningen University, Netherlands

Jeff Offutt

George Mason University, USA

Yvan Labiche

Carleton University, Canada

Vittorio Cortellessa

University of L'Aquila, Italy

Markus Bohlin

RISE SICS Västerås, Sweden

Björn Lisper

Mälardalen University, Sweden

Mika Mäntylä

University of Oulu, Finland

Hadi Hemmati

University of Calgary, Canada

Bestoun S. Ahmed

Czech Technical University in Prague, Czech Republic

James H. Hill

Indiana University-Purdue University Indianapolis, USA

Wasif Afzal

Mälardalen University, Sweden

Pasqualina Potena

RISE SICS Västerås, Sweden

Heinz Schmidt

RMIT University, Australia


Nils Müllner

Mälardalen University, Sweden

Sahar Tahvili

RISE SICS Västerås, Sweden

General sponsors

organizations that support us


  • Aros Congress Center (Room 107), Munkgatan 7, Västerås