Repensando el Modelo de Calidad en Uso

Varias veces en este blog hemos escrito sobre este modelo de QiU, su utilización y tambien sobre cierta confusión que hay sobre su significado y contenido.

Bien, hoy el modelo se está replanteando y existen diversas alternativas propuestas.

En los siguientes párrafos transcribo el paper presentado en el 1st International Workshop on Experience with SQuaRE Series and Its Future Direction, co-located with 26th Asia-Pacific Software Engineering Conference (APSEC 2019). Putrajaya, Malaysia, Dec 2, 2019.

El paper completo se puede ver en http://ceur-ws.org/Vol-2545/.

El trabajo es solamente una propuesta para discusión, pero lo que es indudable es que este modelo debe revisarse dada su importancia en el contexto de los modelos de calidad propuestos por la IS/IEC 25000.

Revisiting the Quality in Use model

Abstract— This paper examines concepts in the Quality in Use model proposed by SQuaRE standards. A more detailed differentiation is made about the results of the interaction with the system in a production environment and propositions are made for future revisions of the model in the standard.

I.          Introduction

In many cases, Quality in Use model is highly, or only, associated with usability. This vision is at least partial.

This paper proposes re-examining one of the more important SQuaRE quality models, the Quality in Use model.

Basic concepts are reviewed, and several suggestions are offered to be considered on future revisions of the model.

A.    The Quality in Use model concept

When a system runs in a production environment, considerations about its quality could be based at least on:

1.      Usability of the system (if a human user exists)

2.      Generation of the correct expected outputs

3.      Fulfilling of expected positive outcomes (results)

4.      Avoidance of possible negative outcomes (consequences)

If these conditions are met within acceptable values the system will be considered as a quality (enough) system.

Usually people describe desired solutions as required outcomes and an unrefined request about how these outcomes are obtained.

These elements, at a very early stage, are the first outline to the Quality in Use model for the system because they contain quality concepts expected and valuable for the customer.

This Quality in Use model will then depict the behaviour of the system in a production environment compared with target behaviour established at modelling stage.

It should be noted this paper is focused on the final user but that, as stated by [4] ISO/IEC 25010:2011, “Other stakeholders, such as software developers, system integrators, acquirers, owners, maintainers, contractors, quality assurance and control professionals, will also be concerned with the quality.”

II.         ISO/IEC 25010 definitions

The ISO/IEC 25010:2011 defines Quality in Use as the degree to which a product or system can be used by specific users to meet their needs to achieve specific goals with effectiveness, efficiency, and satisfaction and freedom from risk in specific contexts of use.

Quality is modelled through five characteristics related to outcomes of interaction with a system: effectiveness, efficiency, satisfaction, freedom from risk and context coverage.

For Quality in Use model the term usability refers to the subset of quality in use composed of effectiveness, efficiency, satisfaction, and context of use coverage.

Figure 1 is a simplified graphical explanation of Quality in Use model and its relationship with a product in operation.

                                                               Figure 1 Graphical explanation of quality in use model

                                                                                       System in a production environment sequence:

                                                                                           1 Product execution:

                                                                                 a. System started, with or without human interaction

                                                                                           2. Resultant outcomes:

                                                                                 a. Outputs produced

                                                                                 b. Usability attained

                                                                                 c. Results achieved

                                                                                 d. Consequences derived

                                                                                           3.  System running in a specific context of use

 III.   Reviewing components of the Quality in Use model

A.    Interactions and outcomes

Different measures are provided by the model for usability (effectiveness, efficiency, and satisfaction that includes user experience and ergonomics). All these measures apply if a human user, primary, secondary or indirect exists.

The model also considers consequences and results, explained later. Figure 2 shows the measures proposed for the production environment.

                                                                       Figure 2. Measures proposed for the production environment.

(*) Output is outside Quality in Use model proposition but is generated as an outcome during execution on a production environment; for that reason, measures from ISO/IEC 25023:2016, as functional suitability and appropriateness, and measures from ISO/IEC 25024:2015 for Data Quality could be used to measure quality of Output produced

B.    Outcomes

Outcomes are goals, projected or attained, that are the main reason of the existence of the system.

Some clarification is required to distinguish between the different outcomes: outputs, results and consequences:

·         Outputs: Things, virtual or real, produced or changed as effect of the execution of the system.
As mentioned above Output is outside Quality in Use model proposition but is generated as an outcome during execution on a production environment, for that reason quality measures from ISO/IEC 25023:2016 could be used.

·         Results: Short-term or long-term effects obtained from the use of the system.
It is assumed that they are related to needs fulfilled and are positive. The quality measures could compare the target value expected for each result and the actual value obtained during executions, one or many.

·         Consequences: ISO/IEC 25022:2016 defines consequence as an outcome emerging from a negative risk (not an opportunity).
Enumerated risks include threats to economic status, human life, health, or the environment. The quality measures could compare the target value expected for each risk or consequence and the actual value obtained during executions.

Definitions for Outputs, Results, Consequences and Outcomes concluded or obtained from [6], [7], [8], [1].

C.    Points to consider on interactions and outcomes

Many existing systems operate exchanging activities with other systems, and subsequently all of them share responsibility for the results.

Organizations or individuals using the system will perceive quality without an exact discrimination of quality of each intervenient component.

-     Should Quality in Use model offer properties and measures for this kind of systems of systems?

Today we are developing systems that “see”, “hear” and “know” where they are located, making decisions and executing actions in actual environments based on incoming data and, for example, previously learned behaviours.

-     Will there be a quality measure for bias of the decisions of the algorithms? Which will be the target value?

-     Will there be a quality measure for ethics of the solution? Which will be the target value?

-     Will it be acceptable to call these “interactions”?

D.   Context of use

ISO/IEC 25022:2016 defines context of use as users, tasks, equipment (hardware, software and materials), and the physical and social environments in which a system, product or service is used.

[SOURCE: ISO 9241‑11:1998, 3.5, modified — With “product” replaced by “system, product or service”.]

The proposed quality property associated to context is Coverage. Completeness and Flexibility are the measures proposed for this property.

Such properties and measures are oriented towards changes in user skills or capabilities, type of users, and several target scenarios. These is correct for some types of systems where contexts can be enumerated.

E.    Points to consider on context of use

-   Is Coverage as proposed by the Quality in Use model applicable to highly variable / unknow contexts?

-   How is quality measured in these cases? How are completeness and flexibility measured?

-   Context changes could require user actions or produce diverse outcomes. How could we measure the quality of the produced response to the new context?

IV.        Conclusion

Quality in Use model is a very important model whose elements, as mentioned, are identified at the very beginning of the development process and completed throughout the product life cycle.

The paper rises several questions about meaning and content of the current Quality in Use model. These questions will require a deep discussion and opinions exchange, but some key ideas are briefed below.

·    The model should be detached from the usability centred idea and the value of the model should be emphasised. Possibly a new name for the model should be considered showing the relationship between the model and the operation of the system in diverse and mutating production environments. A suggestion could be Production Quality Model.

·    ISO/IEC 25022:2016 establishes that the external measures can only be used during “the testing stages of the life cycle process and during any operational stages”. Included in this definition are environments identical to production used for testing process.  For that reason, external measures that apply to production environment should be moved to Quality in Use model obtaining a complete picture of system execution.

·    Current and emerging systems should be considered by the Quality in Use model, as new forms of input, algorithms, processing and communications are constantly evolving and new ones appearing. Nevertheless, the correct level of abstraction should be maintained, and the new characteristics applicable to these kinds of systems must be considered. These characteristics could be allocated on a Technical Report suitable to the specific kind of ICT product without compromising the level of abstraction of the general model.

·    Measures applicable to Usability, Results and Consequences need to be analysed and/or defined in accordance to the kind of ICT product.

·    Current Service Quality model should be re-examined considering at least two possibilities. First, if it really is a different model or it is part of the Quality in Use model. Second, consider the Service Quality model as an specialization of the Quality in Use model.

·    The fundamental concept of Context should be re-examined. Current Quality in Use model includes two basic measures for context, Completeness and Flexibility. The measures embrace important ideas but with completely different meaning depending on the kind of systems. For this reason, context measures should be defined using for example a Technical Report mechanism as stated previously.

As a final thought, Quality in Use model appears at the very beginning as quality expectations about the future product and at the end as outcomes: outputs, results and consequences, of the use of the actual product in Production environments.

Between expectations and production, other models need to be developed, Product Quality Model and Data Quality Model, stating the quality characteristics required on the product. Fulfilment of first expectations and actual behaviour of product in Production conform the user perceived quality.  

References

[1]      ISO/IEC 25022:2016, Systems and Software engineering Measurement of quality in use)

[2]      ISO/IEC 25023:2016, Systems and Software engineering   measurement of system and software product quality.

[3]      ISO/IEC 25024 Systems and software quality requirements and evaluation (SQUARE) - Measurement of data quality

[4]      ISO/IEC 25010:2011. ISO/IEC 25010:2011 Systems and Software engineering  System and software quality models

[5]      ISO 9241‑11:1998 has been revised as ISO9241-11:2018

[6]      Deborah Mills-Scofield It’s Not Just Semantics: Managing Outcomes Vs. Outputs Harvard Business Review, 2012

[7]      Koopman and Fratrik, How Many Operational Design Domains, Objects, and Events?, Safe AI 2019 talk, 2019

[8]      INTRAC 2015, Outputs, Outcomes and Impact