Varias
veces en este blog hemos escrito sobre este modelo de QiU, su utilización y
tambien sobre cierta confusión que hay sobre su significado y contenido.
Bien,
hoy el modelo se está replanteando y existen diversas alternativas propuestas.
En los siguientes párrafos transcribo el paper presentado en el 1st International Workshop on Experience with SQuaRE Series and Its
Future Direction, co-located with 26th Asia-Pacific Software Engineering
Conference (APSEC 2019). Putrajaya,
Malaysia, Dec 2, 2019.
El
paper completo se puede ver en http://ceur-ws.org/Vol-2545/.
El
trabajo es solamente una propuesta para discusión, pero lo que es indudable es
que este modelo debe revisarse dada su importancia en el contexto de los
modelos de calidad propuestos por la IS/IEC 25000.
Revisiting the Quality in Use model
Abstract— This paper examines concepts in the
Quality in Use model proposed by SQuaRE standards. A more detailed
differentiation is made about the results of the interaction with the system in
a production environment and propositions are made for future revisions of the
model in the standard.
I.
Introduction
In many
cases, Quality in Use model is highly, or only, associated with usability. This
vision is at least partial.
This
paper proposes re-examining one of the more important SQuaRE quality models,
the Quality in Use model.
Basic
concepts are reviewed, and several suggestions are offered to be considered on
future revisions of the model.
A. The Quality in Use
model concept
When a system
runs in a production environment, considerations about its quality could be
based at least on:
1. Usability of the system (if a human user
exists)
2. Generation of the correct expected outputs
3. Fulfilling of expected positive outcomes
(results)
4. Avoidance of possible negative outcomes
(consequences)
If these
conditions are met within acceptable values the system will be considered as a
quality (enough) system.
Usually
people describe desired solutions as required outcomes and an unrefined request
about how these outcomes are obtained.
These
elements, at a very early stage, are the first outline to the Quality in Use
model for the system because they contain quality concepts expected and
valuable for the customer.
This
Quality in Use model will then depict the behaviour of the system in a
production environment compared with target behaviour established at modelling stage.
It should
be noted this paper is focused on the final user but that, as stated by [4] ISO/IEC
25010:2011, “Other stakeholders, such as software developers, system
integrators, acquirers, owners, maintainers, contractors, quality assurance and
control professionals, will also be concerned with the quality.”
II.
ISO/IEC 25010 definitions
The ISO/IEC 25010:2011 defines Quality in Use as the degree
to which a product or system can be used by specific users to meet their needs
to achieve specific goals with effectiveness, efficiency, and satisfaction and
freedom from risk in specific contexts of use.
Quality is
modelled through five characteristics related to outcomes of interaction with a
system: effectiveness, efficiency, satisfaction, freedom from risk and context
coverage.
For
Quality in Use model the term usability refers to the subset of quality in use
composed of effectiveness, efficiency, satisfaction, and context of use
coverage.
Figure 1 is a simplified graphical explanation of Quality in Use model and its relationship with a product in operation.
Figure 1 Graphical
explanation of quality in use model
System in a production environment
sequence:
1 Product
execution:
a.
System started, with or without human interaction
2. Resultant
outcomes:
a.
Outputs produced
b.
Usability attained
c.
Results achieved
d.
Consequences derived
3. System running in a specific context of use
III. Reviewing components of the Quality in Use model
A. Interactions and outcomes
Different
measures are provided by the model for usability (effectiveness, efficiency,
and satisfaction that includes user experience and ergonomics). All these
measures apply if a human user, primary, secondary or indirect exists.
The model
also considers consequences and results, explained later. Figure 2 shows the
measures proposed for the production environment.
Figure 2. Measures
proposed for the production environment.
(*)
Output is outside Quality in Use model proposition but is generated as an
outcome during execution on a production environment; for that reason, measures
from ISO/IEC 25023:2016, as functional suitability and appropriateness, and
measures from ISO/IEC 25024:2015 for Data Quality could be used to measure
quality of Output produced
B. Outcomes
Outcomes are
goals, projected or attained, that are the main reason of the existence of the
system.
Some
clarification is required to distinguish between the different outcomes:
outputs, results and consequences:
·
Outputs:
Things, virtual or real, produced or changed as effect of the execution of the
system.
As mentioned above Output is outside Quality in Use model proposition but is generated as an outcome during execution on a production environment, for that reason quality measures from ISO/IEC 25023:2016 could be used.
As mentioned above Output is outside Quality in Use model proposition but is generated as an outcome during execution on a production environment, for that reason quality measures from ISO/IEC 25023:2016 could be used.
·
Results:
Short-term or long-term effects obtained from the use of the system.
It is assumed that they are related to needs fulfilled and are positive. The quality measures could compare the target value expected for each result and the actual value obtained during executions, one or many.
It is assumed that they are related to needs fulfilled and are positive. The quality measures could compare the target value expected for each result and the actual value obtained during executions, one or many.
·
Consequences:
ISO/IEC 25022:2016 defines consequence as an outcome emerging from a negative
risk (not an opportunity).
Enumerated risks include threats to economic status, human life, health, or the environment. The quality measures could compare the target value expected for each risk or consequence and the actual value obtained during executions.
Enumerated risks include threats to economic status, human life, health, or the environment. The quality measures could compare the target value expected for each risk or consequence and the actual value obtained during executions.
Definitions
for Outputs, Results, Consequences and Outcomes concluded or obtained from [6],
[7], [8], [1].
C. Points to consider on interactions and outcomes
Many existing systems operate exchanging
activities with other systems, and subsequently all of them share
responsibility for the results.
Organizations or individuals using the system will perceive quality without an exact discrimination of quality of each intervenient component.
- Should
Quality in Use model offer properties and measures for this kind of systems of
systems?
Today we
are developing systems that “see”, “hear” and “know” where they are located,
making decisions and executing actions in actual environments based on incoming
data and, for example, previously learned behaviours.
- Will
there be a quality measure for bias of the decisions of the algorithms? Which
will be the target value?
- Will
there be a quality measure for ethics of the solution? Which will be the target
value?
- Will
it be acceptable to call these “interactions”?
D. Context of use
ISO/IEC 25022:2016
defines context of use as users, tasks, equipment (hardware, software and
materials), and the physical and social environments in which a system, product
or service is used.
[SOURCE: ISO 9241‑11:1998, 3.5, modified — With “product”
replaced by “system, product or service”.]
The proposed
quality property associated to context is Coverage. Completeness and
Flexibility are the measures proposed for this property.
Such
properties and measures are oriented towards changes in user skills or
capabilities, type of users, and several target scenarios. These is correct for
some types of systems where contexts can be enumerated.
E.
Points to consider on context
of use
- Is
Coverage as proposed by the Quality in Use model applicable to highly variable
/ unknow contexts?
- How
is quality measured in these cases? How are completeness and flexibility
measured?
- Context
changes could require user actions or produce diverse outcomes. How could we
measure the quality of the produced response to the new context?
IV.
Conclusion
Quality in
Use model is a very important model whose elements, as mentioned, are
identified at the very beginning of the development process and completed
throughout the product life cycle.
The paper
rises several questions about meaning and content of the current Quality in Use
model. These questions will require a deep discussion and opinions exchange,
but some key ideas are briefed below.
· The model should be detached from
the usability centred idea and the value of the model should be emphasised.
Possibly a new name for the model should be considered showing the relationship
between the model and the operation of the system in diverse and mutating
production environments. A suggestion could be Production Quality Model.
· ISO/IEC 25022:2016 establishes that
the external measures can only be used during “the testing stages of the life
cycle process and during any operational stages”. Included in this definition
are environments identical to production used for testing process. For that reason, external measures that apply
to production environment should be moved to Quality in Use model obtaining a
complete picture of system execution.
· Current and emerging systems should
be considered by the Quality in Use model, as new forms of input, algorithms,
processing and communications are constantly evolving and new ones appearing.
Nevertheless, the correct level of abstraction should be maintained, and the
new characteristics applicable to these kinds of systems must be considered.
These characteristics could be allocated on a Technical Report suitable to the
specific kind of ICT product without compromising the level of abstraction of
the general model.
· Measures applicable to Usability,
Results and Consequences need to be analysed and/or defined in accordance to
the kind of ICT product.
· Current Service Quality model should
be re-examined considering at least two possibilities. First, if it really is a
different model or it is part of the Quality in Use model. Second, consider the
Service Quality model as an specialization of the Quality in Use model.
· The fundamental concept of Context
should be re-examined. Current Quality in Use model includes two basic measures
for context, Completeness and Flexibility. The measures embrace important ideas
but with completely different meaning depending on the kind of systems. For
this reason, context measures should be defined using for example a Technical
Report mechanism as stated previously.
As a final
thought, Quality in Use model appears at the very beginning as quality
expectations about the future product and at the end as outcomes: outputs,
results and consequences, of the use of the actual product in Production
environments.
Between
expectations and production, other models need to be developed, Product Quality
Model and Data Quality Model, stating the quality characteristics required on
the product. Fulfilment of first expectations and actual behaviour of product
in Production conform the user perceived quality.
References
[1]
ISO/IEC
25022:2016, Systems and Software engineering Measurement of quality in use)
[2]
ISO/IEC
25023:2016, Systems and Software engineering
measurement of system and
software product quality.
[3]
ISO/IEC
25024 Systems and software quality requirements and evaluation (SQUARE) - Measurement
of data quality
[4]
ISO/IEC
25010:2011. ISO/IEC
25010:2011 Systems and Software engineering
System and software quality models
[5] ISO 9241‑11:1998 has been revised as
ISO9241-11:2018
[6]
Deborah
Mills-Scofield It’s Not Just Semantics: Managing Outcomes Vs. Outputs Harvard
Business Review, 2012
[7] Koopman and Fratrik, How Many
Operational Design Domains, Objects, and Events?, Safe AI 2019 talk, 2019
[8]
INTRAC
2015, Outputs, Outcomes and Impact
