INTEGRATION OF RISK ANALYSIS METHODS IN AEROSPACE RESEARCH PROJECTS

Sarah Francisca de Souza Borges

Aeronautics Institute of Technology, Brazil

E-mail: sarahfsborges@gmail.com

Mischel Carmen Neyra Belderrain

Aeronautics Institute of Technology, Brazil

E-mail: carmen@ita.br

Moacyr Machado Cardoso Junior

Aeronautics Institute of Technology, Brazil

E-mail: moacyr@ita.br

Diogo Silva Castilho

Aeronautics Institute of Technology, Brazil

E-mail: castilhods@msn.com

Submission: 10/14/2020

Revision: 1/8/2021

Accept: 3/8/2021

ABSTRACT

Organizations are exposed to several types of risks, such as environmental, legal, operational, financial, and technological; that are subjected to epistemic uncertainty. In this context, a contemporary issue is how to deal with accidents, with greater difficulty in understanding the sociotechnical system, due to its complex and dynamic characteristics, in an attempt to prevent accidents based on components’ behavior. Although, for most complex systems and projects, a record of the exposure to hazards is incomplete or nonexistent, especially when it is highly innovative. This study developed a risk analysis framework for complex aerospace research projects by integrating different methods: problem structuring, safety control action analysis, and prioritization of results. Three methods are proposed: (1) Soft Systems Methodology (SSM) for initial review and understanding of the problem situation, and preliminary identification of hazards and losses; (2) Systems-Theoretic Process Analysis (STPA), to identify Unsafe Control Actions (UCAs) and their causal scenarios; and (3), Preferences Sorting Technique by Similarity to Ideal Solution (TOPSIS Fuzzy) for prioritization of the UCAs and mitigating causal scenarios. This proposal was applied to the Liquid Propulsion Injection Systems Laboratory (CEPROS), and, through the SSM, 7 hazards and 4 losses were found. On the other hand, the STPA method found 15 loops with 48 UCAs and 106 causal scenarios. In the end, it is recommended that the Decision Maker establishes a cut-off criterion, that is, a Hierarchy of Management and Control of the identified UCAs. The proposed methods follow the line of sociotechnical systems, considering the difficulty of the decision-maker for risk analysis in aerospace research projects. Thus, this work presents a structure of different methods covering the entire risk management process, increasing the difficulty in fulfilling the mission due to the level of complexity of the project, and supporting strategies for coordinated decision-making.

Keywords: Risk analysis; Complex problem; Problem structuring.

1. INTRODUCTION

Organizations are exposed to several types of risks, such as environmental, legal, operational, financial, and technological; that are subjected to epistemic uncertainty, present mainly in projects with complex characteristics (multiple decision-makers, multiple perspectives). For those projects, a holistic initial risk analysis is essential. Although, for most complex systems, data of exposure to hazards is incomplete or non-existent due to its innovative character.

In this context, a contemporary issue is how to understand the sociotechnical system with complex and dynamic characteristics to better deal with hazards, in an attempt to prevent accidents. An essential problem in modeling complex sociotechnical systems as a chain of events is that the dependencies (interactions) between components are not adequately considered. These interactions can be multiple, non-linear, and simultaneous. Thus, in the analysis of the systemic model, with a complex and dynamic context, we seek to explain the variability and resonance of activities, with an emphasis on preventive actions that consider the ability to adapt to the organizational pressures (Carbognin, 2017).

According to ISO 31000 (ABNT, 2018), a standard guide, the risk management process has three phases: Definition of scope, context, and criteria; Risk assessment process; and Risk treatment. In these phases, the decision-maker responsible for a project has to select a method, or methods, to mitigate incidents.

In the study of risk analysis methods, since the 1931 Heinrich’s Domino Theory, researchers consider that accidents occur from multiple variables, receiving significant developments. Within this concept, new methods emerged incorporating systemic thinking with a qualitative analysis, such as Accimap by Jens Rasmussen (1997), Functional Resonance Analysis Method (FRAM) by Erik Hollnagel (2004), and Systems-Theoretic Accident Model and Processes (STAMP) by Nancy Leveson (2004).

In this study, we evaluated which methods would serve to identify a higher number of potential causes to prevent accidents in complex systems. STAMP and its derivative technique for hazard analysis, the Systems-Theoretic Process Analysis (STPA), received considerable notoriety. The STPA is based on the evaluation of the interactions between the controllers and controlled processes, not limited to the analysis of possible failures of a component or operators’ errors. In contrast with other hazard identification methods, STPA aims to map control actions to derive more hazardous scenarios and identify more causal factors, being recommended for software development projects, system design, and analysis of human behavior (Leveson, 2011).

Although, STPA, does not reveal which method could be used initially to identify the hazards and losses, and how to prioritize mitigating measures. This, from a management point of view, makes implementation more difficult.

Thus, as a starting point for structuring a complex problem, an exploratory method that allows the researcher to better understand the problem under analysis is essential. In this article, Problem Structuring Methods (PSM) are used to support the understanding of problematic situations, the identification of hazards and losses, and identification of relevant systems that would need intervention. Soft Systems Methodology (SSM) was selected because, since its inception, it was proposed in the administrative area for a preliminary understanding of the problematic situations with a systemic view.

This first analysis, based on the SSM, allows a better application of the STPA. However, there was still a lack of a method for prioritizing defenses. In this phase, Multiple-Criteria Decision Method (MCDM) was used for prioritization according to criteria and weights defined by the researcher. Among MCDMs, it is proposed the use of the TOPSIS Fuzzy method, which is useful for a treat the epistemic uncertainty. That is, there is no specific method or historical data, thus requiring a qualitative analysis.

Also, the TOPSIS method itself aggregates data, providing group decision analysis. In this way, multiple alternatives are analyzed according to the selected criteria. By the Fuzzy method, the intrinsic uncertainty in the decision is considered (initially using linguistic variables to explain the alternative, and then the numerical transformation of the results is made). According to Sodhi and Tadinada (2012), the classification of alternatives is considered when it is closest to the Fuzzy Positive Ideal Solution (FPIS) and the furthest away from the Fuzzy Negative Ideal Solution (FNIS).

Therefore, at the end of this article, we propose a risk management structure for complex aerospace research projects to integrate different risk management methods in order to structure the problem, identify hazards, and prioritize defenses.

2. MATERIAL AND METHODS

According to the Project Management Institute (PMI, 2013), project risk originates from intrinsic uncertainty, in which organizations and stakeholders are willing to accept at different levels, and seeks to reduce the probability and impact of events adverse.

Risk analysis standards are useful for understanding the analysis process and responsibilities, from general definitions to operations for an organization, such as ISO 31000, NR-9, NR-13, among others.

Although standards exist to maintain a higher level of safety during operations, these are often considered to be general, open to many interpretations. An important contemporary issue for analysts and risk managers is how to deal with accidents (or more broadly, "unintended consequences") in complex systems, such as electrical, mechanical, biological, computational, economic, or political. Such complex systems are affected by generalized uncertainty, which can lead to surprising behavior during their operation (Bjerga, Aven & Zio, 2016).

According to ISO 31000 (ABNT, 2018), risk is the uncertainty in the objectives, and risk prevention seeks to mitigate or control adverse events. Besides, risk analysis is the process of understanding its nature and determining its level, as well as the necessary treatment, for example, removing the source of risk, changing the probability and consequences.

As shown in Figure 1, there is a risk management process, from the definition of the scope to the treatment of risks.

Figure 1: Risk management process.

Source: ABNT NBR ISO 31000 (ABNT, 2018).

In the study of risk prevention and accident investigation methods, most of them fell in a sequential and deterministic chain of events. In which, the causality of the accident is described as a chain of failure events and human errors that lead to the event with real loss. Among them: FMEA (Failure Mode and Effects Analysis), FMECA (Failure Modes, Effects and Criticality Analysis), FTA (Fault Tree Analysis), ETA (Event Tree Analysis), HAZOP (Hazard and Operability study), and Cause and Consequence Analysis. Such models are limited in their ability to deal with complex system accidents (resulting from interactions between components and not just individual failures), software-related accidents, highly complex human decision-making, or migration of the system to an accident over time (Leveson, 2013). Within this concept, new methods have emerged that incorporate systemic thinking; the most cited are Accimap, STPA, and FRAM (Carbognin, 2017).

Table 1 highlights some differences identified between those methods and the criteria for selecting one of them. It is possible to observe that no method covers the entire risk management process, but the STPA method came closer to the defined premises.

Table 1: Comparison of methods for the risk management process.

Method / characteristic	Preliminary structuring of the problem	Identification of hazards and accidents	Quantitative assessment of failure rates	Qualitative assessment of hazards and accidents	Identification of unsafe control actions	Identification of causal scenarios	Identification of defenses / protection barriers	Prioritization of alternatives
FMECA		X	X			X	X
FMEA		X	X			X	X
HAZOP		X	X			X	X
Risk matrix			X					X
FTA		X	X			X
ETA		X	X			X
Accimap	X	X		X		X
STPA		X		X	X	X	X
FRAM	X	X		X		X

In this sense, the STPA method stood out in meeting the most considerable number of premises for complex sociotechnical systems, especially due to the need to identify and establish control actions to prevent hazards and losses (before denominated accidents). However, it was noted the lack of a method for initial context analyses and the necessity of results' prioritization.

2.1. SSM

The methods of Soft Operational Research have come to be known as PSM and developed independently from the mid-1960s onwards. These innovations accompanied an expanded criticism of traditional or Hard Operational Research (Mingers & Rosenhead, 2004; Rosenhead & Mingers, 2001). The difference between Hard Operational Research and Soft Operational Research is that the first one considers the existence of very well defined problems, in the world itself, while the second shifts the idea of a problem to the perception of the observer in its process of investigating the world (Ensslin, 2002).

To understand complex problems, the PSMs are essential, and according to Mingers and Rosenhead (2004), they have a feature by multiple actors, multiple perspectives, challenging to measure and conflicting interests, intangible importance, and critical uncertainties.

Besides, Mingers and Rosenhead (2004) affirm that the most well-known PSMs methods are: Strategic Options Development and Analysis (SODA), Strategic Choice Approach (SCA), and SSM. Although, there are other methods, such as Robustness Analysis, Drama Theory, Viable Systems Model (VSM), and System Dynamics (SD).

Of these, SSM gained greater prominence in the 1980s, according to Water, Schinkel and Rozier (2007), to initiate a debate to create a shared vision and understanding of the context of the problematic situation and creating a consensus. In comparison to Delphi, Brainstorm, SCA, SODA, and others that support the formation of this consensus, SSM is mainly focused on solving the question “what” (and perhaps “why”) and not on the issue “how” to solve.

SSM was developed in the period from 1969 to 1972, by Peter Checkland at Lancaster University (Simonsen, 1994). SSM delivers the tools to create a framework of scientific premises and knowledge through an investigation process, that constitutes a starting point on how to look at the problematic situation. In this way, it makes possible the scientific criticism of the framework within the issue (Hanafizadeh & Mehrabioun, 2018).

2.2. STPA

Even in simple situations, risk coordination is necessary. For example, when multiple individuals are controlling the same process, it can result in two types of unsafe interactions: (1) both controllers assume that the other is carrying out control responsibilities and, as a result, nobody does this or (2) the controllers provide conflicting control actions, with unwanted side effects (Leveson, 2011).

According to Rasmussen (1997), more than finding the causes, it is necessary to find the deep reasons that can lead to the accident. Eventually, any task presents many degrees of freedom to the actors, being essential to approach the requirements analysis and evaluation of the system for the management of behaviors and operations.

The STPA is used to prevent losses. It is based on the STAMP model, which was created to explore the connection between Systems Safety and Systems Engineering.

Like traditional methods, the STPA seeks to identify scenarios that lead to hazards, aiming to mitigate them or to control losses. The significant difference of this method comes from the identification of hazards considering not only the failure of a component or operator but the existing hierarchical relationship seen from different perspectives (Leveson, 2004).

According to Bjerga, Aven and Zio (2016), the variation is often referred to as stochastic uncertainty. The authors discuss the feasibility of using probability in complex socio-technical systems. In the method FRAM, according to Hollnagel (2004), fail probability can be complementary data. Meanwhile, in the STPA, Leveson (2004, 2015), in dynamic systems doesn't make sense to talk about probability because the environment and behavior of operations are in constants change.

Thus, both FRAM and STPA methods produce a potential listing of causal scenarios that provide better results than the classic sequential methods. However, these methods are approaches with a focus on qualitative modeling and description of the system's behavior, giving due attention to dependencies, but without considering, in the end, the uncertainty and probability of occurrence. According to Bjerga, Aven, and Zio (2016), this phase is essential for prioritization. However, there is difficulty in defining the probability in specific models and components, as unique cases of new technologies without a history of data or performance of human variability.

2.3. TOPSIS Fuzzy

A proposal to the gap of prioritization of the actions identified in STPA, and to complete the framework, is the use of MCDM methods have several advantages, such as, allowing the criteria that influenced the decision to be explicitly considered; facilitate the monitoring and visualization of the stages of the process; allow to assess the contribution of each criterion in an isolated and aggregated way conducting to the result of the decision; facilitate the discussion of divergent perspectives of the interest groups and increase the understanding of the elaborated recommendations (Campolina, Soárez, Amaral, & Abe, 2017; Figueira, Greco, & Ehrgott, 2016).

However, the selection of a multicriteria decision model depends on the characteristics and objective of the problem under analysis. Each method can be classified depending on elements such as the type of data or the number of decision-makers involved in the decision process (Costa, 2012). Table 2 presents the MCDM in three categories and it can be observed that TOPSIS is categorized as a support method for the choice and prioritization of alternatives.

Table 2: MCDM problems and methods.

	Choice problems	Prioritization problems	Classification problems
1	AHP	AHP	AHPSort
2	ANP	ANP	UTADIS
3	MAUT/UTA	MAUT/UTA	FlowSort
4	MACBETH	MACBETH	ELECTRE-tri
5	PROMETHEE	PROMETHEE
6	ELECTRE I	ELECTRE III
7	TOPSIS	TOPSIS
8	GOAL PROGRAMMING	DEA
9	DEA

Source: Adapted from Ishizaka and Nemery (2013).

The TOPSIS method was proposed by Hwang and Yoon, in 1981, as a method of multicriteria decision support and it is used to order alternatives based on preferences that lead to an ideal solution. Since the solution called the positive ideal is one that maximizes the benefit criteria and minimizes the cost criteria, the negative ideal solution represents the other way around (Hwang & Yoon, 1981; Picanço et al., 2017).

Two widely explored techniques of the MCDM methods are TOPSIS (Hwang & Yoon, 1981) and TOPSIS Fuzzy (Chen, 2000), the latter being an adapted version of the former. Unlike comparative approaches such as AHP (Analytic Hierarchy Process), ANP (Analytic Network Process), AHP Fuzzy, ANP Fuzzy, the TOPSIS, and TOPSIS Fuzzy methods allow the adoption of an unlimited number of criteria to evaluate an unlimited number of alternatives. Besides, even in comparison to other methods, the simplicity of the mathematical procedures of both contributes to easy analysis and application (Lima Junior & Carpinetti, 2015).

Although these two methods have been developed based on the same principle of proximity to the ideal positive solution, they differ concerning the logic that underlies their mathematical procedures. While TOPSIS uses absolute numerical values in crisp format, and these are manipulated through calculations based on classical logic (or Aristotelian logic), the TOPSIS Fuzzy method incorporates Fuzzy logic to perform algebraic operations with numerical interval values, using together linguistic elements. Because of this, the TOPSIS Fuzzy method is considered an easier model, the process of data collection, the computational effort required, and even the final decisions provided for the same problem (Sodhi & T., 2012).

Fuzzy logic is a tool capable of capturing vague information, in general, described in natural language and converting it to a numerical format that is easy to manipulate. For that, linguistic variables, whose values are called Fuzzy sets, can be sentences and described when a preliminary language is specified, using proper terms (low, medium, high), logical connectives (non-negative, connectives and/or), modifiers (very, little) and delimiters (as parentheses) (Chenci, Rignel, & Lucas, 2011).

The following are the relevant conceptual definitions for understanding the TOPSIS Fuzzy method:

A Fuzzy set in in a universe of speech is is characterized by a function that that maps each element in to a real number in the range [0, 1]. The value of the function is called the degree of membership of in . The closer the value of to the unit, the higher the degree of association of in (Sodhi & T., 2012).

According to Kore, Ravi, and Patil (2017), a Fuzzy triangular number is expressed as a triplet = (𝑎1, 𝑎2, 𝑎3), represented in Figure 2.

Figure 2: Fuzzy triangular system.

Source: Kore, Ravi and Patil (2017).

Being that:

· 𝑎2 gives the maximum degree of with = 1

· 𝑎1 gives the minimum degree of with = 0

· 𝑎1 and 𝑎3 are the lower and upper limits of the area available for assessment or support data.

The membership function of the triangular Fuzzy number is given as Eq. 1.

(1)

3. FRAMEWORK

Thus, a study on the integration of methods for risk analysis in complex systems was carried out, featuring a separate proposal in three phases, the first related to the approach to the description of the system as a whole, the second phase, the analysis of UCAs, and the third phase, the prioritization of UCAs.

This approach emerged in the study of methods considering systemic thinking, in which the STPA method was highlighted for the analysis of unsafe action and the identification of scenarios in different loops and levels of complex systems. And, this framework propose was developed to support decision-making in complex aerospace research projects, considering the prevention of accidents through the application of defenses.

As shown in Figure 3, for each phase of the risk analysis process, a support method is suggested, Table 3 reveals the expected results.

Figure 3: Risk management process and proposed methods.

Source: Adapted from ABNT NBR ISO 31000 (2018).

Table 3: Expected results for each method.

Scope identification, a preliminary list of hazards and accidents	SSM: Holistic view and consensus among the main actors in understanding the problematic situation; identification of the relevant systems with their respective CATWOE and Root Definition (revealing the Transformations, what needs to be intermediated in the system, main actors, restrictions, and world view); Preliminary identification of hazards and accident.
Hazards and accidents identification	STPA: Assembly of a Control Structure (with the hierarchical representation of the system's actors); Identification and analysis of control actions, with respective causal scenarios and defenses.
Risk analysis and assessment	TOPSIS Fuzzy: Establishment of criteria; Analysis of epistemic uncertainty; Data aggregation as it is a group decision; Prioritization of Unsafe Control Actions (UCAs); Validation of defenses.

The framework propose is illustrated in Figure 4, it highlights the phases and stages proposed based on the methods.

Figure 4: Framework proposes.

In addition, the risk treatment phase of ISO 31000 would be in charge of the project manager, after all the risk management analysis, for the implementation of the defenses.

Furthermore, this structure needs to be revised when inconsistency is noted in some phase of the application, as well as between one phase and another. In the end, one should also be aware that the context analysis occurs at a given moment, that is, the systemic analysis needs to be revised in the event of considerable changes (such as the structure of the analyzed organization or project, for example), aiming to be true to reality and preserve the validity of the results.

3.1. Phase 1 - Problem Structuring

The SSM is structured in 7 (seven) stages, as shown in Figure 5. In this article, the context analysis of the problem was sought, which was achieved in the first three stages of this methodology, then the theoretical phases and results will be presented, based on several materials (Bellini, Rech, & Borenstein, 2004; Checkland, 2000; Curo & Belderrain, 2010; Heyer, 2004; Parrilla, Araújo Júnior, Belderrain, Bergiante, & Belderrain, 2018).

Figure 5: SSM processes.

Source: Adapted from Mingers and Rosenhead (2001).

It is important to note that other steps of the SSM were covered by the STPA method, like the hazards and losses that are re-evaluated and related to the unsafe control actions of the system.

3.1.1. Stage 1: Analyse and represent the problem (Rich Figure).

In this proposed stage, the first two steps of SSM are used, “Enter situation considered problematical” and “Express the problem situation”.

These two stages involve deeply the problematic situation and identifying people (actors), culture and norms, through interviews and discussions, observations, brainstorming, and “Rich Figures”.

Rich Figures are an ideal starting point for dealing with disordered situations, seeking to capture everything that is intended to know about the disordered situation without imposing any structure or analysis, launching an "uncovered" image. It portrays all the factors involved, ideas, people, structures, the whole situation analyzed. Besides, subjective elements can be represented, such as characteristics, feelings, conflicts, and prejudices (literally "pre-judgments") (Armson, 2011).

3.1.2. Stage 2: Hazards and losses identification.

Thus, first, research and interviews can be carried out to identify the problematic situation, and later Rich Figures are used to express the different elements pointed out by the main actors, finally listing the hazards and losses of the system.

3.1.3. Stage 3: CATWOE identification.

This stage is fundamental in SSM; it is about “Formulate root definitions of relevant systems of purposeful activity” (table 4).

Table 4: CATWOE of the Relevant System 1: Personnel exposure to risks in the CEPROS Laboratory.

CLIENT	Students and researchers are working on the Project in Laboratory.
ACTORS	Teachers, technicians, students (undergraduate and graduate), Aeronautics Institute of Technology (ITA) Work Security. Other actors: Financier, Ministry of Science, Technology, Innovations, and Communications (MCTIC), Suppliers of pressurized tanks.
Transformation	High exposure to risks in the Laboratory. → Low exposure to risks in the Laboratory.
WELTANSCHAUUNG	University laboratories are work environments whose objectives are focused on teaching and research. Thus, the safety approach under the aspects of the law, rules, and procedures contributes to the awareness of teachers and students about preventive practice.
OWNER	Command of Aeronautics and Rectory of ITA.
ENVIRONMENTAL CONSTRAINTS	- Total financial resources made available by FINEP. - Alignment of the activities schedule, consisting of the purchase of equipment, hiring of technicians, structuring, and planning of activities, tests, and treatment of results. - Safety plan for the operation, to mitigate risks, train personnel, and meet technical standards.
Root definition: A system that serves students and researchers; operated by professors, technicians, students (undergraduate and graduate) and ITA's job security; from high to low exposure to laboratory risks; due to the safety approach under the aspects of the law, rules, and procedures that contribute to the awareness of teachers and students; belonging to the Command of Aeronautics and Rectory of ITA; and operates in compliance with the total financial resources, alignment of the activity schedule and safety plan for the operation.

After drawing the Rich Figures, to define the relevant systems within the Systemic Thinking, the Root Definitions are identified. It is a sentence that describes the ideal system: the proposal, who is involved, who is interested, who it will be, or even be likely to be affected. In other words, root definition means saying what should be done, how it should be done, and why it should be done.

To construct the Root Definitions, the mnemonic acronym CATWOE is used (definition of each element):

· Customer or client: Who are the customers, victims, or beneficiaries?

· Actors: Who are the actors, protagonists, or participants in the system?

· Transformation process: What is transformed by this system (conversion of an input and an output)?

· Weltanschauung (Worldview): What view (perception) of the world covering up the system?

· Owner: Who owns the system (with the power to stop it)?

· Environmental Constraints: What are the (external) environmental restrictions allowed to the system?

These elements are implicit in the Root Definition of the problem. They must be identified to have a clear definition of the structure, limitations of the system, and the necessary transformation. Besides, there may be more than one CATWOE, considering the relevance and distinction of the Relevant Systems.

3.1.4. Stage 4: Analyse how to model the system.

Group discussions are then used to try to agree on a single applicable root definition or decide on several to open further considerations. Thus, it ends with a clear consensus of the entire system, keeping in mind the hierarchical structure of those involved and their responsibilities, and where intermediation will be necessary.

3.2. Phase 2 - Analysis of UCAs.

STPA is an approach developed from the STAMP model, in which the main losses and hazards of the system are preliminarily listed, followed by the design of a Control Structure using a process control model. Besides, systemic thinking is highlighted, with a hierarchical relationship of the whole in a top-down analysis.

3.2.1. Stage 1: Control Structure Modeling.

At this stage, steps 1 and 2 of the STPA are implementing. In step 1 of the STPA method, it is necessary to define the purpose of the analysis, identify hazards, losses and define system boundary (Leveson & Thomas, 2018).

Once the purpose is defined, step 2 of STPA is to model the Control Structure, considering the control actions and responses among the components, having: Controller (who sends the control action, with an algorithm in the case of a machine or a process model in the case of a person), Controlled Process (receives the control action from the Controller and executes), Actuator (will be the intermediary between the Controller and the Controlled Process) and Sensor (after the execution of the ordered action, the Controlled Process sends a message to the Controller).

Figure 6: Loop of the Process Control Model.

Source: Adapted from Leveson (2011).

Thus, it´s necessary to develop a hierarchical control structure, being an instance of the more general concept of system theory. The objective is to have defenses to eliminate or reduce losses (Leveson, 2003). Figure 6 shows a feedback control loop in a simple way; in it, the controller has requirements assigned to apply to the controlled process, which it does, by issuing control actions to change the state of the controlled process. For controllers in a safety control structure, the assigned requirements must ensure that defenses are maintained in the controlled process (Leveson & Thomas, 2018).

3.2.2. Stage 2: Identification of UCAs.

The next stage (such the step 3 of the STPA method) is to identify the UCAs, making the following distinction:

· TYPE 1: A necessary control action for safety is not provided (for example, the air traffic controller does not issue a warning essential to maintain safe separation);

· TYPE 2: An unsafe control action is provided, and leads to a hazard (for example, an air traffic controller issues a hazard that the accident has occurred);

· TYPE 3: A potentially safe control action is provided too late, too early, or out of sequence;

· TYPE 4: A safe control action is interrupted or applied excessively; for example, the pilot performs the required ascension maneuver, but continues after the flight level is reached (Leveson, 2011).

3.2.3. Stage 3: Identification of causal scenarios.

This stage (step 4 of STPA) identifies loss scenarios, possible causes to help analyze unsafe actions. The scenarios can then be used to eliminate some causes or, if it is not possible or practical, to mitigate. Prevention may involve altering any part of the control circuit, or the design of the controlled process, such as control actions, projected feedback, means of communication, among others (Leveson, 2011).

3.2.4. Stage 4: Definition of defenses.

The result of the STPA analysis is a list containing requirements and constraints to avoid unsafe control actions from occurring. These conditions occur when the operation becomes unsafe on a certain context. This list of defenses can be given to the decision-maker (Bjerga, Aven & Zio, 2016) and used to suggest and measure how to reduce or mitigate hazards.

3.3. Phase 3 - Prioritization of UCAs.

Following this proposed method integration, the TOPSIS Fuzzy is applied to prioritize alternatives, which according to Sodhi and Tadinada (2012) and Kore, Ravi, and Patil (2017), the application stages can be summarized as follows.

3.3.1. Stage 1: Defining criteria for evaluating UCAs.

Considering a group with k decision-makers. The decision-making problem in a fuzzy multicriteria group can be consistently expressed by the decision matrix , Eq. 2, with m alternatives and n criteria.

(2)

In this analysis, the alternatives represent the UCAs, and the criteria were divided into probability, impact, and detectability. This concept is similar to the FMEA method, which establishes three indexes to score the risk: Occurrence (defines the frequency of failure); Severity (corresponds to the severity of the failure); and Detection (facility to detect the fault before it occurs) (Amaral, Amaral, & Nunes, 2010).

3.3.2. Stage 2: Scoring aggregation and normalization.

The linguistic variable is defined by a triangular fuzzy function (with the minimum, medium, and maximum values). Since is the number of decision-makers (with , the index the alternative, which in the problem in question are the () and the index the evaluation criteria ().

The aggregation of the individual matrices of the k decision-makers is obtained by Eq. 3, resulting in the aggregate matrix for each alternative, Eq. 4.

(3)

(4)

The weights of each criterion are assigned, , separated into the minimum, medium, and maximum. The aggregation of weights, by the decision-makers to the criteria, is obtained similarly, by Eq. 5 and can be represented by Eq. 6.

(5)

(6)

From the aggregate matrix normalization is performed, resulting in matrix , which is obtained by Eq. 7 and Eq. 8, respectively, for benefit and cost criteria. The elements of .

(7)

Being for the criterion of the most significant benefit, that is, the higher, the better. For example, how greater the detectability of the unsafe action, it’s better.

(8)

And is the opposite, that is, if bigger, its worse. For example, the higher the probability of the occurrence and impact of the unsafe action, the worse it is.

Besides, this method preserves the property of normalization of the fuzzy triangular numbers, belonging to the scale between 0 and 1.

The aggregated and normalized matrix, is weighted by the vector of aggregated weights, according to Eq. 9.

(9)

3.3.3. Stage 3: Prioritization of UCAs.

The determination of FPIS and FNIS is obtained by Eq. 10 and Eq. 11.

FPIS:

, wherein: (10)

FNIS:

, wherein: (11)

Calculation of the Euclidean distance between the UCAs and the FPIS and FNIS. The Euclidean distance between two fuzzy triangular numbers, and , is obtained by Eq. 12. Being , .

(12)

The distance from the UCAs to the ideal positive fuzzy point is obtained by Eq. 13.

(13)

Similarly, the distance to the ideal negative fuzzy point is obtained by Eq. 14.

(14)

As soon, the coefficient of the proximity of the UCAs to the ideal positive fuzzy point is obtained, according to eq. 15.

(15)

Being, ∈ [0,1].

3.3.4. Stage 4: Analysis and validation of defenses.

Thus, with the application of the TOPSIS Fuzzy method, there is the prioritization of the UCAs and possible validation of the model, with a list of defenses already ordered according to the UCAs.

4. RESULTS

Brazil has invested in projects that enable the development and launch of satellites for different purposes, minimizing the dependence on supplier countries and expanding national results. The Laboratory of Injection Systems for Liquid Propellants (CEPROS) was created in 2012, with the support of researchers from ITA. The Institute for Advanced Studies (IEAv), the Institute of Aeronautics and Space (IAE), and the National Institute for Space Research (INPE) created a cooperation network on liquid propulsion, with the objective of developing a combustion chamber powered by ethanol and cryogenic oxidizer.

In this study, systemic mapping was sought to identify the critical areas for the proper functioning of the academic laboratory in question, in the standards and legal requirements. Based on the context of empirical safety analysis of this laboratory and interviews with the two main professors in charge, the SSM, STPA, and TOPSIS Fuzzy methods were applied.

4.1. Phase 1 - Structuring the CEPROS Laboratory problem.

Next, following the methodological structure, the proposed steps were applied and the main results found will be presented.

4.1.1. Stage 1: Analyse and represent the problem (Rich Figure).

The CEPROS Laboratory contains the primary support of professors, students, and researchers from ITA and searches for information on the behavior of components of a liquid propellant combustion chamber of rocket engines. In this project, there is a need for studies on hazard identification, because the professors work in the planning of CEPROS Laboratory procedures, with attention to safety and reports for proper accountability for the Financier.

Checkland's SSM (1981) mentions the use of a Rich Figure (shown in Figure 7) as the most common tool, which was assembled based on the interview with the professors.

Figure 7: Rich Figure.

It is possible to observe that the professors, responsible for the laboratory, contribute to the study of the students, as well as, have to prepare the teaching activities (materials, presentations, correction of exercises and tests, ...), and are responsible for the laboratory activities (research procedures, hiring staff, filling in reports, ...). As this is a public-funded project, the Funder must be regularly accounted for, as he or she evaluates the proposals, results and authorize financial resources. The Support Foundation provides administration resources and hires technicians to work in the laboratory, including a Work Safety Technician who monitors compliance with rules and laws (especially in tests). Similarly, laboratory activity requires Air Force Command authorization, and all activities are monitored at a high level.

4.1.2. Stage 2: Hazards and losses identification.

From all these interactions and study, which is reflected in Rich Figure, it is possible to identify losses (for example, loss of human life and damage of equipment or infrastructure) and hazards (for example, leaking oxygen in the laboratory and contact with a spark) that is reflected in Relevant System 1: Personnel exposure to risks in the CEPROS Laboratory.

4.1.3. Stage 3: CATWOE identification.

This stage uses SSM for “Formulate root definitions of relevant systems of purposeful activity”. This article will present the CATWOE of the transformation “High exposure to risks in the Laboratory → Low exposure to risks in the Laboratory.”

4.2. Phase 2 - Analysis of UCAs for the CEPROS Laboratory.

The XSTAMPP software, version 4.7.3, was used to support the application of the STPA model, an open-source platform for application in the field of safety engineering (Abdulkhaleq, Wagner & Leveson, 2015).

4.2.1. Stage 1: Control Structure Modeling.

In the beginning, the Control Structure was set up, with the identification of 12 actors and 15 loops, based on the information previously collected. In Figure 8, the control loop between Technicians and the Laboratory, in the function of Controller and Controlled Process, is shown.

Figure 8: Control Structure (developed at the XSTAMPP software).

4.2.2. Stage 2: Identification of UCAs.

One of the first steps of the STPA method is shown in Figure 9, and it is possible to observe the UCAs of the Control Action “Start the experiment sequence”.

Figure 9: UCAs of the Control action "Start the experiment sequence" (developed at the XSTAMPP software).

4.2.3. Stage 3 and 4: Identification of causal scenarios and Definition of defenses

Subsequently, it is possible to identify causal scenarios and measures to improve system safety, as shown in Figure 10, which took UCA1 as an example. Due to the "Technician does not trigger the start of the experiment sequence when all machines are ready", there would hazard and causal scenarios, with possible defenses.

Figure 10: Causal scenarios for Control action “Start the experiment sequence” (developed at the XSTAMPP software).

4.3. Phase 3 - Prioritization of CEPROS Laboratory UCAs.

Following, from the UCAs identified by the professors (mainly responsible for the Laboratory), in the loop between Technicians and Combustion chamber for UCA1, the TOPSIS Fuzzy method was used.

4.3.1. Stage 1: Defining criteria for evaluating UCAs.

In Fuzzy theory, conversion scales are applied to transform linguistic terms into fuzzy numbers. Thus, the criteria and alternatives can be classified on a scale of 1 to 9, for example, and the intervals are chosen to have a uniform representation for the Fuzzy triangular numbers. Table 5 presents the five linguistic classifications used to analyze this problem.

Table 5: Linguistic variables based on the Fuzzy Theory.

FUZZY numbers	Evaluation of Alternatives			Weights
	Probability	Impact	Detectability
1,1,3	Very low (E)	Very low (E)	Very high (A)	Very low (VL)
1,3,5	Low (D)	Low (D)	High (B)	Low (L)
3,5,7	Average (C)	Average (C)	Average (C)	Average (AA)
5,7,9	High (B)	High (B)	Low (D)	High (H)
7,9,9	Very high (A)	Very high (A)	Very low (E)	Very high (VH)

For this work, an interview was carried out with 2 (two) professors of the CEPROS laboratory for Fuzzy classification and to define the importance weight of each decision-maker about alternative , and criterion (according to Tables 6 and 7).

For example, for Decision 1 at UCA1, the probability was seen as High (B), impact as Very High (A), and the detectability as Low (D).

4.3.2. Stage 2: Scoring aggregation and normalization.

In this phase, for each , given the values of Tables 6 and 7 with the Professors responses, the minimum, average, and maximum general values of each alternative by criterion, as Eq. 3 and Eq. 4, and for weights Eq. 5 and Eq. 6. According to Sodhi e Tadinada (2012), in this way a Fuzzy multicriteria problem of Group Decision Making can be expressed concisely in matrix format (Table 8), the linear scale transformation is used to transform the alternatives on a comparable scale, resulting in an aggregated Fuzzy decision matrix.

Table 6: Decision-maker 1.

UCA	Probability	Impact	Detectability
UCA1	High (B)	Very high (A)	Low (D)
Weight w	Very high (VH)	Very high (VH)	High (H)

Table 7: Decision-maker 2.

UCA	Probability	Impact	Detectability
UCA1	Very low (E)	Very high (A)	Average (C)
Weight w	Average (AA)	Very high (VH)	Very high (VH)

Table 8: Aggregate Matrix.

UCA	Probability			Impact			Detectability
UCA1	1	4	9	7	9	9	3	6	9
Weight w	3	5	7	7	9	9	3	5	7
	Cost Criterion			Cost Criterion			Benefit Criterion

In this phase, each is found, where each of the results of the Probability and Impact criteria in Table 8 is divided by (this is the overall maximum value) because how greater the Probability and the Impact, than will be the prioritization; against the Detectability criterion, each of the results was divided by (this is the general minimum value), because how lower Detectability, than greater will be the degree of prioritization, the results of this analysis are presented in Table 9.

Thus, is the benefit criterion, which considers the maximum value found in , according to Eq. 7.

And, is the cost criterion, which considers the minimum value found in , according to Eq. 8.

Table 9: Normalized Matrix.

UCA	Probability			Impact			Detectability
UCA1	0,11	0,44	1,00	0,78	1,00	1,00	0,11	0,17	0,33
Weight w	3	5	7	7	9	9	3	5	7
	Cost Criterion			Cost Criterion			Benefit Criterion

A weighted normalized Fuzzy decision matrix is calculated by multiplying the weights of the assessment requirements with a normalized Fuzzy decision matrix , according to Eq. 9 and Table 10.

Table 10. Normalized and Weighted Matrix.

UCA	Probability			Impact			Detectability
UCA1	0,33	2,22	7,00	5,44	9,00	9,00	0,33	0,83	2,33
	Cost Criterion			Cost Criterion			Benefit Criterion

4.3.3. Stage 3: Prioritization of UCAs.

The FPIS and FNIS of the alternatives are defined based on Eq. 10 and Eq. 11. Among the alternatives (UCAs) for each criterion, in the column with the maximum values, the one with the highest value is selected (). In the column with the minimum values, the one with the lowest value () is selected. In this way, the minimum and maximum values are found.

The distance ( and ) of the weight of each alternative of FPIS and FNIS is computed as Eq. 12. The results are shown in Table 11, and for each UCA, the FPIS and FNIS are found. As for the Fuzzy triangular distance, let = (a, b, c) and = (a′, b′, c′), two Fuzzy triangular numbers find the distance between them, according to Eq. 13 and Eq. 14.

Table 11: Calculation of FPIS and FNIS distances.

Criterion	FPIS UCA 1	FPIS UCA 2	FPIS UCA 3	FPIS UCA 4	FNIS UCA 1	FNIS UCA 2	FNIS UCA 3	FNIS UCA 4
Probability	4,74	5,25	5,25	4,06	4,00	2,19	2,19	4,24
Impact	2,05	3,01	3,01	3,01	4,27	3,79	3,79	3,79
Detectability	5,89	6,20	4,89	6,20	1,19	0,65	3,93	0,65
Sum =	12,68	14,45	13,14	13,27	9,46	6,63	9,91	8,68

In the end, the proximity coefficient Cpi is calculated, which represents the distances to the positive ideal solution , and the negative ideal solution simultaneously. The result of the proximity coefficient for each alternative is calculated by Eq. 15.

Table 12 shows the ordering of the UCAs. Since, as the scores of the proximity coefficients for alternatives are numerical values, they can be used to indicate the degree of inferiority or superiority of the alternatives among themselves.

Table 12: Cpi Proximity Coefficient.

UCA	Cpi	Classification
1	0,427	2
2	0,315	4
4	0,395	3
3	0,430	1

4.3.4. Stage 4: Analysis and validation of defenses.

As the STPA method has already been adopted as general defenses, this step is carried out for final validation. For each UCA, a safety action is possible, as shown in Table 13.

Table 13: List of UCAs and defenses.

UCA	Defenses
1 – Technician does not trigger the start of the experiment sequence when all machines are ready.	Monitoring of the responsible professor in the test operations.
2 - Technicians triggers the start of the experiment sequence when he notices an error.	Periodic safety training for Laboratory Technicians, and follow the checklist before starting the activities.
3 - Technician triggers the beginning of the experiment sequence when the operation is interrupted.	The teacher in charge should prepare the procedures before starting the experiment and being attentive to details of operation (including external environments, such as power failure due to grid maintenance).
4 - Technicians triggers the start of the experiment sequence without first checking all the machines.	Establish a list of safety check-in and activities to be performed.

In the end, it is recommended that the Decision Maker establishes a cut-off criterion, that is, a Hierarchy of Management and Control of the identified UCAs. Even though that those responsible for the actions are already established, on a day-to-day basis, teachers and laboratory technicians could monitor compliance with defenses and carry out the collection for more significant benefit to the project.

Thus, the professor responsible for the laboratory and the technicians who monitor daily activities, based on the priority list of the 48 identified UCAs, could verify compliance with preventive action, based on the Pareto diagram, for example. According to this method, 80% of the consequences come from 20% of the causes, helping to address nonconformities, identify points for improvement and define action plans that must be sent first (Pedrosa Filho, 2016). Thus, if the Pareto Diagram concept were adopted, the first 9 UCAs, which represent 20% of the causes of accidents, would be given higher priority.

5. DISCUSSIONS AND CONCLUSION

The control and prevention of hazards need to be constant, and it is imperative to plan and take action, through the STPA method it was possible to establish a roadmap for modeling UCAs in systemic thinking. Nevertheless, this study also addressed the lack of a method for the initial consensus of the problem and final prioritization of actions.

With the presentation of a structure composed of three phases: Structuring the Problem (with the identification of the scope, actors, necessary transformation in the system, hazards, and accidents); Analysis of UCAs (with extensive analysis and assessment of hazards and accidents, setting up a hierarchical structure, identification of UCAs, causal scenarios, and defenses) and Prioritization of UCAs (establishment of criteria, analysis of epistemic uncertainty, aggregation of data when deals with a group decision, prioritization, and validation of actions to mitigate hazards).

Methods for structuring problems were studied since most of the difficulties pointed out in research projects were in structuring models, in the planning and in the administration of the time, costs, and other resources. The selection of the SSM is justified because this methodology is oriented to systemic thinking. Since its beginning, it was designed for preliminary analysis and to bring practical value to the administrators, analyzing the changes that would need to be made in the processes or attitudes.

Besides, to prioritize the results, the TOPSIS Fuzzy method was selected, because starting from qualitative analysis, supporting innovative projects that do not have a historical base with probabilities. Also, to enabling the aggregation of data in the own steps for the group decision and, in the end, prioritization based on criteria and alternatives, which are closer to the ideal solution, take into account the imprecision of human evaluation in the process, with the use of linguistic variables and the Fuzzy logic, which supports the classification alternatives and data integration process.

It is also noteworthy that through the literature review, it was possible to verify which methods would best achieve the objective. Thus, the proposed methods follow the line of sociotechnical systems, considering the difficulty of the decision-maker for risk analysis in aerospace research projects. Therefore, different hazard analysis methods were studied, in which the STPA method stood out for modeling the system, and identifying UCAs, and causal scenarios.

Finally, the integration of the methods, SSM and TOPSIS Fuzzy provides risk analysis that fills the gaps of the STPA method, in the identification of hazards and losses or accidents, addressing the lack of considering risk aspects, that is, incorporating uncertainty into the process, by combining the criteria extracted from the FMEA and the fuzzy technique.

Thus, the objective of this research was achieved, providing a coordinated structure for analyzing the context and refining the information available to decision-makers, and finally prioritizing unsafe actions that could cause more significant damage to the system.

ACKNOWLEDGEMENTS

Acknowledgement to Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) for partial support of this research.

REFERENCES

Abdulkhaleq, A., Wagner, S., & Leveson, N. (2015). A Comprehensive Safety Engineering Approach for Software-Intensive Systems Based on STPA. In de B. R. J. & K. N. (Eds.), 3rd European STAMP Workshop, STAMP EU 2015 (Vol. 128, pp. 2–11). Institute of Software Technology, University of Stuttgart, Stuttgard, Germany: Elsevier Ltd. doi: 10.1016/j.proeng.2015.11.498

ABNT. (2018). Gestão de riscos - Diretrizes, NBR ISO 31000. Associação Brasileira de Normas Técnicas (ABNT).

Amaral, É. H. do, Amaral, M. M., & Nunes, R. C. (2010). Metodologia para Cálculo do Risco por Composição de Métodos. X Simpósio Brasileiro Em Segurança Da Informação e de Sistemas Computacionais. 10(1), 460-473. Retrieved from http://ceseg.inf.ufpr.br/anais/2010/06_artigos_completos/artigo_37.pdf

Armson, R. (2011). Growing wings on the way: systems thinking for messy situations. Axminster: Triarchy Press.

Bellini, C. G. P., Rech, I., & Borenstein, D. (2004). Soft Systems Methodology: uma aplicação no “pão dos pobres” de Porto Alegre. RAE Eletrônica, 3(1), 1-22. doi: 10.1590/S1676-56482004000100007

Bjerga, T., Aven, T., & Zio, E. (2016). Uncertainty treatment in risk analysis of complex systems: The cases of STAMP and FRAM. Reliability Engineering and System Safety. 156(1), 203–209. doi: 10.1016/j.ress.2016.08.004

Campolina, A. G., Soárez, P. C. de, Amaral, F. V. do, & Abe, J. M. (2017). Análise de decisão multicritério para alocação de recursos e avaliação de tecnologias em saúde: tão longe e tão perto? Cadernos de Saúde Pública. 33(10), 1-15. doi: 10.1590/0102-311X00045517

Carbognin, B. (2017). Metodologia de verificação de sequência operacional em completação de poços baseada em interdependências (Universidade Estadual de Campinas-UNICAMP, master thesis). Retrieved from UNICAMP Online Research Acess Service (http://repositorio.unicamp.br/bitstream/REPOSIP/324309/1/Carbognin_Breno_M.pdf)

Checkland, P. B. (2000). Soft Systems Methodology: A Thirty Year Retrospective. Systems Research and Behavioral Science. 17(1), 11-58.

Chen, C.-T. (2000). Extensions of the TOPSIS for group decision-making under fuzzy environment. Fuzzy Sets and Systems. 114(1), 1–9. doi: 10.1016/S0165-0114(97)00377-1

Chenci, G. P., Rignel, D. G., & Lucas, C. A. (2011). Uma introdução a lógica fuzzy. Revista Eletrônica de Sistemas de Informação e Gestão Tecnológica. 1(1), 1-12.

Costa, R. F. P. da. (2012). Utilização de Metodologias Multicritério de apoio à decisão como ferramenta de suporte numa empresa de serviços energéticos (Instituto Superior de Engenharia do Porto, master thesis). Retrieved from https://core.ac.uk/download/pdf/302861858.pdf

Curo, R. S. G., & Belderrain, M. C. N. (2010). Uma aplicaçao do SSM para estruturar o problema da produçao científica de um curso de ensino superior. Conference: XVII Simpósio de Engenharia de Produção. 17(1), 1-11.

Ensslin, S. R. (2002). A incorporação da perspectiva sistêmico-sinergética na metodologia MCDA-Construtivista: uma ilustração de implementação. (Doctoral dissertation). Retrieved from UFSC Online Research Acess Service (https://repositorio.ufsc.br/handle/123456789/82357)

Figueira, J., Greco, S., & Ehrgott, M. (2016). Multiple Criteria Decision Analysis: State of the Art Surveys (1st ed.; G. Salvatore, Ed.). New York: Springer New York. doi: 10.1007/b100605

Hanafizadeh, P., & Mehrabioun, M. (2018). Application of SSM in tackling problematical situations from academicians’ viewpoints. Systemic Practice and Action Research, 31(2), 179–220. doi: 10.1007/s11213-017-9422-y

Heyer, R. (2004). Understanding Soft Operations Research: The methods, their applications and its future in the Defence setting. Australian: Australian Goverment. Retrieved from https://apps.dtic.mil/dtic/tr/fulltext/u2/a428464.pdf

Hollnagel, E. (2004). Barriers and Accident Prevention (1st ed.). Ashgate: Routledge. Retrieved from https://www.tandfonline.com/doi/abs/10.1080/00140130600971077?journalCode=terg20

Hwang, C.-L., & Yoon, K. (1981). Multiple Attribute Decision Making (1st ed.; T. & F. Group, Ed.). Heidelberg: Springer Berlin Heidelberg. doi: 10.1007/978-3-642-48318-9

Ishizaka, A., & Nemery, P. (2013). Multi-Criteria Decision Analysis. Chichester: John Wiley & Sons Ltd. doi: 10.1002/9781118644898

Kore, M. N. B., Ravi, K., & Patil, S. B. (2017). A Simplified Description of FUZZY TOPSIS Method for Multi Criteria Decision Making. International Research Journal of Engineering and Technology (IRJET), 4(5), 2395–56. doi: 2395-0072

Leveson, N. G. (2003). A new approach to hazard analysis for complex systems. Conference of the. System Safety Society. 20(1), 24-34.

Leveson, Nancy G. (2004). A new accident model for engineering safer systems. Safety Science. 42(4), 237–270. doi: 10.1016/S0925-7535(03)00047-X

Leveson, Nancy G. (2011). Engineering a Safer World: Systems Thinking Applied to Safety (Engineering Systems) (1st ed.). Cambridge: MIT Press. doi: 10.1017/CBO9781107415324.004

Leveson, Nancy G. (2013). An STPA Primer. Retrieved from https://fliphtml5.com/sgqs/syzv/basic

Leveson, Nancy G. (2015). A systems approach to risk management through leading safety indicators. Reliability Engineering & System Safety, 136, 17–34. doi: 10.1016/j.ress.2014.10.008

Leveson, Nancy G., & Thomas, J. P. (2018). STPA Handbook. Retrieved from http://psas.scripts.mit.edu/home/get_file.php?name=STPA_handbook.pdf

Lima Junior, F. R., & Carpinetti, L. C. R. (2015). Uma comparação entre os métodos TOPSIS e Fuzzy-TOPSIS no apoio à tomada de decisão multicritério para seleção de fornecedores. Gestão & Produção, 22(1), 17–34. doi: 10.1590/0104-530X1190

Mingers, J., & Rosenhead, J. (2004). Problem structuring methods in action. European Journal of Operational Research, 152(3), 530–554. doi: 10.1016/S0377-2217(03)00056-0

Parrilla, F. R., Araújo Júnior, L. S. de, Belderrain, C. M. N., Bergiante, N. C. R., & Belderrain, M. C. N. (2018). Estruturação do problema da baixa motivação do aluno em uma instituição de ensino superior privada. Revista Gestão Em Engenharia. 5(1), 1–18.

Pedrosa Filho, L. E. (2016). Análise de acidentes de trabalho como ferramenta de gestão de segurança em uma empresa de transporte ferroviário (Universidade Federal de Juiz de Fora-UFJF). Retrieved from UFJF Online Research Acess Service (http://www.ufjf.br/engenhariadeproducao/files/2015/10/luizeduardopedrosafilho.pdf)

Picanço, A. R. S., Jeske, M., Belderrain, C., Neto, L. L. de S., & Bergiante, N. (2017). Ranqueamento de criticidade global de equipamentos por meio de análise de decisão multicritério. Oficina Nacional de Problemas de Corte e Empacotamento, Planejamento e Programação de Produção e Correlatos – ONPCE. 8(1), 1-17.

PMI. (2013). Um guia do conhecimento em gerenciamento de projetos (PMBOK, 6^th edition). Newtown Square: Project Management Institute.

Rasmussen, J. (1997). Risk management in a dynamic society: a modelling problem. Safety Science, 27(2–3), 183–213. doi: 10.1016/S0925-7535(97)00052-0

Rosenhead, J., & Mingers, J. (2001). Rational Analysis for a Problematic World Revisited: Problem Structuring Methods for Complexity, Uncertainty, and Conflict. Chichester: Wiley. doi: 10.1016/j.ejor.2004.03.004

Simonsen, J. (1994). Soft Systems Methodology – An Introduction. Roskilde: Spring.

Sodhi, B., & T., P. (2012). A Simplified Description of Fuzzy TOPSIS. Computing Research Repository – CoRR. 1(2), 1-4.

Water, H. van de, Schinkel, M., & Rozier, R. (2007). Fields of application of SSM: a categorization of publications. Journal of the Operational Research Society. 58(3), 271–287. doi: 10.1057/palgrave.jors.2602156