Identification and characterization of "Black Swans" in technological events in Brazil

Moacyr Machado Cardoso Junior

Instituto Tecnológico de Aeronáutica, Brazil

E-mail: moacyr@ita.br

Submission: 04/09/2018

Revision: 25/09/2018

Accept: 01/10/2018

ABSTRACT

“Black swan” events represent a critical issue in risk analysis. Events with extremely low probability of occurrence are in general discarded from the risk analysis process. This paper aims to identify and characterize four accidents that occurred in Brazil into the following classes: “not a black swan”, “black swan: unknown-unknown”, “black swan: unknown-known” and “black Swan: not believed to occur”, by obtaining from experts the distribution of belief for the real probability of each class. Results showed that, throughout all cases analyzed, the class “black swan: unknown-unknown” was never reported, which means that none of the cases studied were a complete surprise to anyone. The method used was able to assign all accident events to the remaining classes. Probability distribution elicited from experts showed large disagreement among them, and the expected value was considered low. Nevertheless, the elicited distributions can be utilized in future risk analysis as a priori distribution in a Bayesian approach.

Keywords: Black Swan; Expert elicitation; Technological accidents; Risk analysis

1. INTRODUCTION

In risk assessment context, specialists are demanded for the risk assessment process, which involves identification, analysis, evaluation, and decision-making about possible controls and/or mitigating measures from inherent risks of a process, system, equipment, and others (ABNT-ISO, 2009).

In the identification phase, all threats and/or hazards inherent to the system are raised. This phase is crucial because unidentified potential events will not be analyzed and will not be part of the decision-making process. In the risk analysis phase, causes and their sources are evaluated, and the consequences of an unwanted event are estimated. The threefold cause-event-consequence defines one risk, and for each risk, its event likelihood is determined associated to a cause and the consequences of its materialization. In the last phase, risk evaluation, tolerance criteria, or attitude towards risk are compared with values obtained in the risk analysis phase. This comparison helps the decision-making process about risk treatment, as well as its prioritization.

Events described by Aven, (2013) and Flage and Aven, (2015) as Black Swans are characterized by the following attributes: They are considered outliers, that is, they represent extremely improbable events with extreme consequences, and as a rule, after their materialization they are perfectly explicable and predictable by the experts, which represents a paradox.

These events represent a real threat to the risk assessment process, as they will not be part of the analysis, or if they are identified, they will be discarded after that because of their extremely low likelihood. Therefore, new strategies have to be developed to incorporate and maintain these events in risk analysis process as described by (AVEN, 2015).

A fundamental step towards this direction is to understand how these events are perceived by experts. Thus this paper aims to identify and classify different black swan events that occurred in technological accidents in Brazil based on elicitation from experts, and consequently, point to general strategies that could help risk analysts to cope with “Black Swans”.

This paper is structured as follows: Beyond this short introduction and contextualization, are sections: Section 2 with the theoretical foundation about black swan events and elicitation tools. In section 3, the research method is presented. Section 4 contains the results and discussion. Finally in section 5, the final considerations and proposals for new research on this theme are presented.

2. THEORETICAL FOUNDATIONS

2.1. “Black Swans”

Black swan event theory is derived from a metaphor that describes an event that comes as a surprise and has a huge negative impact. The term "Black Swan" comes from the idea that these animals would not exist in nature, which was later contradicted by the discovery of these animals in Australia in 1967 (TALEB, 2010).

According to Aven, (2013), the term was popularized by Taleb, (2007) in his book "The Black Swan", and highlighted three attributes associated with the term: (i) It is an outlier, in the sense that one cannot expect the occurrence of that event based on previous events; (ii) It involves an extreme impact; and (iii) After its occurrence people and experts invent logical explanations, making the event explicable and predictable.

The first attribute is questioned by some researchers because if the event is an outlier in the interpretation of probability theory, even small probabilities of occurrence can be expected to occur given the time-space considered.

The concept of "Black Swan" can be seen in two ways according to Aven, (2013): (i) a rare event with extreme consequences; and (ii) an extreme, astonishing event concerning current knowledge and/or belief, and that the latter may be more appropriate.

"Black Swan" events can be classified into three basic types: (i) events that are completely unknown in the scientific milieu, termed unknown-unknown, as a reference to the fact that they are unknown to risk analysts and science; (ii) Events that are not on the list of risk analysts, but are on the list of experts and/or science (unknown-known); (iii) Events known to risk analysts, but judged to have negligible probability of occurrence and therefore are not amenable to analysis because they are judged as "not believed to occur" (FLAGE; AVEN, 2015).

The authors also cite a few examples of the three types: The use of thalidomide in 1957, a drug that caused congenital malformation of the upper limbs, which was totally unknown to physicians and scientists. The attack on the Twin Towers on September 11 can be considered as type (ii), and finally the tsunami that destroyed the Fukushima nuclear reactor is an example of type (iii).

In Figure 1, the conceptual map for the focal question is presented: what are "Black Swans"? Which summarizes what has been discussed up to this point.

Figure 1: Conceptual map of “Black Swan”.

Another concept is "Perfect Storm" to designate a rare event involving uncertainty, and this is represented by the randomness of joint but known events. It differs from the term "Black Swan" because the latter involves epistemic uncertainty or lack of knowledge, i.e. not only the lack of knowledge of the distribution of probabilities but the ignorance of the phenomenon itself (PATÉ-CORNELLl, 2012). The author concludes that proactive management with early warnings, quick detection, and mainly agile responses enable analysts to comply with Augustine's Law XLV, i.e. "One should expect that the expected can be avoided, but the unexpected should have been expected".

2.2. Probability elicitation

Elicitation is a process of constructing probability distributions from the extraction of beliefs and expert knowledge about one or more uncertainties.

Much of the literature on elicitation is concerned with constructing a probability distribution to model uncertainties when there is insufficient data to construct a model (GARTHWAITE; KADANE; HAGAN, 2005). As an example, the authors cite the case of decision making where the uncertainty regarding the theme must be represented by a distribution of probabilities in order to maximize the expected utility.

People are affected by heuristics and biases in how they respond to uncertainty issues (GARTHWAITE et al., 2005). Some heuristics that influence the elicitation process are described by Burgman, et al., (2006). One of them is the representativeness, that is, as the number of details in a given scenario increases, its probability may only decrease, although due to its representativeness apparent probability grows. Another heuristic cited by the authors is availability, that is, common events, or more likely, or more recent or even those that have been very explored by the media. Finally, the anchorage heuristic states that when a person is asked to estimate a number, percentage or range of values, people anchor in values that have been previously suggested or have arisen from other judgments.

The great advantage of the elicitation process is to use the results for the decision-making process, and in this context, the capture of expert opinion is fundamental as well as the possibility of constructing the prior distribution for inference in a Bayesian process (GARTHWAITE et al., 2005).

Several authors used probability elicitation in different applications like probabilities of explosion in different scenarios (MACDONALD; SMALL; MORGAN, 2008). They asked experts for estimates of the upper (U) and lower (L) bounds of probability, and next asked them to give the mode or most likely value, and then divided the interval [L,U] into six subintervals. The authors found lack of consensus among experts. Predictive elicitation of subjective probability distributions was used to evaluate the effectiveness of Risk Control Option (RCO) for reducing the risk of ship collisions in Australia's Territorial Sea and Exclusive Economic Zone (HOSACK; HAYES; BARRY, 2017).

Ioannou et al., (2017) and elicited 13 international experts on the responses of a generic mid-rise cast-in-place reinforced concrete frame when exposed to different fire intensities, and then asked them to judge the level of response that would be required to cause a given level of damage. Elicitation offers a feasible method to generate evidence for the missing information, but a number of key issues must be analyzed in a real elicitation process, like weighting, aggregation, and others (BOIKE et al., 2010).

In an expert elicitation of climate, energy, and economic uncertainties, (USHER; STRACHAN, 2013) found that while experts agreed on the structure of the uncertain parameters, the shape of the distributions representing their beliefs varied widely, which reflects the different perspectives of the interviewees. Decomposing the structure of the parameters and exploring the influence of dependence on expert responses may help explain some of these differences. However, the pooled beliefs are insensitive to the weighting assumptions that compensate for bias and correlations within and among experts.

One of the usual methods for probability elicitation is quartiles, in which experts are asked to assign the median value for the distribution and then evaluate other points of the distribution. Quartile methods are the best for damping heuristics and biases from experts, according to (GARTHWAITE; KADANE; HAGAN, 2005; MORRIS; OAKLEY; CROWE, 2014)

Aggregation of expert distribution typically rely on very simple combination schemes, such as ascribing equal weight to all the participating experts. (ASPINALL, 2005). Although other forms like Cooke´s classical method, which is a weighted arithmetic average of the experts’ probability distributions, many simulated studies were no better than the simplest simple arithmetic average (CLEMEN, 2008).

3. METHOD

To identify and characterize Black Swan events, an expert elicitation research questionnaire was carried out based on technological accidents segregated by type: Aircraft accident – GOL-Legacy in 2006; Construction Industry - Tim Maia bicycle pathway in 2016 and Metro-SP in 2007; and Fire - Terminal Alemoa in 2015, which are explained below.

3.1. Event: Aircraft Accident - Gol 1907 – Legacy – year: 2006

Gol Flight 1907 was a domestic commercial route, operated by Gol Airlines, using a Boeing 737-8EH. On September 29, 2006, the aircraft departed from the Eduardo Gomes International Airport in Manaus to the Galeão International Airport in Rio de Janeiro and scheduled a stop at the Juscelino Kubitschek International Airport in Brasilia. As it flew over Mato Grosso state, it collided mid-air with an Embraer Legacy 600. All 154 passengers and crew aboard the Boeing 737 died after the aircraft collided in the air and crashed into a closed forest area. However, the Legacy, despite having suffered severe damage to its wing and horizontal left stabilizer, landed safely with its seven uninjured occupants at the Air Base of Cachimbo. CENIPA concluded that the accident was caused by both air traffic controllers and Legacy pilots errors, while the National Transportation Safety Board (NTSB) determined that all pilots acted correctly and were placed on a collision route by a variety of air traffic controller errors (WIKIPEDIA, 2017).

3.2. Event: Construction Industry: Accident on the Tim Maia bicycle pathway in Rio de Janeiro- year: 2016.

An section of the Tim Maia bicycle pathway, which was opened in January 2016 in the São Conrado neighborhood of Rio de Janeiro, collapsed on the morning of January 21. Two men died. According to one witness, a series of strong waves could have hit the bike lane before the crash. "The wave swept the track, which fell apart like paper," the man explained. Another witness said he saw three people fall into the sea after the collapse (UOL, 2016).

According to Oliveira, (2016), one of the main conclusions of the civil engineers involved in the study is that the design of the project failed, because preliminary oceanographic studies of the effects of waves on the structure of the bicycle lane were lacking.

3.3. Event - Construction Industry: Accident in the work of the Metro – SP – year: 2007

On January 12, 2007, the work of the future Pinheiros station of line 4 (yellow) of the São Paulo subway collapsed. The accident, according to the builders responsible for the work, occurred due to the instability of the region's soil, aggravated by the heavy rains that hit the city days before. The event culminated in the deaths of 7 people and a total of 271 people were affected due to the need for resettlement (FOLHA DE SÃO PAULO, 2007).

According to the G1 news portal, the defendants were cleared of any criminal offense in a court decision in May 2016. The prosecutor's office defended in the complaint that the employees were negligent. The complaint states that problems were detected in the tunnel in the month prior to the tragedy and, the day before, the decision of those responsible for the work was to install reinforcing structures. However, the work continued without the installation of those structures (G1, 2016a).

A report in the same news portal stated that the judge said the evidence proved that the accident would probably have happen even with the installation of the reinforcing structures. He also said that there was no indication of the accident and that those responsible took the necessary care.

"Now the accused had no way of predicting the accident, because of all the circumstances. The execution of the work project was within normalcy, all the teams carefully monitored each step of the execution and did not point out any situation that indicated the possibility of an accident."

The prosecution said in 2011 that there was recklessness, malpractice, human and technical error. The process questioned the quality of the material used and the neglect of preventive measures and failures in soil analysis. In other words, for the prosecution, the tragedy could have been avoided if those responsible had alerted the authorities and interrupted the work in time.

3.4. Event: Fire in Alemoa Terminal – year: 2015

The fire at the company Ultracargo began around 10 am on April 2 and was extinguished on April 9, 2015. Six fuel tanks were hit, but nobody was injured. At the start of the fire, the temperature reached 800 °C. Federal Government assistance was required, and fire-fighting products had to be imported to stop the flames. The Environmental Company of the State of São Paulo (Cetesb) fined the company R$ 22.5 million for the fire and Santos City Hall imposed a fine of R$ 2.8 million (G1, 2016b).

According to the same portal, the cause of the accident was a pump connected to the valves that were closed, and due to the pressure caused, the tanks exploded.

"The pump should not have been put into operation because it had the inlet and outlet valves closed, and an operator inadvertently had it run. This caused the pump to run without the circulation of fuel until it generated the explosion by the pressure buildup".

For each of the events listed, experts were chosen based on their professional background, so that the analyzed events were related to their area of expertise. Snowball sampling was used as each expert was invited to nominate another expert who could participate in the survey. Google forms were used to send the questionnaires. A brief contextualization of the research, highlighting what Black Swan events are and the accidental scenario according to the expert's area of knowledge, was presented so that they estimate the probability that the event described is framed in one of the classes: “Not a Black Swan”, “Black Swan: unknown-unknown”, “Black Swan: unknown-known” and “Black Swan: not believed to occur". An example of this form is in Appendix A.

The quartile method was used in the elicitation process, to minimize the expert´s biases and heuristics. The fitting of the individual probability distributions and aggregation was performed with the SHELF: Tools to Support the Sheffield Elicitation Framework (OAKLEY, 2017) software package R (R DEVELOPMENT CORE TEAM, 2017). Equal weights were assigned to the different experts so that the final probability distribution was obtained by the linear aggregation algorithm.

4. RESULTS

The results obtained within the previously described method for each type of technological accident studied will be presented in this section. The quotes from experts are translated from Portuguese.

4.1. Event: Aircraft Accident - Gol 1907 – Legacy – year: 2006

This accident was analyzed by 23 experts, including air traffic controllers, aircraft pilots, and aviation safety experts. Of these, only 1 was discarded from the analysis due to incoherence in the probability elicitation process.

As a general result, the experts classified the accident as shown in Table 1. The vast majority (70.8%) considered this type of event to be perfectly predictable, but with a very low probability, and thus it represents an unbelievable event, classifying it as “Black Swan: not believed to occur". This view can be confirmed in the justifications of the experts, among which we highlight the following:

· Expert A: “I consider it of known causes but of very small probability.”

· Expert B: “I believe that the lack of training and investment has made the possibility of an accident to be underestimated, despite all the statements by the responsible bodies that this care is one of the pillars of the aviation and control system in the country.”

· Expert C: “because it had never happened before this way.”

· Expert D: “With the current technologies of (Traffic Collision Avoidance System – (TCAS), among others, those responsible assume that the collision between two aircraft in flight is something almost impossible to occur and end up denying the risks of this situation.”

· Expert E: “Everyone knew it could happen, but no one believed it would happen.”

· Expert F: “Due to the existence of transponder-type equipment and TCAS for example, this type of accident is known to science and risk analysts, but its likelihood of occurrence is very unlikely.”

Table 1: Classification of the event that occurred with the Aircraft Accident - Gol 1907 – Legacy – year: 2006.

“Not a Black Swan”	“Black Swan: Unknown-Unknown”	"Black Swan: Unknown-Known”	"Black Swan: not believed to occur”
20.8%	0%	8.3%	70.8%

The second class, with 20.8% was classified as not being a Black Swan by the experts, and the justifications were:

· Expert A: “The technological evolution of the air navigation provided an almost perfect route in the middle of the airways with almost no lateral deviation. Then the aircraft, due to the advanced navigation equipment on board, maintaining the same level of flight, passed at the same point.”

· Expert B: “Several reports of hazard (RELPREV), now known as Prevention Reports, completed prior to the accident, already indicated that something like this could occur. I have filled in many.”

· Expert C: “The factors that caused the accident are known, but the conjunctures of the system (human and equipment failures) were determinant for it to occur.”

And finally, the lowest of the groups (8.3%) was classified as being a Black Swan of unknown nature by the analysts and known to the experts.

Some experts reported:

· Expert A: “The indication of TCAS OFF with low visibility could be associated with human error. The coincidence of the routes was predictable by the precision of the technology. It is the case of joining the facts together, which is not trivial.”

· Expert B: “Traffic operators generally knew the risks and avoided them.”

It is noteworthy that no expert considered the event to be of the Unknown-Unknown Black Swan type.

We also obtained the results of the elicitation process of the distribution of subjective probability of the experts in each classification, according to Table 2.

Table 2: Probability distribution elicitation for “p” in the event Aircraft Accident - Gol 1907 – Legacy – year: 2006.

Black Swan type	Expert	Best fitting (least square)	Model parameters
“Not a Black Swan”	A	Beta	𝜶 = 0.9999813 𝜷 = 1.000008
	B	Beta	𝜶 = 1.6925468 𝜷 = 1.196452
	C	Beta	𝜶 = 2.4078168 𝜷 = 1.310733
	D	Normal	µ = 0.1538458 σ = 0.1140464
	E	Beta	𝜶 = 4.3184044 𝜷 = 2.683653
“Black-Swan Unknown-Known”	A	Normal	µ = 0.2499995 σ = 0.1853250
	B	Beta	𝜶 = 0.4265952 𝜷 = 0.5049448
“Black Swan not believed to occur”	A	Beta	𝜶 = 0.8971646 𝜷 = 0.3533580
	B	Beta	𝜶 = 7.6919001 𝜷 =1.5864494
	C	Beta	𝜶 = 1.6352622 𝜷 = 0.6040447
	D	Normal	µ = 0.4285489 σ = 0.2115726
	E	Beta	𝜶 = 0.7982027 𝜷 = 1.4190533
	F	Beta	𝜶 = 1.1738755 𝜷 = 0.6334071
	G	Beta	𝜶 = 0.9999813 𝜷 = 1.0000085
	H	Beta	𝜶 = 0.9999813 𝜷 = 1.0000085
	I	Beta	𝜶 = 0.5234347 𝜷 = 1.2619953
	J	Beta	𝜶 = 0.9999813 𝜷 = 1.0000085
	K	Beta	𝜶 = 0.7329743 𝜷 = 0.3972961
	L	Beta	𝜶 = 0.9999813 𝜷 = 1.0000085
	M	Beta	𝜶 = 0.6203376 𝜷 = 1.0806761
	N	Beta	𝜶 = 0.3533580 𝜷 = 0.8971646
	O	Beta	𝜶 = 0.9999813 𝜷 = 1.0000085
	P	Beta	𝜶 = 0.9999813 𝜷 = 1.0000085

Table 2 presents, for each Black Swan class, the distribution that provided the best fit according to the least squares criterion between the adjusted model and the data of each expert, as well as, the values obtained for the parameters. Three models of probability were used in this adjustment: Beta, Gamma, and Normal distributions that have their functional forms defined by:

Beta: p ~ Beta( α, β), (Eq.1).

where Probability-Density Function - pdf is:

Eq. 1

α, β > 0 ; p ∈ [0;1]

α, β are Beta shape parameters.

Gamma: p ~ Gamma (α, β), (Eq.2).

Eq. 2

α, β > 0; p > 0

α: Gamma shape parameter.

β: Gamma rate parameter.

And Γ: is the gamma function (Eq.3), defined as:

Eq. 3

Normal: p ~ Normal(µ, σ^2)

where:

Eq. 4

µ, σ: represents distribution mean and standard deviation, respectively.

Figure 2 presents, as an example, the result for the elicitation process of the class "not a Black Swan".

In class “not a Black Swan”, four of five best fitting were Beta and only one was Normal distributed, and so we decided to consider Beta distribution for all cases, including “Black Swan: unknown-known” and “Black Swan: not believed to occur”. This decision is possible because the values for “p” are ranged in [0,1] and strictly positive, so Beta is technically better than Normal.

Figure 2: The Beta probability distribution for "not a Black Swan", in Aircraft Accident - Gol 1907 – Legacy – year: 2006, for all experts considered.

In Table 3, the quantiles values for the experts are provided considering Beta distribution. There is not much agreement among experts for “p”, and the class of Black swan considered was not strongly evidenced.

Table 3: Probability “p” that the Aircraft Accident - Gol 1907 – Legacy – year: 2006 was “not a Black Swan”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.

Quantil	Expert
	A	B	C	D	E
0.25	0.250	0.394	0.492	0.0801	0.497
0.50	0.500	0.608	0.676	0.1470	0.628
0.75	0.750	0.796	0.828	0.2380	0.748
0.95	0.950	0.949	0.954	0.3990	0.879

The average of the expected values of experts were calculated, and the following values were obtained: E[p] = 0.5193, for the Black Swan “not believed to occur” type; E[p] = 0.5042 for “not a Black swan”; and E[p] = 0.3644 for “Black Swan: Unknown-Known” type. The aggregation of the expected value of different experts was performed by linear aggregation.

The elicitation process was able to produce the aggregate distribution of the “p” values. These values can be used in the future as a priori distribution for future studies in risk analysis of events of this nature.

The conclusion is that although the experts classified the event into different types of Black Swan, they were not able to produce a “p” value consistent with their classification, denoting the great discrepancy among the experts.

4.2. Event: Construction Industry: Accident on the Tim Maia bicycle pathway in Rio de Janeiro- year: 2016.

This accident was analyzed by 7 experts, including safety engineers, oceanographers, and risk analysts.

As a general result, 43% of the experts judged that this type of event was “not a Black Swan” and 57% classified it as “Black Swan: not believed to occur”. This classification was justified by each group of experts as follows:

For the group that considered it to be “not a Black Swan”:

· Expert A: “The event was not "unknown", I believe it was overlooked by human failure.”

· Expert B: “The event is perfectly predictable by civil engineering.”

· Expert C: “A work on the shoreline should consider the possibility of bad weather and the action of waves, in other words, have there never been waves of this nature that were known to the authorities?”

And for “Black Swan: not believed to occur”:

· Expert A: “There were elements to at least take a sufficient storm surge to project waves to the walkway; The locking system of the chosen structure, while suitable for the intended load, disregarded this possibility.”

· Expert B: “This is a predictable event since there is a known hazard (waves crashing on the rocks and history of the height and intensity that they can reach). With known design data, it would be possible to evaluate, even qualitatively, the occurrence of the event and the intensity sufficient to move the walkway. This evaluation could be done in the design stage, as previously determined, for example, the best type of mooring of the boards to the pillars.”

· Expert C: “I imagine that the work could have been designed and constructed to withstand the scenario, but it was not for its probability was considered low..

· Expert D: “The variation in sea outflow is somewhat predictable.”

For this event, none of the experts considered the Black Swan classification of the Unknown-Unknown or Unknown-Known type.

We also obtained the results of the elicitation process of the subjective probability distribution of the experts in each classification, according to Table 4.

Table 4: Probability distribution elicitation for “p” in the Accident on the Tim Maia bicycle pathway in Rio de Janeiro- year: 2016.

Black Swan Type	Expert	Best fitting (Least square)	Model parameters
“Not a Black Swan”	A	Gamma	Shape = 1.231008 Rate = 4.416544
	B	Beta	𝜶 = 0.9079558 𝜷 = 1.068221
	C	Beta	𝜶 = 1.6925468 𝜷 = 1.196452
“Black Swan: not believed to occur”	A	Beta	𝜶 = 0.5095363 𝜷 = 1.0934487
	B	Beta	𝜶 = 0.9999813 𝜷 = 1.0000085
	C	Beta	𝜶 = 0.8391531 𝜷 = 0.9077006
	D	Beta	𝜶 = 0.5716223 𝜷 = 0.9911040

For the expert fitting group “not a Black Swan”, two had their results of the elicitation process best represented by the Beta distribution and 1 by the Gamma distribution, by the least squares criterion. Again, to obtain the aggregate distribution, the Beta distribution was used for all the members of the groups, resulting in Table 5 and Table 6 with the summarized results of the elicitation process of the groups “not a Black Swan” and “Black Swan: not believed to occur”.

Table 5: Probability – “p” that the Accident on the Tim Maia bicycle pathway in Rio de Janeiro was “not a Black Swan”, fitted by Beta distribution, for the 0.25; 0.50; 0.75; and 0.95 quantiles.

	Expert
Quantil	A	B	C
0.25	0.0924	0.2050	0.394
0.50	0.2140	0.4440	0.608
0.75	0.3880	0.7040	0.796
0.95	0.6600	0.9340	0.949

Table 6: Probability-“p” that the Accident on the Tim Maia bicycle pathway in Rio de Janeiro was a “Black Swan: not believed to occur", fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.

Quantil	Expert
	A	B	C	D
0.25	0.05930	0.25	0.2100	0.08940
0.50	0.23400	0.50	0.4720	0.30000
0.75	0.53100	0.75	0.7460	0.60800
0.95	0.88000	0.95	0.9560	0.91600

Graphical results are also shown in Figures 3 and 4. The linear aggregate results are plotted.

Figure 3: The result of the Beta probability distribution elicitation for the class “was not Black Swan”, for the 3 experts considered.	Figure 4: The result of the Beta probability distribution elicitation for the class “Black Swan: not believed to occur”, for the 4 experts considered

Considering the expected value of the aggregate distribution of the experts, for “not a Black Swan” class, the expected value was E[p] = 0.43; and for the “Black Swan: not believed to occur”, the expected value was E[p] = 0.416 for the value of “p”. The fact once again demonstrates that despite the classification given by the experts for each group, they were not able in the elicitation process to make this evidence clear, as the probability “p”, that the event is of the class considered for the two cases is low. Although general fitting is much better for “Black Swan: not believed to occur”, in which the differences among distributions of experts are lower (Figure 4).

4.3. Event - Construction Industry: Accident in the work of the Metro – SP – year: 2007

This accident was analyzed by 9 experts, including civil engineers, geologists, and experts in risk analysis.

As general results, 44% of experts judged that this accident was “not a Black Swan”, 33% classified as “unknown-known type of Black Swan”, and 22% as “Black Swan not believed to occur”. The justifications of each expert for classification are transcribed as follows:

“Not a Black Swan”:

· Expert A: “The risk of collapse exists in the engineering environment and an action plan must be in place in case of heavy rains.”

· Expert B: “All the conditions of the terrain could have been preliminarily evaluated by the technicians considering the normal conditions and with soil wet and other variables.”

· Expert C: “Although it is an extreme event, it is not unknown to risk analysts and science.”

· Expert D: “There was no (presented) calculation worksheet, execution without control, in the placements of hangers.”

And for the group that considered it a “Black Swan of the type not believed to occur”:

· Expert A: “It is possible that the analysts did not believe in the possibility of the accident, and although the workers noticed the risks, they did not carry on because they also believed that it would not happen.”

· Expert B: “The soil characteristics were known, and were not monitored in an apparently adequate way.”

Finally, the “unknown-known Black Swan” classification was justified as follows:

· Expert A: “I think the engineer in charge did not see the need to call a geologist, but this may have seen signs of overload.”

· Expert B: “It is possible that it is not a Black Swan, because in this type of work collapse is one of the main risks and risk analysts are expected to have identified it. However, it is also possible that they were unaware of particular soil characteristics of the region, which favored the occurrence of the catastrophe, hence an Unknown-Known Black Swan.”

· Expert C: “Many times work is carried out with unqualified technical staff, and/or the risks of accidents are diminished by its directors determined to increase of the profit from the work. That is, very low risks are greatly amplified in this scenario.”

We also obtained the results of the elicitation process of the subjective probability distribution of the experts in each classification, according to Table 7.

Table 7: Probability distribution elicitation for “p” in the Metro–SP Accident – year: 2007.

Black Swan Type	Expert	Best fitting (Least square)	Model parameters
“Not a Black Swan”	A	Beta	𝜶 = 0.6809183 𝜷 = 1.1100828
	B	Beta	𝜶 = 0.3878157 𝜷 = 0.6960759
	C	Beta	𝜶 = 0.3978971 𝜷 = 0.6276588
	D	Beta	𝜶 = 0.9999813 𝜷 = 1.0000085
“Black Swan - not believed to occur”	A	Beta	𝜶 = 0.8238680 𝜷 = 1.0816977
	B	Beta	𝜶 = 0.9498462 𝜷 = 0.4723515
“Black Swan - Unknown-Known”	A	Beta	𝜶 =0.8105315 𝜷 =1.1023002
	B	Beta	𝜶 =0.3765031 𝜷 =0.6307464
	C	Beta	𝜶 =0.9999813 𝜷 =1.0000085

For all three classes of Black Swan, the Beta distribution was the one that presented the best fit according to least square criterion. Again, to obtain the aggregate distribution, the Beta distribution was used for all members of the groups, resulting in Tables 8, 9, and 10 with the summarized results of the group elicitation process, “Not Black Swan”, “Black Swan: not believed to occur”, and “Black Swan: unknown-known” type, respectively.

Table 8: Probability-”p” that the Metro – SP Accident was “not a Black Swan”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.

	Expert
Quantil	A	B	C	D
0.25	0.117	0.0453	0.0569	0.25
0.50	0.329	0.2570	0.3010	0.50
0.75	0.613	0.6430	0.7010	0.75
0.95	0.904	0.9590	0.9740	0.95

Table 9: Probability-”p” that the Metro – SP Accident was “Black Swan: not believed to occur”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.

Quantil	Expert
	A	B
0.25	0.172	0.434
0.50	0.405	0.756
0.75	0.675	0.943
0.95	0.925	0.998

Table 10: Probability-”p” that the Metro – SP Accident was “Black Swan: unknown-known”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.

Quantil	Expert
	A	B	C
0.25	0.165	0.0469	0.25
0.50	0.393	0.2760	0.50
0.75	0.664	0.6810	0.75
0.95	0.920	0.9720	0.95

Figures 4, 5, and 6 present the individual expert and aggregate fitting.

Figure 4: The result of the Beta probability distribution for the class "was not Black Swan", for the 4 experts considered.	Figure 5: The result of the Beta probability distribution for the class "Black Swan – not believed to occur", for the 2 experts considered.

Figure 6: The result of the Beta probability distribution for the class "Black Swan Unknown-Known", for the 3 experts considered.

Figure 6. The result of the Beta probability distribution for the class "Black Swan Unknown-Known", for the 3 experts considered.

Considering the expected value of the aggregate distribution of the experts, for the class “not a Black Swan”, the expected value is E[p] = 0.40, and for the “Black Swan: not believed to occur”, the expected value is E[p] = 0.55. Finally, the “Black Swan: unknown-known” resulted in E[p] = 0.43 for “p”. Once again, despite the classification given by the experts for each group, they were not able in the elicitation process to make this evidence clear, because the probability “p”, that the event is of the class considered for the three cases is low. However, in all three cases, the distributions show a similar behavior.

4.4. Event - Fire in Alemoa Terminal – year: 2015

This accident was analyzed by 10 experts, including civil engineers, safety engineers, and experts in risk analysis.

As general results, 20% of experts judged that this accident was “not a Black Swan”, 30% classified as “unknown-known type of Black Swan”, and 50% as “Black Swan not believed to occur”. The justifications of each expert for classification are transcribed as follows:

“Not a Black Swan”:

· Expert A: “This type of installation is already considered a fire risk, lack of preparation, equipment, and products to extinguish fire in the beginning.”

· Expert B: “There are several similar/equivalent events that can be identified in the world.”

And for the group that considered it a “Black Swan of the type not believed to occur”:

· Expert A: “Totally predictable to happen.”

· Expert B: “Tank fire is possible, though unlikely.”

· Expert C: “The risk is known, but very low, a sequence of adverse events is required to occur.”

· Expert D: “It was an extreme event in which the company did not have a fire prevention and improvement plan due to its low probability of occurrence.”

· Expert E: “Every storage location for flammable material must always contain contingency plans, such as containment barriers, distance between them.”

Finally, the “unknown-known Black Swan” classification was justified as follows:

· Expert A: “The probable reason was a human fault.”

· Expert B: “probability of occurrence is difficult.”

· Expert C: “Known risks; but not controlled.”

We also obtained the results of the elicitation process of the subjective probability distribution of the experts in each classification, according to Table 11.

Table 11: Probability distribution elicitation for “p” in the Alemoa terminal fire – year: 2015.

Black Swan Type	Expert	Best fitting (least square)	Model parameters
“Not a Black Swan”	A	Beta	𝜶 = 0.6203376 𝜷 = 1.080676
	B	Beta	𝜶 = 7.6919001 𝜷 = 1.586449
“Black Swan - not believed to occur”	A	Beta	𝜶 = 1.0832013 𝜷 = 0.8756585
	B	Beta	𝜶 = 0.4543275 𝜷 = 0.8756081
	C	Beta	𝜶 = 0.9429176 𝜷 = 0.4615730
	D	Beta	𝜶 = 1.1100828 𝜷 = 0.6809183
	E	Beta	𝜶 = 2.3025908 𝜷 = 0.9229045
“Black Swan - Unknown-Known”	A	Beta	𝜶 =0.9999813 𝜷 =1.0000085
	B	Beta	𝜶 =0.6569452 𝜷 =0.8231489
	C	Beta	𝜶 =0.9877243 𝜷 =1.1428279

For all three classes of Black Swan, the Beta distribution was the one that presented the best fit according to least square criterion. Tables 12, 13, and 14 present the summarized results of the group elicitation process, “Not Black Swan”, “Black Swan: not believed to occur”, and “Black Swan: unknown-known” type, respectively.

Table 12: Probability-”p” that the Alemoa terminal fire was “not a Black Swan”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.

Quantil	Expert
	A	B
0.25	0.0983	0.762
0.50	0.3040	0.853
0.75	0.5970	0.920
0.95	0.9030	0.975

Table 13: Probability-”p” that the Alemoa terminal fire was “Black Swan: not believed to occur”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.

Quantil	Expert
	A	B	C	D	E
0.25	0.310	0.0557	0.438	0.387	0.569
0.50	0.574	0.2510	0.763	0.671	0.761
0.75	0.809	0.5890	0.947	0.883	0.898
0.95	0.970	0.9260	0.998	0.989	0.983

Table 14: Probability-”p” that the Alemoa terminal fire was “Black Swan: unknown-known”, fitted by Beta distribution, for 0.25; 0.50; 0.75; and 0.95 quantiles.

Quantil	Expert
	A	B	C
0.25	0.25	0.149	0.219
0.50	0.50	0.414	0.451
0.75	0.75	0.724	0.700
0.95	0.95	0.959	0.926

Figures 7, 8, and 9 present the individual expert and aggregate fitting.

Figure 7: The result of the Beta probability distribution for the class "was not Black Swan", for the 2 experts considered.	Figure 8: The result of the Beta probability distribution for the class "Black Swan – not believed to occur", for the 5 experts considered.

Figure 9: The result of the Beta probability distribution for the class "Black Swan Unknown-Known", for the 3 experts considered.

Considering the expected value of the aggregate distribution of the experts, for the class “not a Black Swan”, the expected value is E[p] = 0.59, and for the “Black Swan: not believed to occur”, the expected value is E[p] = 0.58. Finally, the “Black Swan: unknown-known” resulted in an E[p] = 0.47 for “p”. Once again it was demonstrated that despite the classification given by the experts to each group, they were not able in the elicitation process to make this evidence clear, because the probability “p”, that the event was of a considered class for the three cases is low. In the last two cases, the distributions show a similar behavior among experts.

5. FINAL REMARKS

This article reached its objectives of identifying and classifying different types of “Black Swans”. For all four events analyzed, it was possible to extract from the different experts the classification of “not a Black Swan”, “unknown-known Black Sawn”, and “Black Sawn not believed to occur”. For all cases, “unknown-unknown Black Swan” was never reported by any of the experts. It is possible to conclude that events never represent completely unknown events, both by risk analysts and science, that is, none of the events studied represents a surprise to anyone. This fact contradicts type (ii) – an extreme, astonishing event concerning current knowledge and/or belief, as stated in Aven, (2013).

One of the attributes of Black swan events is that after their occurrence, people and experts provide logical explanations, making an event hitherto unknown into a perfectly predictable event, which is in agreement with (AVEN, 2013).

Most of the times, experts classified events as “Black Swan not believed to occur”, which means that the events are in general well known, but since their probability is judged to be extremely low, the events are discarded in risk analysis. This leads to the conclusion that the analyzed events are sometimes classified as real outliers and this represents an issue to risk analysis.

The elicitation process showed large disagreement among experts. The expected value, meaning the most probable, was consistently low throughout all cases analyzed. This issue could represent a failure in elicitation, which will need further research to confirm or discard. However, we still believe the process was useful and could serve as a basis for future risk analysis studies, such as a priori belief distribution in a Bayesian analysis.

Future works should analyze other events in different contexts, and maybe analyze Black Swan occurrence using Bayesian belief networks, that is, studying factors that affect expert beliefs in the elicitation process.

6. GENERAL ACKNOWLEDGMENTS

We are very grateful to all anonymous experts who spent time in responding to the specific forms, providing their valuable contribution to the accomplishment of this work.

REFERENCES

ABNT-ISO. (2009) NBR-ISO 31000: Gestão de Riscos – Princípios e Diretrizes.

Aspinall, W. P. (2005) Chapter 2 Structured Elicitation of Expert Judgment for Probabilistic Hazard and Risk Assessment in Volcanic Eruptions. In Mader, H.M.; et al. (Eds.), Statistics in volcanology. Special Publication of IAVCEI (p. 1–31) Geological Society of London.

Aven, T. (2013) On the meaning of a black swan in a risk context. Safety Science, v. 57, p. 44–51. Available: https://doi.org/10.1016/j.ssci.2013.01.016

Aven, T. (2015) Implications of black swans to the foundations and practice of risk assessment and management. Reliability Engineering and System Safety, v. 134. Available: https://doi.org/10.1016/j.ress.2014.10.004

Bojke, L.; Claxton, K.; Bravo-Vergel, Y.; Sculpher, M.; Palmer, S.; Abrams, K. (2010) Eliciting distributions to populate decision analytic models. Value in Health, v. 13, n. 5, p. 557–564. Available: https://doi.org/10.1111/j.1524-4733.2010.00709.x

Burgman, M.; Fidler, F.; Mcbride, M.; Walshe, T.; Wintle, B. (2006) ACERA Project 0611 Eliciting Expert Judgments. Risk Analysis, v. 1 p. 71.

Clemen, R. T. (2008) Comments: Comment on Cooke’s classical method. Reliability Engineering and System Safety, v. 93, n. 5, p. 760–765. Available: https://doi.org/10.1016/j.ress.2008.02.003

Flage, R.; Aven, T. (2015) Emerging risk – Conceptual definition and a relation to black swan type of events. Reliability Engineering & System Safety, v. 144 (August) Available: https://doi.org/10.1016/j.ress.2015.07.008

FOLHA DE SÃO PAULO. (2007) Entenda como aconteceu o desabamento nas obras do metrô em SP. São Paulo. Retrieved from http://www1.folha.uol.com.br/folha/cotidiano/ult95u130487.shtml

G1. (2016a) Justiça inocenta 14 réus por cratera que matou 7 no metrô de São Paulo. São Paulo. Retrieved from http://g1.globo.com/sao-paulo/noticia/2016/10/justica-inocenta-14-reus-por-cratera-que-matou-7-no-metro-de-sao-paulo.html

G1. (2016b) MP divulga laudo e pede multa de R$ 3,6 bilhões por incêndio na Ultracargo. G1. Santos. Retrieved from http://g1.globo.com/sp/santos-regiao/noticia/2016/11/mp-divulga-laudo-e-pede-multa-de-r-36-bilhoes-por-incendio-na-ultracargo.html

Garthwaite, P. H.; Kadane, J. B.; Hagan, A. O. (2005) Statistical Methods for Eliciting Probability Distributions. Journal of the American Statistical Association, v. 100, n. 470, p. 680–701.

HOSACK, G. R.; HAYES, K. R.; BARRY, S. C. (2017) Prior elicitation for Bayesian generalised linear models with application to risk control option assessment. Reliability Engineering and System Safety, n. 167, p. 351–361. https://doi.org/10.1016/j.ress.2017.06.011

IOANNOU, I.; ASPINALL, W.; RUSH, D.; BISBY, L.; ROSSETTO, T. (2017) Expert judgment-based fragility assessment of reinforced concrete buildings exposed to fire. Reliability Engineering and System Safety, n. 167, p. 105–127. https://doi.org/10.1016/j.ress.2017.05.011

MACDONALD, J. A.; SMALL, M. J.; MORGAN, M. G. (2008) Explosion probability of unexploded ordnance: Expert beliefs. Risk Analysis, v. 28, n. 4, p. 825–841. https://doi.org/10.1111/j.1539-6924.2008.01068.x

MORRIS, D. E.; OAKLEY, J. E.; CROWE, J. A. (2014) A web-based tool for eliciting probability distributions from experts. Environmental Modelling and Software, v. 52, p. 1–4. https://doi.org/10.1016/j.envsoft.2013.10.010

OAKLEY, J. (2017) SHELF: Tools to Support the Sheffield Elicitation Framework. Retrieved from https://cran.r-project.org/package=SHELF

OLIVEIRA, N.; (2016) Crea-RJ: falha de projeto é uma das causas do desabamento da Ciclovia Tim Maia. EBC Agência Brasil. Rio de janeiro. Retrieved from http://agenciabrasil.ebc.com.br/geral/noticia/2016-05/crea-rj-falha-de-projeto-foi-uma-causas-do-desabamento-de-ciclovia-no-rio

Paté-Cornell, E. (2012) On “Black Swans” and “Perfect Storms”: Risk Analysis and Management When Statistics Are Not Enough. Risk Analysis, v. 32, n. 11, p. 1823–1833. https://doi.org/10.1111/j.1539-6924.2011.01787.x

R DEVELOPMENT CORE TEAM. (2017) R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.r-project.org/

Taleb, N. N. (2010) The Black Swan: The Impact of the Highly Improbable (2nd ed.) Random House.

UOL. (2016) Trecho de ciclovia inaugurada em janeiro desaba no Rio. São Paulo. Retrieved from https://noticias.uol.com.br/cotidiano/ultimas-noticias/2016/04/21/ciclovia-inaugurada-em-janeiro-em-sao-conrado-desaba-no-rio.htm

Usher, W.; Strachan, N. (2013) An expert elicitation of climate, energy and economic uncertainties. Energy Policy, v. 61, p. 811–821. Available: https://doi.org/10.1016/j.enpol.2013.06.110.

Appendix A: Questionnaire on Air Aircraft Accident - Gol 1907 – Legacy – year: 2006, translated from Portuguese.

"Black Swans" are characterized by the following attributes: They are outliers, that is, represent extremely improbable events associated with extremes consequences and after their occurrence, they are explicable and predictable.

"Black Swan" events can be classified into three basic types: (i) events that are completely unknown in the scientific milieu, termed unknown-unknown, as a reference to the fact that they are unknown to risk analysts and also to science; (ii) Events that are not on the list of risk analysts but are on the list of experts and/or science (unknown-known); (iii) Events known to risk analysts, but judged to have negligible probability of occurrence and therefore are not amenable to analysis since they are judged to be "not believed to occur"

In this context, you are asked to evaluate the event described below.

Event: Aircraft Accident - Gol 1907 – Legacy – year: 2006

Gol Flight 1907 was a domestic commercial route, operated by Gol Airlines, using a Boeing 737-8EH. On September 29, 2006, the aircraft departed from the Eduardo Gomes International Airport in Manaus to the Galeão International Airport in Rio de Janeiro with a scheduled a stop at the Juscelino Kubitschek International Airport in Brasilia. As it flew over Mato Grosso state, it crashed in the air with an Embraer Legacy 600. All 154 passengers and crew aboard the Boeing 737 died after the aircraft collided in the air and crashed into a closed forest area, while the Legacy, despite having suffered severe damage to its wing and horizontal left stabilizer, landed safely with its seven uninjured occupants at the Air Base of Cachimbo. CENIPA concluded that the accident was caused by both air traffic controllers and Legacy pilots errors, while the National Transportation Safety Board (NTSB) determined that all pilots acted correctly and were placed on a collision route for a variety of air traffic controller errors.

Q.1. How do You classify the event described?

a) Not a Black swan;

b) A Black swan: Unknown-unknown;

c) A Black swan: Unknown-known;

d) A Black Sawn: not believed to occur;

Q.2. Justify the choice made in (Q.1).

Q.3. Now, assuming that “p” is the probability that the event described is in the category you choose in (Q.1). Assign a value from 1 to 10 (1=minimum; 10=maximum) that matches your belief that “p” is in the following ranges.

Ranges for “p”	1	2	3	4	5	6	7	8	9	10
(0.00-0.25]
(0.25-0.50]
(0.50-0.75]
(0.75-1.00]