Reaching Consensus with VICA-ELECTRE TRI: A Case Study

This paper reports on an application of the VICA-ELECTRE TRI to a real world case study. We used the model VICA (Visual, Interactive and Comparative Analysis of individual opinions) to aid a cooperative group achieving a collective solution to a multicriteria sorting problem (student allocation to classes on a language school). Based on the individual results initially obtained, the model provides tools for the visualization of results and their comparison. It also points out means to seek or build a consensus on a solution, by guiding the changes of parameters and/or preference revisions in different possible ways of interaction between the members of the group. After a brief description of the model, we present the procedures followed, from the meeting plan to the case analysis.


Introduction
According to Raiffa et al. (2002, p. 389), the more complex and important a decision, the greater the likelihood that a cooperative working group will be designated to analyze and tackle the problem. This brings benefits i.e. a wider range of expertise; incentive for people to engage in organization processes; support to the solution and ownership of the decision, but it also adds complexity to the situation. In regards to offering help for people when they try to make decisions in groups, they state: "One could call it science in the service of the art of human interaction".
B João Paulo Costa jpaulo@fe.uc.pt It is becoming increasingly more common for important and complex decisions within a modern organization to be made by a group of individuals and not by one single individual. Many of these groups seek solutions collectively constructed, through processes which consider the many significant aspects of the problem, in order to achieve transparency and effective results. The decision context we considered is the one where a cooperative group (Dias and Clímaco 2005) wants or needs to reach a consensual result and therefore has people willing to contribute with their own point of view, to learn about other members' perspectives and to improve jointly the understanding of a relevant issue. Besides being multiperson, the approach is also multicriteria, since it explicitly considers the evaluations of the alternatives in each criterion of a set. Cai et al. (2012) argue that although the group decision has been frequently adopted in the highly complex environment of today's world, the problems of multi-criteria sorting in the group decision context are worthwhile endeavors that have not been sufficiently studied. The inherent complexity of the problem suggests the need for tools and methodologies intended to provide support in such a construction. This work applies the model VICA (Visual, Interactive and Comparative Analysis) to support the building of consensus in a multicriteria and multiperson process dealing with a sorting problem.
The application of the VICA-ELECTRE TRI (Bezerra et al. 2008(Bezerra et al. , 2014 model has its starting point on the explicit consideration of the performances of the alternatives on each criterion, the different points of view of the individuals and consequently, their different results. The group learning process is facilitated through tools modelled to analyze performances and results in several forms, i.e. partial, overall, individual or aggregated, in a comparative, interactive or evolving way. Furthermore, means to achieve a consensual result are signaled through modification of parameters or concessions of group members in different possible processes of use. In our view, as in Hodgkin et al. (2005), the main goal of Multicriteria Decision Aiding (MCDA) is not to find the right answer to an issue, but to facilitate decision makers through a learning process about a problem. Extensive sensitivity or robustness analyses (Belton and Stewart 2003) can be helpful in achieving this goal, but they are time-consuming and difficult to communicate, especially when one considers the multi-dimensional complexity of the problem. The VICA-ELECTRE TRI model seeks to address this challenge in the way it structures the data and uses interactive visualization to present the information, synthesizing and reflecting back judgments made and possible solutions.
The case study reported in this paper consisted on assigning 20 students of a Brazilian language institution to 5 classes (levels), according to 6 criteria. The board of this institution has 4 people which are the decision makers of the case study. They have different preferences regarding the criteria relative importance, including different evaluation scale measures. After applying the VICA-ELECTRE TRI model, implemented as an application on Microsoft Excel and supported by an analyst in a face to face meeting, the board of the institution reached a consensual solution for the assignment problem. In order to apply VICA-ELECTRE TRI to this real assignment problem, particular search and construction consensus processes where developed and an application plan with five stages was established. While many approaches and methods for dealing with the sorting problem exist, not many studies use the feedback effect (Vetschera 1991) for dealing with this problem in MCDA, like VICA-ELECTRE TRI (Bezerra et al. 2008(Bezerra et al. , 2014 does. The context was considered in Melo (2005), where decision makers have different explicit preferences that can change during the interactions with the group. Melo (2005) addressed the problem with the aid of a prototype, the TriGdist. This system provides mechanisms for measuring the differences between individual positions, which are then presented in the form of the changes that a DM would have to make to obtain the same result as another DM. Han and Ahn (2005) and Herrera-Viedma et al. (2002 present interactive procedures to achieve preference consensus that also consider the feedback effect. The procedure, proposed by Han and Ahn (2005), points out the direction in which modifications in individual preferences should be made in order for consensus to be achieved. Herrera-Viedma et al. (2002 present an automatic process designed to guide groups toward consensus, based on the use of a measure of consensus to identify the group position and in a measure of proximity that locates the distance of each DM from the collective opinions. More recently, Leyva-López et al. (2017) describe a web-based decision support system providing a feedback mechanism to help decision makers to change their preferences in order to achieve consensus in ranking problems using ELECTRE III. They extend Herrera-Viedma et al. (2007) consistency and consensus measures in order to cope with outranking relations and the use of different group coordination modes.
Regarding the sorting problem, we can highlight Dias and Clímaco (2000), Dias and Mousseau (2003) and Damart et al. (2007), which address the problem from a different perspective: robustness analysis, parameter inference and the aggregation and disaggregation approach, using the IRIS DSS. Since Mousseau and Slowinski (1998) proposed an interactive optimization model to infer the ELECTRE TRI parameters, many other works have used this case-based preference elicitation (Vetschera et al. 2010). In the work we present, we apply the direct elicitation procedures in order to obtain the required preference information from the DM, as in Neves et al. (2008).
As stated in Bezerra et al. (2014), VICA-ELECTRE TRI approach has similarities with the one presented in Vetschera (1991), in that both use feedback and change of individual preferences to achieve acceptable consensus. The main dissimilarity is that one was applied to MAUT-like methods, and VICA is applied to an outrank-like method, incorporating shared veto. An additional difference is that an explicit group evaluation is not attempted in the original work (which focus on a single pair of DMs), whereas VICA consensus building iterates through "group-centred" solutions. Finally, rather than considering a C-consensus for a pair of individuals (proposed in Vetschera 1991, as consensus on the top C results), VICA uses a different definition called G-consensus (with G being defined iteratively by the group process) describing full agreement for G group member evaluations. Moreover, the VICA methodology relies on the principle that, to help the search for consensus in a group decision process, it is necessary to provide, explicitly or implicitly, some sort of information about the comparison of individual preferences and associated outputs, allowing for several revision interactions. Additionally, the provided information should be presented in a visual and interactive form in order to decrease the cognitive burden upon the DMs.
The case study reported in this paper, applied the VICA-ELECTRE TRI model, implemented as an application on Microsoft Excel and supported by an analyst in a face to face meeting. Using search and consensus-building processes custom-adapted for the case, VICA-ELECTRE TRI proved to be effective on reaching a consensus solution in a real collaborative group.
The following section briefly presents a MCDA sorting problem and the ELECTRE TRI. Section 3 is devoted to explaining the VICA-ELECTRE TRI model, its formulations and main features. This section also presents the search and consensus-building processes developed to the case study. Then, Sect. 4 reports on the application plan and the case study and Sect. 5 presents some conclusions.

The ELECTRE TRI Method
There are many approaches and methods for handling sorting problems; a review of them is included in Zopounidis and Doumpos (2002). We choose the original version of the method ELECTRE TRI (Yu 1992), named ELECTRE TRI-B in Almeida- Dias et al. (2012) and Kadziński et al. (2015), as the central pillar of the VICA model. It is a well-known method of the MCDA area, designed specifically to address sorting problems. As well as other methods of the ELECTRE family, its preference system is modelled through outranking binary relations (Roy and Bouyssou 1993;Figueira et al. 2005Figueira et al. , 2013. Considering a set of alternatives A = {a 1 , a 2 , . . . , a i , . . . , a n }, the ELECTRE TRI assignment of each alternative to a class is the result of comparisons between their performance and profiles b h ∈ B = {b 1 , b 2 , . . . , b h , . . . , b k−1 }, in each criteria j = 1, 2, . . . , t. The profiles are lower and/or upper limits located between k ordered categories. The method requires partial concordance indices c j (a i , b h ) to be calculated, followed by a general concordance index C(a i , b h ), discordance indices D j (a i , b h ) and a credibility index σ (a i , b h ), for the outranking relation a i Sb h ("a i is at least as good as b h "), taking into account the performance g j (a i ) and g j (b h ) in each criterion. The calculation of the above indices employs the parameters associated with a pseudo criterion: indifference threshold q j , preference threshold p j , and veto threshold v j . Once these parameters are defined, the partial concordance indices c j (a i , b h ) can be obtained: for all criteria. In order to perform this aggregation, the relative importance, w j , of each criterion must be defined.
The veto thresholds define discordance indices, D j (a i , b h ), which may prohibit the statement "a i outranks b h ".
When the general concordance index is combined with the discordance indices, the credibility index, σ (a i , b h ), can be obtained: The credibility index is a measure belonging to the interval [0, 1]. Therefore, in order to establish an outranking relation a i Sb h or ¬a i Sb h a cutting level λ must be defined. This cutting level is the lowest value belonging to [0.5,1], which is compatible with the statement a i Sb h . Only when σ (a i , b h ) ≥ λ the statement a i Sb h is valid. Finally, two possible procedures exist for assigning alternatives to categories. The pessimistic procedure compares an alternative with b k−1 , b k−2 , . . . , b 1, from the best to the worst profile and then assigns it to the best category so that a i Sb h . The optimistic procedure compares an alternative first with b 1 , then with b 2 ,…, b k−1 , from the worst to the best profile, then assigns it to the worst so that b h Sa i and ¬a i Sb h , where σ (a i , b h ) < λ.

The Group Decision Support Model
In VICA-ELECTRE TRI the evaluation of each individual DM d , is represented by his/her set of credibility indices σ d (a i , b h ), which, in this paper, is referred to as the Matrix of Individual Credibility, according to person d, d = 1, 2, . . . , M. The elements of the credibility matrix, with values equal to or greater than a cutting level (implying the outranking relation a i Sb h ) are identified by numbers and colors/patterns associated with the person who created them. Thus, each member d of the group has a set of assignments corresponding to its vector W d (the criteria relative importance vector of DM d ). The configurations representing these assignments are shown in Fig. 1. Overlapping the configurations (comparison matrix-depicted on the right of Fig. 1), it is easy to compare both assignments and detect the disagreements on assigning the alternatives. These configurations change as a DM modifies its preferences, data or other parameters. The comparison matrix at Fig. 1 shows that a 1 is allocated to class 2. Although both DMs agree that a 2 is at least in class 2, DM 1 places a 2 in class 3 while DM 2 places it in class 2. The disagreement is greater for alternative a 3 : both agree that a 3 is at least in class 2 but DM 1 places it two classes above, in class 4, while DM 2 places it in class 2. Both allocate a 4 and a 5 at least to class 3 while DM 2 places them in class 4 but DM 1 in class 3. Faced with this disagreements, the DMs can either classify the alternatives at the lowest common category, that is, a 1, a 2 and a 3 are allocated to class 2, and a 4 and a 5 are allocated to class 3, or revise their parameters, as explained in the next sub-sections, trying to achieve a less discordant state. Naturally, in a full group setting, the assignments of other group members will have to be considered as well.

In Search of One Solution for the Group: The G-Consensus
In this work we identify as G-consensus the result obtained by a significant subgroup of members composed of a representative majority of the whole, capable of assigning each alternative into one of the classes. If M is the number of group members and G ∈ N, then M/2 < G ≤ M is the necessary majority for having G-consensus. Thus, we have:

lack of consensus) for a i Sb h and for ¬a i Sb h
The results of assigning a i to category K h according to group (a i G → K h ) can also be represented by a configuration, built from the V (a i , b h ) matrix, in which distinctive formats highlight the V (a i , b h ) values leading to a i Sb h, and those leading to ¬a i Sb h , counting all the DM d results.
The group results can also be presented in a table column with the numbers of the category to each alternative a i assignment, when G-consensus exists for a i . Without G-consensus a final assignment is impossible (a i G →?). In a case in which there are just two individuals, DM 1 and DM 2 , whose credibility indices are σ 1 (a i , b h ) and σ 2 (a i , b h ), respectively, agreement will exist between DM 1 and DM Consensus therefore depends not only on the proximity of the credibility indices but also on the cut threshold.
The flowchart of the search group process towards consensus is presented in Fig. 2. After setting up the framework, that is defining an initial structure for the problem (setting the alternatives, the criteria, the criteria relative importance for each DM and other parameters), and arriving at the first individual results with one common arbitrary threshold cut, G is defined originally as M and a set L of possible values for λ is defined. For each of the values λ ∈ L , is determined whether or not a G-consensus solution exists for the entire set of alternatives. If none is found, the value of G is reduced and the search repeated. Once G-consensus solutions are found, the search process can stop, and the selection by the group of one of the G-consensus solutions may potentially complete the process. In the absence of that solution, or if those available prove to be unsatisfactory, or in the case that the group wishes to continue looking for other solutions, it can instead start a consensus-building ("construction") process trying to get close to the central DM. The construction process and the concept of central DM are detailed in the next sub-section.

Constructing One Solution for the Group: Revising Criteria Relative Importance
Let us now analyze how agreement between assignments relates to the difference between the credibility indices of the DMs. The indices are equal to C(a i , b h ) if we consider that q j and p j are shared and the v j is infinite (ignoring veto). Thus, for the analysis we can consider the difference between the general concordance of DM 1 and DM 2 : It can be observed here that, in this case, if the veto effect is ignored (v j is infinite), the shared evaluation of the alternatives and the profiles leads to a situation in which the differences between the indices σ d (a i , b h ) are a function of (proportional to) the differences between the importance levels of the criteria. Taking into consideration the v j (active and defined by the group) as well as the other thresholds, we can analyze the difference between indices σ d (a i , b h ) of DM 1 and DM 2 , now under the influence of the veto effect (E) (Bezerra et al. 2014, for details): and indices will be the same for both DMs and the difference between the σ (a i , b h ) indices for DM 1 and DM 2 will be a function of the differences between their C(a i , b h ) indices. These differences are not proportional, but a direct function of the difference between the parameters of relative importance and so it is possible to achieve consensus by approximating the DMs' relative importance preferences. Therefore, without the veto effect, or with shared v j , reducing the difference between the relative importance of the criteria will approximate the credibility indices and bring consensus. Within the model, this can be obtained on a directed, pair-bypair basis, in which person o seeks to approach person p and vice versa, through α concessions.
In the event that W d = w d j is the relative importance vector of criteria j according to person d, the preference changes can be undertaken by way of α o→ p and α p→o concessions between people: Using the "Data table" 1 feature of Microsoft Excel, combinations of concessions α between two people that might lead to G-consensus can easily be computed and visualised, either for a single alternative or for the entire set A. In this case, minimal concessions required for consensus can be identified with a high level of precision, considering: bilateral concessions(α o→ p , α p→o ); unilateral concessions based on DM o 0, α p→o ; or unilateral ones based on DM p α o→ p , 0 .
It is also possible to verify the necessary individual concessions for consensus, around what we call the central DM. In this case, the DM d with the vector W d whose greatest Euclidian distance from the vectors W i , i = d, is the smallest is identified, and then the minimum necessary concessions are calculated from each DM i to that (central) DM. These calculations can use the Euclidian distance in R t : Note that for different cutting levels and other sets of thresholds, the needed concessions on the relative importance parameters in order to achieve consensus may be different. Thus, it might be helpful to combine the construction process through concessions with the search of other scenarios, or different versions of the model, modifying parameters as λ and G.
The construction process (see Fig. 3 for flowchart), includes the presentation of the minimum concessions necessary on the part of each DM to reach a unanimous consensus, focused on each of the solutions encountered. In order to do this, each solution is compared with one of the individual results. Thus, the existence of central DMs who do not need to make concessions in order to arrive at consensus can be identified. The results of these DMs then come to fulfil the role of guiding to the modifications of

The Context and Planning for the Case Study
A private Brazilian language teaching institution has 20 new students of English and needed to sort these students according to their overall levels of language skills. Until then, students went through an unstructured informal assessment. The board of the organization believe that inadequate programs for skill levels can harm the performance of students and hinder the practice of teaching and the organization's reputation. Thus, they want to improve their process offering different programs according to the different student's level, allocating the students to programs that are appropriate to their stage so they can evolve in a more structured progression. An initial conversation with the general manager set out a proposal in terms of class structure and criteria to deal with the problem. In this proposal, there were four program classes for allocating students, from the most basic to the most advanced, they are: beginner; pre-intermediate; intermediate; and advanced. Note that in this case, there is no upper or lower limit to the number of students in classes, the main characteristics of each class is the type of program offered, and not the physical constitution of the students' group.
Regarding the set of criteria for evaluation, six core competencies were considered: speaking; listening; writing; reading; structuring (grammar); and translating. The organization obtained the scores through different instruments (i.e. test, interview) with different scales. Below we explain each of the criteria, performance tests and scales used.
1. Speaking Refers to the ability to express themselves through comprehensive speech. The evaluation of the criteria is made through informal interview with a native English-speaking teacher, from which the domain level can be considered: Very Good (5); Good (4); Regular (3); Low (2) or Very low (1). 2. Listening The ability to listen and understand what was said correctly. The evaluation is done by presenting short audios, followed by multiple-choice questions to test comprehension. Altogether, there are 10 questions, so that the student can have a score between 0 and 10. 3. Writing Analyses the student's ability to express themselves in written form, it also observes aspects, such as vocabulary and spelling. From a theme provided, the student writes a text with at least 200 words, the assessment goes from 0 to 100%. 4. Reading Concerns the reading skills and reading comprehension. The test for this assessment is the submission of a text, followed by 10 questions with multiple choices to verify the correct interpretation of the text. In this case, the score can range from 0 to 10. 5. Structuring (grammar) Refers to the ability to formulate and recognize correct grammatical structures. The test consists of a series of statements to be marked as true or false and to complete sentences by filling in the blank spaces. The score in this test can also vary from 0 to 10. 6. Translating Evaluates the ability of translating from another language into Portuguese (native language). The assessment requires the translation of a text with approximately 200 words. The possible values range between 0 and 100%.
Therefore, all 20 students went through a complete evaluation, consisting of: one interview; one essay; one translation; and one test with 30 questions, divided into sections with 10 questions each (reading, listening and structuring). The students' performance is shown in Table 1. A planned process was established, for this particular case study, with five stages for the application of VICA-ELECTRE TRI as follows: I. Problem Understanding and Basic Configuration Made by the analyst after collecting the essential information, this stage aims to prepare the analyst to lead the first meeting, summarizing the problem and setting the group goal and a common language. II. Group Meeting-Initial Session i. Validation of the Structure of the Problem Occurs at the beginning of the meeting, when the analyst and the group discuss each concept of the problem: classes, criteria and performance evaluation of the alternatives; ii. Structuring for the Method At this stage, the analyst presents the ELECTRE TRI method to the group, so they can understand and define together values for the parameters of the method: the profiles, indifference and preference thresholds and cutting level; iii. Display of the Alternatives Performances The group is shown a performance graph and analyze some of the alternative performances in order to demonstrate the need to establish criteria relative importance in order to obtain an overall assessment. At the end of this initial phase of the meeting, the DMs establish their individual preferences for the criteria relative importance in private sections, if necessary, with the support of the analyst.

III. Analysis of Possible Paths for Consensus
Considering the initial results, the analyst (this step only involves the analyst) applies the procedures for seeking and building consensus, in order to find possible paths to be presented in the final phase of the meeting. IV. Final Session In this final phase of the meeting, the analyst presents to the group the initial results and gradually introduces the possible modifications for consensus, interacting with DMs and the tool, following the steps: i. Presentation of individual results and their comparisons in pairs; ii. Verification of the acceptability of G-consensus solutions with different cutting levels; and iii. Presentation of the concession mechanism to seek consensus solutions. The analyst can point the minimal concessions needed to achieve consensus. V. Final Report Aiming to structure the analysis, the group was questioned on a set of five propositions about the contributions that the model could bring to the process. Thus, at the end of the application, we were able to analyze the evidence found during the study regarding the following statements: i. Learning and Understanding Provides opportunities for the group to acquire more knowledge about the problem, method and results; ii. Effectiveness and Future Impact Strengthens the perception that the decision will be effectively implemented and that it will influence future ways of deciding; iii. Transparency and Conviction Promotes transparency in how the results are obtained and reinforces the belief in the quality of decisions; iv. Climate and Participation Encourages the maintenance of cooperative spirit during the process and contributes to the participation of all; and v. Structure and Flexibility: Provides sufficient functional support (technological) for structuring the problem, allowing to easily modify, revise or make changes to the parameters and variables.

Structuring the Problem for the Group Decision
The group was composed of four decision makers (board members of the school) that, although holding a high level of education, were not familiar with any MCDA methods. Table 2 lists a profile summary of the DMs. The face-to-face meeting of the group, with all decision makers sitting around a table, where they could view the graphs and tables of VICA-ELECTRE TRI model on a 40" monitor, was conducted in two parts: the first one lasted 1 h and 30 min, with the second lasting 2 h and 15 min. We videotaped the meeting, with consent from all the participants, to facilitate observation and further analysis.
In the introduction, we established that everyone could ask any question and make suggestions or observations at any time of the meeting. Initially, there was a brief explanation about the problem as well as the approach of the ELECTRE TRI method to deal with it. We clarified the goal of the group and the process they would need to follow in order to achieve it. The group agreed on the framework proposed for structuring the problem, that is the group agreed on the use of the ELECTRE TRI method with  the 6 criteria, and the 4 classes of programs where the alternatives (students) would be allocated. The first shared task for the group was to define a set of profile performances to establish the limits for each class. To do this, we combined two displays, the classes' spreadsheet, where the values of the profiles were introduced, and the performance graph showing the representative areas of these classes and the alternative performances ( Fig. 4 presents this display, depicting alternative a 5 ). The DMs visualized the effects of the modifications of the values for the profile performances, decreasing or increasing the representative areas of classes, according to the change of the boundaries. The DMs expressed appreciation for the graphic appeal.
The discussion about the performances of the profiles took longer than expected. The DM 2 , asked whether the model offered some kind of support for defining the

DM Scales
Normalized Lower bound of class g 1 g 2 g 3 g 4 g 5 g 6g 1 g 2 g 3 g 4 g 5 g We offered the analysis of the performance of students in percentiles for support and mentioned the possibility of calculating averages or voting, however, the decision makers chose to follow the suggestion of the DM 4 , discussing the issue in conceptual terms to reach an agreement. The whole process took about 20 min. The DM 3 put forward the opinion that the criterion whose tests were harder should have less demanding profiles to make it easier for the students to reach higher classes. An observation that demonstrated a good understanding of the meaning of these profiles and its relationship with the evaluations on the criteria. This understanding prevailed in the group, influencing the final definition of the profile performances, making them higher for criterion 4 (reading), whose evaluation was considered easier than those of the criterion 2 (listening) and criterion 5 (structuring). Thus, although the three criteria have equal scales from 0 to 10, the performance subsets for the profiles b h ∈ B = {b 3 , b 2 , . . . , b 1 } differ (as seen on Table 3). They considered the same logic for the two other criteria whose scales were in the range [0%, 100%], as the criterion 6 (translating) was considered easier than the criterion 3 (writing). Please see Table 3 for details. In addition to profiles the DMs were asked to set other parameters for the ELECTRE TRI, the first of which was a cutting level, which could be revised along the process. The DM 3 suggested using the average value within the possible range [0.5, 1], which was accepted by all the others without further discussion and thereby we set up λ = 0.75.
After explaining the indifference, preference and veto thresholds, the decision makers asked if they really had to use them. Defining these thresholds requires some cognitive effort, which decision makers did not seem very willing to make (in the opinion of the analyst) so we felt that we should not impose the parameters. Since the use of the veto is optional and the method accepts values equal to zero for p j and q j (quite common in the reported cases of the literature), we defined q j = 0; p j = 0; and an infinite v j (therefore ignoring veto).
The definition of the criteria relative importance, according to each DM, was also needed. We informed the group that the model offers some assistance for this task, but the DMs did not see any difficulty in assigning those values and preferred to establish them on paper and individually. DM 3 even stated: "it's faster this way". Although we asked them to define their preferences individually, the group members discussed amongst themselves the values of their preferences, which may have contributed to the assigning of similar values to the criteria relative importance. Table 4 presents the details.

Analysis: Raising the Possible Consensus Paths
After collecting the values of the relative importance of the criteria the group would have a 30-40 min interval. In the meantime, we studied the possible paths for a consensus solution, using the flowcharts for search (Fig. 2), and for construction (Fig. 3). Figure 5 shows the individual results produced and the situation in terms of Gconsensus for G = 4 (unanimity). Of the 20 alternatives, only three alternatives had differing classifications with λ = 0.75. The expectation was that, because of the similarity among the criteria relative importance and some perceived consistencies in the score data (i.e. students tending to do well in all or any of the tests, see Table 5), there could be a couple of DMs with the same result. However, this was not the case. There were disagreements in all the pairwise comparisons, as follows: the disagreement between DM 1 and DM 2 are for the alternatives (a 5 , a 6 , a 16 ); between DM 1 and DM 3 for (a 6 , a 16 ); between DM 1 and DM 4 for (a 5 ,); between DM 2 and DM 3 for (a 5 ); between DM 2 and DM 4 for (a 6 , a 16 ); and, finally between DM 3 and DM 4 for (a 5 , a 6 , a 16 ). These disagreements can be seen in Fig. 5.
The analysis of possible paths for consensus began with the search for a solution with G = 3, keeping the initial λ = 0.75. Following the flowchart of Fig. 2, we changed the G parameter and it was seen that the majority could not resolve the presented differences, since in all of them there were 2 DMs assigning alternatives to upper classes and 2 who were assigning to lower classes.

Fig. 5 Initial individual results
First, we searched for the central DM, verifying both the Euclidean distance between the vectors of relative importance of the criteria, as well as the number of different alternatives in comparisons between the individual results. We identified two central DMs, equally distant to the others. Then we used the combined concessions data table to see the necessary concessions to be made in order to reach consensus between these central DMs, DM 2 and DM 3 . The comparison between them pointed only one difference, regarding the assignment of the alternative a 5 . The data table with the concessions combinations for consensus pointed out an important asymmetry: for consensus, it would be necessary to make only a small concession α 2→3 = 2% from DM 2 to DM 3 , however, from DM 3 to DM 2 it would be needed a concession α 3→2 = 100%. Actually, their weight vectors were very close and a small change in W 2 would bring the result of DM 2 to match up to DM 3 , keeping λ = 0.75, as shown in Table 5.
To reach unanimity, we still would need from DM 1 and DM 4 2% concessions towards the DM 3 . That is why we also verified the corresponding modifications needed from these DMs, as shown in Tables 6 and 7. The described analyses provided some possible paths for consensus, and now we needed to introduce them to the group.

Building the Group Consensus
We presented to the DMs the Fig. 5 with the visual representation of the individual results, also pointing out the alternatives in consensus and the divergences. Early on, the DM 3 asked which of the DMs were closer to each other in terms of results. We said that, actually, they were all very close to each other. After a brief explanation, the DMs demonstrated an immediate understanding of the visual representations for the assignments shown in the displays of the individual results.
It was explained that with the cutting level of 0.75, it was not possible to obtain a unanimous solution classification for all alternatives, but consensus solutions could be obtained by majority for other λ values such as 0.70 or 0.76, being less or more stringent in the assignment, respectively. However, the group preferred to keep the initial parameter, understanding that it would not be good to "change the rules" that they had already decided. We considered this an interesting observation, because although the possibility of varying the initial parameters established, according to preferences was explained, the group demonstrated a certain resistance to changing parameters already discussed. In addition, we noticed a certain curiosity to know how they could have further discussions for a better understanding about the highlighted differences, within the initial settings of the model.
The group was informed about the differences that, although few in number, could not be resolved by a majority without changing the cutting level. In all the three cases, 2 DMs assigned alternatives in upper classes, while 2 DMs assigned them in contiguous lower classes. Thus, keeping λ = 0.75, consensus building could start with checking the possibilities of concessions between DMs.
As the identified central DMs, DM 2 and DM 3 , found their preferences very close, with only one divergent alternative, we proposed to the DM 2 that she concede α 2→3 = 2%, corresponding to the modifications listed on the Table 2, or DM 3 made the α 3→2 = 100%, which would make her vector of relative importance of the criterion equal to that of DM 2 . The proposed concessions would eliminate the divergence in a 5 , one way or the other. The DM 2 asked to see the performance of that particular student in the performance graph before deciding if she would agree or not on making the 2% concession. Figure 4 shows the display of the performances of a 5 . The discussion lasted longer than expected, since even after analyzing the performance of the student in each criterion; the DMs were unable to reach an agreement.
The discussion involved not only the relative importance of the criteria, but also others issues i.e.: whether it would be better or worse for the business to be more or less rigorous in the classification of the students, the student satisfaction about the resulting assignment and the final impact on the organization's reputation. It turns out that, even though they were very close in terms of relative importance of the criteria, the DM 3 attributed only 5% of relative importance to the criterion 6-translating, while DM 2 set it at 10%, at the expense of the criterion 4 reading, more important for DM 3 . In addition, the DM 2 was concerned that student feels "undervalued" by a very severe classification (in her opinion a 5 should be assigned in class 3), while the DM 4 suggested that the "principles" of the classification should prevail over the interests of the business.
After more than 20 min of debate about the relative importance of criteria, the DM 4 intervened and proposed to make concessions for the central DMs, asking to what direction he should do it in order to reach consensus. On this proposal the DM 1 commented: "that's the spirit!", and DM 2 and DM 3 silenced the discussion. We informed DM 4 that this was optional, meaning that he could concede toward DM 2 or DM 3 depending on his preferences, in other words, on what criteria would he be more willing to modify and the direction of this modification (increasing or decreasing its relative importance), since, in that case the two DMs had the same (Euclidean) distance from him. After a brief assessment, the DM 4 decided to walk towards the DM 3 . Figure 7 shows the comparison between the results DM 3 and DM 4 with λ = 0.75. Note that the pair shows the same differences observed for the group a 5 , a 6 , and a 16 .
The data table for the verification of concessions between DM 3 and DM 4 for consensus showed the same asymmetry between DM 2 and DM 3 . The concession needed from DM 4 was α 4→3 = 2% and from DM 3 was α 3→4 = 100%. The DM 4 decided to make the concession to check how the decision of the group would evolve from that point.
After this concession, we checked if there was a group G-consensus for G = 3 and we found that it was the S3 solution, corresponding to the DM 3 result. The group decided to conclude the search and construction process, considering that they had a final solution to the problem. Even after that, the DM 2 made one last (unsuccessful) attempt to convince the DM 4 to change his decision conceding towards her direction,  arguing that, between them, there were only two divergent alternatives and not 3, as compared to DM 3 . Table 8 presents the results of the case study. After obtaining the final solution, the group had a brief discussion regarding the future use of the model for sorting new students. After that, we asked the decision makers to respond on a paper to a single question: From your perspective, did the model help the group in coming to a solution to the problem? Why? And if so How? Below we present the answers of each DM: DM 1 answer Yes, Because it facilitated the perception that a consensus solution can set a fair evaluation, eventually improving the students learning process. Many gaps in the students' performance were visually apparent during the process. The system stressed the importance of automation in order to provide information to management. It was a careful approach and we got a fair classification, meeting with the expectations/goals of the educational institution. DM 2 answer Yes, The tool enables the consensus in such a way that, it offers solutions on how to bring the individual decisions together. When the percentage to concede is small, it is easier to agree to it. Moreover, only one or some of the decision makers may need to make the concessions and the existence of more than one possibility to achieve consensus makes it more feasible. The method allows decision makers to check these possibilities and accurately measure how much each participant will have to concede. DM 3 answer Yes, It is important to point out that the consensus is made easier with the software and in doing so, it veers the discussion towards the perception of the criteria, how they were chosen, and the importance of each. I noticed that in the discussion about the differences, the mathematical model comes to assist and facilitate the final decision. Visually observing the results is important for the final solution, as it facilitates the decisions. Regarding the translating criterion, for example, it was important to discuss its relevance along with the other objectives and competencies. DM 4 answer Yes, the model shows clearly, in a visually explicative way, the discrepancies in the data of each item (student). As well as a numerical alternative to finding a common decision that takes into account each member's profile, considering their wishes and desires, allowing them to concede with clarity and credibility the needed percentage over the criteria relative importance, in seeking a final consensus.
We could speculate that the group has been generous in the provided answers, considering they knew that the work would be used in a scientific study and it would be nice to emphasize the positive aspects of the model. However, an evidence that they were expressing their actual opinions is shown in the fact that the final result was not reached easily, but only after a lot of work and discussion, where the DMs demonstrated that they were seriously committed to achieving a mutual agreement for a good solution.

Analyst Comments
According to Yin (2010: 138), the Participant Observation is a special mode of observation in which the researcher is not just a passive observer. Instead, he or she can assume various roles in the case study situation and really participate in the process. Its advantages include the fact that it allows information to be captured from an internal point of view. This section intends to report the observations of the analyst during the case study, with no claim to make scientific generalizations, but rather as a complementary scientific methodology in order to collect more evidence to verify the propositions of the work.
To inform the reader about the decision context and to characterize the group, the five perspectives proposed in Marakas (2003) guided us: structure of group; roles within; process; style; and norms of the group. Regarding, the structure and roles, the group consisted of 4 decision makers-business partners, engaged in an informal process, that was developed during a face to face meeting broken up into two sessions. As for the style and standards, we can say that the group is very informal and spontaneous, with little discipline and virtually no rules. However, everyone was very committed to the work, with two of them (DM 2 and DM 3 ) seemly to be more interested in defending their opinions in relation to the other two (DM 1 and DM 4 ) who were more focused in reaching a consensus.
All the following considerations were made taking into account the observations collected during the case study regarding the propositions of Sect. 4, stage V. One by one, we will seek to explain the evidences to confirm or challenge those statements. While we are aware that more studies are required in order to get to generalizable conclusions, we however considered it crucial to analyze and discuss the present case study, in order to take lessons for the following ones.
The first proposition: learning and understanding proposes that the VICA model could provide ways for the group to acquire more knowledge about the problem, the method and the resulting solution. Soon after the explanation of the case, the structure of the model and the ELECTRE TRI method, it did not take long for the group signaled through questions and comments to have a good understanding of the problem, the method and the visual displays. It is observed in the speech of DM 3 :"the consensus is made easier with the software and in doing so, it veers the discussion towards the perception of the criteria, how they were chosen, and the importance of each […]". Our interpretation of this opinion is that the automation of calculations and the structuring of the problem allowed the cognitive group effort to be placed on the fundamental structuring concepts.
The performance chart (Fig. 7) proved especially useful for the DMs to obtain a better understanding of the task, and on three separate occasions they requested its presentation to "check" the model assignment against the relative performance of the alternatives in each criteria. This proved effective for a deeper understanding of the problem, but not so much to bring consensus, as the discussion that was expected to be guided by the criteria importance, was expanded to involve the direct result of some assignments. As the differences were not so many, it was possible to analyze the divergent alternatives one by one without problems, however different cases may present greater challenges.
The second proposition: effectiveness and future impact must be analyzed from two different aspects. As for the effective implementation of the decision obtained, there is no doubt that the group was confident in effectively adopting it. Regarding to future impacts, the issue deserves further discussion. Although they have obtained a consensual solution for the assignment of all 20 alternatives, it was pointed out at the end of the process, that the group had not exactly one vector of the criteria relative importance, but rather a range or as DM 1 mentioned: "a grey area of possibilities". This realization led to conclude that for future decisions with new students or groups of students, we would need to explore within this range. In technical terms, the future group decision would always require a sensitivity analysis of the criteria relative importance to decide assertively. This was an important learning experience in the case study: due to this "grey area of possibilities", it is possible to have the feeling that there is no definitive model for future tasks.
The third proposition concerns the transparency and conviction on the decision. Overall, there was no misgivings about the method or model calculations. For illustration on the subject, the DM 4 says: "allowing them to concede with clarity and credibility", referring to the way the concessions are made and the modifications are seen. It was clear that DMs' lacked the interest to having a deeper understanding of the ELECTRE TRI formulas, they focused only in the general idea of the comparisons, the credibility index and the cutting level. This unwillingness to fully understand the calculations could result in some distrust, low credibility and ultimately harm the transparency of the decision. We believe this problem was overcome during the pro-cess; transparency and credibility were conquered along the way, more especially after checking the assignment of some alternatives in consensus with their performances in each criterion, viewed in the performance chart. Therefore, it was important in the case that the DMs could verify if examples of assignments proposed by the model matched reasonably their holistic assignment expectation.
The fourth proposition relates to how the model could contribute to a good climate and participation. In this specific case, it was evident that the participation varied among the DM. Some were more active than others in their contribution to the discussion. As for the atmosphere, there were a few tense moments, but they eventually dissipated as the meeting progressed. It turns out that it was difficult for the DM 2 to understand the asymmetry of the concessions needed, as we can see in this comment: "how come, I only have to give 2% and she needs to concede 100%?". Although the differences between DM 2 and DM 3 were very small, only on one alternative and two criteria with different relative importance, it was almost impossible to promote agreement between them. We believe it happened not only due to this asymmetry of the concessions, but also to the strong commitment to the established principles, because even after the analysis of the alternative performance on each criterion they disagree. Therefore, we cannot say that, in this case study, the model contributed positively to maintaining the cooperative spirit, not during the whole process and not between all the DMs. However, at end the model was able to promote the achieving of a solution based on a rich discussion of the problem, in a participatory manner.
The proposition five concerns the structure and flexibility of the tool. We attempted to verify if the instrument offered sufficient functional support (technological) for structuring the problem, allowing one to easily make modifications, revisions or changes to the parameters/variables in order to support achieving consensus. Although the numerical experiments had already presented evidence that the tool resources are sufficiently effective, the case study provided additional insights. Some comments highlight the participants' perception in this regard. The DM 4 says: "The model shows clearly, in a visually explicative way […]". The DM 3 highlights: "Visually observing the results is important for the final solution." In addition, DM 1 points, "The system stressed the importance of automation in order to provide information to management." On the behavioral aspect, the approach for the consensus-building process requires the cooperative spirit allowing the group to evolve, as it depends on the willingness to be open towards others opinions. This can be difficult for some people with a more negotiator or competitive profiles. However, for others, more cooperative or just seeking for faster results, it is much easier to concede and the consensus building process happens easier with them.
About the participation of the analyst, it was observed that it is not hard to see through the tool paths (in terms of seeking and building) that can lead to a particular solution. In other words, the analysis can provide enough information so one can veer members toward a specific group result, in a manipulative manner. Therefore, in future applications, we consider important to seek ways to structure the process flow without unduly interference of the analyst, which should be aware of the potential problem and try to avoid bias.
To complete the observations, it is worth mentioning that more time for data analysis might had avoided the lengthy discussion between DM 2 and DM 3 . It happens that in this particular case, we realized retrospectively that it would have been easier to promote the concessions from DMs to a central DM, instead of seeking consensus among the closest central DMs. However, it was not possible to identify a valid general rule for future cases. The case study contributed to corroborating some of the propositions, while for others there was no definite evidence.

Conclusions
We presented the application of VICA-ELECTRE TRI to help the decision of a group on a real problem. The issue at hand was the assignment of twenty students of a language course in predefined and ordered categories, according to six criteria of evaluation with different scale measures, taking into account the preferences of each of the four members of the group. The model provided tools to analyze performances and results, in a comparative, interactive and evolving way. In order to establish an accessible way to reduce the complexity of the multidimensional problem, it provided visual information about the state of each element of the group in the decision process, what effectively supported the group to reach consensus for a collective solution.
Future developments of this study may include research in order to find a model of consensus for future decisions, addressing the issue of the "grey area of possibilities" for the criteria relative importance. Another potential extension concerns approaching the case where the performances of the alternatives are considered differently among DMs, adapting for this use, the indifference and preference thresholds. Finally, we believe that we can apply the principles of VICA for use in other MCDA methods, as well as to evaluate and compare the model's contribution in differently structured processes.