Michalski R., Grobelny J. (2007), Computer-aided subjective assessment of factors disturbing the occupational human performance. Occupational Ergonomics, IOS Press, 7, 27-42
In the paper Subjective Overall Workload Assessment (SOWA) method was proposed and described. A computer information system supporting the process of generating appropriate assessment questionnaires and speeding up the analysis of collected data was presented. Next, the SOWA was practically applied for assessing the subjective workload of persons employed in a Polish sport textile industry company. The obtained results from 18 employees were presented and analysed. Finally, possible advantages, disadvantages, limitations, future works, and a range of applications of the SOWA were discussed.
workload, subjective evaluation, AHP, software
During the occupational work the human being is subjected to various factors disturbing in harmless and efficient job execution. In order to classify and assess these factors and, as a consequence, increase employees’ performance and work safety, many standards, methods and techniques were developed. Among them, substantial part was devoted to workload evaluation. According to Meshkati et al.  they can be generally divided into three groups: performance based techniques, subjective methods and direct physiological measures or, as stated by Chang and Chen , into two categories: subjective and objective. Initially subjective questionnaires were used for assessing these workload aspects that were considered difficult to evaluate objectively. Usually researches focused on mental demands and psychological workload of employers e.g. Cooper-Harper Scale , Bedford Scale [27, 28], Workload Profile , Multiple Resources Questionnaire , Integrated Workload Scale , the mental workload index in supervisory tasks , or scenario-based workload assessment method . One of the best known methods in this area is the Subjective Workload Assessment Technique (SWAT) [19, 26]. The authors distinguished three basic dimensions that allow for determining mental demands: (1) time load, (2) mental effort load, and (3) psychological stress load. Each of these dimensions is rated by the worker on a three level description scale. Next by means of the conjoint measurement and scaling techniques the workload is assessed.
Subjective methods were also used for comprehensive workload evaluation e.g. [6, 36]. Researchers have developed many such tools and some of them have been used in practice for many years. In this area NASA Task Load Index (TLX)  is widely spread. The load assessment in this tool is carried out in six dimensions: (1) mental demand, (2) physical demand, (3) temporal demand, (4) performance, (5) effort and (6) frustration level. The final workload index value depends on the scores given by an employee on a 20 step bipolar scale for the six aspects, and on pairwise comparisons of these dimensions. One of the latest proposals which take into account multidimesional nature of workload was presented by Hwa S. Jung and Hyung-Shik Jung . The major advantage of their method named Overall Workload Level (OWL) is employment of Analytic Hierarchy Process (AHP) developed by Thomas Saaty [31, 32]. In the AHP method, by means of pairwise comparisons of available variants and some relatively simple but arduous calculations, one obtains a vector of weights that allows for setting the hierarchy of importance of the analysed items. The big advantage of the AHP is the possibility of controlling the ratings conformance during the pairwise comparisons by calculating the inconsistency ratio (IR). One of the main drawbacks of this tool is a rapidly growing number of comparisons along with an increasing number of decision variants. The authors of the OWL employed AHP for obtaining the workload index.
There are of course advantages and disadvantages of applying subjective evaluation approaches. On one hand statistically significant correlations with objective results were reported e.g. , and on the other they take into account individual differences in subjective perception of the same work conditions e.g. . So, in order to make the comprehensive workload assessment the best solution is to employ both group of methods, which took place quite often e.g. [2, 9, 11, 20, 23, 34]. The thorough review of the application of subjective measures in ergonomics together with the in depth discussion are presented in the work  followed by a series of commentaries e.g. [4, 10, 14, 16, 22, 37]. In turn, detailed analyses, presentation of virtues and constraints as well as areas of applications of subjective workload evaluation methods can be found, for instance, in the following works [24, 29, 35].
The main objective of this work is to present the Subjective Overall Workload Assessment (SOWA) method with the supporting software, and show in what way it may be applied in practice. The SOWA method was based on the OWL tool with some properties taken from other subjective techniques. The application of the proposed method may make identification of the most important factors influencing workload much easier. Improvements of work conditions in the specified areas give the chance for decreasing the number of various ailments, reducing number of errors made, and improving job satisfaction as well as increasing employees’ efficiency.
2. The SOWA method
In the proposed method, the subjective workload evaluation includes four fundamental dimensions: (1) manual material handling, (2) material work environment, (3) body posture and movement, and (4) mental demand. Each of these dimensions is characterised by several attributes. The detailed structure is presented in Table 1.
The presented method is very similar to the OWL tool. The decomposition of the first three dimensions is almost identical – with some minor names changes. The main difference concerns the mental demand, which in the SOWA method consist of three parameters: time load, mental effort load and psychological stress load, whereas the authors of the OWL method did not decompose their fourth dimension at all. The structure of the mental demand in SOWA is based on three attributes that constitute main areas of the mental demand evaluation in the SWAT technique.
The person being examined is asked to rate each of the specified workload dimensions’ parameters on the 20 step bipolar scale. As opposed to the OWL tool, where five step scale was used, in the presented method it was decided to apply a rating scale similar to the one used successfully in the NASA index. In order to determine weights for individual workload attributes, the AHP technique was used. Subjects first make pairwise comparisons of parameters within the confines of each dimension and then the same procedure is used for calculating weights for the workload areas. Products of these weights and obtained earlier parameters scores are summed up in every dimension. These values are then multiplied by dimensions’ weights. The sum of the four weighted numbers gives us the overall perceived workload assessment index (OPWAI).
Taking advantage of the AHP technique for determining the importance hierarchy of workload components, allows for calculating the inconsistency ratio for every dimension. It is assumed that a IR value less than 0.1 indicates a high comparisons consistency, whereas IR values equalled approximately 0.5 signify considerable incoherence.
3. Program description
In order to help researchers and health and safety specialists to take full advantage of all the merits of the proposed method, specialised supporting software named SOWA was designed and developed. The purpose of the this computer system is to maximize the performance of questionnaire data input, make AHP calculations automatically, and to present the analyses results clearly and comprehensibly. Simultaneously, some of methodological recommendations concerned with constructing questionnaires were taken into consideration during the development process.
3.1. General characteristics
The presented computer system was based on a Microsoft Access database and developed in a Microsoft Visual Basic 6.0 environment. All necessary information regarding carried out examinations, such as personal details, parameter ratings, comparison results along with inconsistency ratios are gathered in a database file. For proper operation the SOWA application requires Microsoft Word 97 and Microsoft Excel 97 or higher installed on a target computer.
In order to conduct subjective workload evaluation by means of the software it is necessary to enter basic information about an examination (Fig. 1) and then generate and print out a required number of assessment questionnaires (Fig. 2).
Next, the printed out forms are filled out by examined persons. The obtained data are entered to the computer program and after that, one can calculate and analyse the results in the form of weighted ratings and inconsistency ratios. The obtained results can also be exported to a Microsoft Excel workbook. The general diagram presenting the way of applying the SOWA system for perceived workload evaluation is shown in Fig. 3.
A main program window is illustrated in Fig. 4. Apart from general information on the given examination, there are two lists of generated questionnaires. The first one relates to forms, from which data were entered to the software database, in the second there are unfilled yet documents. The data input and their modification is possible by double clicking the selected item.
3.2. Questionnaires generation
The questionnaire generation function is run by means of the Generate button (Fig. 4). A general structure of questionnaires is illustrated in Fig. 5.
Every of the generated documents consists of four parts. The questionnaire begins with questions about such personal details as: age, gender, marital status, education, body height and weight as well as information regarding a workplace: position, department, job seniority and information on possible additional working hours. Next, the subject is asked to rate individual attributes of the analysed workload dimensions (Fig. 6).
Consecutive two parts are concerned with pairwise comparisons (Fig. 7). First, the person filling in the form compares parameters within the confines of the individual dimensions, and next makes pairwise comparisons of these dimensions.
For the sake of methodological recommendations concerned with experimental design  in every of the generated document, order of the following components is set randomly by the described computer system:
- parameters in part II,
- parameters comparisons in part III,
- dimensions comparisons in part IV.
Moreover, the order of the individual dimensions in part III of a questionnaire is also set at random. In the document’s footer there are: an examination identification number and a questionnaire identification number (Fig. 7). Because of these IDs it is possible to use the original order of the components during entering data for a specified questionnaire. Such a solution speeds up decidedly the data input.
3.3. Data input
Because the order of some of questionnaire components is drawn for every generated document, the data from a form has to be entered to the corresponding records in a database. The identification numbers located in the footer of a specified questionnaire are used for finding appropriate program entries.
The personal data are inserted using a dialogue window illustrated in Fig. 8. In Fig. 9 a dialog window of entering parameters scores obtained from the persons taking part in the examination was shown. The way the individual attributes are presented in the program is identical to the one visible in a printed out form. The number located on the left hand side of the parameter name specifies the parameter position in a list, while the number on the right is its identifier in the structure from Table 1. The order of attributes in the computer system matches the one in the given questionnaire.
Fig. 10 presents a screen shot of a dialog window used for inputting data regarding pairwise comparisons of attributes as well as workload dimensions. In the window header the dimension number is shown, and next to the parameters’ names there are identification numbers taken from Table 1. Also in this case, the order of comparisons and parameters locations on the screen are consistent with those, printed out on the specified questionnaire (Fig. 7).
3.4. Results presentation
The SOWA software enables the user to present analysis results in the form shown in Fig. 11. Obtained during pairwise comparisons weights for individual parameters along with calculated weighted ratings are visible in the lower part of the dialog window. All scores together with weights are normalized and take a value between zero and one.
For every dimension as well as for dimension comparisons the computer program calculates an inconsistency ratio and displays it under the given list of ratings. Additionally, the software computes an average value of the IR for a questionnaire, which appears in the upper right corner of the dialog window (Fig. 11). Buttons Survey, Parameters and Comparisons run appropriate dialog windows that enable to modify values entered from a questionnaire.
Selecting the Filled in questionnaire checkbox causes that a given questionnaire is treated as if all information was input to the database of the proposed computer system. Such a questionnaire is displayed in the upper list of filled in questionnaires in the main window of the application (Fig. 4).
3.5. Export to MS Excel
All of the data calculated by the software can be exported to Microsoft Excel workbooks after clicking the button Export. Together with the result tables, the program automatically generates appropriate graphs for all of the dimensions included in the SOWA method. The charts contain both relative weights of the individual dimension attributes, and weighted values (Fig. 12–16).
It is possible to export the analyses for multiple subjects at once by selecting in the filled in questionnaires list (Fig. 4) more than one row – keeping the Ctrl key pressed. In this case the results for an individual questionnaire are put in a separate sheet of a single MS Excel workbook. The last sheet of this workbook contains average data computed from all of the exported subjects’ questionnaires.
4. Practical application of the SOWA method: a case study
Eighteen subjects took part in the assessment process. The participants were employed in a company operating in the sport textile industry. The firm was located in Poland, and specialized in designing and manufacturing of sport swimming trunks and costumes. The enterprise employed 50 persons, from which 25 workers were directly engaged in the manufacturing process – mostly seamstresses. The questionnaires were filled in by 14 seamstresses, one seamstress assistant, a seamstress foreman, a cutter, and one cutter assistant. Fourteen of the subjects were married (78%), and among the examined persons there was only one man working as a cutter. Two workers had primary education and two – secondary (high school or equivalent), the rest (14 persons, 78%) were graduates of vocational schools. The detailed characteristics of the analysed employees are presented in Table 2.
Generally, the procedure followed the scheme presented in Fig. 3. After generating the appropriate number of questionnaires, the assessment of the subjective workload was administered during the working hours in groups of several employees. The subjects were informed about a purpose and a detailed range of the study. They were instructed how to fill in the questionnaires and assured about the total anonymity of their answers. The whole procedure for an individual group lasted 20 minutes at most, and usually took less than 15 minutes. Volunteers were allowed to ask questions anytime during the study and they received appropriate explanations and assistance immediately. After collecting all of the questionnaires, the data were entered into the software and then analysed.
The largest value of the overall perceived workload assessment index was registered for the cutter and amounted to 79% of a maximum possible rate. A very high score was also received for the seamstress assistant – 72%. The lowest (mean) value of OPWAI was obtained for seamstresses – 19% (min = 12.4%, max = 34.0%, SD = 6.7%). Similar percentages were calculated for the cutter assistant (26%) and seamstress foreman (22%). Taking into consideration the values calculated for dimensions, it can be noted that the highest shares in the total value of OPWAI was attributed to the manual material handling for the seamstress assistant (48%) and the mental demand component for the cutter (59%). In the case of seamstresses, the least important factor was the MMH – the relative share amounted to 2% (min = 0.5%, max = 7.3%, SD = 2.2%), while the rest three dimensions were assessed almost equally (MWE: min = 3.1%, max = 11.7%, SD = 2.7%; BPM: min = 1.8%, max = 13.9%, SD = 3.6%; MDE: min = 2.6%, max = 17.6%, SD = 4.6%). The data about the overall workload index and mean IR values for every analysed post are presented in Table 3 and, together with calculated weights, illustrated in Fig. 12–16.
It is interesting that seamstresses, a seamstress foreman, and cutter assistant indicated that work environment has the greatest impact on their workload, whereas the weighted score for this dimension occurred to be comparatively small. This means that the MWE component was perceived by these employees as an important factor but its attributes were rated very low. From the Fig. 13, 14, and 16 it can be observed that the structure of relative weights for these workers is very similar: the biggest weights were obtained for MWE, then for BPM, and the MMH and MDE were considered as the least important factors. Although the differences in weights were substantial, the obtained weighted values were little diverse. On the other hand, the seamstress assistant and cutter clearly pointed out one specific dimension. In the case of seamstress assistant, the highest weight and weighted score was registered for the MMH factor, while the cutter showed decidedly that the mental demand related to his activities influenced his workload the most.
The weighted scores for every evaluated attribute allow for further in depth analysis within the confines of the given dimension. Thus, the breakdown showed in Table 4 is very helpful in identifying the most influential workload parameters. As it was already mentioned, the largest weighted scores were given by the seamstress assistant for the manual material handling and by the cutter for the mental demand, so it is justified to study the detailed distribution of these ratings. For instance, it is easy to notice that the seamstress assistant indicated that the frequency of carrying a load (57.5%) had a biggest impact on his workload in the MMD factor, whereas the cutter pointed decidedly to a time load (78.7%).
4.4. Discussion of the results
In order to improve the working conditions in the analysed company, the employer should take some steps especially in areas indicated by the highest obtained weighted scores. This analysis – maybe accompanied by some additional objective workload assessment techniques – may improve the workforce performance, help in reducing the number of errors and accidents and, finally, increase the satisfaction from a work. However, one must not forget that the computed average IR (Table 3) values indicated that the pairwise comparisons were substantially inconsistent. This could result from a fact that in some cases the participants encountered difficulties in distinguishing between the dimension variants because, for example, the given group of factors was of no importance to the workers’ perceived workload. In such a situation, further examination may be necessary and one should be very careful in drawing conclusions. Moreover, the presented analyses were based on single representants of seamstress assistants, seamstress foremen, cutters and cutter assistants, so in order to obtain more representative results, it is required to make additional workload assessments in these posts. It is also recommended to repeat the workload evaluation process in future on a regular basis in order to detect possible problems, all the more that the filling in the forms is not a time consuming activity. Having information about the subjective workload assessment over a longer period of time will give the employer an opportunity to analyse the trend, and can also be treated as a kind of subjective indicator of changes in the existing work conditions.
5. General discussion and conclusions
The proposed SOWA method is relatively simple and moderately time consuming. Apart from the possibility of in-depth analysis of perceived workload in the distinguished areas it enables the researcher to assess the degree of pairwise comparisons coherence. It is possible thanks to the application of the AHP method and inconsistency ratio analyses.
The presented software allows for gathering, in one computer system, information from many various examinations. Moreover it considerably improves and speeds up making the analyses. The proposed computer system is fairly simple and its operation should not cause significant problems, at the same time it enables taking advantage of the AHP technique along with its merits, without the necessity of making arduous and laborious computations. Furthermore, the program takes into account some of methodological recommendations by means of ensuring random order of filling out crucial questionnaire components.
Some of the shortcomings and constraints of the software include the lack of standard indicators corresponding to the obtained overall subjective workload values, which should tell the researcher whether the workload is e.g. higher than in other companies in the same branch of industry. Although the substantial part of the SOWA is based on the OWL method, which was partly validated in the machine processing industry by examining the correlations with objective physiological and epidemiological measurements, the introduced changes require renewed procedures confirming the usefulness of the presented technique. Without this, it is not possible to determine how well the obtained values predict the intensity of the workload. Thus, it should be stressed that in order to convert the proposed framework into the validated instrument that could be applied to a comparative analysis across various industries, empirical evaluation should be conducted and appropriate statistical tests employed to show the reliability and validity of the method. For the sake of these constrains, the calculated outcomes should rather be compared to other workplaces in the given firm. It is also possible to analyse changes in the perceived workload in the course of time.
As in any method based on a questionnaire, there is always a danger of receiving unreliable information from subjects, which may lead to wrong conclusions. Using the described method in practice requires at least basic knowledge about the AHP technique.
The SOWA method along with the computer software may be used for example by companies providing services concerned with workload assessment or health and safety specialists employed in enterprises for systematic monitoring of changes in a workload.
The presented supportive software system will be further developed and improved. The future works will encompass the following:
- Export of the results both for a single and a group of questionnaires to an external file in additional formats, other than supported in this version MS Excel workbooks.
- Verification of the method and software during making analyses of real workload problems.
- Putting additional questions to the generated questionnaire concerned with the subject physical activity after the working hours. Such information will enable to make more thorough workload analyses and will facilitate identification of problems in the given company.
- Adding a possibility of filling up the questionnaire by an employee directly in the SOWA program. This option would allow for carrying out examinations solely in an electronic form in cases where it is possible to seat the subjects at the computer unit.
The described SOWA method as well as the software supporting its application probably can not fully replace objective workload evaluation techniques. However, it may be assumed that the proposed methodology along with the program will become in future a complementary tool for carrying out ergonomic analyses.
 J. Annett, Subjective rating scales: science or art?, Ergonomics 45 (2002), 966–987.
 P. Averty, C. Collet, A. Dittmar, S. Athenes, E. Vernet-Maury, Mental workload in air traffic control: An index constructed from field tests, Aviation Space and Environmental Medicine 75 (2004), 333–34.
 D.B. Boles, L.P. Adair, The Multiple Resources Questionnaire (MRQ), in: Proceedings of the Human Factors and Ergonomics Society, 45, 2001, pp. 1790–1794.
 K.A. Brookhuis, D. Waard, On the assessment of (mental) workload and other subjective qualifications, Ergonomics 45 (2002), 1026–1030.
 J. Brzeziński, Elementy metodologii badań psychologicznych, Państwowe Wydawnictwo Naukowe, Warszawa, 1980.
 S.Y. Chang, T.H. Chen, Discriminating relative workload level by data envelopment analysis, International Journal of Industrial Ergonomics 36 (2006), 773–778.
 H.A. Colle, G.B. Reid, Context effects in subjective mental workload ratings, Human Factors 40 (1998), 591–600.
 G.E. Cooper, R.P. Harper, The use of pilot ratings in the evaluation of aircraft handling qualities, NASA Ames Technical Report NASA TN-D-5153, NASA Ames Research Center, Moffett Field, CA, 1969.
 A. DiDomenico, M.A. Nussbaum, Interactive effects of mental and postural demands on subjective assessment of mental workload and postural stability, Safety Science 43 (2005), 485–495.
 C.G. Drury, Measurement and the practising ergonomist, Ergonomics 45 (2002), 988–990.
 R. Frey, K. Decker, L. Reinfried, G. Klosch, B. Saletu, P. Anderer, H.V. Semlitsch, D. Seidler, A.N. Laggner, Effect of rest on physicians’ performance in an emergency department, objectified by electroencephalographic analyses and psychometric tests, Critical Care Medicine 30 (2002), 2322–2329.
 A. Gregoriades, A.G. Sutcliffe, Automated assistance for human factors analysis in complex systems, Ergonomics 49 (2006), 1265-1287.
 S.G. Hart, L.E. Staveland, Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research, in: Human Mental Workload, P.A. Hancock, N. Meshkati, eds, Elsevier, Amsterdam, 1988, pp. 139–183.
 H. Heuer, Measurements serve a purpose: a note on a possible use of natural measurements, Ergonomics 45 (2002), 1015–1017.
 H.S. Jung, H.-S. Jung, Establishment of overall workload assessment technique for various tasks and workplaces, International Journal of Industrial Ergonomics 28 (2001), 341–353.
 H. Kanis, Can design supportive research be scientific? Ergonomics 45 (2002), 1037–1041.
 S. Levin, D.J. France, R. Hemphill, I. Jones, K.Y. Chen, D. Rickard, R. Makowski, D. Aronsky, Tracking workload in the emergency department, Human Factors 48 (2006), 526–539.
 D.Y. Lin, S.L. Hwang, The development of mental workload measurement in flexible manufacturing systems, Human Factors and Ergonomics in Manufacturing 8 (1998), 41–62.
 A. Luximon, R.S. Goonetilleke, Simplified subjective workload assessment technique, Ergonomics 44 (2001), 229–243.
 R. Ma, D.B. Kaber, Presence, workload and performance effects of synthetic environment design factors, International Journal of Human-Computer Studies 64 (2006), 541–552.
 N. Meshkati, P. Hancock, M. Rahimi, Techniques in mental workload assessment, in: Evaluation of human work. A practical ergonomics methodology, J. Wilson, E. Corlett, eds, Taylor & Francis, London, 1992, pp. 605–627.
 J. Michell, Do ratings measure latent attributes?, Ergonomics 45 (2002), 1008–1010.
 S. Miyake, Multivariate workload evaluation combining physiological and subjective measures, International Journal of Psychophysiology 40 (2001), 233–238.
 J.B. Phillips, D.B. Boles, Multiple resources questionnaire and workload profile: Application of competing models to subjective workload measurement, in: Proceedings of the Human Factors and Ergonomics Society, 48, 2004, pp. 1963-1967.
 L. Pickup, J.R. Wilson, B.J. Norris, L. Mitchell, G. Morrisroe, The Integrated Workload Scale (IWS): A new self-report tool to assess railway signaller workload, Applied Ergonomics 36 (2005), 681–693.
 G.B. Reid, T.E. Nygren, The subjective workload assessment technique: A scaling procedure for measuring mental workload, in: Human mental workload, P.A. Hancock, N. Meshkati, eds, Elsevier, Amsterdam, 1988, pp. 185–218.
 A.H. Roscoe, The practical assessment of pilot workload, AGARD-AG-282, Advisory Group for Aerospace Research and Development, Neuilly Sur Seine, France, 1987.
 A.H. Roscoe, G.A. Ellis, A subjective rating scale assessing pilot workload in flight. A decade of practical use, Technical Report 90019, Royal Aerospace Establishment, Farnborough, Royal Aerospace Establishment, UK, 1990.
 S. Rubio, E. Díaz, J. Martín, J.M. Puente, Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA-TLX, and Workload Profile Methods, Applied Psychology: An International Review 53 (2004), 61–86.
 K. Ryu, R. Myung, Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic, International Journal of Industrial Ergonomics 35 (2005), 991–1009.
 T.L. Saaty, The analytic hierarchy process, McGraw Hill, New York, 1980.
 T.L. Saaty, The analytic hierarchy process, RWS Publications, Pittsburgh, 1996.
 P.S. Tsang, V.L. Velazquez, Diagnosticity and multidimensional subjective workload ratings, Ergonomics 39 (1996), 358–381.
 S. Usui, Y. Egawa, Psycho-physiological analysis of mental workload at an elevated work place, Japanese Psychological Research 44 (2002), 152–161.
 M.A. Vidulich, P.S. Tsang, Techniques of subjective workload assessment: A comparison of SWAT and NASA-Bipolar methods, Ergonomics 29 (1986), 1385–1398.
 S.S. Yeung, A. Genaidy, J. Deddens, S. Sauter, The relationship between protective and risk characteristics of acting and experienced workload, and musculoskeletal disorder cases among nurses, Journal of Safety Research 36 (2005), 85–95.
 M.S. Young, N.A. Stanton, It’s all relative: defining mental workload in the light of Annett’s paper, Ergonomics 45 (2002), 1018–1020.