Rafal Michalski, DSc, PhD, Eng, Wroclaw, Poland
Home Publications Software Research Teaching Consulting Contact 
Major articles (JCR)Other articlesCitationsExternal bibliometric databases
This is the preprint version of the following article:
Ulutas B.H., Ozkan N.F., Michalski R. (2019). Application of hidden Markov models to eye tracking data analysis of visual quality inspection operations. Central European Journal of Operations Research.
Cited (JCR): 0, Other Cites: 0


Visual inspection is used in many areas due to the potential high costs of in-spection error such as injury, fatality, loss of expensive equipment, scrapped items, rework, or failure to procure repeat business. This study presents an application of Hidden Markov Models (HMM) to fixations’ sequences analy-sis during visual inspection of front panels in a home appliance facility. The eye tracking data are gathered when quality control operators perform their tasks. The results support the difference between expert and novice operator. Moreover, the article demonstrates four HMMs with two and three hidden states both for novice and experienced operators and provides analysis and discussion of the outcomes.


human factors, visual inspection, human visual behavior, hidden Markov models, eye tracking technology

1. Introduction

Inspection is defined as a deliberate, in-depth, exacting process that requires more than mere looking or scanning (See, 2012). Drury and Prabhu (1992) point out preci-sion, depth, and validity are critical elements of the definition. Inspection processes require a large amount of mental processing, concentration, and information trans-mission, along with extensive use of both short-term and long-term memory (Gall-wey, 1982).

Eye tracking studies help to analyze the human visual activities. An eye-tracker is a device for measuring eye positions and eye movement (Jacob & Karn, 2003). There are several applications of eye-tracking technologies such as, web usability, advertising, sponsorship, package design, and automotive engineering. In general, commercial eye-tracking studies focus on a sample of consumers and the eye-tracker is used to record the activity of the eye. Michalski and Grobelny (2016) present re-sults of the eye-tracking data for box package designs. Michalski (2017) analysis digital control panels in terms of effectiveness and efficiency. Ozkan and Ulutas (2017a) use eye-tracking to assess knowledge and behavior of medicine information leaflet. Ozkan and Ulutas (2017b) provide results for evaluation of medicine leaflets on screen. Mobile eye-trackers can be used in manufacturing environments such as introduced by Ozkan and Ulutas (2016) that utilize data as an indicator of cognitive workload for forklift drivers. Ulutas and Ozkan (2017a) compare fixation data from expert and novice operators conducting visual control activities for ceramic tile inspection. Apart from manufacturing environments, tower crane operator’s eye tracking data to assess their performance is a pioneering application in construction sites (Ulutas and Ozkan, 2017b).

Khasawneh et al. (2003) attract attention to the inspection task during quality control, particularly the search portion and use of eye-tracking technology. See (2015) provides the first empirical data to address the reliability of visual inspection for precision manufactured parts. In a recent study, See et al. (2017) summarize the factors influencing inspection performance and suggest to explore new research areas in terms of visual inspection by considering automated inspection. Gramo-padhye et al. (1997) focus on importance of visual inspection and the principles of effective training. Speed (2015) summarizes the issues associated with quantifying expert, domain-specific visual search behavior in operationally realistic environ-ments.

By examining a series of fixations and saccades it is possible to generate infor-mation about the attention shifting process that accompanies visual task executions. The fixation position depends on the location of the previous fixation. Ellis and Stark (1986) suggest that scan paths can be modeled by a stochastic first-order Mar-kov process. Observable eye movements can be associated with overt attention while hidden states from HMM are coupled with shifting covert attention (Findlay and Gilchrist, 2003). HMM tools allow for finding stochastic relations between observa-tions and hidden states and, thus, are very useful in broadening the basic knowledge about attentional visual processes.

Two-states HMM that corresponds to covert attention (local and global) regis-tered during visual analysis of printed advertisements saccades’ lengths are illustrat-ed by Liechty et al. (2003). Chuk et al. (2014) identify two specific attention patterns (holistic and analytic) for the analysis of the face recognition task. During textual information search, Simola et al. (2008) HMM scan paths investigation provide evidence for existing three consecutive hidden states (scanning, reading, and the answer). Grobelny and Michalski (2016) estimate four HMMs with three hidden states for the digital control panels examined.

One of the applications of HMMs in the industrial management context is con-cerned with fall detection and localization of the operator. Kianoush et al. (2015) use HMMs to monitor various worker’s postures and detect a fall event by analyzing the changes in the radio-frequency signal quality.

Given more and more accessible mobile eye-trackers that are becoming increas-ingly accurate, a more ecological approach is started being more appreciated in psy-chology of human visual behavior (e.g., Tatler et al. 2011). Lukander et al. (2017) describe a number of further studies in this field. They review eye-tracking experi-ments where visual behavior is recorded during everyday activities outside research laboratories. One of the directions in this trend focuses on providing classifications and models based on eye-tracking data. For example, Greene et al. (2012) use a line-ar discriminant classifier to verify if the complex mental states to be inferred from eye tracking data as it was suggested by Yarbus (1967) whereas Coutrot et al. (2018) provide a method for scan path modeling and classification using variational HMMs and discriminant analysis. Similar methods, also involving HMMs are also em-ployed by Kit and Sullivan (2016) for naturalistic eye movement behavior, while an interesting and comprehensive review and comparative analysis is provided by Boisvert and Bruce (2016).

Though a number of papers recently have been published that involve HMMs in eye-tracking data analysis, most of them focus on classification problem. Moreover, studies on how these models can be used in work environment analyses are very scarce. Among the few research in this area, Sodergren et al. (2010) utilize data of subjects using eye-tracker when performing natural orifice translumenal endoscopic surgery. Unfortunately, HMMs are only used for profiling subjects and the authors did not include any substantial analysis of participants’ visual attentional behavior. Another possible application of employing HMMs in the industrial environment may be related to automatic face recognition. A number of studies published in this domain are rather concerned with improving the effectiveness of this method in general (e.g., Chuk et al., 2014), than to examine if and how it might be useful in the industrial environment.

Based on the accessible literature, it can be said that data obtained by use of eye-tracking technologies in real life manufacturing environments are not analyzed by use of HMM. Therefore, this study focuses on the visual activity of quality inspec-tion operator tasks. Basic research question is defined as the qualitative differences between subjects’ visual activities and the analyses are carried out in the HMM per-spective. Second section summarizes the basics of HMM. Following section ex-plains the control panel experiment. Then, the outcomes of the HMM simulation results are discussed and the paper is concluded with directions for future studies.

2. Hidden Markov Models brief overview

Markov models, also called as processes or chains, are introduced by a Russian mathematician Andrei Andreyevich Markov in 1913. Generally, they deal with states where the transitions between states specify probabilities of their occurrences. An HMM is a statistical Markov model in which the system being modeled is assumed to be a Markov process with unobserved (i.e., hidden) states. The HMM can be rep-resented as the simplest dynamic Bayesian network. The detailed HMMs specifica-tions can be found in various sources. One of the most comprehensive explanation is provided by Rabiner (1989).

A discrete, first order HMM typically comprises of a set of hidden states S = {S1, S2, ..., SN}, a group of M observations for every state V = {v1, v2, ..., vM}, a states’ transition probability matrix which provides likelihoods of switching from state i to state j: A = {aij}, the emission probabilities matrix B = {bj(k)}, where bj(k) = P[vk at t | qt = Sj,]; 1 ≤ j ≤ N, 1 ≤ k ≤ M, and the starting probability distribu-tion π = {πi}, where πi = P[q1 = Si ], 1 ≤ I ≤ N, and qt denotes the state at time t. HMM models are often defined by a set of three probability matrices λ = (A, B, π) that produces the sequence of O = O1O2  OT observations, where every Ot is one of the V symbols, whereas T is the total length of the observation sequence. In section 5 of the current paper, the HMMs parameters (A, B, π) are estimated based on obser-vation sequences derived from the eye-tracking data, assume number of states, and a vocabulary corresponding to the defined areas of interests (AOIs).

3. Method

3.1. Company overview

The facility in concern is located in Turkey and is structured in a 10000m² closed and 12640m² total area. Currently, there are 24 white-collar employees and 222 blue-collar personnel. More than 400000 tumble driers are usually manufactured every year, 99% of which are exported. The facility has three assembly lines. Two of these lines has 22 workstations and the third one has 6 workstations. The products undergo functional and electrical tests after the assembly is completed.

3.2. Inspection area in concern

Depending on the model of the tumble drier, the plastic control panel may be pro-duced in different colors (i.e., white, silver, and black). Each front panel requires to be controlled by a quality control operator before it is transferred to the related as-sembly workstation. The inspection area in concern is presented in Fig. 1. The pic-ture comes from a Tobii Pro Glasses Analyzer snapshot that is the original viewpoint of the control operator.

Fig. 1. Quality control area from the point of an operator’s view

Main steps of the visually based quality control are explained as follows based on the numbering in Fig.1:

  1. Related information is printed on the front panels in another department of the facility, carried to the inspection area in deep cartoon boxes, and placed on the left hand side of the control operator.
  2. The smaller cartoon box keeping the digital cardboards that need to be mounted to the backside of the panels are placed in front of the control operator. The box is replaced with a new one and seal of the new box needs to be opened approxi-mately after 60 panels.
  3. Operator takes one front panel from the deep box and places it on the control ta-ble front side up. After visual inspection of front side is completed, panel is turned backside and backside of the panel is inspected. Two screws are taken from box (4) and mounted to the back side. If a defect (i.e., scratch, faded color) is detected, a colored tape (5) is sticked on the panel and separated to the area (8) that is on the left hand side of the control table.
  4. If the panel meets the defined specifications, it is first placed on the area (6), then placed in the carton box (7) that is on the right hand side of the control table to be send to the related assembly workstation. When the box is full, it is carried to the material handling area and an empty box brought.

3.3. Visual inspection task description and identification of areas of interests (AOI)

The visual inspection of the front control panel in concern is represented in Fig. 2. In eye-tracking studies, they are usually called areas of interests (AOI) and they are defined as the main independent variable in the present research. Fig.3. illustrates the areas on the panel that are of special interests.

Fig. 2. Panel in concern

Fig. 3. Representation of AOIs and their abbreviations

Details of AOIs characteristics are summarized in Table 1. The quality control engineers in the facility define highest importance for the area named as 01_BN because it includes information of the brand and model of the tumble drier. Faded or not aligned painting in this area cannot be tolerated. Likewise, the area that is named as 06_DP that lies in the middle of the panel and keeps the digital panel screen have high importance because any scratch in this area has a potential to be quickly real-ized by the customer. The space area (03_SA) attracts attention of the customer if there is scratch. Therefore, needs to be inspected carefully.

The control operators are asked to focus on the printed information on the two sides of the black digital section. These areas include several printed information including, brand name, loading capacity, energy class, and program setting infor-mation. The printing may be faded or not aligned properly in these areas. Therefore, adequate time is required during visual inspection. 07_RH have lowest importance that lies on a smaller area compared to other AOIs defined.

The visual activities related to AOIs are examined and analyzed separately for one novel and one experienced female quality control operators. We take advantage of the fixation sequences registered within the confines of the defined AOIs to model the possible hidden visual quality inspection task states by HMMs.

	Table 1. Characteristics of AOIs and their abbreviations for the panel in concern
	AOI no	Abbreviation	Definition	Possible defect	Importance
	1	01_BN	Brand name	Faded or not aligned 	High
	2	02_BT	Brand type	Faded or not aligned 	Medium 
	3	03_SA	Space area	Scratched 	High 
	4	04_LE	Information Left Top 	Faded or not aligned 	Medium 
	5	05_LP	Information Left Bottom	Faded or not aligned	Medium
	6	06_DP	Digital panel screen	Scratched	High 
	7	07_RH	Information Right 	Scratched	Low
	8	08_RK	Information Right 	Faded or not aligned 	Medium
	9	09_RE	Information Right 	Faded or not aligned	Medium
	10	10_PS	Program setting information	Faded or not aligned 	Medium

3.4. Data collection procedure

Several data can be obtained by use of an eye-tracker and evaluated for the purpose in concern. Data such as; fixation (the length of a fixation is usually an indication of information processing or cognitive activities), saccade (the rapid eye movement occurring during reading and when viewing visual patterns), smooth pursuit (slow eye movements that stabilize the images of a slow moving target on or near fovea), and change in the pupil size (increases in response to light, emotional stimuli, atten-tion, working memory workload, pupillary unrest) are recorded by use of an eye tracking glasses that has the components such as illuminators, cameras, and the pro-cessing unit containing the image detection, 3D eye model and gaze mapping algo-rithms. In this study, participants’ eye movement data are collected through a mobile eye-tracker (Tobii Pro Glasses 2) that is represented in Fig. 4.

Fig. 4. Representation of the eye-tracker used


The study is conducted in Eskisehir, Turkey, during January-May 2016 daytime shift at a home appliance facility. Two participants who represent the population for the visual inspectors are considered. The female participants who are 25 and 30 years old have no visual problem such as near, far sighted, or astigmatic. Each par-ticipant is tested individually in the workplace represented in Fig.1. They were not extra paid for their participation in the examination. The sequence of the test is as follows:

  1. Introduction of the nature and aim of the test.
  2. Calibration of the Tobii Pro Glasses 2 device for participant’s gaze.
  3. Performance of the inspection task for 60 panels to identify possible defects as summarized in Table 1.

The experimental eye-tracking data are first elaborated in the Tobii Pro Glasses Analyzer software, version 1.95 (https://www.tobiipro.com/learn-and-support/downloads-pro/), then exported to text files and further processed in in TIBCO Statistica 13.3 package (TIBCO Software Inc., 2017). The first order, discrete time, HMMs are estimated according to the Baum-Welch procedure (Baum, 1972). The functions elaborated by Murphy (1998, 2005) are used to calculate estimates of initial, transition, and emission probabilities. The maximum number of iterations are set at 1000 whereas the convergence threshold equaled 0.0001. All the HMM com-putations are performed in a 64 bit version of Matlab (9.5.0) R2018b.

4. Overview of the recorded visual activity

The eye-tracker is used when experienced and novice control operator are inspecting one panel at a time. Table 2. summarizes the metrics for ten AOIs in concern gener-ated by Tobii Analyzer software. Fixation count, total fixation duration, visit count, average visit duration, and time to first fixation data for experienced and novel oper-ators are provided. During the inspection performed by the novel operator, no signif-icant data are recorded for 09_RE and 10_PS AOIs.

	Table 2. Eye-tracking metrics for the operators’ visual activity 
		Experienced operator	Novice operator
	AOI	Fixation Count	Total Fixation 
	Duration*	Visit Count	Average Visit
	Duration*	Total Visit
	Duration*	Time To First
	Fixation	Fixation Count	Total Fixation
	Duration*	Visit Count	Average Visit
	Duration*	Total Visit
	Duration*	Time To First
	01_BN	67	1.34	11	0.12	1.34	21.49	33	0.66	8	0.08	0.66	1.62
	02_BT	18	0.36	3	0.12	0.36	32.37	24	0.48	5	0.10	0.48	1.34
	03_SA	15	0.30	6	0.05	0.32	4.54	17	0.34	9	0.04	0.34	1.30
	04_LE	50	1.00	18	0.06	1.00	4.08	57	1.14	13	0.09	1.14	0.56
	05_LP	54	1.08	20	0.05	1.08	4.62	46	0.92	13	0.07	0.92	1.20
	06_DP	340	6.80	35	0.29	10.10	3.74	191	3.82	14	0.57	7.92	0.00
	07_RH	25	0.50	10	0.05	0.52	13.98	2	0.04	1	0.04	0.04	7.00
	08_RK	21	0.42	6	0.07	0.42	20.17	6	0.12	1	0.12	0.12	6.80
	09_RE	16	0.32	4	0.08	0.32	40.83	–	–	–	–	–	–
	10_PS	114	2.28	12	0.19	2.34	18.62	–	–	–	–	–	–
	Mean	72	1.44	12.5	0.11	1.78	16.5	47	0.94	8	0.14	1.45	2.48
	Std. Dev.	105	2.10	10.3	0.08	3.17	13.5	65.8	1.32	5.7	0.19	2.83	2.98
	* Times are provided in seconds

Heat maps are meant to visualize how fixation duration is distributed. Red color represents the area where the operator spends more time during inspection. Yellow and green colors represents shorter duration periods respectively. Fig.5 represents the heat map for the experienced operator and Fig.6 for the novice operator. It is clear that experienced operator is capable of inspecting the areas that are determined as to have the highest importance (01_BN, 03_SA, and 06_DP ). However, as also supported with the data from Table 2, novice operator completely misses to inspect 09_RE and 10_PS AOIs.

Fig. 5. Heat map for the experienced operator

Fig. 6. Heat map for the novice operator

5. Modeling visual activity by HMMs for novice and experienced operators

5.1. Simulation experiment

As the research outcomes of Hayashi (2003) on novice and expert pilots suggest, they might use a qualitatively different visual strategies. A simulation experiment is designed and conducted to verify the best number of hidden states necessary to mod-el the visual behavior of novice and expert workers. Overall, 10 conditions were examined. They were differentiated by five possible hidden states (from 2 to 6) and two types of workers (WT: novice and expert): 5 (States) × 2 (WT).

It is well known that the HMM estimation results depend on initial probabilities. Thus, to compensate for this, 100 single simulations are run for each experimental condition with random starting values. Each and every model is evaluated by Akaike’s (1973) Information Criterion (AIC), and Schwarz’s (1978) Bayesian Infor-mation Criterion (BIC). Obtained in current experiment values for these criteria, e.g., their means, standard deviations (in brackets), minimum values, as well as log-likelihoods are presented in Table 3.

	Table 3. The HMM simulation results for expert and novice workers.
	Standard deviations are given in brackets

	experience	No of
	states	AIC	BIC	Log-likelihood
				Mean	Min	Mean	Min	Mean	Max
	1	Expert	2	229	(5.6)	221	279	(5.6)	271	-88.7	(2.82)	-84.6
	2		3	248	(4.9)	237	328	(4.9)	317	-81.9	(2.46)	-76.4
	3		4	273	(4.3)	267	388	(4.3)	381	-76.6	(2.13)	-73.3
	4		5	304	(4.6)	291	457	(4.6)	444	-71.9	(2.29)	-65.5
	5		6	339	(4.4)	327	534	(4.4)	522	-67.3	(2.18)	-61.3
	6	Novice	2	96	(1.2)	96	121	(1.2)	120	-22.1	(0.59)	-22.0
	7		3	121	(1.4)	119	161	(1.4)	159	-18.4	(0.71)	-17.4
	8		4	152	(1.5)	150	209	(1.5)	207	-16.0	(0.77)	-15.0
	9		5	187	(2.4)	184	263	(2.4)	260	-13.7	(1.18)	-12.2
	10		6	229	(2.7)	226	325	(2.7)	322	-12.5	(1.35)	-10.8

Given that smaller values of AICs and BICs signify more appropriate models. Ob-tained results unanimously suggest two-states HMMs irrespective of the operator’s experience. Such a finding is somewhat surprising as it is in contrast with Hayashi (2003) results for the pilots eye movement data registered while landing. It is pro-posed that models with larger number of states better explain the visual behavior of more experienced pilots.

These two criteria are known for their bias towards more parsimonious models favoring solutions with as smaller number of parameters to be estimated. Such a property may lead to selecting models with too few states to adequately reflect the observed data structure. On the other hand, the log-likelihood itself increases along with the larger number of states suggesting the need for more and more parameters in models. Naturally, this approach may deliver solutions fitting the noise. Unfortu-nately, the present simulation experiment outcomes do not provide clear indication on how many states to choose. Though, the smallest values of AICs and BICs suggest two-states HMMs both for the novice and expert operators. One may have doubts whether the smallest possible number of states is enough to explain the registered shifts in visual attention. Therefore, we present and analyze both two- and three-states HMMs in the next section.

5.2. Analysis of HMMs

According to the AIC and BIC criteria, the HMMs with the smallest possible number of hidden states are considered. For comparison purposes, estimations of HMMs for the three states models are also presented. From among a hundred models computed for a given number of states, the models which have AIC and BIC values close to their minima and log-likelihoods near maximum values are selected. Further, the models that provide logical and meaningful interpretation in the examined context are considered. Table 4 contains two and three states HMMs determined as the most appropriate for representing the visual attention shifts registered for novice and ex-pert operators. Second rows in each table contain initial states probabilities (π), con-secutive two (or three) rows include probabilities of transitions between each pair of states (A) whereas the rest of the table presents emission probabilities (B).

Analyzing two-states HMMs emission likelihoods it is quite obvious that visual inspections performed by the novice worker ignore some AOIs (03_SA. 07_RH. 08_RK. 09_RE. 10_PS). In contrast, the HMM for the expert shows attention shifts to all the predefined AOIs. This outcome is fully consistent with the qualitative analysis of the obtained heat maps presented in Figs. 5 and 6.

Another difference between these models is visible in transition probability ma-trices. High values on the diagonal suggest that the attention is very likely to stay in the same state. This may be interpreted as being more focused on the currently per-formed subtask. Despite high diagonal values, there are still considerable probabili-ties of switching to the other state. This indicates that the experienced worker jumps between the two states from time to time. On the other hand, the novice operator transitions imply different pattern of attention switching strategies. In both cases, the two-states HMMs shows that the operator starts with the second state related with searching for scratched surfaces. However, the less experienced operator is more prone to shift to the first state than to stay in the current one. Next, the novice operator is likely to stay in this first state, and there is very little chance to come back to the second state.

The three-state HMMs demonstrated in Table 5 confirm considerable differences in visual attention shifts performed by both operators. Examining three-state mod-els, one may notice similar phenomenon as in two-states models, i.e., visual inspec-tions of the less experienced operator produced no fixations in a number of AOIs as opposed to the expert user’s model where emission probabilities in all AOIs were estimated.

The fact that no fixations are observed in many regions of the visually examined panel for the less experienced operator, may result from using peripheral vision, which cannot be registered by eye-trackers. Such a general view of the whole panel may be associated with the zoom-lens model (Eriksen & James, 1986). Though here, the second phase of the model is not detected. In turn, the visual attention behavior of the experienced operator suggest rather applying the spotlight strategy (Posner et al., 1980) for a thorough and systematic visual inspection of panel quality.

Likewise in two-states HMMs, we can presume also for the three-states HMMs that the states indicate searching for some specific types of flaws, namely, searching for scratches, misalignments, and faint elements. The analysis of the models from Table 5 together with information from Table 1 suggests that the S3 state may re-flect scratches identification. However, a more detailed analysis of this proposal shows that it is not fully compliant with information gathered from operators, espe-cially in terms of searching for scratches. Since identified AOIs are differentiated by their importance to the manufacturer, the presented hidden states from models may correspond to some combination of defect type and importance components of the quality control process.

	Table 4. Two states HMMs for the experienced and novice operators
		Experienced	Novice

	S1	S2
		π	0.0000	1.0000	0.0003	0.9997
		S1	0.8970	0.1659	0.9999	0.7374
		S2	0.1030	0.8341	0.0001	0.2626
		01_BN	0.0000	0.1384	0.2135	0.0000
		02_BT	0.1442	0.0000	0.2135	0.0000
		03_SA	0.0000	0.0277	0.0000	0.0000
		04_LE	0.0000	0.1384	0.0000	0.2076
		05_LP	0.0000	0.1384	0.2961	0.1273
		06_DP	0.0140	0.5205	0.2769	0.6651
		07_RH	0.1442	0.0000	0.0000	0.0000
		08_RK	0.1442	0.0000	0.0000	0.0000
		09_RE	0.1442	0.0000	0.0000	0.0000
		10_PS	0.4091	0.0367	0.0000	0.0000
	Table 5. Three states HMMs for the experienced and novice operators
		Experienced	Novice

	S1	S2
	S3	S1	S2
		π	0.0000	0.6471	0.3529	0.0000	0.3750	0.6250
		S1	0.8882	0.2082	0.0000	0.6092	0.6755	0.0000
		S2	0.1118	0.7913	0.0000	0.0000	0.3245	0.2500
		S3	0.0000	0.0005	1.0000	0.3908	0.0000	0.7500
		01_BN	0.0000	0.2603	0.0000	0.4387	0.0000	0.0000
		02_BT	0.0000	0.0000	0.1026	0.4387	0.0000	0.0000
		03_SA	0.0000	0.0521	0.0000	0.0000	0.0000	0.0000
		04_LE	0.0000	0.0527	0.2046	0.0000	0.3676	0.0000
		05_LP	0.0000	0.2603	0.0000	0.1226	0.6324	0.0000
		06_DP	0.0750	0.2937	0.6420	0.0000	0.0000	1.0000
		07_RH	0.0895	0.0000	0.0507	0.0000	0.0000	0.0000
		08_RK	0.1769	0.0000	0.0000	0.0000	0.0000	0.0000
		09_RE	0.1769	0.0000	0.0000	0.0000	0.0000	0.0000
		10_PS	0.4817	0.0809	0.0000	0.0000	0.0000	0.0000

6. Conclusions

6.1. Summary of the findings

In the current research we have presented and analyzed eye-tracked data of quality assurance workers performing visual inspection tasks in their natural working envi-ronment. As mentioned in the introduction of this article, such an approach is being increasingly popular in research involving both eye-tracking techniques (Tatler et al. 2011; Lukander et al., 2017) and HMMs (Kit and Sullivan, 2016; Boisvert and Bruce, 2016).

We examined basic eye-tracking metrics, qualitative heat maps, and compared them with obtained HMMs. We analyzed their visual attention patterns and searched for differences between novice and expert operators. The simulation studies aimed at providing the most appropriate number of hidden states to capture visual registered visual characteristics. Our results in this aspect did not confirm results from a previ-ous study of Hayashi (2003), where experienced pilots exhibited more hidden states than novice ones. In our case, the simulation outcomes suggested the same number of hidden states for both novice and experienced operators.

Further, two and three HMMs were estimated and thoroughly analyzed. These models showed that despite the same number of hidden states, these two types of operators employed considerably different visual strategies. It seems that experts used the spotlight strategy (Posner et al., 1980), which well corresponds to a thor-ough and systematic visual inspection whereas inexperienced workers’ visual behav-ior involved much more global assessment of the entire panel associated with pe-ripheral vision and the first phase of the zoom-lens model (Eriksen & James, 1986).

The identified differences between attention shifting strategies may constitute a basis for optimizing the quality control operations. The provided models are logical-ly interpreted in terms of specific sub goals accompanying the visual inspection tasks.

The paper confirms how the modern equipment for mobile eye-tracking may be practical in examining operations in a manufacturing facility. The gathered data can be analyzed in a classic way for specified AOIs and given visual activity measures. However, taking advantage of more sophisticated approach like HMM, one may try to discover, not so obvious, human visual behavior patterns. In the present study, it is illustrated that even for a quite limited sample of eye-tracking data, such a meth-odology may provide qualitatively different results for novice and experienced qual-ity assurance operators.

6.2. Limitations and future research

There are, naturally, specific limitations concerning the current investigation. First, the eye tracking data are registered only for one novice and one experienced em-ployee. Though, they repeated the same experimental task multiple times, one should rather not draw any far reaching conclusions about the general differences between expert and inexperienced operators.

Since it is, usually, difficult to obtain a consent to examine workers by eye-trackers in a real environment, most of the experiments in the literature, related with use of an eye-tracker are conducted in a laboratory environment. In the current study, data are collected as inspectors conduct their routine tasks, in authentic work-ing conditions. In this context, our contribution might be interesting for researchers and practitioners, since it is more ecologically valid than lab investigations.

As far as we know, neither lab nor factory based studies similar to ours, were car-ried out to analyze humans’ visual attention recorded during the inspection process. Therefore, despite the above mentioned limitation, we think that the present research is still valuable as it provides some insight in this regard. The presented outcomes may be used as a benchmark in future studies.

From the perspective of the presented and described Markov models, the collect-ed data were sufficient. Participants examined 60 panels, and the visual inspection tasks were relatively simple. Thus, we were able to estimate and provide significant models uncovering specific patterns that help to better understand subjects’ atten-tional behavior.

However, modelling human behavior is usually troublesome and may be very dif-ficult. Selecting a meaningful model that can be reasonably interpreted, seems to be a kind of art even for a simple experimental setup like the one in the present re-search. It resembles, to some degree, the procedures used in factor analysis (rota-tions, oblique rotations, decisions about the number of factors, selecting which fac-tor loadings should be assign to which factors, etc.). Likewise, in classic regression analysis, the researcher has to decide how many dependent variables to include, whether or not leave the variates that are correlated with each other, or to what ex-tent a given variable has to add up to the whole model in terms of R square to keep it in the regression equation. Similar problems occurred in the present study. For each simulation experiment condition there were 100 models produced. Some of the good models, in terms of the information criteria, are trivial and cannot be used as an explanation of human visual behavior. Moreover, the model selection criteria em-ployed are not very helpful since there is no optimal number of states. Additionally, there are qualitatively different models with similar values of the goal function (or criterion). Probably, taking advantage of some other measures in this regard may provide more concrete suggestions.

The extension of the present investigation may be focused on automatic identifi-cation of novice and experienced operators. It is also possible to apply HMMs algo-rithms to the selection process of the most promising candidates for positions where specific visual strategies are required. Further studies may also assess whether oper-ator gender has an influence on visual behavior or not.

Ulutas & Ozkan (2017a) attract attention to the use of digital cameras and image processing algorithms that may be helpful in inspection tasks. However, worker’s decision making capability, and the ability to tolerate some type of defects are the main reasons for many firms to still rely on human visual inspection. On the other hand, automated control methods may not suitable or too costly for inspection activ-ities such as panel inspection. Some future directions may, however, involve auto-mated intelligent procedures combined with human direct supervision. The present-ed findings may be treated as next step towards better understanding of how humans perform visual inspection tasks. This could also facilitate the development of better algorithms and intelligent procedures for automated control systems.

6.3. Final remarks

Despite the limitations and problems outlined in the previous section, the HMMs seem to be an interesting way of modeling human visual behavior while performing work operations based on visual inspection.

The study also attracts attention to the eye-tracking techniques that are useful to support inspector training and improve subsequent performance by linking failure modes to visual search activity like the problem of “looking but not seeing” (Muczynski & Gucma, 2013). The models obtained in this study, are certainly quali-tatively different for novice and expert operators. Gaining the insight into how ex-perts perform visual inspection tasks may help in developing appropriate training procedures for novice operators or enhancing existing ones. As a result, the overall efficiency and effectiveness of work operations can be improved.

In general, HMMs allow to conduct analyses of hidden states that can be associat-ed with the so called covert attention (Findley & Gilchrist, 2003), whereas the clas-sic scan path examination deals only with manifestations of overt attention changes.


1. Akaike H (1973). Information theory as an extension of the maximum likelihood theo-ry. In: Petrov. BN and Csaki. F (eds) Second International Symposium on Information Theory. pp 267–281. Akademiai Kiado. Budapest.

2. Baum, L. E. (1972). An inequality and associated maximization technique in statistical estimation for probabilistic functions of Markov processes. In Shisha. O. (Ed.). Proceed-ings of the 3rd Symposium on Inequalities. University of California. Los Angeles. pp. 1-8.

3. Boisvert, J. F. G., & Bruce, N. D. B. (2016). Predicting task from eye movements: On the importance of spatial distribution, dynamics, and image features. Neurocomputing, 207, 653–668. doi: 10.1016/j.neucom.2016.05.047

4. Chuk, T.. Chan, A. B., Hsiao, & J. H. (2014). Understanding eye movements in face recognition using hidden Markov models. Journal of Vision, 14(11), 1-14. doi:10.1167/14.11.8

5. Coutrot, A., Hsiao, J. H., & Chan, A. B. (2018). Scanpath modeling and classification with hidden Markov models. Behavior Research Methods, 50(1), 362-379. doi: 10.3758/s13428-017-0876-8

6. Drury, C.G., & Prabhu, P.V. (1992). Human factors in test and inspection. In G. Salvendy & W. Karwowski (Eds.). Handbook of Human Factors in Advanced Manufacturing. New York: Wiley.

7. Ellis, S. R., & Stark, L. (1986). Statistical Dependency in Visual Scanning. Human Fac-tors: The Journal of the Human Factors and Ergonomics Society, 28(4), 421-438. doi: 10.1177/001872088602800405

8. Eriksen, C.W., & James, J.D.S. (1986). Visual attention within and around the field of focal attention: A zoom lens model. Perception & Psychophysics, 40(4), 225-240. doi:10.3758/BF03211502

9. Findlay, J.M., & Gilchrist, I.D. (2003). Active vision. The psychology of looking and see-ing. New York: Oxford University Press.

10. Gallwey, T.J. (1982). Selection tests for visual inspection on a multiple fault type task. Ergonomics. 25, 1077-1092.

11. Greene, M. R., Liu, T., & Wolfe, J. M. (2012). Reconsidering Yarbus: A failure to pre-dict observers’ task from eye movement patterns. Vision Research, 62, 1-8. doi: 10.1016/j.visres.2012.03.019

12. Grobelny, J., & Michalski, R. (2016). Applying hidden Markov models to visual activity analysis for simple digital control panel operations. Proceedings of 37th International Conference on Information Systems Architecture and Technology. ISAT 2016. Part III. Advances in Intelligent Systems on Computing. 523. doi: 10.1007/978-3-319-46589-0_1

13. Gramopadhye, A.K., Drury, C.G., & Prabhu, P.V. (1997). Training Strategies for Visual Inspection. Human Factors and Ergonomics in Manufacturing, 7(3), 171-196.

14. Hayashi, M. (2003). Hidden Markov Models to identify pilot instrument scanning and attention patterns. In IEEE International Conference on Systems. Man and Cybernetics. 2003, 3,. 2889-2896. doi: 10.1109/ICSMC.2003.1244330

15. Jacob, R.J.K., & Karn, K.S. (2003). Commentary on Section 4 - Eye Tracking in Hu-man-Computer Interaction and Usability Research: Ready to Deliver the Promises. In J. Hyönä. R. Radach, & H. Deubel (Eds.). The Mind’s Eye (pp. 573–605). Amsterdam: North-Holland.

16. Khasawneh, M.T., Kaewkuekool, S., Bowling, S.R., Desai, R., & Jiang, X., Duchowski, A. T., & Gramopadhye, A.K. (2003). The Effect of eye movements on visual inspection performance. Proceedings of Industrial Engineering Research Conference. Oregon.

17. Kianoush, S., Savazzi, S., Vicentini, F., Rampa, V., & Giussani, M. (2015). Leveraging RF signals for human sensing: Fall detection and localization in human-machine shared workspaces. In 2015 IEEE 13th International Conference on Industrial Informat-ics (INDIN) (pp. 1456–1462). doi: 10.1109/INDIN.2015.7281947

18. Kit, D., & Sullivan, B. (2016). Classifying Mobile Eye Tracking Data with Hidden Markov Models. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (pp. 1037–1040). New York, NY, USA: ACM. doi: 10.1145/2957265.2965014

19. Liechty, J., Pieters, R., & Wedel, M. (2003). Global and local covert visual attention: Evidence from a bayesian hidden Markov model. Psychometrika, 68(4), 519-541. doi: 10.1007/BF02295608

20. Lukander, K., Toivanen, M., & Puolamäki, K. (2017). Inferring Intent and Action from Gaze in Naturalistic Behavior: A Review. International Journal of Mobile Human Com-puter Interaction, 9(4), 41-57. doi: 10.4018/IJMHCI.2017100104

21. Markov, A.A. (1913). An example of statistical investigation of the text Eugene Onegin concerning the connection of samples in chains. (In Russian.) Bulletin of the Imperial Academy of Sciences of St. Petersburg 7(3):153-162. Unpublished English translation by Morris Halle. 1955. English translation by A.Y. Nitussov, L: Voro-pai, G. Custance, & D. Link, 2006. Science in Context 19(4):591–600.

22. Michalski, R., & Grobelny, J. (2016). An eye tracking based examination of visual at-tention during pairwise comparisons of a digital product’s package. UAHCI 2016. Part I. LNCS 9737. 430-441. doi: 10.1007/978-3-319-40250-5_41

23. Michalski, R. (2017). Information Presentation Compatibility in the Simple Digital Control Panel Design – Eye-Tracking Study. International Journal of Occupational Safe-ty and Ergonomics. doi: 10.1080/10803548.2017.1317469

24. Muczynski, B., & Gucma, M. (2013). Application of eye-tracking techniques in human factor research in marine operations: Challenges and methodology. Scientific Journals, 36, 116-120.

25. Murphy, K. (1998. 2005). Hidden Markov Model (HMM) Toolbox for Matlab. www.cs.ubc.ca/~murphyk/Software/HMM/hmm.html

26. Ozkan F.. Ulutas B. (2017a). Assessing knowledge and behavior of medicine leaflet use for university students. 3rd International Researchers. Statisticians and Young Statisti-cians Congress (IRSYSC 2017). 24-26 May 2017. Konya. Turkey.

27. Ozkan F., Ulutas B., (2016). A framework to assign forklift drivers to relevant duties based on physical and cognitive factors. 1st Global Conference of Industrial Engineering. 14-15 July 2016. Istanbul. Turkey.

28. Ozkan F., Ulutas, H.B. (2017b). Using eye-tracking data to evaluate medicine infor-mation leaflets on-screen. Journal of Mathematics and Statistical Science, 3(12), 364-376.

29. Posner, M.I., Snyder, C.R., Davidson, B.J. (1980). Attention and the detection of sig-nals. Journal of Experimental Psychology: General, 109(2), 160-174. doi:10.1037/0096-3445.109.2.160

30. Rabiner, L.R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257-286. doi: 10.1109/5.18626

31. Schwarz, G. (1978). Estimating the Dimension of a Model. The Annals of Statistics, 6(2), 461-464. doi: 10.1214/aos/1176344136

32. See, J.E. (2012). Visual inspection: A review of the literature (Report SAND2012-8590). Albuquerque. NM: Sandia National Laboratories.

33. See, J.E. (2015). Visual Inspection Reliability for Precision Manufactured Parts. Human Factors, 57(8), 1427-1442. doi: 10.1177/0018720815602389

34. See, J.E., Drury, C.G., Speed, A., Williams, A., Khalandi, N. (2017). The Role of Visu-al Inspection, In the 21st Century. Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting. 262-266.

35. Simola, J., Salojärvi, J., & Kojo, I. (2008). Using hidden Markov model to uncover pro-cessing states from eye movements in information search tasks. Cognitive Systems Re-search, 9(4), 237-251. doi: 10.1016/j.cogsys.2008.01.002

36. Sodergren, M. H., Orihuela-Espina, F., Clark, J., Darzi, A., & Yang, G.-Z. (2010). A hidden markov model-based analysis framework using eye-tracking data to characterise re-orientation strategies in minimally invasive surgery. Cognitive Processing, 11(3), 275-283. doi: 10.1007/s10339-009-0350-3

37. Speed, A. (2015). Visual search in operational environments: Balancing operational constraints with experimental control. In Proceedings of 9th International Conference. AC 2015. Held as Part of HCI International 2015. Los Angeles. CA. USA. August 2-7. 2015. 528-536. doi: 10.1007/978-3-319-20816-9_51.

38. Tatler, B. W., Hayhoe, M. M., Land, M. F., & Ballard, D. H. (2011). Eye guidance in natural vision: Reinterpreting salience. Journal of Vision, 11(5), 5-5. doi: 10.1167/11.5.5

39. TIBCO Software Inc. (2017). Statistica (data analysis software system). version 13. http://statistica.io.

40. Ulutas, H.B., Ozkan, F. (2017a). Assessing visual control activities in ceramic tile sur-face defect detection: an eye-tracking study. International Journal of Intelligent Engi-neering Informatics, 5(4), 342-351.

41. Ulutas, H.B., Ozkan N.F. (2017b). From the Point of a Tower Crane Operator's View: Use of an Eye-Tracker in Construction Sites. The Japanese Journal of Ergonomics. 53 (Supplement2). S616-S619 (Special issue of 2nd Asian Conference on Ergonomics and Design (ACED). 1-4 June 2017. Chiba. Japan).

42. Yarbus, A. L. (1967). Eye Movements and Vision. Springer Science+Business Media: New York.

© 2012-2024 RafalMichalski.com, Rafal Michalski All rights reserved.Last modified on: 2019-07-19HTML 4.0! standard CSS standard!