A pilot study on automated quantitative grading of facial functions

1. Introduction

Facial paralysis (FP) is a condition which affects both facial symmetry and function. It can be congenital or acquired due to various causes for instance following nerve injury [1, 2]. FP patients are affected to different degrees, with symptoms possibly including speech impairment, the inability to drink and eat normally [1, 3] and the inability to perform facial expressions correctly [2]. Hence there is need for a quantitative clinical grading tool to evaluate the severity of the FP condition and to quantify the loss of facial function as well as the effectiveness of medical treatment or rehabilitation.

In the past several decades, a large number of facial grading systems have been proposed and are comperehensively reviewed in [1, 3, 4]. They are divided into two main categories; traditional systems and computer-assisted grading systems [3]. Most of the traditional grading systems such as House-Brackmann, Nottingham, and Sunnybrook systems are based on comparing distances between specific facial landmarks at rest and during performing various facial exercises. Although these systems are widely used and accepted by clinicians, they have several limitations. In general, they are subjective because their scoring criteria depend on the physician’s observations and measurements. Furthermore, most of them require significant time for identifying and manually marking facial landmarks.

In recent years, several computer-assisted facial grading systems have been proposed [1, 3] and are comprehensively reviewed in [5, 6]. They include video recording, Photoshop [7], image processing and classification approaches to provide objective, quantitative, and repeatable assessment of facial function. One such system is SmartEye Pro-MME, it tracks lip movements after identifying and capturing lip landmarks [8]. The Recognition of Smile Excursion (RoSE) scale uses face photos to automatically evaluate the symmetry of the smile in FP patients [9]. However, for both systems, the physician has to select the landmarks of interest manually and the program automatically calculates the scores [8]. Although these computer-assisted grading systems provide rapid repeatable results in a quantitative form, their general acceptance as a clinical tool is limited by their requirements of high cost specialized equipment as well as the lengthy time required to manually identify landmarks on the patient’s face or photo.

More recently, 3D depth sensors, generally used for motion detection in commercial games, have started being used as a clinical tool for motion detection in medical applications. Recent reviews show that 3D depth sensors like the Microsoft Kinect® [10] have been used in musculoskeletal virtual rehabilitation systems [11-13]. A small number of studies have used the Kinect depth sensor for face applications. These include the recognition of basic facial expressions [14] and the classification of different facial exercises [15]. In U.K., ongoing research uses the Kinect depth sensor for facial rehabilitation of patients following stroke [16].

The work presented here lies within the framework of a comprehensive Automated Facial Grading (AFG) system initially presented by the authors in [17, 18] which employs the Kinect v2 (Microsoft Inc., USA) and the SDK 2.0 for automatic capturing of facial landmarks. The AFG system includes three different assessment modules; resting symmetry grading, voluntary movements’ grading, and standard grading scales. The resting symmetry assessment module was presented in [17] while its modified extension was described in [18]. The current work focuses on developing and testing the voluntary movements’ assessment module. The AFG system is fast, independent of user subjectivity and cost effective enough for widespread clinical use.

2. Methods

The assessment module presented here is used for grading the function of three facial features; eyebrows, eyes, and mouth while performing four selected facial exercises; raising eyebrows, closing eyes, smiling and opening the mouth. The AFG system automatically captures the facial landmarks in real time using the Kinect v2 as previously described [17, 18]. The eighteen facial landmarks selected are shown in Fig. 1. The distances between landmarks used for voluntary movements’ grading are shown in Fig. 2 and the processing methodology is described below.

Fig. 1The eighteen facial landmarks selected for the AFG system

Fig. 2Distances used in voluntary movements’ assessment

3. Results

As an initial proof of concept, grading of the four voluntary facial functions using the AFG system was tested on six normal subjects and two facial paralysis patients. In order to simulate facial paralysis, each normal subject was instructed to perform the four functions in a specific way not necessarily to their full ability. The scores and the corresponding grades were computed for each side independently and are shown in Table 1. We have validated the system by testing on the two FP patients and correlating the evaluation with their clinical assessment. The experimental procedures involving human subjects described in this paper were approved by the Neurophysiology department council, El Kasr El-Aini, Faculty of Medicine, Cairo University. Facial paralysis patients participated in this research have to sign a patient consent form.

Test subjects 1 and 2 were instructed to perform the four facial voluntary movements to their maximum ability and the results show high scores for both sides. Subject 3 was instructed not to raise his eyebrows fully and his results show a grade of 4 for both the right and left sides. Subjects 4 and 5 were instructed not to perform any movement and their scores are around zero in all the movements for both sides. Subject 6 was instructed to close the left eye only and achieved a score of 03 % for the right eye. He was also instructed to move only the left side while smiling and hence his score was 00 % in right side smiling. In addition, he only slightly opened the mouth, and his score was 34 %.

Test subject 7 is a 52 years old FP Patient. Electromyogram (EMG) examination and nerve conduction studies showed evidence of complete lesion affecting the right facial nerve with no evidence of regeneration. When he was asked to perform the four exercises using our AFG system, he raised his eyebrows and opened his mouth in a moderate rate as shown in Table 1. But he could not close his right eye, it was always opened. He was suffering a severe right FP as diagnosed using needle EMG. Test subject 8 is a 58 years old FP Patient. EMG examination showed evidence of severe partial lesion affecting the left facial nerve. When she was examined using our AFG system, she slightly raised her eyebrows and opened her mouth. Her left eye was opened, she could not close it. It was diagnosed as a severe left FP patient. A sample report is shown in Fig. 3. For each of the four movements, it shows both the percentage of successful accomplishment of the task as well as the grade for the left and right sides individually.

4. Discussion

Although several computer based systems for FP grading have been presented in the literature [6, 7], there is still no system which has been widely accepted by clinicians. Such a system should be accurate, quantitative, automated, and independent of user variability. Moreover, it should be fast, easy to use, and inexpensive. The assessment module designed and implemented in this study is part of ongoing work to create a comprehensive AFG system as previously reported by the authors [17, 18]. This module grades the function of three facial features; eyebrows, eyes, and mouth while performing four selected facial exercises; lifting the eyebrows, closing the eyes, smiling and opening mouth. The first three movements are found in traditional scales. However, opening the mouth is an additional movement included here to better assess mouth function.

Fig. 3Voluntary movement assessment report generated using the AFG System

Automated capture of the 3D locations of eighteen facial landmarks in real time using the Kinect for Windows v2 eliminates the need for manually placing markers on the subject’s face as was required in other grading systems [19]. In addition, it eliminates the need for time-consuming feature recognition and classification algorithms which have been used in other computer-assisted systems [6]. Another significant advantage of the AFG system is that the distances computed do not depend on facial orientation or tilting.

Basing the grading of FP on distances between landmarks is one of the standard accepted methods used in traditional grading systems. However, in the AFG system, distances are computed automatically, accurately and objectively within a very short period of time. It should be noted that evaluating the smiling function is not based simply on the distance between the two mouth corners. It is actually based on the distances between each mouth corner and the tip of the nose as shown in Fig. 2. This has the advantage of reporting the excursion of each side of the mouth individually. In addition, referring the mouth corners to a fixed facial landmark avoids the possibility of erroneous results resulting from mouth distortion in FP patients.

Preliminary testing of the AFG system was done in six normal subjects and two FP patients. The small number of test subjects is a limitation of this study. However, it showed promising results as shown in Table 1 and there is ongoing work to test it on more FP patients. The detailed report shown in Fig. 3 gives quantitative grades for the four functions for both the right and left sides independently. This allows the AFG system to be used for both unilateral and bilateral FP. This work is still in progress. More landmarks points and additional voluntary movements are being added to the system.

5. Conclusions

This work describes the design, implementation, and testing of the voluntary movements grading module of a comprehensive AFG system for FP. The system utilizes the Kinect v2 sensor to detect and capture eighteen facial landmarks in real time without the need to place markers on the subject’s face. The functions of three facial regions, the eyebrows, eyes and mouth, are evaluated by quantitatively grading four voluntary movements: raising the eyebrows, closing the eyes, smiling, and opening the mouth. Grading is based on movement of landmarks during each exercise relative to their resting positions and scores are assigned as percentages of normal limits of motion. The left and right sides are graded independently allowing the AFG system to be used for both bilateral and unilateral FP. Preliminary results are promising and testing on more FP patients is being performed. The AFG system is suitable for clinical use because it is fast, easy to use, low cost, quantitative, and automated.