1. Introduction
With the recent development of mobile devices such as smartphones, demand for multimedia content consumption is no longer limited by time and space. A massive amount of content is currently produced and consumed compared to the past. Along with this trend, the need to measure user satisfaction and the degree of content immersion has emerged as an important issue [1], and providing an optimized/automated solution to measure user satisfaction as well as to collect the entailed big data has emerged as a lucrative business opportunity. The currently available solutions for measuring content immersion are typically question-based surveys [2,3] including redefining the evaluation scope [4] to improve the quality of surveys. However, these non-automated methods have limitations on collecting large amounts of data.
Recently, in order to overcome the limitations of the existing survey-based studies, some studies have attempted to measure content immersion using biological signals; however, because this approach involves the use of high-end multichannel electroencephalogram (EEG) equipment [5,6], its implementation is not practical in actual environments where mobile users consume multimedia content.
Conversely, by using a portable EEG device, large amounts of data can be obtained easily without any content consumer intervention. Although the EEG device enables analysts to objectively gather information on consumer attention, the extraction of brainwaves relevant to the content immersion measurement is a complex technological implementation. This is because portable EEG devices are used normally in non-ideal conditions (i.e., implemented with non-adhesive electrodes and a single channel with a battery, and contaminated by noise from wireless communication operations).
In this study, the authors developed a system to measure the level of a user’s content immersion in entertainment multimedia content with a portable EEG sensor device that can automatically and objectively observe brain states, whereas the existing methods rely on a manual and subjective conventional approach based on surveys or interviews. To address the noise problems from a portable EEG sensor, the authors specifically used the online singular spectrum analysis (SSA) algorithm [7,8], which is a powerful real-time noise filtering method for raw signals that have high SNR(signal-to-noise ratio) ratio and a limited number of sensor channels because the other existing filtering algorithms are mostly used in multi-channel sensors and offline processing environments. The authors also used the median of medians algorithm to find the loss signal value. As a result, the authors could remove noise and isolate brainwaves from the noise-ridden signal output of the portable EEG device. The frequency analysis of the pre-processed EEG signal was used to measure the degree of user content immersion. This study conclusively demonstrates that content immersion measurement using a portable EEG device performed as well as expected, presenting positive experimental results.
This paper consists of five sections: Section 1 presents an overview of the study, Section 2 explains the related work, Section 3 introduces the proposed solution approach, Section 4 presents the experimental results, and Section 5 concludes the paper with a summary.
2. Related Work
One of the most significantly studied areas related to content quality is improvement in survey questions and survey configuration to gather accurate and meaningful feedback [2-4]. However, a recent trend is to avoid the survey methods used previously that prompt the user to provide a response, causing inconvenience, and instead measure various biological signals that occur when a user views the content. There is a movement towards developing a method that automatically measures the quality of the content [9-12]. These studies are limited to laboratory environments. Therefore, this study suggests using a portable device that is applicable to any mobile environment, so feedback is not limited to a lab environment.
Analytical studies on human brainwaves and concentration are also an active topic. Brainwaves are electrical signals that are generated from the activity of the neurons in the brain, and are important biological signals that can be used to evaluate the activity state of the human brain. As shown in Table 1, the activity state of the brain can be categorized on the basis of the brainwave frequency ranges.
According to Table 1, brainwaves during the concentration state are defined by sensory motor rhythm (SMR), beta, and high beta brainwaves because concentration measurement is a critical issue in neurofeedback training for improving brain-computer interfaces (BCIs), user evaluation, and other applications [13]. A study confirmed the concentration state experimentally [14,15]. It was reported that there is a high correlation between the concentration state of the brain and the beta, SMR, and high beta waves.
Frequency ranges of brainwaves and the characteristics of the activity state of the human brain
Research on noise reduction at the EEG site is also under way. EEG studies in the past used adhesive or implanted electrodes to obtain signals in a lab experiment environment, and there was no particular mention regarding interfering noise that could occur in everyday environments. Previous studies on EEG noise removal did not consider the removal of external noises (e.g., user’s arbitrary actions and white noise), and instead, it focused on the removal of conflicted noise signals occurring during EEG measurements such as separating and removing biological signals; this could only be executed using multichannel EEG methods [16,17].
Portable EEG measuring devices have been introduced recently, and studies on removing other unwanted biological noise from single electrodes are underway. For example, methods to remove electrooculography (EOG) and electrocardiography (ECG) signals from the EEG signal are being studied.
3. Proposed Signal Processing
In this section, the authors explain how to acquire EEG signals, measure content immersion, and handle exceptional signal patterns.
3.1 EEG Signal Acquisition and Content Immersion Measurement
The authors assume EEG data is collected over wireless communications from mobile devices in practical situations (i.e., not lab-testing environments). Therefore, not only are unwanted biological signals removed, which were included in the EEGs in the previous studies, but the signal loss due to other everyday noise is also removed from the single electrode to extract a pure EEG signal. Later, the signal is separated into frequency ranges to numerically compute the immersion. The process steps are summarized in Fig. 1.
In the filtering process of Fig. 1, first, the original EEG is collected in real time. At this time, the device used for collecting the EEG should not restrict the freedom of time and space. To meet this requirement, a portable device must be used and it must be a device that has non-adhesive electrodes and can amplify up to 1 μV potential in the 0.1–100 Hz band, which is a general requirement of the EEG; the authors used the NeuroSky MindSet product and a Samsung Galaxy Note 3 device in the experiment. The collected data (Fig. 2) shows that the EEG signal is mixed with other biological signals and noise due to errors in the mobile device.
Original EEG signal mixed with other biological signals and noise.
The next step is the signal recovery process. As shown in Fig. 2, a steep slope change is observed at the point where the signal is lost. So, it is necessary to remove these parts in order to restore the original signal, and the first differential is used for this purpose. By calculating the first differential of this signal, the steep slope is eliminated according to Eq. (1). This is followed by signal restoration through integration, as shown in Fig. 3, to extract the primary filtered data.
After Step 2, most of the noise due to signal loss has been removed, but this signal still contains excessive noise from unwanted biological signals such as EOG, EEG, and other electronic device noise. Thus, noise filtering was necessary on the third step using an online SSA algorithm that was effective for real-time processing on a single electrode [7]. After filtering, the pure EEG signal is obtained as shown in Fig. 4.
Signal after applying the online SSA algorithm.
Finally, because the baseline dropped to approximately −400 due to the previously applied algorithm, by removing the signal below 0.1 Hz to return the baseline to 0, the final signal is extracted as shown in Fig. 5.
Moreover, as reported in [14], when the subject is in a state of concentration, the alpha wave decreases by an average of 2.38%, and the beta, SMR, and high beta waves increase by 4.16%, 6.47%, and 7.49%, respectively. The numerical calculation used in the study is as follows:
3.2 Configuration for Exception Handling
The proposed method is intended to be implemented not in a lab environment but in actual mobile environments. Consequently, many technical problems in handling exceptional signal patterns must be addressed. The first is defective connectivity of the electrode; if the electrodes of an EEG sensor device are not perfectly placed on the subject’s head, device connectivity and its working may be incomplete.
When using adhesive and implanted electrodes, there is only a small chance of defective connectivity; however, the portable EEG adopted in this study uses a non-adhesive electrode. Therefore, in the event of sudden movement, yawning, or even head scratching, the adhesion state of the portable EEG can momentarily change, creating constant states of defective connectivity (Fig. 6).
Defective connectivity state signal.
The defective connectivity state is a result of the amplification of the 60-Hz waveform of other electronic devices; however, it is a good rule of thumb to exclude the section with defective connectivity in order to derive accurate data. It is possible to filter the signal using a notch filter that filters out only the 60-Hz waveforms. Therefore, wavelength analysis is carried out before executing the filtering process, and if there is more than a 10-fold 60-Hz waveform factor compared to other waveforms as shown in Eq. (3), the measurement values of the corresponding section are excluded.
The second exception taken into consideration is the case where a subject is not viewing the content. In ideal conditions, the test (measuring the brainwaves of subjects) is conducted under the assumption that the subject is concentrating on the content; however, in actual situations, there are many reasons why the subject is not paying attention to the content. A possible situation is when the subject leaves the area of content consumption. In the case where the subject takes off the portable EEG device and moves away, a defective connectivity signal can be transmitted. Such a situation is excluded in the analysis. In the case where the subject physically leaves the EEG signal receiver device, the signal goes to 0. However, this exceptional case can be easily determined, as shown in Eq. (4), and excluded from the analysis.
If the subject does not leave the area of consumption and does not pay attention to the content, several other data points must be calculated to evaluate in comparison with the baseline. This is carried out using the relationship between the number of blinks and concentration. If the subject blinks too much, it means the subject has no focus on the content; that signal part will be excluded [18]. By numerically scoring the number of blinks, the measurement can be used as a numerical score along with Eq. (2) in the evaluation of immersion. However, in the event of an excess of an average of more than 40 blinks per minute as shown in Eq. (5), it is thought that the subject is not watching the content and is engaged in a different activity. Consequently, the corresponding signal part is therefore excluded. The blinking determination signal disappears if the proposed filtering method is applied; the numerical data for blinking provided by the hardware can be used for evaluation.
The case where the subjects have their eyes open but are not focusing on the content and enter a delusional state must also be considered as an exception case. In Table 1, the Theta brainwave is defined as the brainwave associated with delusion. As reported in [19], Theta waves are released in a dreamlike state where the eyes can move in a paradoxical sleep state while thinking about other things. To confirm the reported clue, a comparison is performed between the measured results of the brainwaves of when the subject is forced to think about something else, when they are immersed in the visual stimulation, and when they doze off. The result for the comparison is that Theta waves take up more than 30% of brainwaves, as shown in Fig. 7.
Ratio comparison of theta waves.
When measuring content immersion, theta waves increase. In other words, the fact that “the subject is not watching the content and thinking about something else” can be included as a factor when calculating the immersion score. However, that is neither a test environment solely to examine the personal state of the subject, nor is it an environment set up solely for content viewing. Therefore, for the signal parts with a theta ratio of more than 30% (Eq. (6)), it is determined that the contents are not being watched and the signal part is excluded.
The final situation that is handled as an exception is when the subject falls asleep. Similar to the delusion numerical value configured in the previously discussed exception, it can be determined that the drowsiness is caused because the content lacks a degree of immersion. However, because the study is fundamentally conducted only on situations that the subjects actually watch the content, those signal parts are excluded. The Alpha to Theta brainwave ratio has already been verified in an existing study [20]. It can be confirmed that compared to that of an alert state, the ratio was significantly lower in the drowsy state, as shown in Fig. 8.
Ratio of theta wave to alpha wave.
However, in the case where the numerical value drops below 2, owing to the involvement of excessive personal discrepancy for a comparison of absolute values, the study defined the state where the content is first engaged as the alert state, and the state where the theta to alpha brainwave ratio drops to 80% of the alert state as the drowsy state. Moreover, as shown in Eq. (7), the values during the drowsy state are excluded from the test values.
The reason for this exception to be configured and excluded in this manner can be seen clearly when examining the actual numerical data from the test. Fig. 9 shows the brainwave values of the subjects with an average content immersion numerical score of 0.74 who fall asleep during content consumption.
Numerical immersion score of sleeping subjects.
Data comparison before and after the removal of exception parts.
From the data of Fig. 9, it can be observed that the immersion brainwave numerical data of the subjects is especially high during sections where they fall asleep. This is because once the subject enters the rapid eye movement (REM) state of sleep, the brainwave state becomes similar to that of someone who is awake, which causes increased Beta brainwave activity [13,21]. Therefore, if that signal part is not removed, the data will be distorted. Fig. 10 shows a comparison of the exception parts before and after they are removed.
Because the subjects are very tired and unable to concentrate on the content, this should generally be predictable data that show a lower than average immersion score. When the exception parts are included, the data during the sleep state and the distorted numerical measurements should be considered. The corresponding results show a higher immersion score. On the other hand, when the exception parts are excluded, the results show that the immersion score is lower than the average score predicted for the subjects.
Therefore, in the test conducted in the study, before calculating the numerical immersion score of Eq. (2), the exception situations of Eqs. (3)–(7) are determined, and accordingly, the data with those conditions are excluded before conducting the experiment.
4. Experiment
This section explains the experimental settings and presents the analyzed results.
4.1 Experiment Setup
To verify the validity of the immersion brainwave measurement calculated using Eq. (2), the subjects were exposed to entertaining movie content and boring movie content, and using the proposed Eq. (2), the EEG was collected and a correlation analysis was conducted.
To determine whether the contents were entertaining or boring, the top three movies based on NAVER movie ratings with more than 300 reviewers with a standard deviation of 95% ± 5.65% were selected as entertaining content, and the last three movies were categorized as boring content. These movies were shown to the subjects without giving any prior information on the movies.
After the subjects watched the movies, scores on the entertaining factor of the movies were surveyed, and then they were compared with those obtained through previous immersion measurement methods.
4.2 Experiment Results
The experiment results indicated that when engaging content with no prior information is given, as shown in Fig. 11, the immersion brainwave measurement was identical for all content for the first 10 minutes at an average of 0.06, showing that the brainwave was maintained at a higher level than the general state. When the subjects watched a funny movie, the immersion was either maintained or increased. However, when the subjects watched a boring movie, the immersion dropped dramatically to a negative value.
By numerically representing the data according to Eq. (2), a comparison of the scores obtained from the EEG measurement, the survey, and the NAVER movie ratings were similar, as shown in Table 2.
The comparison in Table 2 shows that the immersion score computed by the proposed method is as highly accurate as the previous survey methods and the NAVER ratings. In addition, the proposed method has advantages over of existing methods. When an additional non-automated survey was carried out, 5 minutes was required on average to fill out the survey questionnaire; however, in the case of using the proposed method, no additional time was required to objectively and accurately measure the degree of immersion. The properties of automation, short time consumption, and no user disturbance in survey, which the proposed method can support but the others cannot, are highly critical in mobile environments.
Immersion brainwave values while watch a movie.
Comparison of immersion scores
5. Conclusion
This paper confirmed that it is possible to measure the degree of content immersion with the EEG of user brainwaves. Brainwaves are extracted from the portable EEG device signals. Because the portable EEG device consists of a non-adhesive electrode and is powered by a single channel battery, raw signals from the device are highly contaminated by noise owing to wireless communication operations in the environment. We addressed the noise problems in this study. Our proposed method is an automated technique that significantly reduces the time required for collecting user survey feedback, which is easily applicable to large-scale big data analysis. Our proposed method of collecting and analyzing information from brainwave signal scans significantly contribute to the design of a user-adaptive content recommendation system.
Acknowledgement
This research was supported by Next-Generation Information Computing Development Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (No. 2012M3C4A7033345).