1. Introduction
Signage, a simple outdoor billboard, has evolved into a digitized and interactive platform. It is being integrated with the latest IT technologies, such as artificial intelligence and virtual reality.
This paper proposes an intelligent digital signage platform using an intelligent content determination system based on edge computing. The proposed system consists of a server and an edge. The server manages the data, learns the recommendation model used as the content determination system, and uses the learned content determination system to determine the products to be advertised in real time. As a content decision system, a recommendation list was derived by predicting the probability of purchase for selected products using a wide-and-deep learning-based recommendation model.
The wide and deep recommendation model combines individual wide and deep models, each of which learns different types of input values. The wide model is a product probability prediction model that uses logistic regression functions. The deep model expresses the product purchase status of each user in binary, adds an embedded product matrix and user features as input, and learns a dependent variable expressing the product purchase status of each user in binary. If several users exist in the advertisement range, a matrix is created by using a list recommended to the users in the advertisement range to select the product with the highest probability of purchase. The selected product is displayed to the user through signage. The system proposed in this paper determines the content using the model trained at the edge without communicating with the server. It is well suited for digital signage that requires immediate response to multiple users. This paper suggests a direction toward achieving a custom signage system suitable for several people, rather than a conventional one-to-one or one-way signage system.
2. Related Research
Digital signage involves the use of digital technology to provide various types of services to various displays or billboards for advertising in public or commercial spaces [1-3]. Digital signage functions by displaying various media content both indoors and outdoors. In other words, information is provided through outdoor advertisements in real time, allowing interaction with consumers [2-5]. In addition, digital signage uses a display combined with a camera to identify basic information such as the gender and age of a person within the advertising range and to display customized advertisements [4-6]. The display not only conveys one-sided information of a person, but also aims to provide a customized view of the content that the user wants to see.
Artificial intelligence technology is at the core of intelligent digital signage. It is being developed rapidly for predicting consumer behavior, and several types of daily data are used to predict consumer activity. An artificial neural network is used for extracting meaningful information from the data, and the types of learning data and layers constituting the artificial neural network vary depending on the purpose [7]. Advances in semiconductor technology have shown that the accuracy of neural networks can be considerably improved by enabling the rapid learning of large amounts of data [8].
In the past, artificial intelligence was used to transfer all the data to the data center or cloud for processing [9,10]. However, recently, edge computing has attracted considerable attention [11,12]. Edge computing processes data directly at the edge where they are generated, rather than processing them centrally. This allows us to analyze and utilize data in real time by reducing the amount of time required to transmit data to and from the cloud [10]. Recently, signage systems have evolved into digital signage systems that can communicate with users beyond simple outdoor billboards [4]. As such, digital signage systems aim to provide customized content to several users in real time, which needs to be studied further.
2.1 Digital Signage
Digital signage is a mixture of similar concepts according to display type, network type, etc. However, it can be defined as “fused media that deliver various contents and messages through a digital display in a specific space” [1-4]. Digital signage is mainly composed of hardware (display), software (content determination system), content, and network, and it is classified according to the type of display and installation location (indoor or outdoor). The concept and scope of digital signage have been expanding due to IT convergence and display diversification. Digital signage includes telescreens (an evolution of converged media and network), smart signage (which combines smart devices and various contents), media facades (which utilize the entire building as a display), and projection of the screen (rear projection). Humanoid digital signage and narrowcasting, which broadcasts to viewers who are clustered locally and hierarchically through displays, are also being developed [2,3].
In recent years, Indian signage has been considerably developed and used in various fields such as recognizing the gender and age of a user through face recognition technology to display customized advertisements or issue discount coupons [13]. Such digital signage has a significant influence on a society that is close to digital, and previous studies have investigated factors that influence signage [14].
Table 1 compares the edge-based digital signage proposed in this paper with existing research cases. First, a study using emotion recognition technology proposed a meaningful content determination system by collecting the emotion data of users, scoring them, and using them as weights when scheduling advertisements [5]. In another study, we proposed a content determination system based on gender classification by collecting user gender data through image processing technology. In addition, we provided a direction for real-time recommendation using edge computing by collecting and transmitting the gender data to a Raspberry Pi environment [15].
Digital signage research cases
2.2 Intelligent Recommendation Model
A recommendation system or recommendation model finds a list that might be of interest to a specific user at a given time. It collects user information, information on the list to be recommended, and information outside the system in some cases, and reflects it to provide a list desired by a specific user at a specific time [16-18]. Three types of recommendation systems exist: collaborative-filtering-based recommendation model, content-based recommendation model, and a hybrid model that combines them. Various techniques have been proposed to implement them.
First, collaborative filtering predicts the items to be selected by a user in the future based on the taste information collected from several users [16]. It is classified into user-based filtering, which groups users who have similar preferences and recommends the group preference products, and item-based filtering, which recommends other items with high preference association with the items purchased in the past [17].
The content-based recommendation method is based on information search technology. It analyzes the similarity between item-item or item-user preferences and recommends the item to the user. The collaborative-filtering-based and content-based recommendation methods have their respective advantages and disadvantages. The content-based recommendation method can recommend items whose evaluation is unclear, but there is a tendency toward excessive specialization. The collaborative-filteringbased recommendation method shows high serendipity, but the user is not recommended items whose evaluation is unclear. Recently, a hybrid recommendation system that can effectively utilize various types of information, maximize the advantages of the two aforementioned methods, and compensate for their disadvantages has been proposed [18-21]. Hybrid recommendations are categorized into four broad categories [22]. The first category involves combining the results of several recommendation techniques into various forms. It may be a method of combining the results of several recommendation techniques into a single result using a weighted average sum or a method of selecting and using a recommendation technique suitable for a situation or mixing various variables of each recommendation technique. The second category is a model in which a user profile that uses the attributes of items rather than the evaluation information is constructed to use the same information simultaneously for content-based recommendation and collaborative filtering. The third category involves using the features of collaborative filtering for content-based recommendations. A representative example is the use of a topic model, a technique for reducing content-based information [23]. The final model is a single model that uses collaborative filtering and content-based recommendation simultaneously [24].
In this study, the content decision system to be applied to the signage system uses a recommendation system that combines the results of two recommendation models with different data as inputs. To the best of our knowledge, this is the first study to propose a hybrid recommendation system.
Similar to the wide and deep recommendation model used for content decision in this study, the recommendation system using machine learning has been studied extensively [25-28]. A previous study investigated the characteristic that affects the behavior, which is derived from deep matrix factorization under the assumption that there is a potential characteristic that the developer or user does not recognize [19]. This was used to improve the accuracy of the recommended model. As computing power improves and data are diversified, the types of data available for recommendation systems are increasing. The content decision system used in this study is one such recommendation model. Similar studies have suggested a recommendation algorithm using appreciation environment data, such as weather, season, and time, for achieving performance improvement. This study assumes various characteristics as factors of music listening. Each feature was set to multidimensional weights, and the effects of the factors of music listening were analyzed using multiple regression analysis [20]. Another study proposed a recommendation system using long short-term memory (LSTM) and embedding technology [21]. This study embedded movie genres as a product category. The LSTM_WT model was applied to the LSTM model by employing a weight combining method that combines the embedding matrix and the projection matrix of the film.
As listed in Table 2, the recommendation system is used in different fields with various algorithms and types of data. Among them, the recommendation system using deep learning is attracting attention. The recommendation performance of deep learning is proportional to the complexity of data quality and network structure.
Comparison of learning models
In other words, the amount of computation is proportional to performance, which is closely related to real-time operation. The wide and deep recommendation model used in this study consists of a simple three-layer deep neural network (DNN) and multiple regression equations. We used a wide and deep recommendation model that can recommend a minimum amount of computation for application to the real-time recommendation pursued in this study [29-31].
2.3 Edge Computing
As data originate from several smart devices, it is not efficient to send all the data from the device to the cloud for processing or analysis. Approaches began to appear at or near where the data were generated [32,33]. The attempt to process important data in real time by computing at the edge where data is generated is called edge computing [33]. IDC defines edge computing as “a mesh network consisting of micro data centers of about 10 square meters or less, which processes or stores critical data locally and sends all received data to a central data center or cloud storage repository” [32].
In addition, prominent IT companies or experts on edge computing predicted that half of all the data generated by 2020 would be stored, processed, analyzed, and utilized at the edge [33]. In addition, several market forecasting agencies and economic journals, such as the Forbes magazine and Gartner, have suggested that edge computing is already a major trend or a change that should be noted [34,35].
3. Digital Signage based on Intelligent Predictive Model
3.1 Introduction to Data
Fig. 1 shows the data used for the deep learning model for correlation prediction and prediction. The table on the left shows the amount of Social Network Service buzz collected by daily income amount, payment number, and consumption-related keywords. The table on the right lists the public data showing the daily dust concentration (PM2.5) in the Cheonan region [36].
In fact, studies have been conducted on the effect of fine dust concentration on off-line consumption [37-39]. Based on these studies, fine dust was selected from the weather data [40]. The study analyzed the effect of fine dust on domestic retail sales, which were measured by the sales volume of large retailers. Consequently, it was shown that an increase in fine dust (PM2.5) by 10 μg/m3 reduces the sales of large retailers by approximately 2%. In the present study, we used the calculation of the correlation coefficient and basic regression analysis.
3.2 Correlation Analysis
Fig. 2 shows the process of analyzing the correlation between unstructured data, SNS data, consumption data, and climate data. Normalization was used to analyze the collected data and to organize the data on duplicate dates. Pearson correlation was used to analyze the correlation between the consumption, SNS, and fine dust data.
Correlation analysis process.
The correlation between fine dust concentration data and consumption data using Pearson correlation coefficient is expressed as:
A list of defined variables is shown in Table 3.
The correlation analysis using Pearson correlation coefficient with 2017 data shows that the consumption rate can be lowered as Max PM2.5 is increased, and the maximum value of PM2.5 has a greater influence on consumption than the average value does (Fig. 3).
Proposed partition-based device discovery.
3.3 Intelligent Recommendation Model
Fig. 4 shows a schematic of the intelligent digital signature system from the user’s point of view. A user such as a camera is used to identify a user near a space where a digital signage is installed, that is, a user who has entered the advertisement coverage. The advertisements will be sent to the edge device whose advertisement range contains the user.
User data such as the age, gender, and location of the user, and category data about products within the advertising range are sent to the recommendation model at the edge to predict the probability of purchase and consider all the users within the advertising range. This is the process of advertising through signage for products with high sales probability.
The system consists of a two-level DNN model. As shown in Fig. 5, it consists of a product candidate group generation model for predicting the products to be purchased by existing users and a DNN-based recommendation model for predicting the purchase probability of the candidate products. First, the product candidate group generation model embeds the basic information and product information of the user and uses them as input to the DNN. Using this information, the model extracts several products for the user to purchase. In the product candidate selection step, the amount of calculation in the recommendation step, which is the next step, is reduced by filtering the users within the corresponding advertisement range by using the user information and the location information of the user.
System from the user's point of view.
In the recommendation stage, the final selected product is derived by using the wide and deep recommendation model (Fig. 6), which learns the user’s previous purchase history, product metadata, and basic user information.
The wide and deep recommendation model has been chosen because the recommendation using a logistic regression model lacks serendipity, a phenomenon in which only the top products are continuously displayed. In addition, when using the DNN to solve this phenomenon, various products are displayed, but a product not suitable for the user at all may also be shown. We used the wide and deep recommendation model to avoid the two disadvantages of recommendation: over-consolidation and underfitting. The wide model uses a logistic regression model to create recommendation algorithms and returns detailed prediction results through training. However, this model always makes the same recommendation to the user, resulting in the a for mentioned lack of serendipity. Hence, the deep model is used together with the wide model to overcome the above issue. The deep model is a neural network model that generalizes specific products into higher categories (e.g., coffee, instant food, snacks) and recommends them. In this study, we used a deep model consisting of a simple DNN and a wide and deep recommendation model using a wide model as a content decision system for minimizing computation.
3.4 Probability and Method of Predicting Purchase Probability
The wide and deep model differs from other models in that it has two types of input data as shown in Fig. 7. In other words, it analyzes the result of inputting different types of data. Accordingly, discontinuous data such as category (classification) are suitable for learning. For predicting user-specific purchase history using Doc2Vec, user characteristics such as region, age, and gender were used as input values for the deep model through embedding.
In this study, we selected the most recent purchase history as test data for each user and selected the purchase history up to that point as learning data. If the user made less than one purchase, it was excluded from learning.
Referral process using the wide and deep recommendation model.
4. Implementation of Intelligent Digital Signage System
4.1 Recommendation System Construction
In this study, the products for the advertisement are selected using the wide and deep recommendation model. This model considers the merits of using both wide and deep recommendation models and overcomes their limitations. First, in the wide model, logistic regression is used to predict the products that can be purchased by a user. In the deep model, we recommend a DNN using embedded vectors.
First, the purchase history data were vectorized to learn the wide and deep recommendation model. If the purchase history data are changed to one-hot encoding during the vectorization process, the column will have as many labels as the number of traded items (N). This indicates that N products were traded. It is a classic deep learning approach to use this data simply as input, and as it is a one-hot method, the features purchased for the “apple” data cannot be utilized [41].
In this study, we perform embedding of the data with Gensim’s Doc2Vec technology to determine the distributed expression in deep learning [42] (Fig. 8). Item2Vec is an embedding technology that applies Doc2Vec used in NLP series to products.
Fig. 9 shows the vector product for 56 products and 17 categories and data consisting of [Item Name, Category] to create the input data of the wide model. Here, 896 columns are created for entering the data in the wide category, which is a combination of 56 item types and 17 upper category types. As there are four categories of 896 degrees (T, T), (T, F), (F, T), and (F, F), up to 3,584 columns are produced after stacking.
In the case of the neural network model, that is, the deep model, Fig. 10 is generalized and classified as a non-alcoholic beverage, thereby increasing the width of the recommendable product. The inputs of the deep model combine and dummy user feature data. In the dummy process, embedding technique is embedded using TensorFlow’s embedding_lookup function. In this case, categorical to num is used because the categorical variable must be entered as a factor rather than one-hot encoding.
5. Experiment and Result
Fig. 11 shows a table illustrating the experimental procedures and experimental environments to demonstrate the necessity of edge computing in digital signage. The learning speed comparison based on the same dataset in each environment is presented in Fig. 11.
To evaluate the performance of the wide and deep recommendation model used as a content determination system, we recommend the top 10 products having a score of 0.5 or higher and determine the precision of the top 10 recommendation list. In other words, we consider the hit rate: (number of hits/number of recommendations) by comparing the recommendation list with the actual purchase history of the user. Then, the precision is calculated as shown in Eq. (2) and Table 4 to determine the performance of the recommender system.
Experimental environment.
For example, the purchase history of the actual user (ID: 14674) is compared with the list of items recommended by the above recommendation model and the actual purchased items.
From Table 5, it can be observed that, although there are no products that lead to actual purchases in the recommendation list, the recommendation list is related to the actual purchased products and does not contain irrelevant products. The recommendation of chocolate to users who bought candy can be observed.
In addition, several prior experiments showed the highest precision for the top 10 recommendations near epoch 10 (Fig. 12). In addition, after experimenting by changing the number of items included in the recommendation list, it was observed that 25.94% of three recommendations and 16.28% of 10 recommendation led to actual purchases from the recommended list.
Table 6 compares the time it takes to display the same intelligent digital signage service at the edge and server. The service at the edge did not require video data transmission; hence, it was approximately twice as fast as the service in the cloud environment with a better performance.
Service rate comparison (unit: second)
6. Conclusion
In this paper, we proposed an intelligent digital signage system based on edge computing and a trained intelligent advertisement recommendation model. The proposed system consists of a server and an edge. The server manages the data, trains the advertising recommendation model, and uses the learned advertising recommendation model to determine the product to be advertised in real time. The advertisement recommendation model consisted of selecting products and predicting purchase probability. In the product candidate generation stage, the size of the candidate set was reduced by filtering out users within the advertising range using user information and user location information. We used the wide and deep recommendation model to derive the recommendation list by predicting the purchase probability of the selected products. Finally, the most suitable advertisement was selected using the predicted probability of purchase of the cluster within the advertisement range. The proposed system does not communicate with the server and determines the advertisement using the model trained at the edge. It is well suited for digital signage that requires immediate response to multiple users. In this study, we constructed a digital signage system in edge and cloud environments. Consequently, it was demonstrated that the service based on edge computing should be used in the proposed signage system because it was approximately twice as fast as the service in the cloud. In future studies, we will investigate ways to reduce various network delay times and propose systems suitable for digital signage that needs to provide real-time processing and service to multiple users by using parallel computing.
Acknowledgement
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (No. NRF-2021R1A2C2011966).