Analyzing Customer Experience in Hotel Services Using Topic Modeling

1. Introduction

Everyday users post or review a large volume of comments, reviews, blog posts, or online news [1,2]. This data is stored in unstructured texts [3]. The problem is that with a significant amount of textual content, not all of it can be read. Therefore, an approach is needed to “summarize” this data into profound characteristics, such as the subject of a comment or an opinion, or online reviews of the product or the service being discussed, in other words, the “topics” users are referring to.

The topic modeling approach has been used by many researchers to analyze data and extract information in many fields such as tourism, bioinformatics, accommodation, education, and online sales [2-10]. Topics of opinion when making comments or providing feedback on the company’s products or services are highly diverse, they provide information and capture the habits and behavior of individuals or online communities. However, owing to the features of online networks, the subjects of the content of messages have not been created beforehand or, in other words, the topic being discussed on the network forum is implicit [4,7]. Therefore, discovering the topic and understanding the content of users’ exchanged messages is a challenging problem [8].

In this study, we were able to successfully detect hidden topics that our guests were referring to about hotel services so that we could capture the issues of interest to our users. With the problems identified, we can retain customers, improve existing customers’ satisfaction, or increase conversion rates when applying a business strategy that matches target products and services with the right customers.

The rest of this paper is organized as follows: Section 2 discusses related studies, focusing on analyzing text data and user opinions in the field of hotel services, latent Dirichlet allocation (LDA) topic modeling, and labeling topic techniques. Specific research methods applied are presented in Section 3 and the results are discussed in Section 4. Finally, Section 5 presents the conclusions with an outline of future work.

2. Related Work

In previous studies [1,2,4-8], the authors studied and experimented on unstructured data. This data mainly comprised of comments from customers’ feedback on the company’s products and services through online channels. Analyzing users’ comments in the field of hotel services has also been studied [1,2,5,11-13]. The topic model [14,15] is a useful tool for identifying hidden topics from online customer reviews or social networking data [6,9,11,12,15-20]. Labeling the topics, which is a difficult problem, was mentioned in [7] and [21].

2.1 Analyzing Customer Experience Approaches to Text Mining

Text data mining refers to the process of extracting high-quality information from text. This is an important part of data mining and knowledge discovery [8]. It handles a set of documents and extracts meaningful information from the text, and creates the implicit information using various data mining algorithms such as statistics and machine learning.

Among the tasks of document data mining, opinion mining is a difficult task that has attracted much attention recently. This is one of the most active research topics and has been widely studied in data, web, and text mining [13]. User experience is the feeling that customers receive when using a company’s products or brands. User opinion analysis focuses on opinions, feelings, reviews, attitudes, and emotions from written language.

Previously, due to time and technology limitations, customer opinions often had to be surveyed by questionnaire. However, with the development of the Internet and information technology, customer feedback can be quickly collected online and this reduces the time and manpower required to better understand customer experience. The growing importance of analyzing user comments coincides with the growth of social media, such as reviews, forum discussions, blogs, and social networks such as Facebook and Twitter. In particular, in the era of digital development, we now have a significant amount of data recorded in digital form for analysis [2,13,20].

2.2 Analyzing Customer Experience in Hotel Services

Currently, the hotel and restaurant industry has undergone continuous growth and profound development throughout the world, which is recognized by international organizations such as the World Bank and the World Tourism Organization [2]. A number of articles have emerged on several methods used in research to extract comments from hotel reviews. Most of these approaches are based on natural language processing techniques, resources, and vocabulary approaches based on machine learning [1,5]. In [13], the authors developed a hybrid model to implement opinion mining on multilingual social media texts and explore differences of opinions of cross-cultural customers. The results show that there is a difference in preferential hotel services between Vietnamese and American customers. Based on the results of this study, hotels should improve customer satisfaction by diversifying services according to customers from different cultures.

In other studies [2,5,11], the authors also analyzed customer experience in tourism by analyzing opinions according to the topic model [2,5] and characteristics extraction techniques including knowledge-based and aspect-based approaches [11]. With these methods, the author proposes a number of important tools for tourists when deciding which hotel to stay in and which restaurants or places to visit. These articles focus on discovering opinions applied to data from travel rating websites. The studies [2,5,11] describe how the emotion analysis results of text reviews can be visualized using Google Maps, enabling users to easily discover good hotels and places to stay. More advanced features are also provided, such as visual images and interactions [11]. Some applications from the research [5] also need to be experimented and applied, particularly the applications of (1) forecasting and proposing ratings of products and services, (2) topic analysis and interpretation, and (3) proposal ratings for reviewers.

2.3 LDA Topic Modeling

In machine learning and natural language processing, the topic model is a type of statistical model to explore abstract “topics” that occur in a collection of documents. The topic model is a text-mining tool frequently used to explore the semantic structures hidden in the body of documents [9,10]. In the topicclassification problem, the LDA algorithm [9] is one of the more favored model classes because of its fast calculation efficiency, providing accurate results based on the generative statistical model (Fig. 1). The results returned by LDA include textual and each word distribution. Textual distribution is a combination of a mixed distribution of a fixed number of topics, and each word distribution represents the level of contribution to the document through its representation in topics.

Definitions used in the LDA model:

Word is the most basic unit of the LDA. A word is determined by an index in the dictionary with values of [TeX:] $$1,2, \ldots, \mathrm{V} . w_{i}$$ is expressed as a one-hot vector [TeX:] $$w_{i} \in \mathbb{R}^{\mathbf{v}}$$ such that the i-th element of the vector is 1 and the remaining elements are 0.

The document is a combination of N words denoted by [TeX:] $$\mathbf{w}=\left(w 1, w_{2}, \ldots, w N\right).$$ Each vector represents one word in a sentence.

The corpus is a combination of M documents denoted by [TeX:] $$\boldsymbol{D}=\left\{w_{1}, w_{2}, \ldots, w_{M}\right\}.$$

Latent topics are determined based on the distribution of words and mediate the representation of documents by topic. The number of topics was predefined, and the symbol was K.

A formal description of the LDA model is shown in Table 1.

In the LDA model, there are two continuous iterative processes: the topic selection process and the word selection process. The initial parameters of the process were and . Then, we calculate the joint distribution of topic and the distribution of words according to topic w. To infer topics in the corpus, the model creates an imaginative process whereby texts are created in a mechanism that can infer and reverse that process. The probability distribution of the document is created in a random mix of topics, in which each topic is determined by the distribution across all words. LDA [9] assumes a generative process for a corpus D consisting of M documents, as follows:

Each document has a length [TeX:] $$N_{i},$$ with N∼Poisson [TeX:] $$(\xi).$$

(2) Choose ∼Dir().

(3) Choose [TeX:] $$\varphi \sim \operatorname{Dir}(\beta)$$

(4) For each of the [TeX:] $$\mathcal{W}_{i j}$$ of document:

(a) Choose a topic [TeX:] $$z_{i j}$$ –multinomial [TeX:] $$\left(\theta_{i}\right)$$.

(b) Choose a word [TeX:] $$w_{i j}$$ from [TeX:] $$p\left(w_{i j} \mid z_{i j}, \beta\right),$$ a multinomial conditional probability on the topic [TeX:] $$z_{i j}.$$

First, we assume that the number of hidden topics is known and equal to K; therefore, K will also specify the number of dimensions of the Dirichlet distribution. Second, we determine the probability of the word parameterized by [TeX:] $$\mid \beta \in \mathbb{R}^{K x V}, \text { with } \beta_{i j}=p\left(w_{j}=1 \mid z_{i}=1\right).$$ Then, these parameters are fixed. Finally, we calculate the probability density function of topics for each document once we know the α parameter according to the Dirichlet distribution:

Then, the general probability distribution of the topic mixture with the set N topic z and the set N words w with , is known as

The component [TeX:] $$p(\theta \mid \alpha)$$ is a mixture topic probability distribution corresponding to the document when the Dirichlet distribution α parameter is known in advance. The other side [TeX:] $$\prod_{i=1}^{N} p\left(z_{n} \mid \theta\right) p\left(w_{n} \mid z_{n}, \beta\right)$$ is the probability distribution when determining the distribution mixture topic and Dirichlet distribution parameter . If we take the marginal probability of a document by integrating over and taking the sum over z, we obtain

Finally, we calculate the probability of the corpus based on the marginal probability from a single document:

From the probability equation on the whole document, the expectation maximization (EM) method is applied to estimate the parameters and and then calculate [TeX:] $$\theta, z, \varphi, w.$$

2.4 Labeling of Topics

Because a word can be distributed among different topics, it is difficult to identify the label of each topic and distinguish the differences among topics. Research in [7] and [21] has shown that there are two main ways to label topics: (1) asking experts in specific fields to label the topics (this is very difficult to achieve because of time and cost constraints) and (2) automatic labeling of topics. In [7], a method of automatic labeling of topics was used based on training data. First, the authors created a list of topics, including labels for product marketing. They then manually chose documents that belonged to only one topic; they selected approximately 500 documents for each topic. Finally, they used an LDA model with this corpus. As a result, the list of distributions of each word belonging to a specific topic has already been labeled. In the next step, they calculated similar measures between the training topics and the discovered ones, including cosine similarity, overlapping similarity, mutual similarity, and dice similarity plus Tanimoto and Jaccard coefficients. These measures helped to compare topics and obtain labels for detected topics. They then chose the maximum value for the results.

3. Research Methods

3.1 Overview of Research Model

Opinion mining is a classification process with three main levels: document level, sentence level, and aspect level [13]. Document-level opinion mining aims to classify a document that expresses an opinion on a specific topic [4]. At this level, a document is considered a basic unit of information. To analyze the model in this study, customer reviews or comments from online websites were collected, and then those written in English were extracted. Data preprocessing was performed carefully using a Python package. After obtaining the clean data set, the model was trained until the optimal number of topics (K) were selected, and K is the input for building the LDA model. The output of the LDA model is the set of topics and keywords, and a model is built to label the extracted topics. Finally, hidden topics were identified and visualized using the pyLDAvis package [22]. The flow in Fig. 2 proposes an experimental model for collecting, processing, and analyzing customer experience.

3.2 Data Collection

The experimental corpus in this study was collected from websites of companies in hotel services, including https://www.agoda.com/ and https://www.booking.com/. To collect data, we researched application programming in Python to access the website’s API and collect user reviews and articles on the review tabs. The JSON data were converted to a CSV data frame format, and then topic extraction analysis was performed on the collected corpus. Some extracted attributes for analysis include hotel_id, review_comments, rating, and review_date.

A total of 99,322 user comments were collected and used as input data for analyzing the model, further details are listed in Table 2.

3.3 Data Preprocessing

Preprocessing data is one of the most important steps in data mining, especially in the extraction of text data, because there are many differences in text-based content on electronic media channels on the Internet

In online services, when providing feedback, users often repeat a number of special characters to emphasize their messages. These words can make it difficult to analyze reviews, to avoid this problem, these special characters should be removed. Punctuation marks that are not meaningful in the corpus were also deleted. Uppercase characters are converted to lowercase characters, numbers, and spaces, and stop words are removed. Users on social networks and online services often use a mix or match of misspelled words and unexpected or intentional mistakes. In particular, there have been recent users accessing online services via mobile phones and may ignore grammar and spelling rules, use of abbreviations, emoticons, and more concise sentences. Therefore, the data preprocessing stage is critical and determines the accuracy of the model. Fig. 3 shows the preprocessing data flow before being built into the LDA model.

3.4 Building the LDA Model

3.4.1 Creating the bigram and trigram for the model

The LDA model uses the input as a co-occurrence matrix of words (n-grams). Bigrams are two words that frequently occur together in documents. Trigrams are three frequently occurring words. To calculate the frequency of copper appearing on these matrices, bigrams and trigrams were created. The function phrases() in Gensim is useful for building bigrams and trigrams. In the coding process, min_ counts the minimum frequency for a given word included in the grams, and the threshold value is input. In the next step, the stop-words are removed and only the words that are tagged with the category [‘NOUN’, ‘ADJ’, ‘VERB’, ‘ADV’] are filtered out. These stop words in the English language are already included in the nltk package.

3.4.2 Creating the dictionary and corpus

The dictionary and corpus are the two main inputs for the LDA model. The Gensim package was used to create the data. After processing, we obtained a corpus, that is; a set of pairs (index, frequency) that encodes the documents about the index specified in the dictionary along with the frequency of their occurrence in the text. To convert back from index to vocabulary, we used the dictionary id2word.

3.4.3 Building a LDA model

LDA describes documents as a mixture of topics with certain probabilities. For the purpose of paragraphs (documents) is represented by a number of topics, which are then represented by a small set of words (with the weight corresponding to each descending word). The main parameters specified in the Gensim LDA model are the number of topics (num_topics), the number of documents to be included in each training batch (chunksize), and the number of training sessions (passes). After the model is trained, the results can be saved to the folder for later use.

The Perplexity and Coherence Score are indicators that describe the quality of the model. It can be used in the best number of topics, which is consistent with the data. Perplexity is based on the logarithm of the maximum likelihood function (MLE), which lowers the perplexity of the model quality as much as possible. On the contrary, the model with the higher coherence score will be a better model. In this study, the authors experimented with the topic number 14 (with the corresponding Perplexity = -9.0373 and Coherence Score = 0.6392) as input parameters for the model. Fig. 4 shows the correlation between the coherence score and the number of topics in the experimental process.

4. Experimental Results

4.1 Identifying the Topics and Dominant Topics

After training the LDA model, we determined the distribution of documents by topics and represented topics according to the distribution of words. Table 3 shows keywords with probability of Topics 0, 2, 5, 9, and 13.

For each of the above topics, the keywords are expressed according to the probability of appearing from high to low, for example in Topic 0 the word “bus” has the highest probability of 0.1642, similar to Topic 9 the word “room” appears the most frequently with a probability of 0.0529.

After identifying the topic along with the corresponding probabilities of the word in that topic, we identify the predominant topics in the documents, i.e., the ones with the highest probability rate. The format_topics_sentences() function is used to determine the dominant topic in a document. Table 4 shows the dominant topics with percentages in the document.

4.2 Inferring the Label of Topics

LDA is a generative approach, in which for each topic, the model simulates the probabilities of word occurrences as well as the probabilities of topics within the document. The top 10 words for each topic are displayed in Fig. 5.

We can observe the following meaningful matches. Topic 0 deals with customers’ transportation to arrive or leave the hotel. Topic 2 appears to be related to the price and cost of renting a room at the hotel. Topic 13 appears to be about the quality of meal services in restaurants and hotels.

An open source library, MALLET, which is a topic analysis toolkit based on the LDA model, was developed by McCallum and used to analyze the reviews not only chosen from the corpus but also those already labeled [7]. After the detailed list of training topics were collected, we executed the topic labeling technique according to the similarity measures mentioned in Section 2.4. The labels for the discovered topics were identified. For instance, in Fig. 6, the label of Topic 5 is “people” because it has the highest values. Equally, the label of Topic 9 is “room services.” From the results of the keywords extracted from Fig. 5, we can infer the labels of the corresponding topics, as shown in Fig. 6.

4.3 Visualizing and Interpreting Results

The pyLDAvis package was developed to work effectively with Jupyter notebooks and to be useful in visualizing the LDA model results with interactive charts [22], as shown in Fig. 7. Each bubble on the left side represents the topic. The larger the bubble, the more popular the topic is. A good topic model will consist of fairly large, non-overlapping bubbles scattered in the chart instead of being grouped in a quadrant. A model with too many topics often has many small bubbles overlapping. The words and bars on the right-hand side are updated when one of the bubbles is moved. These words are prominent keywords that constitute the chosen topic.

By using the pyLDAvis package that is designed to support interactively: (1) we can manually select each topic to identify the most frequently and/or relevant terms and use different values for the λ parameter. This can be helpful when attempting to assign a human interpretable name or “meaning” to each topic; (2) discovering the intertopic gap plot can help to learn how topics relate to each other, including potential higher-level structures among topic groups.

5. Conclusion and Future Work

The main contribution of this article is that a novel method was presented for analyzing customer experience using a topic model based on a corpus for the hotel sector. The results show that the proposed research method is effective with the goal of applying the topic model to discover user comments. This study used data collected from e-commerce sites such as Agoda and Booking from 2012 to 2019, and can act as a representative to experiment with the model. The extracted keywords corresponded to each topic. The latent topic set was found to reflect the issues that customers are often interested in relating to hotel services. Establishing these topics is one of the best approaches to better understand customer experience. Thus, the proposed technique is more closely aligned with human intuition on topics and keywords. The results of this work can be applied to support decision-making systems for improving service quality as well as business development in the field of hotel services.

Future work will concentrate on proposing a model for collecting and classifying customer comments in real time, and the classification results will be forwarded to real-time analysis systems in which each comment will be recorded with the temporal factor. The user opinion analysis system will have the ability to analyze negative and positive changes. Customer issues will be addressed over time to help businesses quickly develop appropriate strategies, deal with crises in a timely manner, or identify and improve factors that enhance customer satisfaction.