A Development of LDA Topic Association Systems Based on Spark-Hadoop Framework

Kiejin Park and Limei Peng
Volume: 14, No: 1, Page: 140 ~ 149, Year: 2018
10.3745/JIPS.04.0057
Keywords: Association Analysis, Hadoop, LDA (Latent Dirichlet Allocation), Spark, Topic Model
Full Text:

Abstract
Social data such as users’ comments are unstructured in nature and up-to-date technologies for analyzing such data are constrained by the available storage space and processing time when fast storing and processing is required. On the other hand, it is even difficult in using a huge amount of dynamically generated social data to analyze the user features in a high speed. To solve this problem, we design and implement a topic association analysis system based on the latent Dirichlet allocation (LDA) model. The LDA does not require the training process and thus can analyze the social users’ hourly interests on different topics in an easy way. The proposed system is constructed based on the Spark framework that is located on top of Hadoop cluster. It is advantageous of high-speed processing owing to that minimized access to hard disk is required and all the intermediately generated data are processed in the main memory. In the performance evaluation, it requires about 5 hours to analyze the topics for about 1 TB test social data (SNS comments). Moreover, through analyzing the association among topics, we can track the hourly change of social users’ interests on different topics.

Article Statistics
Multiple requests among the same broswer session are counted as one view (or download).
If you mouse over a chart, a box will show the data point's value.


Cite this article
IEEE Style
Kiejin Park and Limei Peng, "A Development of LDA Topic Association Systems Based on Spark-Hadoop Framework ," Journal of Information Processing Systems, vol. 14, no. 1, pp. 140~149, 2018. DOI: 10.3745/JIPS.04.0057.

ACM Style
Kiejin Park and Limei Peng, "A Development of LDA Topic Association Systems Based on Spark-Hadoop Framework ," Journal of Information Processing Systems, 14, 1, (2018), 140~149. DOI: 10.3745/JIPS.04.0057.