A Development of LDA Topic Association Systems Based on Spark-Hadoop Framework


Kiejin Park, Limei Peng, Journal of Information Processing Systems Vol. 14, No. 1, pp. 140-149, Feb. 2018  

https://doi.org/10.3745/JIPS.04.0057
Keywords: Association Analysis, Hadoop, LDA (Latent Dirichlet Allocation), Spark, Topic Model
Fulltext:

Abstract

Social data such as users’ comments are unstructured in nature and up-to-date technologies for analyzing such data are constrained by the available storage space and processing time when fast storing and processing is required. On the other hand, it is even difficult in using a huge amount of dynamically generated social data to analyze the user features in a high speed. To solve this problem, we design and implement a topic association analysis system based on the latent Dirichlet allocation (LDA) model. The LDA does not require the training process and thus can analyze the social users’ hourly interests on different topics in an easy way. The proposed system is constructed based on the Spark framework that is located on top of Hadoop cluster. It is advantageous of high-speed processing owing to that minimized access to hard disk is required and all the intermediately generated data are processed in the main memory. In the performance evaluation, it requires about 5 hours to analyze the topics for about 1 TB test social data (SNS comments). Moreover, through analyzing the association among topics, we can track the hourly change of social users’ interests on different topics.


Statistics
Show / Hide Statistics

Statistics (Cumulative Counts from November 1st, 2017)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.




Cite this article
[APA Style]
Park, K. & Peng, L. (2018). A Development of LDA Topic Association Systems Based on Spark-Hadoop Framework . Journal of Information Processing Systems, 14(1), 140-149. DOI: 10.3745/JIPS.04.0057.

[IEEE Style]
K. Park and L. Peng, "A Development of LDA Topic Association Systems Based on Spark-Hadoop Framework ," Journal of Information Processing Systems, vol. 14, no. 1, pp. 140-149, 2018. DOI: 10.3745/JIPS.04.0057.

[ACM Style]
Kiejin Park and Limei Peng. 2018. A Development of LDA Topic Association Systems Based on Spark-Hadoop Framework . Journal of Information Processing Systems, 14, 1, (2018), 140-149. DOI: 10.3745/JIPS.04.0057.