Design of a Log Analysis System Based on Hadoop

被引：0
作者：
Li, Jingmin [1 ]
机构：
[1] Wuyi Univ, Sch Comp Sci, Jiangmen 529020, Guangdong, Peoples R China
来源：
INDUSTRIAL ENGINEERING, MACHINE DESIGN AND AUTOMATION (IEMDA 2014) & COMPUTER SCIENCE AND APPLICATION (CCSA 2014) | 2015年
关键词：
Hadoop; Hive; Cloud Computing; Log Analysis;
D O I：
暂无
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
User behavior analysis is the hotspot of the internet information technology. Through the analysis of the web log, we can analyze users' operation behavior. However, most of the data of logs are expanding greatly day by day, and the log file size is usually GB level. The traditional single computing node is difficult to process such large amounts of data. So this paper presents a log analysis system based on Hadoop. The system architecture is composed by Flume, HBase and Hive. It will improve the system's computing power and scalability.
引用
收藏
页码：249 / 256
页数：8
相关论文
共 9 条
  • [1] Afrah A., 2008, DISTR SMART CAM 2008, P1
  • [2] Hadoop Architecture and Its Issues
    Alam, Anam
    Ahmed, Jamil
    [J]. 2014 INTERNATIONAL CONFERENCE ON COMPUTATIONAL SCIENCE AND COMPUTATIONAL INTELLIGENCE (CSCI), VOL 2, 2014, : 288 - 291
  • [3] BAI J, 2013, NAT COMP ICNC 2013 9, P1166
  • [4] Gunarathne T., 2010, Proceedings of the 2010 IEEE 2nd International Conference on Cloud Computing Technology and Science (CloudCom 2010), P565, DOI 10.1109/CloudCom.2010.107
  • [5] Kalagiakos P., 2011, 2011 5th International Conference on Application of Information and Communication Technologies (AICT), P1, DOI DOI 10.1109/ICAICT.2011.6110925
  • [6] LI Jing-min, 2010, 2 INT C INF SCI ENG, P1417
  • [7] Lin XQ, 2013, 2013 5TH IEEE INTERNATIONAL CONFERENCE ON BROADBAND NETWORK & MULTIMEDIA TECHNOLOGY (IC-BNMT), P273, DOI 10.1109/ICBNMT.2013.6823956
  • [8] Vora M. N., 2011, Proceedings of the 2011 International Conference on Computer Science and Network Technology (ICCSNT), P601, DOI 10.1109/ICCSNT.2011.6182030
  • [9] Dynamic load balancing for distributed network management
    Yoshihara, K
    Isomura, M
    Horiuchi, H
    [J]. INTEGRATED NETWORK MANAGEMENT VIII: MANAGING IT ALL, 2003, 118 : 277 - 290
1