IMPLEMENTATION OF COMBINER FOR EFFICIENT BIG DATA PROCESSING IN HADOOP MAPREDUCE FRAMEWORK

Abstract

Big Data is a data that cannot be processed or analyzed by using traditional systems such as relational databases and data warehouses. The Big Data can be structured, unstructured or semi-structured. An open source, large data processing framework called Hadoop is been used to process large amounts of data. However, processing jobs in Hadoop is time-consuming. In this paper I have proposed a Hadoop Online Aggregation Technique to decrease response time of Hadoop by executing the job partially. In Online Aggregation of Hadoop MapReduce an early result are made available to the user before job completion. In online aggregation, Hadoop system processes a query in an online fashion. The processing of Hadoop is also benefited by adding the combiners in MapReduce paradigm. The proposed technique can reduce the total computing time taken by the Hadoop MapReduce framework.