Big Data

A Comparison of different Big Data Processing Platforms

ClusterExpected VolumeBenchmark hardwareProject Hardware requirements
CoresRAM# nodesDisk
Source6 Million records / month

~ 3 records per second

HDFS6 million/month1 namenode, 20 datanodes, 2 CPU/node, 64GB RAM/node16G1 : Master

3: Slaves

120% of 6G


Kafka4 topics

6 million/month per topic

1 nodes @ 4 GB RAM, 1 CPU,200 GB disk each

16,000 Msg/sec

14G124 GB/month
Kafka Connector for HBASEN/A
Kafka Connector for HDFSN/A
Logstash6 million/month1 node, 3.75GB RAM, 1 CPU Cores

180 Events/sec

Hbase6GB/month7 nodes, 32gb RAM, 8 CPU cores

60240 req/sec – 200 req/sec



Hassan Askari