Blog Post

Big Data
Hassan Askari 0 Comments

A Comparison of different Big Data Processing Platforms

Cluster Expected Volume Benchmark hardware Project Hardware requirements
Cores RAM # nodes Disk
Source 6 Million records / month

~ 3 records per second
HDFS 6 million/month 1 namenode, 20 datanodes, 2 CPU/node, 64GB RAM/node 1 6G 1 : Master

3: Slaves

 120% of 6G

=7.2GB/month
Kafka 4 topics

6 million/month per topic

 1 nodes @ 4 GB RAM, 1 CPU,200 GB disk each

16,000 Msg/sec

 1 4G 1 24 GB/month
Kafka Connector for HBASE N/A
Kafka Connector for HDFS N/A
Logstash 6 million/month 1 node, 3.75GB RAM, 1 CPU Cores

180 Events/sec

 1 4G 1
Hbase 6GB/month 7 nodes, 32gb RAM, 8 CPU cores

60240 req/sec – 200 req/sec

 1 8G 1 4GB/month

Related Posts

♥1

Author

Hassan Askari