We are a startup named QuickoLabs based out of Bangalore, India. Our product SearchEnabler, is on-demand SEO software which crawls and analyzes user’s website to provide recommendations, helping them improve their website ranking in search engine results.
Our goal is to make SEO easy, affordable & measurable for start-ups and small businesses. To realize our goal, we wanted to ensure minimum cost is incurred in our operations without compromising on product capability.
Today our infrastructure holds more than 8TB of data collected from web and processes nearly 250 GB of data everyday. It consists of more than 700 Million unique URLs and analyzed more than 35 million webpages. This numbers will grow quickly as customer base increases.
Our infrastructure currently manages:
2 Applications Servers
5 Cassandra Nodes
4 Task Trackers
9 Data Nodes
We have used following open source software’s to setup 24×7 crawling, distributed storage and processing :
- Hadoop HDFS
- Cassandra NoSQL Storage
- Hadoop Map-Reduce Tasks
- Pig Scripts
- Zookeeper for co-ordination
- Apache Nutch Search Engine
- Text Processing Through Lucene