![](\"\/wp-content\/uploads\/2018\/05\/horton-overload.png\")
<\/p>\n<\/p>\n
Series: The Hidden Challenges of Putting Hadoop and Spark in Production<\/strong><\/em><\/p>\n A recent Gartner survey estimates that only 14% of Hadoop deployments are in production. We\u2019re not surprised. We\u2019ve been in many conversations with companies that have been piloting Hadoop to bolster their analytic capabilities beyond relational databases. These companies want to use new distributed processing frameworks, like Apache Spark, which allow them to process data more efficiently, especially data that is inconsistently or variably structured.<\/em><\/p>\n The pilots typically involve finding a few existing servers within the datacenter and repurposing them to support Hadoop. There is often minimal infrastructure investment for trying out these new technologies, and sometimes, it\u2019s not even an official project.<\/em><\/p>\n But the stories often continue similarly: The pilot shows promising results, and a series of meetings leads to a set of potential use cases. A project is established, the clock starts ticking, and the realization sets in: <\/em><\/p>\n Building a production-grade configuration for Hadoop is a non-trivial exercise. Whether on-premises in a local data center, or in the public cloud, putting Hadoop in production requires expertise and experience to get right.<\/em><\/p>\n Common challenges fall into a few important categories, which we\u2019ll explore in this blog series:<\/em><\/p>\n Choosing the Right Infrastructure for Hadoop<\/strong><\/span><\/p>\n First, you need to know what type of machines will support the Hadoop cluster. In the single-engine database world (Oracle, GPDB, Vertica, etc.), this is a constrained problem. Yet in the Hadoop world, it is not as simple. That\u2019s because the Hadoop ecosystem has multiple processing engines and components from multiple vendors and open source projects. The list is long: Apache, Hortonworks, Cloudera, Spark, Impala, MapReduce, etc. The different engines are optimized for different types of workloads.<\/p>\n One company we spoke with started their pilot using MapReduce (M\/R) as the primary engine. However, they decided that Cloudera Spark\u2019s in-memory capabilities made it the faster, more appropriate processing engine for the problem they needed to solve. That\u2019s when they learned the hard way that the optimal infrastructure for a M\/R dominant Cloudera Hadoop configuration is different<\/em> from a Spark dominant one due to the additional memory requirements. That meant they needed to reconfigure the existing hardware by buying new memory modules, complicating and slowing down the project.<\/p>\n Keeping up with a (Very) Rapidly Evolving Ecosystem<\/strong><\/p>\n The underlying technologies in the Hadoop ecosystem are also rapidly evolving and forking-off as new projects start and vendors add in their own components. This makes things even harder to manage and plan for. Consider not long ago, in 2015, Impala (Cloudera\u2019s SQL engine for Hadoop), did not exploit multiple cores. That\u2019s a big deal when you\u2019re trying to optimize performance. Building a configuration with the best price for performance for that Impala release meant lots of machines with very few cores. But today Cloudera Impala exploits multiple cores \u2013 so the new configuration for maximum performance at the lowest cost would be fewer machines with more cores. Totally different. That\u2019s just one example of how rapidly technology is evolving.<\/p>\n If you\u2019re trying to keep up with that in your datacenter, you\u2019re constantly optimizing. It becomes incredibly difficult to choose the right type of server for on-premises deployments and manage capacity planning. In enterprise environments, servers are typically expected to last a minimum of 3 years, with the norm being closer to 5 years. Without a crystal ball, many companies overbuy hardware. They get really powerful machines \u201cjust to be safe,\u201d even though that power and capacity might sit idle for years.<\/p>\n In our next blog<\/a>, we\u2019ll talk more about a closely related subject: Hadoop performance, sizing and scaling.<\/p>\n Original artwork by Carlos Joaquin in collaboration with Cazena. In the first installment of our series on The Hidden Challenges of Putting Hadoop and Spark in Production, we explore the infrastructure selection process. This series was inspired after we read a recent Gartner survey, which estimates that only 14% of Hadoop deployments are in production. We\u2019re not surprised….<\/span>\n
\n
\nApache\u00ae, Apache Hadoop, Hadoop\u00ae<\/a>, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software <\/a>Foundation<\/a>in the United States and\/or other countries.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
\n <\/p>\n","protected":false},"author":12,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[20,22,46],"class_list":["post-2472","post","type-post","status-publish","format-standard","hentry","category-blog","tag-cloudera","tag-data-science","tag-technical"],"_links":{"self":[{"href":"https:\/\/cazenasite.com\/index.php?rest_route=\/wp\/v2\/posts\/2472","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cazenasite.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cazenasite.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cazenasite.com\/index.php?rest_route=\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/cazenasite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2472"}],"version-history":[{"count":0,"href":"https:\/\/cazenasite.com\/index.php?rest_route=\/wp\/v2\/posts\/2472\/revisions"}],"wp:attachment":[{"href":"https:\/\/cazenasite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2472"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cazenasite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2472"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cazenasite.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2472"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}