![](\"\/wp-content\/uploads\/2018\/05\/Yellow-question.png\")
<\/p>\n<\/p>\n
Series: <\/strong><\/em>The<\/strong><\/em> Hidden Challenges of Putting Hadoop and Spark in Production<\/strong><\/em><\/p>\n A recent Gartner survey estimates that only 14% of Hadoop deployments are in production. We\u2019re not surprised. We\u2019ve been in many conversations with companies that have been piloting Hadoop to bolster their analytic capabilities beyond relational databases. Common challenges fall into a few important categories, which we explore in this blog series:<\/em><\/p>\n Hadoop in the Cloud: Flexible, but Challenging<\/strong><\/span><\/p>\n Many enterprises are asking tough \u2013 but reasonable \u2013 questions about Hadoop, Spark and many other new data technologies:<\/p>\n How do I get a functional production environment more quickly? What do I buy vs. build? How can services help?<\/p>\n How do I deal with the shifting open-source software landscape in a sustainable manner? Do I bet on the current promising processing engine \u2013 how do I hedge my bets?<\/p>\n As our series introduction stated, one survey estimated that only 14% of Hadoop deployments are in production.That\u2019s because in most corporate environments, production-ready means that the cluster is secure and has the operational processes established for access, governance, compliance and service-level agreements (SLAs.) At the most basic level, companies must have a plan for how they will monitor infrastructure, software components and overall system health.<\/p>\n But they also need to plan for processes like patching and upgrading, which is complex with an ecosystem like Hadoop, Spark and related technologies, evolving at a rapid pace. Ensuring that you keep a record of the hundreds of configurations that have been applied to each layer of the stack is mandatory if you want to recover from a disaster…<\/p>\n This is especially challenging in enterprises, which is why we often hear of initial production Hadoop environments taking nine to 12 months (at best!). It\u2019s the operational processes and maintenance that can turn into a much bigger headache than the software deployment.<\/p>\n Leveraging the cloud for Hadoop: Helpful, not a panacea<\/strong><\/p>\n Deploying Hadoop and Spark in the cloud means that there is no long-term commitment to either the server type that you choose for your data nodes nor the number of nodes that you start with. There is little need to have a configuration that is configured beyond the project(s) that are currently funded and are in motion. This implies that sizing your cluster can be based on known variables for existing projects rather than trying to look out into the future and extrapolate usage. That\u2019s helpful.<\/p>\n A second advantage is that you have flexibility if you get the initially sizing incorrect. Taking away a node as you have over provisioned is a viable option. Try doing that on premises! The cloud gives you a lot of flexibility when it comes to the infrastructure layer, but<\/strong> you still need to lay down the software in the right manner and keep that overall configuration operational.<\/p>\n Configuring and Managing Hadoop in the Cloud<\/strong><\/p>\n Configuring Hadoop and Spark in the cloud, and keeping it operational, is time-consuming and where the \u201cas a service\u201d model starts to really show its value. If you\u2019ve not yet run your own production Hadoop environment, ask a friend. There\u2019s a growing understanding of the care and feeding of a cluster, cloud or not. <\/p>\n That\u2019s why having automated processes<\/strong>, which can ensure that configurations can be repeated and verified, are a critical step for a production environment. You must have processes to conform to the various compliance attestations that production environments need to have\u2013and it\u2019s not easy to DIY:<\/p>\n And if all of that sounds scary or techie to you, are you really ready to DIY in the cloud?<\/p>\n All of these operational responsibilities are necessary for Hadoop in the cloud. They are time consuming, and can take away from valuable staff time for analytics. All of this does not even touch on security which is a topic all by itself.<\/p>\n\n
\n\n