{"id":2450,"date":"2015-05-09T00:00:00","date_gmt":"2015-05-09T00:00:00","guid":{"rendered":""},"modified":"2021-01-08T21:13:14","modified_gmt":"2021-01-08T21:13:14","slug":"devops-killer-drag-big-data","status":"publish","type":"post","link":"https:\/\/cazenasite.com\/?p=2450","title":{"rendered":"DevOps: The Killer Drag for Big Data"},"content":{"rendered":"

By Prat Moghe, Cazena Founder<\/em><\/p>\n

As enterprises seek to drive faster big data outcomes, cloud offers a promising solution for agility. Indeed, public cloud infrastructure is, in many cases, far cheaper and faster to deploy than on-premises alternatives. Yet, cloud big data deployments have proven complex for many enterprises, and few companies are ready to call systems officially in production. Reasons range from compliance concerns to integration issues, but there\u2019s a much bigger problem lurking.<\/p>\n

The real challenge holding back production big data cloud deployments has less to do with the infrastructure or platform as a service (PaaS) capabilities: It is the pervasive lack of\u00a0DevOps\u00a0skills for big data.<\/strong><\/p>\n

We estimate that 70-80% of the cost of any production big data environment is driven by DevOps resources. These skills are hard to find and retain. They are extremely expensive. Companies often underestimate the amount of work they will need to do to keep systems current and secure, and the time that they will need to do it. Thus, relying on DevOps can put a huge and unnecessary drag on data projects, slowing down potential returns, hindering analytic agility and throwing new variables into ROI equations. Addressing the DevOps drag in big data should be a critical imperative for all enterprises that want faster business outcomes. <\/strong><\/p>\n

Let me provide a few simple examples for illustration.<\/p>\n

Cost: Cloud is cheap. DevOps is expensive. <\/strong><\/p>\n

\"DevOpsDragSummary8.jpg\"<\/p>\n

A common goal of most production big data environments is to deliver self-service capabilities to data scientists and analysts, so that they can reduce preparation work and focus on analysis.<\/p>\n

For a new team, let us assume a typical starter PaaS platform with Hadoop\/Spark on AWS supporting production workloads: 10 normalized compute nodes plus object store, data processing PaaS, and nominal data ingest\/egress. The licensing for this works out to about $80K a year. For larger PaaS footprints supporting multiple groups, say a medium-sized 50 node environment, total cloud costs work out to about $320K\/year at current prices.<\/p>\n

However, to deliver this big data platform capability requires expertise in development, optimization, ongoing operations and support. The problem isn\u2019t just financial; it\u2019s also difficult to hire these teams, depending on your location and requirements.<\/p>\n

In fact, recent\u00a0Big Data DevOps Team Salary Research uncovered some interesting facts: The\u00a0salaries for a\u00a0five person big data\u00a0team\u00a0cost up to $629,000 –\u00a0$1,186,000\u00a0annually, depending on seniority,\u00a0industry and location.\u00a0<\/strong><\/p>\n

While you may be able to cross-train, cloud\u00a0and big data technologies require specific skill sets and coordination across activities. Different areas of expertise need to come together, at a minimum:<\/p>\n

1. Cloud DevOps<\/strong> to administer cloud accounts and resources, and manage the cloud infrastructure.<\/p>\n

2. Hadoop Platform Administrator<\/strong> to provision and tune Hadoop\/Spark nodes, with attached data stores and centralized object store required to deliver workload performance.<\/p>\n

3. Cloud Security Architect<\/strong> to administer security controls such as encryption, key-management, identities and role-based access control, as well as establish and ensure compliance controls.<\/p>\n

4. Data Management Lead<\/strong> to manage and administer data ingestion, data governance and logging as well as manage user access from a variety of data engineering, machine learning and SQL tools.<\/p>\n

5. Data Production Ops<\/strong> to cover first and second-line alerting, support, root-cause analysis and upgrade\/patching\/validation issues. This is also a catch-all capability required for technical issues like sprint tracking, billing, SLA monitoring and management.<\/p>\n

While you might not need all of these as individual, full-time-employees to start, every area is important. For our basic starter environment example, we will likely need at least two superhero DevOps pros to cover all these skills, which will cost about $200K each or ~$400K annually. For a medium, scaled out environment (50+ nodes), individuals will be needed to fill all five roles, at a minimum, particularly since there will be multiple applications and end-user groups accessing these systems. Let\u2019s assume these are heroes (not multi-role superheroes), in the mid range of salaries, that cost ~$150K annually each. With five people, this team will cost about $750K\u00a0annually. <\/strong><\/p>\n

Now let\u2019s do the math. This means that 70-80% of your overall costs are driven by DevOps. Cloud PaaS\/IaaS itself costs less than 20% of the overall investment. <\/strong><\/p>\n

\"ComparisonZ.jpg\"<\/p>\n

 <\/p>\n

The Impact on\u00a0Time to Production and Operational Complexity<\/strong><\/h3>\n

Cloud is fast to cycle and easy to iterate through failure. DevOps is slow to cycle and has high risk to fail due to manual processes and significant configuration and optimization requirements. Spinning up cloud clusters is straightforward, and takes seconds to minutes. However, managing them in production is hard, especially with an ever-changing ecosystem of capabilities and many moving parts. Developing a hybrid data access environment for cloud infrastructure is also complex. Security, compliance processes and governance of users accessing the cloud platform is non-trivial. Taking all this into account, developing a production environment could take 6 months, assuming it\u2019s well-managed. A wrong hire or unforeseen departure is even worse and could extend projects to a year or more.<\/p>\n

Purely as a thought experiment, if we were to define the overall production \u201cdrag\u201d as: [Cost of Investment<\/em>] multiplied by [Time to get to Production<\/em>], a stark picture emerges. Production drag is almost completely defined by DevOps drag \u2013 99.97% of the physical friction is DevOps. This drag completely dominates cloud agility. See the graphic below; the area of the graph represents the drag.<\/strong><\/p>\n

\"DragAreaUpdate.jpg\"<\/p>\n

Takeaways for Cloud and Big Data Projects<\/strong><\/h3>\n