unbelievablethree

Dosya yonetim sistemine giris yapin

{"id":2485,"date":"2018-07-30T00:00:00","date_gmt":"2018-07-30T00:00:00","guid":{"rendered":""},"modified":"2021-01-08T20:39:55","modified_gmt":"2021-01-08T20:39:55","slug":"streaming-data-analytics-kafka-spark-kudu-tutorial","status":"publish","type":"post","link":"https:\/\/cazenasite.com\/?p=2485","title":{"rendered":"Streaming Data Analytics processing with Kafka + Spark + Kudu: A Field Tutorial"},"content":{"rendered":"

<\/p>\n

Cazena\u2019s cloud data lake team carefully tracks the latest architectural approaches and technologies against our customer\u2019s current requirements. We\u2019ve seen much more interest in real-time streaming data analytics with Kafka + Apache Spark + Kudu. Companies are using streaming data for a wide variety of use cases, from IoT applications to real-time workloads, and relying on Cazena\u2019s Data Lake as a Service as part of a near-real-time data pipeline. Here is what we learned about stream processing with Kafka, Spark and Kudu in a brief tutorial.<\/p>\n

This is an example of a streaming data analytics use case we see frequently:
\n <\/p>\n

1. Stream data in from a Kafka cluster to a cloud data lake, analyze it, and expose processed data to end users and applications. This requires a cloud data lake with embedded analytic engines, such as Cazena\u2019s Cloudera-powered Data Lake as a Service.<\/p>\n

2. Furthermore, once the data is streamed in, merge historical and new data in-stream.<\/p>\n

3. Finally, performance is key! The system needs to be able to ingest, transform, and write data quickly, i.e. with sub-second, end-to-end latencies would be nice. Analytical result data should be available for immediate consumption, and end-user queries should run quickly too.<\/p>\n

One can come up with a bunch of possible complicated architectures for this use case in Hadoop.<\/p>\n

For example, we could use Hbase for fast random reads and writes. Or, we could store data in a Parquet-formatted table on HDFS and use Impala for interactive analytics. Then, those tables will be query-able in Spark and Hive, and we could use a total of four different execution engines on a single cluster, taking advantage of the strengths of each. Plus we would need to make sure to tune them all to run well under load.<\/p>\n

If that sounds like an unappealing and time-consuming project, may we could recommend something simpler?<\/p>\n

Using Cazena as Part of a Near Real-Time Data Analytics Pipeline<\/strong><\/p>\n

We\u2019ve architected Cazena\u2019s Data Lake as a Service to support near-real-time streaming data analytics use cases for demanding projects. In addition to us spending many hours benchmarking new technologies for our platform — we’ve added automation, workload optimization and built-in DevOps. <\/p>\n

Cazena\u2019s End-to-End Data Stream Processing and Analytics Architecture<\/strong><\/em><\/p>\n

\"EngBlog2.jpg\"<\/p>\n

We’ve found that there is a lot to like about a combination of Spark Streaming (ETL) and Kudu (storage layer) tool kit for this workload:<\/p>\n