Data Engineering Foundations Part 2: Building Data Pipelines with Kafka and Nifi

Colleagues, the “Data Engineering Foundations Part 2: Building Data Pipelines with Kafka and NiFi” program introduces you to creating data pipelines at scale with Kafka and NiFi. You learn to work with the Kafka message broker and discover how to establish NiFi dataflow. You also learn about data movement and storage. All software used in videos is open source and freely available for your use and experimentation on the included virtual machine. Learn Kafka topics, brokers, and partitions, implement basic Kafka usage modes, Kafka producers and consumers with Python, KafkaEsque graphical user interface, core concepts of NiFi, NiFi flow and web UI components, direct data movement with HDFS, HBase with Python Happybase and Sqoop for database movement. Skill-based lessons address: 1) Working with the Kafka Message Broker - Kafka message broker concept and describes the producer-consumer model that enables input data to be reliably decoupled from output requests. Kafka producers and consumers are developed using Python, and internal broker operations are displayed using the Kafkaesque graphical user interface, 2) Working with NiFi Dataflow - Lesson 8 begins with a description of NiFi flow-based programming and then provides several examples that include writing pipeline data to the local file system, then to the Hadoop Distributed File System, and finally to Hadoop Hive tables. The entire flow process is constructed using the NiFi web Graphical User Interface. The creation of portable flow templates for all examples is also presented, 3) Big Data Movement and Storage - moving data to and from the Hadoop Distributed File System. Hands-on examples include direct web downloads and using Python Pydoop to move data. Basic data movement between Apache HBase, Hive, and Spark using Python Happybase and Hive-SQL. Finally, movement of relational data to and from the Hadoop Distributed File System is demonstrated using Apache Sqoop.

Enroll today (teams & executives are welcome): https://tinyurl.com/bdpz4vf 

Download your free Data Science  - Career Transformation Guide.

Explore our Data-Driven Organizations Audible and Kindle book series on Amazon:

1 - Data-Driven Decision-Making  (Audible) (Kindle)

2 - Implementing Data Science Methodology: From Data Wrangling to Data Viz (Audible) (Kindle)

3 - The Upskill Gambit - Discover the 5 Keys to Your Career and Income Security in the Digital Age (Audible) (Kindle) 

Much career success, Lawrence E. Wilson - AI Academy (share with your team)