HDP Developer: Apache Spark 2.3 Training

• Scala Introduction
• Working with: Variables, Data Types, and Control Flow
• The Scala Interpreter
• Collections and their Standard Methods (e.g. map())
• Working with: Functions, Methods, and Function Literals
• Define the Following as they Relate to Scale: Class, Object, and Case Class
• Overview, Motivations, Spark Systems
• Spark Ecosystem
• Spark vs. Hadoop
• Acquiring and Installing Spark
• The Spark Shell, SparkContext

LABS

• Setting Up the Lab Environment
• Starting the Scala Interpreter
• A First Look at Spark
• A First Look at the Spark Shell

• RDD Concepts, Lifecycle, Lazy Evaluation
• RDD Partitioning and Transformations
• Working with RDDs Including: Creating and Transforming
• An Overview of RDDs
• SparkSession, Loading/Saving Data, Data Formats
• Introducing DataFrames and DataSets
• Identify Supported Data Formats
• Working with the DataFrame (untyped) Query DSL
• SQL-based Queries
• Working with the DataSet (typed) API
• Mapping and Splitting
• DataSets vs. DataFrames vs. RDDs

LABS

• RDD Basics
• Operations on Multiple RDDs
• Data Formats
• Spark SQL Basics
• DataFrame Transformations
• The DataSet Typed API
• Splitting Up Data

• Working with: Grouping, Reducing, Joining
• Shuffling, Narrow vs. Wide Dependencies, and Performance Implications
• Exploring the Catalyst Query Optimizer
• The Tungsten Optimizer
• Discuss Caching, Including: Concepts, Storage Type, Guidelines
• Minimizing Shuffling for Increased Performance
• Using Broadcast Variables and Accumulators
• General Performance Guidelines

LABS

• Exploring Group Shuffling
• Seeing Catalyst at Work
• Seeing Tungsten at Work
• Working with Caching, Joins, Shuffles, Broadcasts, Accumulators
• Broadcast General Guidelines

• Core API, SparkSession.Builder
• Configuring and Creating a SparkSession
• Building and Running Applications
• Application Lifecycle (Driver, Executors, and Tasks)
• Cluster Managers (Standalone, YARN, Mesos)
• Logging and Debugging
• Introduction and Streaming Basics
• Spark Streaming (Spark 1.0+)
• Structured Streaming (Spark 2+)
• Consuming Kafka Data

LABS

• Spark Job Submission
• Additional Spark Capabilities
• Spark Streaming
• Spark Structured Streaming
• Spark Structured Streaming with Kafka