HDP Developer: Apache Spark 2.3 Training

Scala Introduction

Working with: Variables, Data Types, and Control Flow

The Scala Interpreter

Collections and their Standard Methods (e.g. map())

Working with: Functions, Methods, and Function Literals

Define the Following as they Relate to Scale: Class, Object, and Case Class

Overview, Motivations, Spark Systems

Spark Ecosystem

Spark vs. Hadoop

Acquiring and Installing Spark

The Spark Shell, SparkContext

LABS

Setting Up the Lab Environment

Starting the Scala Interpreter

A First Look at Spark

A First Look at the Spark Shell

RDD Concepts, Lifecycle, Lazy Evaluation

RDD Partitioning and Transformations

Working with RDDs Including: Creating and Transforming

An Overview of RDDs

SparkSession, Loading/Saving Data, Data Formats

Introducing DataFrames and DataSets

Identify Supported Data Formats

Working with the DataFrame (untyped) Query DSL

SQL-based Queries

Working with the DataSet (typed) API

Mapping and Splitting

DataSets vs. DataFrames vs. RDDs

LABS

RDD Basics

Operations on Multiple RDDs

Data Formats

Spark SQL Basics

DataFrame Transformations

The DataSet Typed API

Splitting Up Data

Working with: Grouping, Reducing, Joining

Shuffling, Narrow vs. Wide Dependencies, and Performance Implications

Exploring the Catalyst Query Optimizer

The Tungsten Optimizer

Discuss Caching, Including: Concepts, Storage Type, Guidelines

Minimizing Shuffling for Increased Performance

Using Broadcast Variables and Accumulators

General Performance Guidelines

LABS

Exploring Group Shuffling

Seeing Catalyst at Work

Seeing Tungsten at Work

Working with Caching, Joins, Shuffles, Broadcasts, Accumulators

Broadcast General Guidelines

Core API, SparkSession.Builder

Configuring and Creating a SparkSession

Building and Running Applications

Application Lifecycle (Driver, Executors, and Tasks)

Cluster Managers (Standalone, YARN, Mesos)

Logging and Debugging

Introduction and Streaming Basics

Spark Streaming (Spark 1.0+)

Structured Streaming (Spark 2+)

Consuming Kafka Data

LABS

Spark Job Submission

Additional Spark Capabilities

Spark Streaming

Spark Structured Streaming

Spark Structured Streaming with Kafka