Beginning Apache Spark 3 : with DataFrame, Spark SQL, structured streaming, and Spark machine learning library / Hien Luu.
Luu, Hien, author.
2021
QA76.9.D3
Available Online

Linked e-resources

Linked Resource
Online Access
Concurrent users
Unlimited
Authorized users
Authorized users
Document Delivery Supplied
Can lend chapters, not whole ebooks

Details

Title
Beginning Apache Spark 3 : with DataFrame, Spark SQL, structured streaming, and Spark machine learning library / Hien Luu.
Author
Luu, Hien, author.
Edition
Second edition.
ISBN
9781484273838 (electronic bk.)
1484273834 (electronic bk.)
1484273826
9781484273821
DOI
https://doi.org/10.1007/978-1-4842-7383-8
Published
New York : Apress, 2021.
Language
English
Description
1 online resource
Item Number
10.1007/978-1-4842-7383-8 doi
Call Number
QA76.9.D3
Dewey Decimal Classification
005.7
Summary
Take a journey toward discovering, learning, and using Apache Spark 3.0. In this book, you will gain expertise on the powerful and efficient distributed data processing engine inside of Apache Spark; its user-friendly, comprehensive, and flexible programming model for processing data in batch and streaming; and the scalable machine learning algorithms and practical utilities to build machine learning applications. Beginning Apache Spark 3 begins by explaining different ways of interacting with Apache Spark, such as Spark Concepts and Architecture, and Spark Unified Stack. Next, it offers an overview of Spark SQL before moving on to its advanced features. It covers tips and techniques for dealing with performance issues, followed by an overview of the structured streaming processing engine. It concludes with a demonstration of how to develop machine learning applications using Spark MLlib and how to manage the machine learning development lifecycle. This book is packed with practical examples and code snippets to help you master concepts and features immediately after they are covered in each section. After reading this book, you will have the knowledge required to build your own big data pipelines, applications, and machine learning applications. What You Will Learn Master the Spark unified data analytics engine and its various components Work in tandem to provide a scalable, fault tolerant and performant data processing engine Leverage the user-friendly and flexible programming model to perform simple to complex data analytics using dataframe and Spark SQL Develop machine learning applications using Spark MLlib Manage the machine learning development lifecycle using MLflow Who This Book Is For Data scientists, data engineers and software developers.
Note
Includes index.
Access Note
Access limited to authorized users.
Available in Other Form
Print version: 9781484273821
Linked Resources
Online Access
Record Appears in
Online Resources > Ebooks
All Resources
Intro
Table of Contents
About the Author
About the Technical Reviewers
Acknowledgments
Introduction
Chapter 1: Introduction to Apache Spark
Overview
History
Spark Core Concepts and Architecture
Spark Cluster and Resource Management System
Spark Applications
Spark Drivers and Executors
Spark Unified Stack
Spark Core
Spark SQL
Spark Structured Streaming
Spark MLlib
Spark GraphX
SparkR
Apache Spark 3.0
Adaptive Query Execution Framework
Dynamic Partition Pruning (DPP)
Accelerator-aware Scheduler
Apache Spark Applications

Spark Example Applications
Apache Spark Ecosystem
Delta Lake
Koalas
MLflow
Summary
Chapter 2: Working with Apache Spark
Downloading and Installation
Downloading Spark
Installing Spark
Spark Scala Shell
Spark Python Shell
Having Fun with the Spark Scala Shell
Useful Spark Scala Shell Command and Tips
Basic Interactions with Scala and Spark
Basic Interactions with Scala
Spark UI and Basic Interactions with Spark
Spark UI
Basic Interactions with Spark
Introduction to Collaborative Notebooks
Create a Cluster
Create a Folder

Create a Notebook
Setting up Spark Source Code
Summary
Chapter 3: Spark SQL: Foundation
Understanding RDD
Introduction to the DataFrame API
Creating a DataFrame
Creating a DataFrame from RDD
Creating a DataFrame from a Range of Numbers
Creating a DataFrame from Data Sources
Creating a DataFrame by Reading Text Files
Creating a DataFrame by Reading CSV Files
Creating a DataFrame by Reading JSON Files
Creating a DataFrame by Reading Parquet Files
Creating a DataFrame by Reading ORC Files
Creating a DataFrame from JDBC

Working with Structured Operations
Working with Columns
Working with Structured Transformations
select(columns)
selectExpr(expressions)
filler(condition), where(condition)
distinct, dropDuplicates
sort(columns), orderBy(columns)
limit(n)
union(otherDataFrame)
withColumn(colName, column)
withColumnRenamed(existingColName, newColName)
drop(columnName1, columnName2)
sample(fraction), sample(fraction, seed), sample(fraction, seed, withReplacement)
randomSplit(weights)
Working with Missing or Bad Data
Working with Structured Actions

Describe(columnNames)
Introduction to Datasets
Creating Datasets
Working with Datasets
Using SQL in Spark SQL
Running SQL in Spark
Writing Data Out to Storage Systems
The Trio: DataFrame, Dataset, and SQL
DataFrame Persistence
Summary
Chapter 4: Spark SQL: Advanced
Aggregations
Aggregation Functions
Common Aggregation Functions
count(col)
countDistinct(col)
min(col), max(col)
sum(col)
sumDistinct(col)
avg(col)
skewness(col), kurtosis(col)
variance(col), stddev(col)
Aggregation with Grouping
Multiple Aggregations per Group

Browse Subjects

Show more subjects...

Statistics

from
to
Export