Learning-PySpark - GitLite

KO | EN

gitlite — search

#javascript #python #typescript #llm #android #machine-learning #rust #nodejs #macos #java #windows #pytorch #security #ios

Learning-PySpark

(★ 345)

Code repository for Learning PySpark by Packt

File Explorer

Download Latest Version (.zip)

- HelloWorldFromPySpark.ipynb
- - db.lck
  - dbex.lck
  - README_DO_NOT_TOUCH_FILES.txt
  - service.properties
- LearningPySpark_Chapter02.ipynb
- - airport-codes-na.txt
  - departuredelays.csv
  - README.md
- LearningPySpark_Chapter03.ipynb
- LearningPySpark_Chapter04.ipynb
- LearningPySpark_Chapter05.ipynb
- - db.lck
  - dbex.lck
  - README_DO_NOT_TOUCH_FILES.txt
  - service.properties
- LearningPySpark_Chapter06.ipynb
- LearningPySpark_Chapter07.ipynb
- LearningPySpark_Chapter08.ipynb
- LearningPySpark_Chapter09.ipynb
- stateful_streaming_word_count.py
- streaming_word_count.py
- structured_streaming_word_count.py
- - setup.py
- calculatingGeoDistance.py
- launch_spark_submit.sh
.gitattributes
.gitignore
LICENSE
README.md

# Installation Guide

1. Get the code

git clone https://github.com/PacktPublishing/Learning-PySpark

Downloads the entire project code from GitHub to your computer.

cd Learning-PySpark

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

⚠️ This is a large repository, so this method may point to an internal sub-package rather than the actual core product. Check the full README as well.

pip install .

Installs the package published on PyPI directly — no need to clone the source.

jupyter notebook

Launches Jupyter in your browser so you can open and run the notebook (.ipynb) files.

✅ If it runs without errors and prints output in the terminal, it worked.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Learning PySpark
This is the code repository for [Learning PySpark](https://www.packtpub.com/big-data-and-business-intelligence/learning-pyspark?utm_source=github&utm_medium=repository&utm_content=9781786463708), published by Packt. It contains all the supporting project files necessary to work through the book from start to finish.

## About the book
Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark.

You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command.

By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.

## Instructions and Navigation
All of the code is organized into folders. Each folder starts with a number followed by the application name. For example, Chapter 03.

The code will look like the following:
          
        data_key = sc.parallelize( 
             [('a', 4),('b', 3),('c', 2),('a', 8),('d', 2),('b', 1), 
             ('d', 3)],4) 
        data_key.reduceByKey(lambda x, y: x + y).collect()

### Software requirements:
For this book you need a personal computer (can be either Windows machine, Mac, or Linux). To run Apache Spark, you will need Java 7+ and an installed and conﬁgured Python 2.6+ or 3.4+ environment; we use the Anaconda distribution of Python in version 3.5, which can be downloaded from https://www.continuum.io/downloads.

The Python modules we randomly use throughout the book come preinstalled with Anaconda. We also use GraphFrames and TensorFrames that can be loaded dynamically while starting a Spark instance: to load these you just need an Internet connection. It is fine if some of those modules are not currently installed on your machine – we will guide you through the installation process.

### Note:
Chapter 11 and Bonus Chapter 02 does not contain code files.

## Related Products:
* [Python Machine Learning](https://www.packtpub.com/big-data-and-business-intelligence/python-machine-learning?utm_source=github&utm_medium=repository&utm_content=9781783555130)

* [Spark for Python Developers](https://www.packtpub.com/big-data-and-business-intelligence/spark-python-developers?utm_source=github&utm_medium=repository&utm_content=9781784399696)

* [Python Data Analysis Cookbook](https://www.packtpub.com/big-data-and-business-intelligence/python-data-analysis-cookbook?utm_source=github&utm_medium=repository&utm_content=9781785282287)

### Suggestions and Feedback
[Click here](https://docs.google.com/forms/d/e/1FAIpQLSe5qwunkGf6PUvzPirPDtuy1Du5Rlzew23UBp2S-P3wB-GcwQ/viewform) if you have any feedback or suggestions.

### Download a free PDF

<i>If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost.<br>Simply click on the link to claim your free PDF.</i>
<p align="center"> <a href="https://packt.link/free-ebook/9781786463708">https://packt.link/free-ebook/9781786463708 </a> </p>