Pyspark Foundation for Data Engineering | Beginners

Data Engineering, PySpark, Coding exercise

Created by : Rated :4 738 views

Last Updated: 2022-12-09 05:12:46 English Course Type: Self Learning

What you will learn :
SparkSession and imports
Spark DataFrame and its characteristics
Syntax and example
Print results
Understanding the data
Number of records
Columns in dataFrame
Describe a DataFrame
Schema of a DataFrame
Create a new column
Arithmetic operations on Data
Change column data type
Create a column with integer as constant
Rounding of digits
Sorting operation
Drop columns
Rename columns
Create a column with string as constant
Conditional Statements
Changing case of a column
Filter operations
Grouping and aggregations

This course will prepare you for a real world Data Engineer role (basics)!

Learn to code PySpark like a real world developer. Here our major focus will be on Practical applications of PySpark and bridge the gap between academic knowledge and practical skill.

In this course we will get to know and apply few of the most essential and basic functions in PySpark, that are used frequently in scripting for any project based on PySpark.

About PySpark:

Learn the latest Big Data Technology - Spark! And learn to use it with one of the most popular programming languages, Python!

One of the most valuable technology skills is the ability to analyze huge data sets, and this course is specifically designed to bring you up to speed on one of the best technologies for this task, Apache Spark! The top technology companies like Google, Facebook, Netflix, Airbnb, Amazon, NASA, and more are all using Spark to solve their big data problems!

Spark can perform up to 100x faster than Hadoop MapReduce, which has caused an explosion in demand for this skill! Because the Spark 2.0 DataFrame framework is so new, you now have the ability to quickly become one of the most knowledgeable people in the job market!

What you will learn :

SparkSession and imports
Spark DataFrame and its characteristics
Syntax and example
Print results
Understanding the data
Number of records
Columns in dataFrame
Describe a DataFrame
Schema of a DataFrame
Create a new column
Arithmetic operations on Data
Change column data type
Create a column with integer as constant
Apply what we know
Rounding of digits
Sorting operation
Drop columns
Rename columns
Create a column with string as constant
Conditional Statements
Changing case of a column
Filter operations
Grouping and aggregations

Prerequisites :

Some basic programming skills (Not Mandatory)
Will to implement theoretical knowledge in practical.

Who this course is for:

Beginners who want to learn Big Data or experienced people who want to transition to a Big Data role
Big data beginners who want to learn how to code in the real world
Aspiring candidates for data engineering role

Some basic programming skills (Not Mandatory).
Will to implement theoretical knowledge in practical.

Beginners who want to learn Big Data or experienced people who want to transition to a Big Data role
Big data beginners who want to learn how to code in the real world
Aspiring candidates for data engineering role

Section 1 : Unpublished Section 24 Lectures 00:00:00
Lecture 1 :
01 - Introduction to the course Preview
Lecture 2 :
02 - SparkSession and Imports
Lecture 3 :
03 - Spark DataFrame and its characteristics
Lecture 4 :
Syntax and example
Lecture 5 :
Print operation
Lecture 6 :
Understanding the data
Lecture 7 :
Number of records in DataFrame
Lecture 8 :
Columns present in DataFrame
Lecture 9 :
Summary of a DataFrame
Lecture 10 :
Get schema of a DataFrame
Lecture 11 :
Create a new column in a dataframe
Lecture 12 :
Arithmetic operations on columns
Lecture 13 :
Change column Data Types by casting
Lecture 14 :
Create a column with integer constant
Lecture 15 :
Application of the learnings
Lecture 16 :
Rounding operations using bround
Lecture 17 :
Sorting operation
Lecture 18 :
Drop columns of a dataframe
Lecture 19 :
Rename a column
Lecture 20 :
Create a column with String constant
Lecture 21 :
Conditional Statements
Lecture 22 :
Changing Case of a column
Lecture 23 :
Filter operations
Lecture 24 :
Grouping and Aggegrations

How do i access the course after purchase?
It's simple. When you sign up, you'll immediately have unlimited viewing of thousands of expert courses, paths to guide your learning, tools to measure your skills and hands-on resources like exercise files. There’s no limit on what you can learn and you can cancel at any time.
Are these video based online self-learning courses?
Yes. All of the courses comes with online video based lectures created by certified instructors. Instructors have crafted these courses with a blend of high quality interactive videos, lectures, quizzes & real world projects to give you an indepth knowledge about the topic.
Can i play & pause the course as per my convenience?
Yes absolutely & thats one of the advantage of self-paced courses. You can anytime pause or resume the course & come back & forth from one lecture to another lecture, play the videos mulitple times & so on.
How do i contact the instructor for any doubts or questions?
Most of these courses have general questions & answers already covered within the course lectures. However, if you need any further help from the instructor, you can use the inbuilt Chat with Instructor option to send a message to an instructor & they will reply you within 24 hours. You can ask as many questions as you want.
Do i need a pc to access the course or can i do it on mobile & tablet as well?
Brilliant question? Isn't it? You can access the courses on any device like PC, Mobile, Tablet & even on a smart tv. For mobile & a tablet you can download the Learnfly android or an iOS app. If mobile app is not available in your country, you can access the course directly by visting our website, its fully mobile friendly.
Do i get any certificate for the courses?
Yes. Once you complete any course on our platform along with provided assessments by the instructor, you will be eligble to get certificate of course completion.
For how long can i access my course on the platform?
You require an active subscription to access courses on our platform. If your subscription is active, you can access any course on our platform with no restrictions.
Is there any free trial?
Currently, we do not offer any free trial.
Can i cancel anytime?
Yes, you can cancel your subscription at any time. Your subscription will auto-renew until you cancel, but why would you want to?

Akash Pawar,

1382 Course Views

2 Courses

Akash Pawar has completed Bachelor of Technology in Electronics Engineering from National Institute of Technology (NIT, Rourkela) and has years of experience as a professional data enginner and trainer for ETL operations using PySpark. Over the course of his career he has developed a skill set in analyzing data and he hopes to use his experience in teaching and data engineering to help other people learn the power of programming, the ability to analyze data, and the skills needed to present the data in clear and beautiful visualizations. Currently he works as the Data Engineer in Fractal Analytics.