Pandas

Introduction to pandas:

Pandas is a powerful and popular Python library for data manipulation and analysis. Whether you're a data scientist, analyst, or just someone interested in exploring data, Pandas provides an easy-to-use and efficient framework to work with structured data. In this tutorial, we will cover the basics of Pandas and guide you through the process of data analysis step by step.

Prerequisites:

Before you get started, you should have a basic understanding of Python programming. If you are new to Python, you can find many beginner-friendly tutorials available online. Additionally, make sure you have Pandas installed on your system. You can install it using 'pip'. 

pip install pandas

 

Importing Pandas

Once you have Pandas installed, you can import it into your Python script or Jupyter Notebook using the import statement.

In this line of code, we import Pandas and give it an alias pd, which is a common convention used by the Pandas community. Using the alias pd makes it easier to call Pandas functions later in the code. 

import pandas as pd

 

Before diving deep into Pandas, let's get to know about DataFrames and Series in Pandas.

DataFrames:

A DataFrame is a two-dimensional, tabular data structure, similar to a spreadsheet or database table. It consists of rows and columns, where each row represents a data entry, and each column represents a specific attribute or feature of the data. DataFrames can hold data of different data types and efficiently handle missing values, making them ideal for working with real-world datasets.

 

Key features of Pandas DataFrame:

• Rows and Columns: Each row in a DataFrame represents a separate data entry, while each column represents a specific attribute or feature of the data.

• Labeling: Both rows and columns in a DataFrame are labeled, which allows for easy and intuitive data selection and manipulation.

• Heterogeneous Data Types: Unlike NumPy arrays, Pandas DataFrames can hold data with different data types in different columns. This makes it convenient to work with real-world datasets, which often have a mix of numerical, categorical, and text data.

• Missing Data Handling: DataFrames can handle missing or NaN (Not-a-Number) values gracefully, making it easier to work with incomplete datasets.

• Data Alignment: When performing operations on DataFrames, Pandas automatically aligns data based on the row and column labels, which simplifies many data manipulations.

• Indexing and Slicing: Pandas provides various methods for selecting specific rows and columns from a DataFrame, allowing for powerful data filtering and slicing.

• Data Transformation: DataFrames support a wide range of data transformation and manipulation functions, such as filtering, sorting, merging, grouping, and aggregating.

Series:

A Series, on the other hand, is a one-dimensional labeled array. It can be thought of as a single column of a DataFrame and serves as the building block for constructing DataFrames. Series can store homogeneous data and provide labeled indexing, enabling easy data manipulation and alignment.

 

Aside from Pandas DataFrame, there are a few other important concepts in Pandas that you may encounter.

 

• Index : The Index is an immutable array-like structure that labels the rows and columns of a DataFrame or Series. It helps with data alignment and allows for efficient data retrieval. 

• GroupBy: The GroupBy operation in Pandas allows you to group data based on specific criteria, such as values in one or more columns. This is useful for data aggregation and applying functions to different groups of data. 

• MultiIndex: A MultiIndex is an index with multiple levels, allowing you to work with higher-dimensional data in Pandas. 

• Panel: A Panel is a three-dimensional data structure in Pandas, but it is less commonly used compared to DataFrames.

 

Loading Data into a Pandas DataFrame:

 Pandas provides several methods to read data from different file formats, such as CSV, Excel, SQL databases, etc. For this example, we'll load data from a CSV file.

# Assuming you have a CSV file named 'data.csv' in your current directory 

# You can load the data into a DataFrame like this:

df = pd.read_csv('data.csv')

In this code snippet, we use the pd.read_csv() function to read the CSV file and store the data in a Pandas DataFrame called df. The read_csv() function is a powerful method that can handle various options and configurations, such as specifying a custom delimiter, handling missing values, etc.

Now, you have successfully loaded your data into a Pandas DataFrame, which is a tabular data structure similar to a spreadsheet or a database table. The DataFrame allows you to perform various operations on the data, such as filtering, grouping, aggregating, and visualization.

 To check if the data has been loaded correctly, you can use the head() method, which displays the first few rows of the DataFrame.

 print(df.head())

The head() method prints the top 5 rows of the DataFrame by default. You can pass a number inside the parentheses to see a different number of rows (e.g., df.head(10) will show the top 10 rows).

Importing Pandas and Loading Data:

To begin, import Pandas into your Python script or Jupyter Notebook. Then, we'll load data into a Pandas DataFrame, which is the core data structure in Pandas.

 import pandas as pd

# Load data from a CSV file
df = pd.read_csv('data.csv') # Display the first few rows of the DataFrame
print(df.head())

          Output:

id season city date team1
0 1 2017 Hyderabad 2017-04-05 Sunrisers Hyderabad
1 2 2017 Pune 2017-04-06 Mumbai Indians
2 3 2017 Rajkot 2017-04-07 Gujarat Lions
3 4 2017 Indore 2017-04-08 Rising Pune Supergiant
4 5 2017 Bangalore 2017-04-08 Royal Challengers Bangalore

Exploring the Data:

 Now that we have our data loaded, let's explore it. We can use various methods and attributes to get a sense of the data.

# Get the basic information about the DataFrame

print(df.info())

Output:

<class 'pandas.core.frame.DataFrame'> RangeIndex: 636 entries, 0 to 635 Data columns (total 18 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 id 636 non-null int64 1 season 636 non-null int64 2 city 629 non-null object 3 date 636 non-null object 4 team1 636 non-null object 5 team2 636 non-null object 6 toss_winner 636 non-null object 7 toss_decision 636 non-null object 8 result 636 non-null object 9 dl_applied 636 non-null int64 10 winner 633 non-null object 11 win_by_runs 636 non-null int64 12 win_by_wickets 636 non-null int64 13 player_of_match 633 non-null object 14 venue 636 non-null object 15 umpire1 635 non-null object 16 umpire2 635 non-null object 17 umpire3 0 non-null float64 dtypes: float64(1), int64(5), object(12) memory usage: 89.6+ KB

In Pandas, the df.info() function is used to display a concise summary of a DataFrame's information. It provides valuable insights into the DataFrame's structure, including the number of non-null values in each column, data types, memory usage, and the total number of rows and columns.

# Get summary statistics for numerical columns

print(df.describe())

          Output: 

id season dl_applied win_by_runs win_by_wickets
count 636.000000 636.000000 636.000000 636.000000 636.000000
mean 318.500000 2012.490566 0.025157 13.682390 3.372642
std 183.741666 2.773026 0.156726 23.908877 3.420338
min 1.000000 2008.000000 0.000000 0.000000 0.000000
25% 159.750000 2010.000000 0.000000 0.000000 0.000000
50% 318.500000 2012.000000 0.000000 0.000000 4.000000
75% 477.250000 2015.000000 0.000000 20.000000 7.000000
max 636.000000 2017.000000 1.000000 146.000000 10.000000 umpire3
count 0.0
mean NaN
std NaN
min NaN
25% NaN
50% NaN
75% NaN

max NaN

 In Pandas, the df.describe() function is used to generate descriptive statistics of a DataFrame. It provides a summary of the central tendency, dispersion, and distribution of the numerical columns in the DataFrame. This function is particularly useful during the exploratory data analysis (EDA) phase, as it helps to quickly understand the distribution of data and identify potential issues.

# Get unique values in a specific column

print(df['column_name'].unique())

 Output:

 ['Hyderabad' 'Pune' 'Rajkot' 'Indore' 'Bangalore'

'Mumbai' 'Kolkata'

'Delhi' 'Chandigarh' 'Kanpur' 'Jaipur' 'Chennai' 'Cape Town'
'Port Elizabeth' 'Durban' 'Centurion' 'East London' 'Johannesburg'
'Kimberley' 'Bloemfontein' 'Ahmedabad' 'Cuttack' 'Nagpur' 'Dharamsala'
'Kochi' 'Visakhapatnam' 'Raipur' 'Ranchi' 'Abu Dhabi' 'Sharjah' nan]

# Check for missing values

print(df.isnull().sum())

          Output: 

 id 0

season 0
city 7
date 0
team1 0
team2 0
toss_winner 0
toss_decision 0
result 0
dl_applied 0
winner 3
win_by_runs 0
win_by_wickets 0
player_of_match 3
venue 0
umpire1 1
umpire2 1
umpire3 636
dtype: int64