PanDaS¶

pandas is a powerful library in Python that is intended for data cleaning and analysis.
It contain data structures and data manipulation tools that is intended to make data analysis fast and easy.
pandas is often used along with other numerical computing tools like numpy and data visualization libraries like matplotlib. Later, we will also cover matplotlib when we discuss data visualization in Python.

pandas adopts the same principle of array-based computing just like numpy. For example, pandas also make use of array-based functions and preference of manipulating array data without for loops. The main difference between the two is that:

  • pandas is designed for tabular or heterogeneous data while numpy is intended for homogeneous numerical data.

Given below is an example import statement for pandas where we will refer to it using its shortened name pd.

In [3]:
import pandas as pd

We will mainly talk about the two data structures of pandas: Series and DataFrames

Series¶

A Series is a one-dimensional array-like object containing a sequence of values (similar to numpy) and an associated array of data labels called its index.

A simple way to initialize a Series is to pass a list of values:

In [2]:
series1 = pd.Series([9,3,1,7,8,5])
series1
Out[2]:
0    9
1    3
2    1
3    7
4    8
5    5
dtype: int64

Notice that it is similar to the use of np.array() in NumPy where a list is passed to the array().

We can get the array representation and index object of the Series via its values and index attributes, respectively:

In [3]:
series1.values
Out[3]:
array([9, 3, 1, 7, 8, 5])
In [4]:
series1.index
Out[4]:
RangeIndex(start=0, stop=6, step=1)

Notice that the values of a Series is represented as an array. This means we can perform computations in Series similar to computations we do in numpy arrays.

However, in Series, we can change the index. For example, we can use letters as index instead of numbers. The code below shows how this can be done. We pass a list containing the indices and assign it to the index parameter when creating a Series.

In [5]:
series2 = pd.Series([9,3,1,7,8,5], 
    index=['a','b','c','d','e','f'])
series2
Out[5]:
a    9
b    3
c    1
d    7
e    8
f    5
dtype: int64
In [6]:
series2.values
Out[6]:
array([9, 3, 1, 7, 8, 5])
In [7]:
series2.index
Out[7]:
Index(['a', 'b', 'c', 'd', 'e', 'f'], dtype='object')

Series operations¶

Since values in Series are just array objects, operations on array can also be applied to series objects.

The codes below illustrate some array operations we learn in our study of numpy and this time, applied to pandas Series.|

In [8]:
series2
Out[8]:
a    9
b    3
c    1
d    7
e    8
f    5
dtype: int64
In [9]:
# provide Series where values in data_2 are multiplied by 3
series2 * 3
Out[9]:
a    27
b     9
c     3
d    21
e    24
f    15
dtype: int64

We can also pass a Series as argument to numpy functions.

In [10]:
import numpy as np
np.square(series2)               # series2 is a pandas Series that you can plug in to an numpy function
Out[10]:
a    81
b     9
c     1
d    49
e    64
f    25
dtype: int64

Series Indexing and Splicing¶

In [11]:
series2
Out[11]:
a    9
b    3
c    1
d    7
e    8
f    5
dtype: int64
In [12]:
# access value having index 'e'
series2["e"]
Out[12]:
8
In [13]:
# access part of data_2 with values at index 'a','d','f'
series2[["a","d","f"]]
Out[13]:
a    9
d    7
f    5
dtype: int64
In [14]:
# access part of data_2 where all values are less than 4
series2[series2 < 4]
Out[14]:
b    3
c    1
dtype: int64

Splicing in Series is also similar to splicing in numpy arrays. The slight change is the way we use the index provided after the colon. In numpy arrays, value corresponding to the upper bound provided is not part of the values accessed. In pandas Series, this value is included.

In [15]:
# assign value 0 to index 'a'
series2["c":"f"]
Out[15]:
c    1
d    7
e    8
f    5
dtype: int64

Series Assignments¶

In [16]:
series2
Out[16]:
a    9
b    3
c    1
d    7
e    8
f    5
dtype: int64
In [17]:
# assign value 0 to index 'a'
series2["a"] = 0
series2
Out[17]:
a    0
b    3
c    1
d    7
e    8
f    5
dtype: int64
In [18]:
series2["d":"f"] = -3
series2
Out[18]:
a    0
b    3
c    1
d   -3
e   -3
f   -3
dtype: int64
In [19]:
# assign value 0 to index 'a'
series2[["c","b"]] = [-1,-4]
series2
Out[19]:
a    0
b   -4
c   -1
d   -3
e   -3
f   -3
dtype: int64

Using dictionaries to create Series¶

We can also think of a Series as an ordered dictionary. Thus, a Series can also created by passing a dictionary. Given a dictionary, the associated index will then be the keys in the passed dictionary.

In [4]:
age_dict={"Ellen":27, "Charlie":18, "Ana":20, "Ben":24,  "Dina":29}
age = pd.Series(age_dict)
age
Out[4]:
Ellen      27
Charlie    18
Ana        20
Ben        24
Dina       29
dtype: int64
In [5]:
age["Ellen"]
Out[5]:
27
In [6]:
age[age<25]
Out[6]:
Charlie    18
Ana        20
Ben        24
dtype: int64
In [7]:
age["Charlie":"Ben"]
Out[7]:
Charlie    18
Ana        20
Ben        24
dtype: int64

Exercise¶

Write codes to determine the ff:

  • age of "Ellen"
  • ages less than 25
  • ages of index from "Charlie" to "Ben"
In [21]:
# access the value having index "Ellen"
age["Ellen"]
Out[21]:
27
In [22]:
# access those having age less than 25
age[age < 25]
Out[22]:
Charlie    18
Ana        20
Ben        24
dtype: int64
In [23]:
# access those values from index "Charlie" to "Ben" 
age["Charlie":"Ben"]
Out[23]:
Charlie    18
Ana        20
Ben        24
dtype: int64

The name attribute¶

pandas Series and its index has an attribute name. We can use this to provide descriptive names that can be integrated at other key areas of pandas functionality.

In [8]:
age
Out[8]:
Ellen      27
Charlie    18
Ana        20
Ben        24
Dina       29
dtype: int64
In [9]:
age.name = "Age"
age.index.name = "First Names"
age
Out[9]:
First Names
Ellen      27
Charlie    18
Ana        20
Ben        24
Dina       29
Name: Age, dtype: int64

When we query age, the name of this Series and its associated index are both reflected in the provided info for the Series.

In [11]:
age.info()

<class 'pandas.core.series.Series'>
Index: 5 entries, Ellen to Dina
Series name: Age
Non-Null Count  Dtype
--------------  -----
5 non-null      int64
dtypes: int64(1)
memory usage: 252.0+ bytes
In [ ]:

DataFrame¶

DataFrame represents a rectangular table of data with flexible rows and column indices. It can be thought of as a dictionary of Series all sharing the same index. This data structure is very familiar especially for those that have worked on spreadsheets.

Here is a simple way to create a DataFrame from a dictionary of equal-length lists.

In [26]:
some_dict = {'a':[0,1,2], 'b':[3,4,5]}
df = pd.DataFrame(some_dict)
df
Out[26]:
a b
0 0 3
1 1 4
2 2 5

Notice that keys in the dictionary become column indices while the row indices follow the numerical indexing in lists.

We can also create a Dataframe from multiple Series.

Recall the age Series we had earlier.

In [12]:
age
Out[12]:
First Names
Ellen      27
Charlie    18
Ana        20
Ben        24
Dina       29
Name: Age, dtype: int64

Let's create another series that use the same indices.

In [13]:
province_dict = {"Ellen":"Tarlac", "Charlie":"Cebu", "Ana":"Pampanga", "Ben":"Davao",  "Dina":"Cebu"}
province = pd.Series(province_dict)
province
Out[13]:
Ellen        Tarlac
Charlie        Cebu
Ana        Pampanga
Ben           Davao
Dina           Cebu
dtype: object
In [14]:
people = pd.DataFrame({'age':age, 'province':province})
people
Out[14]:
age province
Ellen 27 Tarlac
Charlie 18 Cebu
Ana 20 Pampanga
Ben 24 Davao
Dina 29 Cebu

Selecting a column/row of a DataFrame¶

A column in a DataFrame can be retrieved as a Series either by dictionary-like notation or by attribute.

In [15]:
people
Out[15]:
age province
Ellen 27 Tarlac
Charlie 18 Cebu
Ana 20 Pampanga
Ben 24 Davao
Dina 29 Cebu
In [16]:
people['age']
Out[16]:
Ellen      27
Charlie    18
Ana        20
Ben        24
Dina       29
Name: age, dtype: int64
In [32]:
people.province
Out[32]:
Ellen        Tarlac
Charlie        Cebu
Ana        Pampanga
Ben           Davao
Dina           Cebu
Name: province, dtype: object

Note that while the dictionary-like notation always works for any column name, the attribute-based can only work when the column name is a valid Python variable name. Also, attribute-like access and tab completion of column names in Jupyter is provided as a convenience.

In [18]:
people.province
Out[18]:
Ellen        Tarlac
Charlie        Cebu
Ana        Pampanga
Ben           Davao
Dina           Cebu
Name: province, dtype: object

Rows can be retrieved by position or name with the special loc attribute.

In [33]:
people.loc['Ellen']
Out[33]:
age             27
province    Tarlac
Name: Ellen, dtype: object

Adding a New Column¶

We can create a new column through assignment.

In [34]:
# create a new column 'debt' and set value to 0
people['debt'] = 0
people
Out[34]:
age province debt
Ellen 27 Tarlac 0
Charlie 18 Cebu 0
Ana 20 Pampanga 0
Ben 24 Davao 0
Dina 29 Cebu 0

Instead of a scalar value, we can also specify an array of values. Note that in such type of assignment, the values in the list and the number of index must be equal.

In [21]:
people['is_married'] = [True, False, False, True, True]
people

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[21], line 1
----> 1 people['is_married'] = [True, False, False, True, True]
      2 people

File ~/anaconda3/lib/python3.11/site-packages/pandas/core/frame.py:3980, in DataFrame.__setitem__(self, key, value)
   3977     self._setitem_array([key], value)
   3978 else:
   3979     # set column
-> 3980     self._set_item(key, value)

File ~/anaconda3/lib/python3.11/site-packages/pandas/core/frame.py:4174, in DataFrame._set_item(self, key, value)
   4164 def _set_item(self, key, value) -> None:
   4165     """
   4166     Add series to DataFrame in specified column.
   4167 
   (...)
   4172     ensure homogeneity.
   4173     """
-> 4174     value = self._sanitize_column(value)
   4176     if (
   4177         key in self.columns
   4178         and value.ndim == 1
   4179         and not is_extension_array_dtype(value)
   4180     ):
   4181         # broadcast across multiple columns if necessary
   4182         if not self.columns.is_unique or isinstance(self.columns, MultiIndex):

File ~/anaconda3/lib/python3.11/site-packages/pandas/core/frame.py:4915, in DataFrame._sanitize_column(self, value)
   4912     return _reindex_for_setitem(Series(value), self.index)
   4914 if is_list_like(value):
-> 4915     com.require_length_match(value, self.index)
   4916 return sanitize_array(value, self.index, copy=True, allow_2d=True)

File ~/anaconda3/lib/python3.11/site-packages/pandas/core/common.py:571, in require_length_match(data, index)
    567 """
    568 Check the length of data matches the length of the index.
    569 """
    570 if len(data) != len(index):
--> 571     raise ValueError(
    572         "Length of values "
    573         f"({len(data)}) "
    574         "does not match length of index "
    575         f"({len(index)})"
    576     )

ValueError: Length of values (5) does not match length of index (6)

We can also add a new column by providing a Series. In the code below, we created a Series and assigned it to a new column named occupation in the DataFrame people. Notice that the indices in created Series does not exactly match the row indices in the DataFrame. In this case, pandas will only consider the indices that match on both structure. Those without match in the DataFrame will have corresponding values of NaN. NaN is a special value interpreted in pandas to mean missing value or NA. Also notice that since "Regine" does not exist in the row index of the people DataFrame, it is simply ignored during the assignment.

In [22]:
occupation = pd.Series(["teacher","dentist","scientist"], index=["Ellen", "Ben", "Regine"])
people['occupation']=occupation
people
Out[22]:
age province occupation
Ellen 27 Tarlac teacher
Charlie 18 Cebu NaN
Ana 20 Pampanga NaN
Ben 24 Davao dentist
Dina 29 Cebu NaN
Regine 22 NaN scientist

Adding a New Row¶

How do we add another row? We can create a dictionary and use the attribute loc as follows:

In [20]:
values = {'age':22, "is_married":False, "occupation":"cashier"}
people.loc['Regine']=values
people
Out[20]:
age province
Ellen 27 Tarlac
Charlie 18 Cebu
Ana 20 Pampanga
Ben 24 Davao
Dina 29 Cebu
Regine 22 NaN

Notice that the value for Regine's province is NaN. As in adding columns, in adding arrows, only those values with a corresponding column index is considered.

Deleting a column/row¶

To delete a column, we can use the drop() method as follows:

In [30]:
people
Out[30]:
age province occupation
Charlie 18 Cebu NaN
Ana 20 Pampanga NaN
Ben 24 Davao dentist
Dina 29 Cebu NaN
In [27]:
people.drop(columns="province")
Out[27]:
age occupation
Ellen 27 teacher
Charlie 18 NaN
Ana 20 NaN
Ben 24 dentist
Dina 29 NaN
Regine 22 scientist

Note that by default, when using drop(), the method will only gives us a version of the dataframe with the dropped index, but the dropped index is still present in the original dataframe. If we want to remove the index from the original dataframe, we need to set in_place=True.

Similar to deleting a column, to delete a row, we use drop().

In the code below, we provided a list to remove multiple rows. We also use the parameter index instead of columns.

In [29]:
people.drop(index=["Regine","Ellen"], inplace=True)
people
Out[29]:
age province occupation
Charlie 18 Cebu NaN
Ana 20 Pampanga NaN
Ben 24 Davao dentist
Dina 29 Cebu NaN

Sorting in a DataFrame¶

We can sort the data values given a column index as follows:

In [41]:
people.sort_values(by='province')
Out[41]:
age province debt is_married occupation
Charlie 18 Cebu 0.0 False NaN
Dina 29 Cebu 0.0 True NaN
Ben 24 Davao 0.0 True dentist
Ana 20 Pampanga 0.0 False NaN

We can also sort the value by index as follows:

In [31]:
people.sort_index(axis=0)
Out[31]:
age province occupation
Ana 20 Pampanga NaN
Ben 24 Davao dentist
Charlie 18 Cebu NaN
Dina 29 Cebu NaN
In [34]:
people.sort_index(axis=1, ascending=True)
Out[34]:
age occupation province
Charlie 18 NaN Cebu
Ana 20 NaN Pampanga
Ben 24 dentist Davao
Dina 29 NaN Cebu

There are actually a lot of methods we can apply on DataFrames. We only cover a small portion here. We will further explore what we can do with these pandas structures later when we discuss data processing techniques in the next module.

Quick Descriptive Statistics¶

In [44]:
people
Out[44]:
age province debt is_married occupation
Charlie 18 Cebu 0.0 False NaN
Ana 20 Pampanga 0.0 False NaN
Ben 24 Davao 0.0 True dentist
Dina 29 Cebu 0.0 True NaN

Let's end this module by providing another good feature of pandas, i.e. the method describe(). This method provides quick statistical description

In [45]:
print(people.describe())

             age  debt
count   4.000000   4.0
mean   22.750000   0.0
std     4.856267   0.0
min    18.000000   0.0
25%    19.500000   0.0
50%    22.000000   0.0
75%    25.250000   0.0
max    29.000000   0.0

Summary¶

  • We were introduced to pandas library
  • We learned: about two data structures in pandas: Series and DataFrames how to select, add, delete and perform operations on Series and DataFrames

Main Reference¶

  1. Wes McKinney - Python for Data Analysis. Data Wrangling with Pandas, NumPy, and IPython-O’Reilly (2017)