> Data science course by AMA & ICAML - Practical Part

Using pandas

What we did so far are very common and basic tasks. Using Python we should recall that there are usually packages that simplify those basic tasks. A famous package for working with tabular data is pandas. Reading a csv file in pandas can be done within a single function call.

import pandas as pd  # pandas renamed to pd

table = pd.read_csv('titanic2.csv', sep=';') # we have to specify the seperator since pandas assumes a comma

Pandas stores the data in a so called data frame.

print(type(table))

<class 'pandas.core.frame.DataFrame'>

This data type has some nice functionalities e.g. a nice textual representation of the data itself. In a notebook, we can simply type the name of the table to access this representation.

table

The function head() will print only the first elements, which is often enough to get an impression of the table.

table.head()

Another very useful feature is the function describe() that will compute several descriptive values like the mean, the quantiles and the number of values.

table.describe(include='all')

Using the function hist(), we can directly compute and show histograms of multiple attributes.

plot = table.hist(column= ['Age', 'Fare', 'Pclass', 'Survived' ], figsize=(15, 10))

Single columns of the table are accessed by using the column name in square brackets or via the dot operator. Each column is stored as a data series which is again some sort of list with special functions e.g. a nice textual representation.

fares = table['Fare']         # Extracting the column 'Fare' (alternatively we could use table.Fare)
print(type(fares))            # This is the data type of the column representation
print(fares.dtype)            # This is the data type of each element in the column
fares.head()

<class 'pandas.core.series.Series'>
float64





0     7.2500
1    71.2833
2     7.9250
3    53.1000
4     8.0500
Name: Fare, dtype: float64

table['Pclass'].dtype

dtype('int64')

Data cleaning and preprocessing

In order to prepare our data for the later processing, we will first remove the attributes, we are not interested in.

cleaned_table = table.drop(columns=['PassengerId', 'Name', 'Ticket', 'Cabin']) # step 1: drop some columns
cleaned_table.describe(include='all')

In a second step, we remove all entries, where not all attributes are present.

cleaned_table = cleaned_table.dropna()
cleaned_table.describe(include='all')

We can see, that the number of entries per attribute is now the same for all attributes.

In the last step we want to convert the categorical entries for ‘Sex’ to numerical values. Similar to ‘survived’ ‘male’ should be represented by 1 and ‘female’ by 0. This is done in the next cell.

sex_types = ['female', 'male']
cleaned_table.Sex = cleaned_table.Sex.astype("category").cat.codes
cleaned_table.describe(include='all')
#cleaned_table.head()

Finally we will save the cleaned table to a new csv file.

cleaned_table.to_csv('titanic_cleaned.csv', sep=';', index=False)