Jose R. Zapata | Jose Ricardo Zapata

Procesamiento de Lenguaje Natural (NLP)

Sun, 28 Sep 2025 00:00:00 +0000

Practical Data Science Project with Python

Sun, 07 Jan 2024 00:00:00 +0000

Last update 20 / Feb / 2026

Project Purpose

Doing data science requires knowledge in different areas (Statistics, Mathematics, Programming, visualization, Machine Learning, etc.), but with practice, one can soon begin to understand the various notions and terminologies of the subject. The best way to gain more experience in Data Science, besides reviewing the literature, is to carry out some practical projects that involve the application of data analysis techniques and Machine Learning to datasets in order to adopt a methodology for carrying out the processes of:

Data extraction
Data cleaning
Exploratory analysis
Feature Engineering
Baseline model creation
Model Selection
- Cross Validation
- Model evaluation

The purpose of this project is to provide an example of a methodology or practical guide for developing a data science project with Python, from data extraction, analysis, and preparation to the evaluation and selection of Machine Learning models. Throughout this project, we will use libraries such as pandas, matplotlib, plotly, seaborn, and scikit-learn to perform tasks of data cleaning, exploratory analysis, Feature Engineering, and predictive modeling.

The most important aspect of the project is the methodology for carrying out the processes of data cleaning, exploratory analysis, feature engineering, model selection, and evaluation that can then be applied to any dataset.

The dataset

The sinking of the RMS Titanic in 1912 remains one of the greatest maritime disasters in history, causing a significant loss of life. More than 1500 passengers and crew members perished that fateful night. Understanding the factors that contributed to survival can provide valuable information about safety protocols and social dynamics during crises.

The dataset we will use in this project is the famous Titanic dataset, which can be downloaded from the Kaggle competition platform. This dataset contains information about 891 passengers aboard the Titanic, including details such as name, age, sex, the class in which they traveled, the number of siblings and spouses aboard, the number of parents and children aboard, the ticket price, the port of embarkation, and whether they survived or not. Using classification algorithms, the goal is to create a supervised classification predictive model that allows estimating the survival of each individual aboard the Titanic.

Streamlit Demo.

Practical guide to developing a data science project with Python, from data extraction, analysis, and preparation to the evaluation and selection of Machine Learning models. (pandas and scikit-learn)
1. Data Download

Data download and variable selection.
2. Data Exploration

Data exploration.
3. Data Analysis (EDA)

Exploratory Data Analysis.
4. Feature Engineering

This chapter covers the feature engineering process, which is the process of selecting and transforming variables to create a predictive model.
5. Baseline Model

Baseline models to later compare with more complex models.
6. Model Selection

Machine learning model selection.
7. Model Interpretation

Machine learning model interpretation.
8. Streamlit Demo

Demo of the machine learning model with Streamlit.

Start

Practical Data Science Project with Python

Sun, 07 Jan 2024 00:00:00 +0000

Last update 20 / Feb / 2026

Project Purpose

Data extraction
Data cleaning
Exploratory analysis
Feature Engineering
Baseline model creation
Model Selection
- Cross Validation
- Model evaluation

The dataset

Streamlit Demo.

Practical guide to developing a data science project with Python, from data extraction, analysis, and preparation to the evaluation and selection of Machine Learning models. (pandas and scikit-learn)

1. Data Download

Data download and variable selection.

2. Data Exploration

Data exploration.

3. Data Analysis (EDA)

Exploratory Data Analysis.

4. Feature Engineering

This chapter covers the feature engineering process, which is the process of selecting and transforming variables to create a predictive model.

5. Baseline Model

Baseline models to later compare with more complex models.

6. Model Selection

Machine learning model selection.

7. Model Interpretation

Machine learning model interpretation.

8. Streamlit Demo

Machine learning model demo with Streamlit.

Start

Univariate Data Analysis

Tue, 27 Feb 2024 00:00:00 +0000

By Jose R. Zapata

Last update: 19/Feb/2026

📚 Import libraries

# base libraries for data science
import sys
from pathlib import Path

import pandas as pd

# configuration to show only 2 decimal places
pd.set_option("display.float_format", "{:.2f}".format)

# print library version for reproducibility

print("Python version: ", sys.version)
print("Pandas version: ", pd.__version__)

Python version: 3.11.10 (main, Sep 27 2024, 20:27:21) [GCC 11.4.0]
Pandas version: 2.1.4

💾 Load data

The dataset has correct data types, fixed in:

Initial data exploration

DATA_DIR = Path.cwd().resolve().parents[1] / "data"

titanic_df = pd.read_parquet(
 DATA_DIR / "02_intermediate/titanic_type_fixed.parquet", engine="pyarrow"
)

📊 Data description

General data information

titanic_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 10 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Unnamed: 0 1309 non-null int64
1 pclass 1309 non-null int64
2 survived 1309 non-null bool
3 name 1309 non-null object
4 sex 1309 non-null category
5 age 1046 non-null float64
6 sibsp 1309 non-null int64
7 parch 1309 non-null int64
8 fare 1308 non-null float64
9 embarked 1307 non-null category
dtypes: bool(1), category(2), float64(2), int64(4), object(1)
memory usage: 75.8+ KB

Ordinal data has to be converted again

Information about the pclass column can be checked in the notebook

Initial data exploration

titanic_df["pclass"] = pd.Categorical(
 titanic_df["pclass"], categories=[3, 2, 1], ordered=True
)

# column Unnamed: 0 is not needed
titanic_df = titanic_df.drop(columns=["Unnamed: 0"])

General information about the data set:

titanic_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 9 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 pclass 1309 non-null category
1 survived 1309 non-null bool
2 name 1309 non-null object
3 sex 1309 non-null category
4 age 1046 non-null float64
5 sibsp 1309 non-null int64
6 parch 1309 non-null int64
7 fare 1308 non-null float64
8 embarked 1307 non-null category
dtypes: bool(1), category(3), float64(2), int64(2), object(1)
memory usage: 56.8+ KB

# size of the dataframe
titanic_df.shape

(1309, 9)

# sample of the dataframe
titanic_df.sample(5)

	pclass	survived	name	sex	age	sibsp	parch	fare	embarked
58	1	False	Case, Mr. Howard Brown	male	49.00	0	0	26.00	S
736	3	False	Coxon, Mr. Daniel	male	59.00	0	0	7.25	S
523	2	True	Oxenham, Mr. Percy Thomas	male	22.00	0	0	10.50	S
1073	3	False	O'Connor, Mr. Maurice	male	NaN	0	0	7.75	Q
518	2	False	Nicholls, Mr. Joseph Charles	male	19.00	1	1	36.75	S

Number of missing values

titanic_df.isnull().sum()

pclass 0
survived 0
name 0
sex 0
age 263
sibsp 0
parch 0
fare 1
embarked 2
dtype: int64

Target Variable = Survived

General statistics of the data set

Numerical variables

titanic_df.describe()

	age	sibsp	parch	fare
count	1046.00	1309.00	1309.00	1308.00
mean	29.88	0.50	0.39	33.30
std	14.41	1.04	0.87	51.76
min	0.17	0.00	0.00	0.00
25%	21.00	0.00	0.00	7.90
50%	28.00	0.00	0.00	14.45
75%	39.00	1.00	0.00	31.27
max	80.00	8.00	9.00	512.33

Categorical variables

# categorical columns description
titanic_df.describe(include="category")

	pclass	sex	embarked
count	1309	1309	1307
unique	3	2	3
top	3	male	S
freq	709	843	914

📈 Univariate Analysis

Target Variable

titanic_df["survived"].value_counts().plot(
 kind="bar", color=["skyblue", "orange"], title="Survived passengers"
);

Numerical Variables

# list of the numerical columns
numerical_columns = list(titanic_df.select_dtypes(include=["number"]).columns)
numerical_columns

['age', 'sibsp', 'parch', 'fare']

age

column = "age"
titanic_df[column].describe()

count 1046.00
mean 29.88
std 14.41
min 0.17
25% 21.00
50% 28.00
75% 39.00
max 80.00
Name: age, dtype: float64

# number of unique values
titanic_df[column].nunique()

titanic_df[column].plot(
 kind="hist", bins=30, edgecolor="black", title=f"{column} histogram"
);

titanic_df[column].plot(kind="density", title=f"{column} distribution");

titanic_df[column].plot(kind="box", title=f"{column} boxplot");

sibsp

column = "sibsp"
titanic_df[column].describe()

count 1309.00
mean 0.50
std 1.04
min 0.00
25% 0.00
50% 0.00
75% 1.00
max 8.00
Name: sibsp, dtype: float64

# number of unique values
titanic_df[column].nunique()

# unique values (because are only a few)
titanic_df[column].unique()

array([0, 1, 2, 3, 4, 5, 8])

# histogram of a column with 7 unique values between 0 and 8
titanic_df[column].plot(
 kind="hist", bins=8, edgecolor="black", title=f"{column} histogram"
);

titanic_df[column].plot(kind="box", title=f"{column} boxplot");

parch

column = "parch"
titanic_df[column].describe()

count 1309.00
mean 0.39
std 0.87
min 0.00
25% 0.00
50% 0.00
75% 0.00
max 9.00
Name: parch, dtype: float64

# number of unique values

titanic_df[column].nunique()

# unique values (because are only a few)
titanic_df[column].unique()

array([0, 2, 1, 4, 3, 5, 6, 9])

titanic_df[column].plot(
 kind="hist", bins=9, edgecolor="black", title=f"{column} histogram"
);

titanic_df[column].plot(kind="box", title=f"{column} boxplot");

fare

column = "fare"
titanic_df[column].describe()

count 1308.00
mean 33.30
std 51.76
min 0.00
25% 7.90
50% 14.45
75% 31.27
max 512.33
Name: fare, dtype: float64

titanic_df[column].plot(kind="kde", title=f"{column} distribution");

titanic_df[column].plot(
 kind="hist", bins=50, edgecolor="black", title=f"{column} histogram"
);

titanic_df[column].plot(kind="box", title=f"{column} boxplot");

Categorical Variables

# list of the categorical columns
categorical_columns = list(titanic_df.select_dtypes(include=["category"]).columns)
categorical_columns

['pclass', 'sex', 'embarked']

pclass

column = "pclass"
titanic_df[column].describe()

count 1309
unique 3
top 3
freq 709
Name: pclass, dtype: int64

titanic_df[column].unique()

[1, 2, 3]
Categories (3, int64): [3 < 2 < 1]

titanic_df[column].value_counts()

pclass
3 709
1 323
2 277
Name: count, dtype: int64

titanic_df[column].value_counts().plot(
 kind="bar", color=["skyblue", "orange", "green"], title=f"{column} value counts"
);

Sex

column = "sex"
titanic_df[column].describe()

count 1309
unique 2
top male
freq 843
Name: sex, dtype: object

titanic_df[column].unique()

['female', 'male']
Categories (2, object): ['female', 'male']

titanic_df[column].value_counts()

sex
male 843
female 466
Name: count, dtype: int64

titanic_df[column].value_counts().plot(
 kind="bar", color=["skyblue", "orange"], title=f"{column} value counts"
);

embarked

column = "embarked"
titanic_df[column].describe()

count 1307
unique 3
top S
freq 914
Name: embarked, dtype: object

titanic_df[column].unique()

['S', 'C', NaN, 'Q']
Categories (3, object): ['C', 'Q', 'S']

titanic_df[column].value_counts()

embarked
S 914
C 270
Q 123
Name: count, dtype: int64

titanic_df[column].value_counts().plot(
 kind="bar", color=["skyblue", "orange", "green"], title=f"{column} value counts"
);

String columns

titanic_df["name"].sample(5)

394 Dibden, Mr. William
755 Davies, Mr. Joseph
237 Robbins, Mr. Victor
459 Jacobsohn, Mr. Sidney Samuel
1136 Rasmussen, Mrs. (Lena Jacobsen Solvang)
Name: name, dtype: object

titanic_df["name"].nunique()

📊 Analysis of Results and Conclusions

Dataset Overview

The dataset contains 1,309 passengers with 9 variables (after dropping the unnamed index column).
Missing values: age has 263 missing values (~20%), fare has 1, and embarked has 2. These will need to be addressed in the data preparation phase.

Target Variable: `survived`

The dataset is imbalanced: more passengers did not survive than survived, reflecting the historical reality of the Titanic disaster.

Numerical Variables

Age

Mean age is 29.88 years (median 28), with a standard deviation of 14.41.
Ages range from 0.17 (infants) to 80 years.
The distribution is right-skewed, with a concentration of passengers between 20 and 40 years old.
There are 98 unique age values and 263 missing values (~20% of the data), which is significant and must be handled carefully during imputation.
The boxplot reveals outliers at the upper end (elderly passengers).

SibSp (Siblings/Spouses aboard)

Most passengers traveled without siblings or spouses (median = 0, 75th percentile = 1).
The distribution is heavily right-skewed, with values ranging from 0 to 8.
Only 7 unique values exist, making this variable behave almost like a categorical feature.

Parch (Parents/Children aboard)

Similar to sibsp, most passengers had no parents or children aboard (median = 0).
Values range from 0 to 9, with 8 unique values.
The distribution is also heavily right-skewed, concentrated at 0.

Fare

Mean fare is $33.30 but with a very high standard deviation of $51.76, indicating large variability.
Fares range from $0 to $512.33.
The distribution is extremely right-skewed: the median ($14.45) is much lower than the mean, indicating that a few very expensive tickets pull the average up.
The boxplot shows many outliers on the upper end, corresponding to first-class luxury cabins.
There is 1 missing value.

Categorical Variables

Pclass (Passenger Class)

3rd class is the most frequent (709 passengers, ~54%), followed by 1st class (323) and 2nd class (277).
The majority of passengers were in the lowest class, consistent with historical records.

Sex

Males significantly outnumber females: 843 males (~64%) vs. 466 females (~36%).
This gender imbalance is important for survival analysis, as the “women and children first” policy was applied during evacuation.

Embarked (Port of Embarkation)

Southampton (S) is the dominant port with 914 passengers (~70%), followed by Cherbourg (C) with 270 and Queenstown (Q) with 123.
There are 2 missing values in this variable.

Key Takeaways

Class disparity: The majority of passengers were in 3rd class, which historically had the lowest survival rates.
Gender imbalance: Nearly two-thirds of passengers were male, yet survival rates were higher for females.
Age distribution: The passenger population was predominantly young adults (20-40 years), with significant missing data that requires careful imputation.
Fare distribution: Highly skewed with extreme outliers, reflecting the vast economic differences between passenger classes.
Family variables (sibsp, parch): Most passengers traveled alone or with very small family groups.
Missing data strategy: The age column requires special attention due to its 20% missing rate; fare and embarked have minimal missing values that can be easily handled.

📖 References

Visualization

EDA

Statistical tests

https://nathanrosidi.medium.com/commonly-used-statistical-tests-in-data-science-93787568eb36

Bivariate Data Analysis

Thu, 24 Oct 2024 00:00:00 +0000

By Jose R. Zapata

Last update: 19/Feb/2026

📚 Import libraries

# base libraries for data science
import sys
import warnings
from pathlib import Path

import pandas as pd
import seaborn as sns

warnings.simplefilter(action="ignore", category=FutureWarning)

# configuration to show only 2 decimal places
pd.set_option("display.float_format", "{:.2f}".format)

# print library version for reproducibility

print("Python version: ", sys.version)
print("Pandas version: ", pd.__version__)

Python version: 3.11.11 (main, Dec 6 2024, 20:02:44) [Clang 18.1.8 ]
Pandas version: 2.2.3

💾 Load data

The dataset has correct data types, fixed in:

Initial data exploration

DATA_DIR = Path.cwd().resolve().parents[1] / "data"

titanic_df = pd.read_parquet(
 DATA_DIR / "02_intermediate/titanic_type_fixed.parquet", engine="pyarrow"
)

📊 Data description

General data information

titanic_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 10 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Unnamed: 0 1309 non-null int64
1 pclass 1309 non-null int64
2 survived 1309 non-null bool
3 name 1309 non-null object
4 sex 1309 non-null category
5 age 1046 non-null float64
6 sibsp 1309 non-null int64
7 parch 1309 non-null int64
8 fare 1308 non-null float64
9 embarked 1307 non-null category
dtypes: bool(1), category(2), float64(2), int64(4), object(1)
memory usage: 75.8+ KB

Ordinal data has to be converted again

information about the pclass column can be chech in the notebook

Initial data exploration

titanic_df["pclass"] = pd.Categorical(
 titanic_df["pclass"], categories=[3, 2, 1], ordered=True
)

# column Unnamed: 0 is not needed
titanic_df = titanic_df.drop(columns=["Unnamed: 0"])

General information about the data set:

titanic_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 9 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 pclass 1309 non-null category
1 survived 1309 non-null bool
2 name 1309 non-null object
3 sex 1309 non-null category
4 age 1046 non-null float64
5 sibsp 1309 non-null int64
6 parch 1309 non-null int64
7 fare 1308 non-null float64
8 embarked 1307 non-null category
dtypes: bool(1), category(3), float64(2), int64(2), object(1)
memory usage: 56.8+ KB

# size of the dataframe
titanic_df.shape

(1309, 9)

# sample of the dataframe
titanic_df.sample(5)

	pclass	survived	name	sex	age	sibsp	fare	embarked
1033	3	True	Moss, Mr. Albert Johan	male	NaN	0	7.78	S
222	1	False	Ovies y Rodriguez, Mr. Servando	male	28.50	0	27.72	C
1070	3	False	O'Brien, Mr. Timothy	male	NaN	0	7.83	Q
952	3	False	Larsson-Rondberg, Mr. Edvard A	male	22.00	0	7.78	S
706	3	False	Caram, Mrs. Joseph (Maria Elias)	female	NaN	1	14.46	C

Number of missing values

titanic_df.isnull().sum()

pclass 0
survived 0
name 0
sex 0
age 263
sibsp 0
parch 0
fare 1
embarked 2
dtype: int64

Target Variable = Survived

General statistics of the data set

Numerical variables

titanic_df.describe()

	age	sibsp	parch	fare
count	1046.00	1309.00	1309.00	1308.00
mean	29.88	0.50	0.39	33.30
std	14.41	1.04	0.87	51.76
min	0.17	0.00	0.00	0.00
25%	21.00	0.00	0.00	7.90
50%	28.00	0.00	0.00	14.45
75%	39.00	1.00	0.00	31.27
max	80.00	8.00	9.00	512.33

Categorical variables

# categorical columns description
titanic_df.describe(include="category")

	pclass	sex	embarked
count	1309	1309	1307
unique	3	2	3
top	3	male	S
freq	709	843	914

📈 Bivariate Analysis

Is important to check the relationship between the target variable and the other variables.

The target variable is Survived

Target vs Numerical Variables

survived vs age

variable = "age"

titanic_df.plot(
 kind="box",
 column=variable,
 by="survived",
 grid=False,
 title=f"Survived vs {variable} boxplot",
);

titanic_df[titanic_df["survived"] == 1][variable].plot(
 kind="kde",
 label="Survived",
 legend=True,
 title=f"Survived vs {variable} density plot",
)
titanic_df[titanic_df["survived"] == 0][variable].plot(
 kind="kde",
 label="Not Survived",
 legend=True,
 xlabel=variable,
 ylabel="Density",
);

survived vs sibsp

variable = "sibsp"

titanic_df.plot(
 kind="box",
 column=variable,
 by="survived",
 grid=False,
 title=f"Survived vs {variable} boxplot",
);

titanic_df[titanic_df["survived"] == 1][variable].plot(
 kind="hist",
 label="Survived",
 legend=True,
 bins=8,
 alpha=0.5,
 title=f"survived vs {variable} density plot",
)
titanic_df[titanic_df["survived"] == 0][variable].plot(
 kind="hist",
 label="Not Survived",
 legend=True,
 bins=8,
 alpha=0.3,
 xlabel=variable,
 ylabel="Density",
);

survived vs parch

variable = "parch"

titanic_df.plot(
 kind="box",
 column=variable,
 by="survived",
 grid=False,
 title=f"Survived vs {variable} boxplot",
);

titanic_df[titanic_df["survived"] == 1][variable].plot(
 kind="hist",
 label="Survived",
 legend=True,
 bins=9,
 alpha=0.5,
 title=f"Survived vs {variable} density plot",
)
titanic_df[titanic_df["survived"] == 0][variable].plot(
 kind="hist",
 label="Not Survived",
 legend=True,
 bins=9,
 alpha=0.3,
 xlabel=variable,
 ylabel="Density",
);

survived vs fare

variable = "fare"

titanic_df.plot(
 kind="box",
 column=variable,
 by="survived",
 grid=False,
 title=f"Survived vs {variable} boxplot",
);

titanic_df[titanic_df["survived"] == 1][variable].plot(
 kind="kde",
 label="Survived",
 legend=True,
 title=f"Survived vs {variable} density plot",
)
titanic_df[titanic_df["survived"] == 0][variable].plot(
 kind="kde",
 label="Not Survived",
 legend=True,
 xlabel=variable,
 ylabel="Density",
);

Target vs Categorical Variables

survived vs pclass

column = "pclass"

(
 pd
 .crosstab(titanic_df[column], titanic_df["survived"], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption("Survived vs pclass Heatmap")
)

Survived vs pclass Heatmap
survived	False	True	All
pclass
3	528	181	709
2	158	119	277
1	123	200	323
All	809	500	1309

titanic_df.groupby(column, observed=True).agg({"survived": "mean"}) * 100

	survived
pclass
3	25.53
2	42.96
1	61.92

2nd class passengers had twice the survival rate of 3rd class and 1st class passengers had even better rates

(
 pd.crosstab(titanic_df[column], titanic_df["survived"]).plot(
 kind="bar", stacked=True, title=f"Survived vs {column} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column], titanic_df["survived"]).plot(
 kind="bar",
 title=f"Survived vs {column} barplot",
 )
);

survived vs sex

column = "sex"

(
 pd
 .crosstab(titanic_df[column], titanic_df["survived"], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption("Survived vs pclass Heatmap")
)

Survived vs pclass Heatmap
survived	False	True	All
sex
female	127	339	466
male	682	161	843
All	809	500	1309

titanic_df.groupby(column, observed=True).agg({"survived": "mean"}) * 100

	survived
sex
female	72.75
male	19.10

(
 pd.crosstab(titanic_df[column], titanic_df["survived"]).plot(
 kind="bar", stacked=True, title=f"Survived vs {column} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column], titanic_df["survived"]).plot(
 kind="bar",
 title=f"Survived vs {column} barplot",
 )
);

survived vs embarked

column = "embarked"

(
 pd
 .crosstab(titanic_df[column], titanic_df["survived"], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption("Survived vs pclass Heatmap")
)

Survived vs pclass Heatmap
survived	False	True	All
embarked
C	120	150	270
Q	79	44	123
S	610	304	914
All	809	498	1307

titanic_df.groupby(column, observed=True).agg({"survived": "mean"}) * 100

	survived
embarked
C	55.56
Q	35.77
S	33.26

(
 pd.crosstab(titanic_df[column], titanic_df["survived"]).plot(
 kind="bar", stacked=True, title=f"Survived vs {column} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column], titanic_df["survived"]).plot(
 kind="bar",
 title=f"Survived vs {column} barplot",
 )
);

Numerical vs Numerical Variables

age vs sibsp

# scatter plot of age vs sibsp

titanic_df.plot(
 kind="scatter",
 x="age",
 y="sibsp",
 title="age vs Sibsp scatter plot",
);

age vs parch

# scatter plot of age vs parch

titanic_df.plot(
 kind="scatter",
 x="age",
 y="parch",
 title="age vs Parch scatter plot",
);

age vs fare

# scatter plot of age vs fare

titanic_df.plot(
 kind="scatter",
 x="age",
 y="fare",
 title="age vs fare scatter plot",
);

sibsp vs parch

# scatter plot of sibsp vs parch

titanic_df.plot(
 kind="scatter",
 x="sibsp",
 y="parch",
 title="Sibsp vs parch scatter plot",
);

sibsp vs fare

# scatter plot of sibsp vs fare

titanic_df.plot(
 kind="scatter",
 x="sibsp",
 y="fare",
 title="Sibsp vs fare scatter plot",
);

parch vs fare

# scatter plot of parch vs fare

titanic_df.plot(
 kind="scatter",
 x="parch",
 y="fare",
 title="parch vs fare scatter plot",
);

Categorical vs Categorical Variables

pclass vs sex

column_1 = "pclass"
column_2 = "sex"

(
 pd
 .crosstab(titanic_df[column_1], titanic_df[column_2], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption(f"{column_1} vs {column_2} Heatmap")
)

pclass vs sex Heatmap
sex	female	male	All
pclass
3	216	493	709
2	106	171	277
1	144	179	323
All	466	843	1309

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar", stacked=True, title=f"{column_1} vs {column_2} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar",
 title=f"{column_1} vs {column_2} barplot",
 )
);

pclass vs embarked

column_1 = "pclass"
column_2 = "embarked"

(
 pd
 .crosstab(titanic_df[column_1], titanic_df[column_2], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption(f"{column_1} vs {column_2} Heatmap")
)

pclass vs embarked Heatmap
embarked	C	Q	S	All
pclass
3	101	113	495	709
2	28	7	242	277
1	141	3	177	321
All	270	123	914	1307

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar", stacked=True, title=f"{column_1} vs {column_2} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar",
 title=f"{column_1} vs {column_2} barplot",
 )
);

sex vs embarked

column_1 = "sex"
column_2 = "embarked"

(
 pd
 .crosstab(titanic_df[column_1], titanic_df[column_2], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption(f"{column_1} vs {column_2} Heatmap")
)

sex vs embarked Heatmap
embarked	C	Q	S	All
sex
female	113	60	291	464
male	157	63	623	843
All	270	123	914	1307

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar", stacked=True, title=f"{column_1} vs {column_2} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar",
 title=f"{column_1} vs {column_2} barplot",
 )
);

sex vs pclass

column_1 = "sex"
column_2 = "pclass"

(
 pd
 .crosstab(titanic_df[column_1], titanic_df[column_2], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption(f"{column_1} vs {column_2} Heatmap")
)

sex vs pclass Heatmap
pclass	3	2	1	All
sex
female	216	106	144	466
male	493	171	179	843
All	709	277	323	1309

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar", stacked=True, title=f"{column_1} vs {column_2} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar",
 title=f"{column_1} vs {column_2} barplot",
 )
);

embarked vs pclass

column_1 = "embarked"
column_2 = "pclass"

(
 pd
 .crosstab(titanic_df[column_1], titanic_df[column_2], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption(f"{column_1} vs {column_2} Heatmap")
)

embarked vs pclass Heatmap
pclass	3	2	1	All
embarked
C	101	28	141	270
Q	113	7	3	123
S	495	242	177	914
All	709	277	321	1307

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar", stacked=True, title=f"{column_1} vs {column_2} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar",
 title=f"{column_1} vs {column_2} barplot",
 )
);

embarked vs sex

column_1 = "embarked"
column_2 = "sex"

(
 pd
 .crosstab(titanic_df[column_1], titanic_df[column_2], margins=True)
 .style.background_gradient(cmap="coolwarm")
 .set_caption(f"{column_1} vs {column_2} Heatmap")
)

embarked vs sex Heatmap
sex	female	male	All
embarked
C	113	157	270
Q	60	63	123
S	291	623	914
All	464	843	1307

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar", stacked=True, title=f"{column_1} vs {column_2} stacked barplot"
 )
);

(
 pd.crosstab(titanic_df[column_1], titanic_df[column_2]).plot(
 kind="bar",
 title=f"{column_1} vs {column_2} barplot",
 )
);

Categorical vs Numerical Variables

pclass vs age

column_cat = "pclass"
column_num = "age"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

pclass vs sibsp

column_num = "sibsp"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

pclass vs parch

column_num = "parch"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

pclass vs fare

column_num = "fare"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

sex vs age

column_cat = "sex"
column_num = "age"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

sex vs sibsp

column_num = "sibsp"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

sex vs parch

column_num = "parch"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

sex vs fare

column_num = "fare"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

embarked vs age

column_cat = "embarked"
column_num = "age"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

embarked vs sibsp

column_num = "sibsp"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

embarked vs parch

column_num = "parch"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

embarked vs fare

column_num = "fare"

titanic_df.plot(
 kind="box",
 column=column_num,
 by=column_cat,
 xlabel=column_cat,
 ylabel=column_num,
 title=f"{column_num} by {column_cat} boxplot",
 grid=False,
);

sns.barplot(
 data=titanic_df,
 x=column_cat,
 y=column_num,
 errorbar="ci",
 capsize=0.1,
 hue=column_cat,
).set_title(f"{column_num} by {column_cat} barplot");

📊 Analysis of Results

Target vs Numerical Variables

survived vs age: The age distributions for survived and not-survived passengers are similar overall, but children (age 0–10) show a noticeably higher survival rate. The median age is slightly lower for survivors. There is no strong linear relationship between age and survival.
survived vs sibsp: Passengers with 1 sibling/spouse aboard had a higher survival rate than those traveling alone (sibsp = 0) or those with many siblings/spouses. Having too many siblings/spouses (sibsp ≥ 3) is associated with very low survival.
survived vs parch: Similar to sibsp, passengers with 1–2 parents/children aboard survived at higher rates than those alone or with large families. Passengers with parch ≥ 4 had very low survival rates.
survived vs fare: Survivors paid significantly higher fares on average. The fare distribution for survivors is right-skewed with a higher median, while non-survivors cluster at lower fares. Fare is strongly associated with survival.

Target vs Categorical Variables

survived vs pclass: 1st class passengers had the highest survival rate (~62%), 2nd class had a moderate rate (~43%), and 3rd class had the lowest (~26%). Almost all Pclass 1 females survived and so did most of Pclass 2. Pclass 3 female survival was around 50%, and almost all (86%) Pclass 3 males did not survive.
survived vs sex: Female passengers had a dramatically higher survival rate (~73%) compared to males (~19%). Sex is the single strongest predictor of survival.
survived vs embarked: Passengers who embarked in Cherbourg (C) had the highest survival rate, followed by Queenstown (Q) and Southampton (S). This is likely confounded by the higher proportion of 1st class passengers embarking at Cherbourg.

Numerical vs Numerical Variables

age vs sibsp / parch: No strong correlation observed. Family size variables are largely independent of age.
age vs fare: Weak correlation. Older passengers do not necessarily pay higher fares.
sibsp vs parch: Weak positive correlation. Passengers with siblings/spouses tend to also have parents/children aboard, but the relationship is not strong.
sibsp vs fare / parch vs fare: Moderate positive correlation. Larger families tend to pay higher total fares.

Categorical vs Categorical Variables

pclass vs sex: The gender distribution is relatively balanced across classes, though 3rd class has a higher proportion of males.
pclass vs embarked: Most Southampton passengers were in 3rd class. Cherbourg had a higher proportion of 1st class passengers, which helps explain the higher survival rate for Cherbourg embarkation.
sex vs embarked: The gender distribution across embarkation ports is relatively similar, with a slight male majority across all ports.

Categorical vs Numerical Variables

pclass vs fare: Fare is very strongly correlated with passenger class — 1st class passengers paid the highest fares, and 3rd class the lowest. This confirms that fare is essentially a proxy for class.
pclass vs age: Age distribution is similar across classes, with no strong relationship between class and age.
sex vs age / fare: Males and females have similar age distributions. However, females tend to have slightly higher fares, likely because a higher proportion of female passengers traveled in 1st and 2nd class.
embarked vs fare: Cherbourg passengers paid the highest average fares (consistent with the higher proportion of 1st class passengers), while Southampton and Queenstown passengers paid lower fares on average.
embarked vs age: Age distributions are similar across embarkation ports.

Key Takeaways

The two variables with the strongest association with survival are sex and pclass. Females and higher-class passengers had significantly better survival outcomes.
Fare is highly correlated with class and acts as a strong proxy for socioeconomic status, which in turn is associated with survival.
Family size matters: Traveling with 1–2 family members (sibsp or parch) improved survival odds compared to traveling alone or in very large groups.
Children (age 0–10) had higher survival rates, consistent with the “women and children first” evacuation policy.
Embarkation port (embarked) is confounded with class: Cherbourg passengers survived more because they were disproportionately 1st class, not because of the port itself.
Age alone is not a strong predictor of survival, except at the extremes (very young children).

💡 Proposals and Ideas

For first baseline model, could use the following features:

sex
pclass
age

and can be done with heuristics or with a simple model like a decision tree.

📖 References

Visualization

EDA

Statistical tests

https://nathanrosidi.medium.com/commonly-used-statistical-tests-in-data-science-93787568eb36

Multivariate Data Analysis

Thu, 24 Oct 2024 00:00:00 +0000

By Jose R. Zapata

Last update: 19/Feb/2026

📚 Import libraries

# base libraries for data science
import sys
from pathlib import Path

import pandas as pd
import seaborn as sns

# configuration to show only 2 decimal places
pd.set_option("display.float_format", "{:.2f}".format)

# print library version for reproducibility

print("Python version: ", sys.version)
print("Pandas version: ", pd.__version__)

Python version: 3.11.11 (main, Dec 6 2024, 20:02:44) [Clang 18.1.8 ]
Pandas version: 2.2.3

💾 Load data

The dataset has correct data types, fixed in:

Initial data exploration

DATA_DIR = Path.cwd().resolve().parents[1] / "data"

titanic_df = pd.read_parquet(
 DATA_DIR / "02_intermediate/titanic_type_fixed.parquet", engine="pyarrow"
)

📊 Data description

General data information

titanic_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 10 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Unnamed: 0 1309 non-null int64
1 pclass 1309 non-null int64
2 survived 1309 non-null bool
3 name 1309 non-null object
4 sex 1309 non-null category
5 age 1046 non-null float64
6 sibsp 1309 non-null int64
7 parch 1309 non-null int64
8 fare 1308 non-null float64
9 embarked 1307 non-null category
dtypes: bool(1), category(2), float64(2), int64(4), object(1)
memory usage: 75.8+ KB

Ordinal data has to be converted again

information about the pclass column can be chech in the notebook

Initial data exploration

titanic_df["pclass"] = pd.Categorical(
 titanic_df["pclass"], categories=[3, 2, 1], ordered=True
)

# column Unnamed: 0 is not needed
titanic_df = titanic_df.drop(columns=["Unnamed: 0"])

General information about the data set:

titanic_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 9 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 pclass 1309 non-null category
1 survived 1309 non-null bool
2 name 1309 non-null object
3 sex 1309 non-null category
4 age 1046 non-null float64
5 sibsp 1309 non-null int64
6 parch 1309 non-null int64
7 fare 1308 non-null float64
8 embarked 1307 non-null category
dtypes: bool(1), category(3), float64(2), int64(2), object(1)
memory usage: 56.8+ KB

# size of the dataframe
titanic_df.shape

(1309, 9)

# sample of the dataframe
titanic_df.sample(5)

	pclass	survived	name	sex	age	sibsp	parch	fare	embarked
1176	3	False	Sage, Mr. Douglas Bullen	male	NaN	8	2	69.55	S
581	2	False	Watson, Mr. Ennis Hastings	male	NaN	0	0	0.00	S
18	1	True	Bazzani, Miss. Albina	female	32.00	0	0	76.29	C
921	3	False	Keefe, Mr. Arthur	male	NaN	0	0	7.25	S
494	2	True	Mallet, Mrs. Albert (Antoinette Magnin)	female	24.00	1	1	37.00	C

Number of missing values

titanic_df.isnull().sum()

pclass 0
survived 0
name 0
sex 0
age 263
sibsp 0
parch 0
fare 1
embarked 2
dtype: int64

Target Variable = Survived

General statistics of the data set

Numerical variables

titanic_df.describe()

	age	sibsp	parch	fare
count	1046.00	1309.00	1309.00	1308.00
mean	29.88	0.50	0.39	33.30
std	14.41	1.04	0.87	51.76
min	0.17	0.00	0.00	0.00
25%	21.00	0.00	0.00	7.90
50%	28.00	0.00	0.00	14.45
75%	39.00	1.00	0.00	31.27
max	80.00	8.00	9.00	512.33

Categorical variables

# categorical columns description
titanic_df.describe(include="category")

	pclass	sex	embarked
count	1309	1309	1307
unique	3	2	3
top	3	male	S
freq	709	843	914

📈 Multivariate Analysis

Is important to check the relationship between the target variable and multiple variables.

The target variable is Survived

Target vs age vs Sex

sns.boxplot(data=titanic_df, x="sex", y="age", hue="survived");

Target vs age vs pclass

sns.boxplot(data=titanic_df, x="pclass", y="age", hue="survived");

Target vs age vs embarked

sns.boxplot(data=titanic_df, x="embarked", y="age", hue="survived");

Target vs sex vs pclass

(
 titanic_df[["sex", "pclass", "survived"]]
 .groupby(["pclass", "sex"], observed=True)
 .mean()
 * 100,
 1,
)

( survived
pclass sex
3 female 49.07
male 15.21
2 female 88.68
male 14.62
1 female 96.53
male 34.08,
1)

almost all Pclass 1 females survive and so do most of Pclass 2. Pclass 3 female survival is 50% and almost all (86%) Pclass 3 males unfortunately do not survive.

sns.catplot(
 data=titanic_df,
 x="sex",
 hue="survived",
 col="pclass",
 kind="count",
 height=4,
 aspect=0.7,
);

sns.catplot(data=titanic_df, x="sex", y="survived", hue="pclass", kind="point");

sns.catplot(
 data=titanic_df,
 x="pclass",
 y="survived",
 hue="sex",
 palette={"male": "b", "female": "m"},
 markers=["^", "o"],
 linestyles=["-", "--"],
 kind="point",
);

sns.catplot(data=titanic_df, x="sex", y="survived", hue="pclass", kind="bar");

Target vs pclass vs fare

sns.catplot(
 x="fare",
 y="survived",
 row="pclass",
 kind="box",
 hue="survived",
 orient="h",
 height=1.5,
 aspect=4,
 data=titanic_df.query("fare > 0"),
).set(xscale="log");

Target vs sex vs pclass vs embarked

(
 titanic_df[["embarked", "sex", "pclass", "survived"]]
 .groupby(["embarked", "pclass", "sex"], observed=True)
 .agg(["count", "mean"])
)

			survived
			count	mean
embarked	pclass	sex
C	3	female	31	0.71
	3	male	70	0.21
	2	female	11	1.00
	2	male	17	0.29
	1	female	71	0.97
	1	male	70	0.40
Q	3	female	56	0.59
	3	male	57	0.12
	2	female	2	1.00
	2	male	5	0.00
	1	female	2	1.00
	1	male	1	0.00
S	3	female	129	0.40
	3	male	366	0.14
	2	female	93	0.87
	2	male	149	0.13
	1	female	69	0.96
	1	male	108	0.31

Numerical vs All Numerical Variables

# list of the numerical columns
numerical_columns = list(titanic_df.select_dtypes(include=["number"]).columns)
numerical_columns

['age', 'sibsp', 'parch', 'fare']

sns.pairplot(titanic_df[numerical_columns], diag_kind="kde");

df = titanic_df[[*numerical_columns, "survived"]]

sns.pairplot(df, hue="survived", diag_kind="kde");

📏 Heuristics baseline model

This is a simple baseline model that will be used to compare the performance of the machine learning models.

The model will be a simple decision tree with the following features:

sex
pclass
age

def calculate_survival_percentage(df, pclass, age_min, age_max=None) -> float:
 """Calculate the percentage of survivors based on the class and age range.

 Args:
 df: DataFrame with the Titanic dataset
 pclass: int with the class of the passenger
 age_min: int with the minimum age of the passenger
 age_max: int with the maximum age of the passenger

 Returns:
 float with the percentage of survivors

 """

 if age_max:
 survivors = len(
 df[
 (df["pclass"] == pclass)
 & (df["age"] > age_min)
 & (df["age"] < age_max)
 & (df["survived"])
 ]
 )
 total = len(
 df[(df["pclass"] == pclass) & (df["age"] > age_min) & (df["age"] < age_max)]
 )
 else:
 survivors = len(
 df[(df["pclass"] == pclass) & (df["age"] > age_min) & (df["survived"])]
 )
 total = len(df[(df["pclass"] == pclass) & (df["age"] > age_min)])

 return round((survivors / total) * 100, 1)


# Calcular y mostrar los porcentajes de supervivencia
print(
 "Pclass 1 survivors above age 60:",
 calculate_survival_percentage(titanic_df, 1, 59),
 "%",
)
print(
 "Pclass 2 survivors above age 60:",
 calculate_survival_percentage(titanic_df, 2, 59),
 "%",
)
print(
 "Pclass 3 survivors above age 60:",
 calculate_survival_percentage(titanic_df, 3, 59),
 "%",
)

print(
 "Pclass 1 survivors between 20-30 age:",
 calculate_survival_percentage(titanic_df, 1, 19, 31),
 "%",
)
print(
 "Pclass 2 survivors between 20-30 age:",
 calculate_survival_percentage(titanic_df, 2, 19, 31),
 "%",
)
print(
 "Pclass 3 survivors between 20-30 age:",
 calculate_survival_percentage(titanic_df, 3, 19, 31),
 "%",
)

Pclass 1 survivors above age 60: 38.5 %
Pclass 2 survivors above age 60: 12.5 %
Pclass 3 survivors above age 60: 16.7 %
Pclass 1 survivors between 20-30 age: 69.8 %
Pclass 2 survivors between 20-30 age: 41.0 %
Pclass 3 survivors between 20-30 age: 25.2 %

📊 Analysis of Results

Sex and Passenger Class (Key Predictors)

sex and pclass are the two variables with the highest correlation to the target variable survived, making them the most important features for predicting survival.
Almost all Pclass 1 females survived (~97% from Cherbourg, ~96% from Southampton) and so did most Pclass 2 females (100% from Cherbourg and Queenstown, ~87% from Southampton).
Pclass 3 female survival drops significantly (~59% from Queenstown, ~71% from Cherbourg, ~40% from Southampton), showing that class is a strong differentiator even among women.
Almost all Pclass 3 males did not survive (~86% mortality), regardless of port of embarkation.
Male survival rates were consistently low across all classes, but Pclass 1 males had the best chance (~40% from Cherbourg, ~31% from Southampton).

Age Interactions

Age is not correlated with pclass: the age distributions across the three classes are similar, with overlapping median values and interquartile ranges.
Age is not correlated with fare: knowing a passenger’s age does not help predict the fare they paid.
The survival rate among children (age 0–10) is notably higher than other age groups, suggesting a “women and children first” policy.
Among passengers above age 60, Pclass 1 had a 38.5% survival rate, while Pclass 2 and Pclass 3 had only 12.5% and 16.7% respectively.
Among passengers aged 20–30, Pclass 1 had a 69.8% survival rate, Pclass 2 had 41.0%, and Pclass 3 had only 25.2%.

Fare and Class

The fare is highly correlated with pclass: higher-class passengers paid substantially more. Within each class, survivors tended to have paid higher fares than non-survivors, especially in Pclass 1.

Port of Embarkation

The survival rate among passengers who embarked in Cherbourg (C) is higher than those from Southampton (S) or Queenstown (Q). This is likely because Cherbourg had a higher proportion of Pclass 1 passengers.
Queenstown (Q) passengers had the lowest overall survival rate, which aligns with the fact that most of them were Pclass 3.

Summary

The multivariate analysis confirms that survival on the Titanic was primarily determined by sex (females had much higher survival rates) and passenger class (higher class meant better survival odds). Age played a secondary role, with children having an advantage. The port of embarkation is a proxy for class composition rather than an independent predictor. These findings suggest that a baseline model using sex, pclass, and age should capture the main patterns in the data.

💡 Proposals and Ideas

For first baseline model, we will use the following features:

sex
pclass
age

and can be done with heuristics or with a simple model like a decision tree.

📖 References

Visualization

EDA

Statistical tests

https://nathanrosidi.medium.com/commonly-used-statistical-tests-in-data-science-93787568eb36

Jose R. Zapata

Fri, 20 Mar 2026 00:00:00 +0000

Data Science Technical Leader at Mercado Libre. Researcher at GIDATIC and Professor at the Faculty of Information and Communication Technologies (TIC), Universidad Pontificia Bolivariana (UPB).

My interests are based around Data science, MLOps and Audio and Music information technologies, which includes Music information retrieval, Machine Learning, CI/CD/CT, Audio analysis and Data analysis. EN ESPAÑOL

Visual Studio Code Configuration for Python, Data Science and Machine Learning

Fri, 20 Mar 2026 00:00:00 +0000

Last updated 20 / Mar / 2026

This post is a living document: it reflects my current VS Code configuration, and it is updated over time as I change my settings, extensions, or tools.

Introduction

Visual Studio Code (VS Code) is a free, lightweight, and highly extensible source code editor developed by Microsoft. It is one of the most popular editors for software development thanks to its wide ecosystem of extensions and its customization capabilities.

However, installing VS Code is not enough: to get the most out of it when working with Python, Data Science and Machine Learning, it needs to be configured properly. A good configuration allows you to:

Code quality: enable linters and automatic formatters that detect errors and maintain a consistent code style.
Unit testing: integrate the test runner directly in the editor to run tests and see results without leaving VS Code.
Debugging: configure the debugger to be able to inspect the program state step by step with the correct environment variables.

VS Code configurations are stored in the .vscode/ folder at the root of the project. Having these files versioned alongside the code ensures that the entire team (or any future collaborator) uses exactly the same configuration.

General configuration — `settings.json`

The settings.json file contains the editor preferences for the project. These settings override the user’s global preferences when working inside the project.

{
 "[python]": {
 "editor.formatOnSave": true,
 "editor.codeActionsOnSave": {
 "source.fixAll": "explicit",
 "source.organizeImports": "explicit"
 },
 "editor.defaultFormatter": "charliermarsh.ruff",
 "editor.rulers": [100]
 },
 "notebook.formatOnSave.enabled": true,
 "notebook.defaultFormatter": "charliermarsh.ruff",
 "notebook.codeActionsOnSave": {
 "notebook.source.fixAll": "explicit",
 "notebook.source.organizeImports": "explicit"
 },
 "[markdown]": {
 "editor.formatOnSave": true,
 "editor.codeActionsOnSave": {
 "source.fixAll.markdownlint": "explicit"
 }
 },
 "files.exclude": {
 "**/__pycache__": true
 },
 "python.languageServer": "None",
 "editor.formatOnPaste": true,
 "notebook.lineNumbers": "on",
 "editor.inlineSuggest.enabled": true,
 "editor.formatOnType": true,
 "git.autofetch": true,
 "python.terminal.activateEnvInCurrentTerminal": true,
 "python.testing.pytestArgs": [
 "tests"
 ],
 "python.testing.unittestEnabled": false,
 "python.testing.pytestEnabled": true,
 "[jsonc]": {
 "editor.defaultFormatter": "vscode.json-language-features"
 },
 "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
 "markdownlint.config": {
 "MD007": {"indent": 4}
 },
 "ruff.configuration": "--config=.code_quality/ruff.toml",
 "ruff.format.preview": true,
 "mypy-type-checker.args": [
 "--config-file=.code_quality/mypy.ini"
 ]
}

Explanation of each section

[python] — Configuration for Python files

"[python]": {
 "editor.formatOnSave": true,
 "editor.codeActionsOnSave": {
 "source.fixAll": "explicit",
 "source.organizeImports": "explicit"
 },
 "editor.defaultFormatter": "charliermarsh.ruff",
 "editor.rulers": [100]
}

editor.formatOnSave: automatically formats the file every time it is saved.
source.fixAll: applies all available automatic fixes on save (removes unused imports, fixes style errors, etc.).
source.organizeImports: sorts and groups imports following Python conventions (stdlib, third-party, local).
editor.defaultFormatter: uses Ruff as the default formatter. Ruff is an extremely fast linter and formatter written in Rust that replaces Flake8, isort, Black, and others.
editor.rulers: displays a vertical line at column 100 to indicate the recommended line length limit.

Jupyter Notebooks configuration

"notebook.formatOnSave.enabled": true,
"notebook.defaultFormatter": "charliermarsh.ruff",
"notebook.codeActionsOnSave": {
 "notebook.source.fixAll": "explicit",
 "notebook.source.organizeImports": "explicit"
}

Applies the same formatting and import organization rules as in .py files, but inside Jupyter notebooks (.ipynb). This maintains code quality also during the exploratory phase.

[markdown] — Configuration for Markdown files

"[markdown]": {
 "editor.formatOnSave": true,
 "editor.codeActionsOnSave": {
 "source.fixAll.markdownlint": "explicit"
 }
}

Enables automatic formatting when saving .md files.
Applies markdownlint corrections on save, ensuring that documentation follows Markdown style conventions.

files.exclude — Hide unnecessary files

"files.exclude": {
 "**/__pycache__": true
}

Hides __pycache__ folders in the VS Code file explorer. These folders contain compiled Python bytecode and are not relevant for development.

python.languageServer — Language server

"python.languageServer": "None"

Disables the built-in Python language server (Pylance). "None" is used when you prefer to rely on Ruff and Mypy for static analysis, avoiding duplicate diagnostics or conflicts between tools.

General editor configuration

"editor.formatOnPaste": true,
"editor.inlineSuggest.enabled": true,
"editor.formatOnType": true,
"notebook.lineNumbers": "on"

editor.formatOnPaste: automatically formats pasted code to match the project’s style.
editor.inlineSuggest.enabled: enables inline autocomplete suggestions (like GitHub Copilot or other suggestion sources).
editor.formatOnType: formats the code as you type.
notebook.lineNumbers: shows line numbers in Jupyter notebooks.

git.autofetch — Automatic Git synchronization

"git.autofetch": true

VS Code automatically fetches remote Git changes in the background, keeping the repository state up to date without needing to run git fetch manually.

python.terminal — Automatic virtual environment activation

"python.terminal.activateEnvInCurrentTerminal": true

Automatically activates the Python virtual environment (.venv) when an integrated terminal is opened in VS Code. This avoids having to activate the environment manually each time.

pytest configuration

"python.testing.pytestArgs": ["tests"],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true

pytestEnabled: enables pytest as the testing framework.
unittestEnabled: disables the unittest runner to avoid conflicts.
pytestArgs: tells VS Code to look for tests in the tests/ folder.

[jsonc] — Formatter for JSON files with comments

"[jsonc]": {
 "editor.defaultFormatter": "vscode.json-language-features"
}

Uses VS Code’s native JSON formatter for .jsonc (JSON with Comments) files, such as settings.json or launch.json themselves.

python.defaultInterpreterPath — Python interpreter

"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python"

Points to the Python interpreter inside the project’s virtual environment (.venv), ensuring that VS Code always uses the correct environment with the project’s dependencies installed.

Markdownlint configuration

"markdownlint.config": {
 "MD007": {"indent": 4}
}

Customizes markdownlint rule MD007 to require 4-space indentation in nested Markdown lists.

Ruff configuration

"ruff.configuration": "--config=.code_quality/ruff.toml",
"ruff.format.preview": true

ruff.configuration: tells the Ruff extension where to find its configuration file (.code_quality/ruff.toml), instead of looking in the project root.
ruff.format.preview: enables formatting rules in preview mode (new features that are not yet stable).

Mypy configuration

"mypy-type-checker.args": [
 "--config-file=.code_quality/mypy.ini"
]

Tells the Mypy extension where to find its configuration file (.code_quality/mypy.ini). Mypy is the Python static type checker that verifies type annotations.

Debugging configuration — `launch.json`

The launch.json file defines the debugging configurations that appear in the “Run and Debug” panel of VS Code.

{
 "version": "0.2.0",
 "configurations": [
 {
 "name": "Python: Current File",
 "type": "debugpy",
 "request": "launch",
 "program": "${file}",
 "console": "integratedTerminal",
 "cwd": "${workspaceFolder}",
 "env": {
 "PYTHONPATH": "${cwd}",
 "VERBOSITY": "DEBUG"
 }
 }
 ]
}

Explanation

version: version of the launch.json schema. Version 0.2.0 is the current standard.
name: name shown in the debug configuration selector (Python: Current File).
type: uses debugpy, the official Python debugger for VS Code.
request: launch indicates that VS Code will launch the program (as opposed to attach, which connects to an already running process).
program: ${file} runs the Python file that is currently open in the editor.
console: integratedTerminal displays the output in VS Code’s integrated terminal, allowing interaction with the program if needed.
cwd: sets the working directory to the project root (${workspaceFolder}), avoiding issues with relative paths.
env: defines environment variables for the debug session:
- PYTHONPATH: points to the project root so Python can correctly import local modules.
- VERBOSITY: sets the logging level to DEBUG to get detailed information during debugging.

Recommended extensions — `extensions.json`

The extensions.json file contains a list of recommended extensions for the project. When someone opens the project in VS Code, the editor automatically suggests installing them.

{
 "recommendations": [
 "alefragnani.project-manager",
 "bierner.emojisense",
 "charliermarsh.ruff",
 "davidanson.vscode-markdownlint",
 "editorconfig.editorconfig",
 "github.copilot-chat",
 "github.vscode-github-actions",
 "GitHub.vscode-pull-request-github",
 "KevinRose.vsc-python-indent",
 "mechatroner.rainbow-csv",
 "ms-python.mypy-type-checker",
 "ms-python.python",
 "ms-python.vscode-pylance",
 "ms-toolsai.jupyter",
 "oderwat.indent-rainbow",
 "redhat.vscode-yaml",
 "streetsidesoftware.code-spell-checker",
 "tamasfe.even-better-toml",
 "usernamehw.errorlens",
 "vivaxy.vscode-conventional-commits",
 "vscode-icons-team.vscode-icons"
 ]
}

Description of each extension

Extension	Description
Project Manager (`alefragnani.project-manager`)	Allows saving and quickly switching between projects without having to navigate the file system.
Emojisense (`bierner.emojisense`)	Autocomplete and preview for emojis in text and code files.
Ruff (`charliermarsh.ruff`)	Ultra-fast linter and formatter for Python. Replaces Flake8, isort, pyupgrade, Black and others in a single tool.
markdownlint (`davidanson.vscode-markdownlint`)	Linting of Markdown files using configurable rules to maintain quality documentation.
EditorConfig (`editorconfig.editorconfig`)	Applies style settings (indentation, encoding, line endings) defined in an `.editorconfig` file, ensuring consistency across editors and operating systems.
GitHub Copilot Chat (`github.copilot-chat`)	GitHub Copilot AI assistant with a chat interface for asking questions about code, generating code, explaining functions, and more.
GitHub Actions (`github.vscode-github-actions`)	Visualization and editing of GitHub Actions workflows directly from VS Code, with autocomplete and validation.
GitHub Pull Requests (`GitHub.vscode-pull-request-github`)	GitHub Pull Request management without leaving VS Code: create, review, comment on, and merge PRs.
Python Indent (`KevinRose.vsc-python-indent`)	Improves the Enter key behavior in Python to correctly indent within structures such as lists, dictionaries, and functions.
Rainbow CSV (`mechatroner.rainbow-csv`)	Colorizes the columns of CSV files with different colors to make reading and error detection easier.
Mypy Type Checker (`ms-python.mypy-type-checker`)	Integrates Mypy in VS Code to verify Python type annotations and detect type errors during development.
Python (`ms-python.python`)	Official Python extension for VS Code. Provides base support: IntelliSense, script execution, virtual environment management, and more.
Pylance (`ms-python.vscode-pylance`)	High-speed Python language server with advanced IntelliSense, type inference, and code navigation.
Jupyter (`ms-toolsai.jupyter`)	Full Jupyter Notebook support in VS Code: create, edit and run cells, manage kernels, and visualize results.
Indent Rainbow (`oderwat.indent-rainbow`)	Colorizes indentation levels with different colors to make reading code with multiple levels of nesting easier.
YAML (`redhat.vscode-yaml`)	Full YAML support with schema validation, autocomplete, and formatting. Very useful for CI/CD configuration files.
Code Spell Checker (`streetsidesoftware.code-spell-checker`)	Spell checker for code and comments. Detects typos in English (and other languages with additional dictionaries).
Even Better TOML (`tamasfe.even-better-toml`)	Full TOML file support with syntax highlighting, validation, and autocomplete. Essential for projects using `pyproject.toml`.
Error Lens (`usernamehw.errorlens`)	Displays errors, warnings, and diagnostics directly on the line of code where they occur, without needing to hover over them.
Conventional Commits (`vivaxy.vscode-conventional-commits`)	Assistant for writing commit messages following the Conventional Commits specification, with autocomplete for type, scope, and description.
vscode-icons (`vscode-icons-team.vscode-icons`)	Adds file and folder icons to the VS Code explorer to visually identify file types quickly.

Data science project template

Thu, 01 Feb 2024 00:00:00 +0000

Base Structure for Data Science Projects

Mon, 30 Oct 2023 00:00:00 +0000

Last updated 7 / Jan / 2024

https://github.com/JoseRZapata/data-science-project-template

Data science projects are often characterized by their complexity and exploratory nature. This can make it difficult to organize and maintain code and data. A well-defined project structure can help data scientists improve the quality of their work, use industrial software development standards, facilitate collaboration, and be scalable.

Initially these projects start with Jupyter notebooks, and after an iteration and refinement process, they are converted into Python scripts. However, Jupyter notebooks are an exploration tool and are not suitable for software development (with good software development practices). Therefore, it is important for data scientists to have a project structure that allows them to iterate quickly on experimentation and then develop complete processes (pipelines) with Python scripts.

To avoid decision fatigue and having to create a data science project structure from scratch, it is recommended to use a project template that contains the software tools and folder structure needed to develop data science projects efficiently, reproducibly, and with good software development practices.

Objective definition:

A data science project aims to solve a problem based on data analysis (Descriptive Analysis), understanding relationships in the data (Inferential Analysis) and creating a prediction model (which could be a Machine Learning model!).

This prediction model at a production level is a software product and as such should be managed as a software project.

How to use this template (Project Template)

If you want to regularly update your project with changes from the original template, it is recommended to use Cruft

These changes can be updates to library versions, new folders or tools to use in the project.

Cruft (Recommended)

Install Cruft with UV

uv tool install cruft

Create a project based on the template:

cruft create https://github.com/JoseRZapata/data-science-project-template

When it is necessary to update the project with changes from the template:

In the Root of the project where the .cruft.json file is located, run:

cruft update

Cookiecutter

Install Cookiecutter with UV:

uv tool install cookiecutter

Create a project based on the template:

cookiecutter gh:JoseRZapata/data-science-project-template

Reference projects

Comparative table of reference project templates:

Link	Pros	Cons	Last updated
Data Science Template	Well-focused elements for data science projects with software development elements	Missing Github Actions to automate CI/CD tasks and some folders	12/Jul/2023
cookiecutter-modern-datascience	Elements focused on data science projects with software development elements	No package manager with poetry to avoid dependency issues nor automation of some processes	10/Aug/2022
cookiecutter-hypermodern-python	Modern elements for developing Python projects	Needs to expand the folder structure for data science projects.	3/Jun/2022
cookiecutter-cruft-poetry-tox-pre-commit-ci-cd	Modern tool configuration for developing Python projects	Not specific for data science projects, needs to expand the folder structure	22/Aug/2023
cookiecutter-data-science	Widely used template with elements focused on data science projects	No Python software development configuration nor package manager with poetry, last update was several years ago	20/Mar/2021

Features of the base project template for data science

Based on my experience and other references, some of the features that a base project for data science should have are:

Dependency management with UV: UV is a modern dependency manager for Python that simplifies the process of installing and managing dependencies, and also manages virtual environments and Python installation. Previously I used Poetry, but UV is faster and replaces both Poetry and pyenv.
Pre-commit hooks with pre-commit: pre-commit is a tool that allows developers to automatically run scripts before making a commit. This can help ensure that the code meets the project’s quality standards.
Code quality:
- Ruff: is a tool that finds and fixes Python usage errors, and currently already replaces Black which formats code and helps ensure the code has a consistent style.
- Mypy is a static type checker for Python that helps find errors before the code runs.
Unit testing and test coverage with PyTest and Codecov:
- PyTest is a unit testing library for Python.
- Codecov is a tool that provides test coverage reports on the code.
CI/CD with GitHub Actions: GitHub Actions is a workflow automation platform that allows developers to automate tasks such as building, testing, and deploying code.
Documentation: Generate static project documentation.
Project template: Libraries are needed to create projects from templates with folders and configuration files.
- Cruft allows creating a project from a cookiecutter and also updating the project with changes from the original template.
- Cookiecutter allows creating projects from templates without updates.
Configuration scripts: Scripts to configure the project and development environment when starting the project and for repetitive commands, for example Makefile.

What the base data science project template does NOT include:

~~pypi:~~ PyPI will not be used to publish packages, since machine learning models and data transformations will be created.
~~nox~~, ~~tox:~~ Tests across different Python versions will not be needed; development will use only one Python version.

Initial base

To achieve this I will create a base project template for data science with the following elements:

Feature	Library	Status
Programming language	Python version 3.9 or higher	✅
Library manager	UV or ~~Poetry~~ or ~~pipenv~~	✅
Virtual environment manager	UV or ~~Poetry~~ or ~~pipenv~~	✅
Testing	Pytest	✅
Hook manager	pre-commit	✅
Auto update pre-commit	github actions	✅
Linter	Ruff	✅
Code formatting	Ruff and ~~Prettier~~	✅
Static type checking	Mypy	✅
Import ordering	Ruff	✅
Code coverage	coverage.py	✅
Test coverage report	codecov	✅
Dependency updates	Dependabot	✅
Security and auditing	Bandit and ~~Safety~~	✅
Scaffolding manager	cruft or Cookiecutter ~~copier~~	✅
Project configuration	OmegaConf or hydra	✅
Python syntax upgrade	Ruff	✅
Documentation	MkDocs or ~~Sphinx~~ or ~~pdoc~~	✅
Base libraries	Pandas, NumPy, scikit-learn, Jupyter	✅
Configuration scripts	Makefile or ~~just~~	✅

Currently Ruff replaces flake8, black, pylint, pyupgrade and isort which format code and help ensure the code has a consistent style. Ruff is very fast and is written in Rust

Additional MLOps elements

Additionally, to achieve a base data science project structure that enables MLOps implementation, the following elements will be included:

Libraries for a Machine Learning project

Feature	Library	Status
Data manager	DVC	❓
Data validation	Great Expectations or pandera	❓
Experiment manager	MlFlow, weights & biases or Neptune	❓
Model manager	MlFlow	❓
Pipeline manager	Kedro , ZenML	❓
Train/test validation	deepcheck	❓
Data integrity validation	deepcheck	❓
Model validation	deepcheck	❓

References

Cookiecutter Templates

References for cookiecutter project templates for data science and software projects:

Cookiecutter Data Science

Cookiecutter with UV

https://github.com/fpgmaas/cookiecutter-uv

Cookiecutter with Poetry

Enviroments

https://www.pluralsight.com/tech-blog/managing-python-environments/

LoFi Album with Python Version Zero

Mon, 19 Jun 2023 00:00:00 +0000

In this post, I will present a project I have been working on: a Python script that automatically generates music videos using artificial intelligence techniques and audio processing. Throughout the article, I will break down the code into sections to explain how it works and how each component contributes to the creation of these videos.

The code was run in Google Colaboratory and can be accessed at the following link:

Install Libraries

In the first code segment, the necessary libraries and tools are installed to run the script:

imagemagick: This tool is used to process and manipulate images in different formats.
The ImageMagick policy file (/etc/ImageMagick-6/policy.xml) is modified to allow reading and writing of images.
Several PyTorch libraries are installed with specific versions:
- torch: This is the main PyTorch package, a Python library for deep learning used in this project to implement artificial intelligence algorithms.
- torchvision: Contains utilities for working with images and applying transformations to them, as well as pretrained computer vision models.
- torchaudio: Provides tools for working with audio data and applying audio transformations.
- torchtext: Used to process and manipulate text data.
- torchdata: Facilitates the loading and preprocessing of datasets.
The xformers library is installed from a specific provided link. This library contains implementations of optimized and efficient transformers for natural language processing and computer vision tasks.
Additional libraries are installed, such as:
- diffusers: Facilitates the implementation of stable diffusion models, which are used in this project to generate images.
- transformers: Contains implementations of transformer-based natural language processing models, such as BERT and GPT.
- ftfy: A library for cleaning and normalizing text, fixing common encoding and formatting errors.
- pydub: Allows manipulation of audio files in different formats.
- accelerate: Provides an API for running deep learning models on multiple devices and hardware accelerators.
The matchering library is installed, which is used to process and master audio files, adjusting the sound and quality of the automatically generated music files.
Finally, pytube is installed, a library that allows downloading YouTube videos in Python.

This code segment is responsible for preparing the working environment, ensuring that all necessary dependencies are installed before starting to work with the main script.

!apt install imagemagick -qq > /dev/null
!cat /etc/ImageMagick-6/policy.xml | sed 's/none/read,write/g'> /etc/ImageMagick-6/policy.xml

!pip install -q torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 torchtext==0.14.1 torchdata==0.5.1 --extra-index-url https://download.pytorch.org/whl/cu116 -U
!pip install -q https://github.com/camenduru/stable-diffusion-webui-colab/releases/download/0.0.17/xformers-0.0.17+b6be33a.d20230315-cp39-cp39-linux_x86_64.whl
!pip install -q -U diffusers transformers ftfy pydub accelerate

!pip install -q matchering

!pip install -q pytube

It is necessary to restart the Google Colab runtime after running the library installation code segment. This can be done from the top menu under Runtime > Restart runtime.

Import Libraries

The necessary libraries are imported for processing images, audio, and using deep learning algorithms. numpy and PIL (Python Imaging Library) are used to work with images and numerical arrays, while pydub, scipy.io.wavfile, and torchaudio are used to manipulate and process audio files. matchering is used for audio mastering. Finally, torch and diffusers provide the tools to implement artificial intelligence models, such as stable diffusion in this case.

import numpy as np
from PIL import Image
import pydub
from scipy.io import wavfile
import io
import typing as T
from pydub import AudioSegment
from IPython.display import Audio, display
import matchering as mg

import torch
from diffusers import StableDiffusionPipeline
import torchaudio

The next step is responsible for loading the model to automatically generate music:

A variable model_id2 is defined containing the path to the pretrained model called “riffusion-model-v1”. Riffusion is an application based on stable diffusion for generating music in real time Riffusion. This model is specifically designed to work with spectrograms, which allows generating music of different styles and genres.
A pipe2 object is created using the StableDiffusionPipeline class. The from_pretrained function loads the pretrained model specified by model_id2 and sets the data type to torch.float16. By using the mixed-precision data type float16, memory consumption can be reduced and the performance of the music generation process can be improved.
Finally, the pipe2 object is moved to the GPU using the to("cuda") method. This allows the model to run on the GPU, taking advantage of its parallel processing capability and accelerating music generation tasks.

model_id2 = "riffusion/riffusion-model-v1"
pipe2 = StableDiffusionPipeline.from_pretrained(model_id2, torch_dtype=torch.float16)
pipe2 = pipe2.to("cuda")

The following functions work together to generate music from a pretrained Riffusion model, converting spectrograms into audio files and vice versa.

get_music(audiofilename, prompt, musicAI_indx, duration=5): This function is used to generate music from a pretrained Riffusion model. The function takes as parameters the audio filename, the text input (prompt), the model index, and the desired duration of the music. The function creates a spectrogram and then converts it into an audio file in WAV and MP3 format. It returns the spectrogram and the name of the MP3 file.
wav_bytes_from_spectrogram_image(image: Image.Image) -> T.Tuple[io.BytesIO, float]: This function takes a spectrogram image as input and returns an io.BytesIO object containing the reconstructed WAV audio file data and the duration of the audio in seconds. The function performs the reconstruction using the inverse Mel transform and the Griffin-Lim algorithm.
spectrogram_from_image(image: Image.Image, max_volume: float = 50, power_for_image: float = 0.25) -> np.ndarray: This function takes a spectrogram image as input and returns a spectrogram magnitude array. The function performs a series of operations on the image, such as inverting, changing the maximum volume, and applying an inverse power curve.
waveform_from_spectrogram(...) -> np.ndarray: This function takes a spectrogram magnitude array as input and returns a reconstructed waveform. The function uses the inverse Mel transform and the Griffin-Lim algorithm to approximate the phase and reconstruct the waveform.
mp3_bytes_from_wav_bytes(wav_bytes: io.BytesIO) -> io.BytesIO: This function takes an io.BytesIO object containing WAV file data as input and returns an io.BytesIO object containing the converted MP3 file data. The function uses the pydub library to perform the WAV to MP3 conversion.

#@title functions
def get_music(audiofilename, prompt, musicAI_indx, duration=5):
 mp3file_name = f"{audiofilename}.mp3"
 wavfile_name = f"{audiofilename}.wav"
 if musicAI_indx == 0:
 if duration == 5:
 width_duration=512
 else :
 width_duration = 512 + ((int(duration)-5) * 128)
 spec = pipe2(prompt, height=512, width=width_duration).images[0]
 print(spec)
 wav = wav_bytes_from_spectrogram_image(spec)
 with open(wavfile_name, "wb") as f:
 f.write(wav[0].getbuffer())


 #Convert to mp3, for video merging function
 wavfile = AudioSegment.from_wav(wavfile_name)
 wavfile.export(mp3file_name, format="mp3")
 return spec, mp3file_name


def wav_bytes_from_spectrogram_image(image: Image.Image) -> T.Tuple[io.BytesIO, float]:
 """
 Reconstruct a WAV audio clip from a spectrogram image. Also returns the duration in seconds.
 """

 max_volume = 50
 power_for_image = 0.25
 Sxx = spectrogram_from_image(image, max_volume=max_volume, power_for_image=power_for_image)

 sample_rate = 44100 # [Hz]
 clip_duration_ms = 5000 # [ms]

 bins_per_image = 512
 n_mels = 512

 # FFT parameters
 window_duration_ms = 100 # [ms]
 padded_duration_ms = 400 # [ms]
 step_size_ms = 10 # [ms]

 # Derived parameters
 num_samples = int(image.width / float(bins_per_image) * clip_duration_ms) * sample_rate
 n_fft = int(padded_duration_ms / 1000.0 * sample_rate)
 hop_length = int(step_size_ms / 1000.0 * sample_rate)
 win_length = int(window_duration_ms / 1000.0 * sample_rate)

 samples = waveform_from_spectrogram(
 Sxx=Sxx,
 n_fft=n_fft,
 hop_length=hop_length,
 win_length=win_length,
 num_samples=num_samples,
 sample_rate=sample_rate,
 mel_scale=True,
 n_mels=n_mels,
 max_mel_iters=200,
 num_griffin_lim_iters=32,
 )

 wav_bytes = io.BytesIO()
 wavfile.write(wav_bytes, sample_rate, samples.astype(np.int16))
 wav_bytes.seek(0)

 duration_s = float(len(samples)) / sample_rate

 return wav_bytes, duration_s


def spectrogram_from_image(
 image: Image.Image, max_volume: float = 50, power_for_image: float = 0.25
) -> np.ndarray:
 """
 Compute a spectrogram magnitude array from a spectrogram image.
 TODO(hayk): Add image_from_spectrogram and call this out as the reverse.
 """
 # Convert to a numpy array of floats
 data = np.array(image).astype(np.float32)

 # Flip Y take a single channel
 data = data[::-1, :, 0]

 # Invert
 data = 255 - data

 # Rescale to max volume
 data = data * max_volume / 255

 # Reverse the power curve
 data = np.power(data, 1 / power_for_image)

 return data


def waveform_from_spectrogram(
 Sxx: np.ndarray,
 n_fft: int,
 hop_length: int,
 win_length: int,
 num_samples: int,
 sample_rate: int,
 mel_scale: bool = True,
 n_mels: int = 512,
 max_mel_iters: int = 200,
 num_griffin_lim_iters: int = 32,
 device: str = "cuda:0",
) -> np.ndarray:
 """
 Reconstruct a waveform from a spectrogram.
 This is an approximate inverse of spectrogram_from_waveform, using the Griffin-Lim algorithm
 to approximate the phase.
 """
 Sxx_torch = torch.from_numpy(Sxx).to(device)

 if mel_scale:
 mel_inv_scaler = torchaudio.transforms.InverseMelScale(
 n_mels=n_mels,
 sample_rate=sample_rate,
 f_min=0,
 f_max=10000,
 n_stft=n_fft // 2 + 1,
 norm=None,
 mel_scale="htk",
 max_iter=max_mel_iters,
 ).to(device)

 Sxx_torch = mel_inv_scaler(Sxx_torch)

 griffin_lim = torchaudio.transforms.GriffinLim(
 n_fft=n_fft,
 win_length=win_length,
 hop_length=hop_length,
 power=1.0,
 n_iter=num_griffin_lim_iters,
 ).to(device)

 waveform = griffin_lim(Sxx_torch).cpu().numpy()

 return waveform


def mp3_bytes_from_wav_bytes(wav_bytes: io.BytesIO) -> io.BytesIO:
 mp3_bytes = io.BytesIO()
 sound = pydub.AudioSegment.from_wav(wav_bytes)
 sound.export(mp3_bytes, format="mp3")
 mp3_bytes.seek(0)
 return mp3_bytes

Functions for Creating Audio from a List of Words

The following functions work together to generate music from a list of words, apply audio effects such as crossfade and fades, and improve audio quality through automatic mastering.

list_get_music(list_variables, texto_prompt, dur=5): This function takes a list of strings, a prompt, and the music duration. For each string in the list, it generates music using the get_music function with the specified prompt and duration. The generated music files are saved with names based on their index in the list.
list_display_audio(list_variables): This function takes a list of strings and plays the audio files previously generated by list_get_music. It uses the IPython.display.Audio module to play the audio files in MP3 format.
n_times(number_segments, n_times=3): This function takes the number of audio segments and the desired number of repetitions. It multiplies the repetitions of each audio segment and saves the results in WAV files.
n_times_segments(number_segments, n_times=3): This function takes the number of audio segments and the desired number of repetitions. It multiplies the repetitions of each audio segment and returns a list of AudioSegment objects with the repeated segments.
audio_segments_concat(songs_list, crossfade=2000, fade_in=2000, fade_out=3000, out_filename="full_song.wav"): This function takes a list of AudioSegment objects, crossfade, fade in, fade out values, and an output filename. It concatenates the audio segments with crossfade, applies fade in at the beginning and fade out at the end, and saves the result in a WAV file.
crear_fades(seg_inicial=0, seg_final=6, fade_in=3000, fade_out=3000): This function takes the indices of the initial and final segments, and the fade in and fade out values. It applies fade in to the initial segment and fade out to the final segment, and saves the results in WAV files.
mejoras_audio(number_segments=7, master_reference='audio.mp4'): This function takes the number of audio segments and a mastering reference file. It converts each audio segment from mono to stereo, applies automatic mastering using the reference file, and saves the results in WAV files. It also removes the temporary files created during the process.

from os import lstat
import os

def list_get_music(list_variables:list,
 texto_prompt: str,
 dur:int=5) -> None:
 """Takes a list of strings and a prompt and generates sounds with riffusion"""

 for number, str_var in enumerate(list_variables):
 prompt = f"{str_var} {texto_prompt}"
 get_music(f"{number}",prompt=prompt, musicAI_indx=0, duration=dur)


def list_display_audio(list_variables:list)->None:
 """display audio of sounds generated by riffusion"""

 for number, _ in enumerate(list_variables):
 display(Audio(f'{number}.mp3'))

def n_times(number_segments:int,
 n_times:int=3) -> None:
 """multiply the repetitions of the audio files and then
 save them"""

 for number in range(number_segments):
 song = AudioSegment.from_wav(f'{number}.wav')
 song = song *n_times
 song.export(f'{number}_times.wav',format='wav')

def n_times_segments(number_segments:int,
 n_times:int=3) -> list:
 """multiply the repetitions of the segments and then
 concatenate them in a list"""

 songs_list = []
 for number in range(number_segments):
 song = AudioSegment.from_wav(f'{number}.wav')
 song = song *n_times
 songs_list.append(song)
 return songs_list

def audio_segments_concat(songs_list:list,
 crossfade:int = 2000,
 fade_in:int = 2000,
 fade_out:int = 3000,
 out_filename:str = "full_song.wav"):
 full_song = songs_list[0]
 for song_segment in songs_list[1:]:
 full_song = full_song.append(song_segment, crossfade=crossfade)

 # fade in and fade out
 full_song = full_song.fade_in(fade_in).fade_out(fade_out)
 #store song
 full_song.export(out_filename,format='wav')
 print(f"File {out_filename} stored")

def crear_fades(seg_inicial:int =0,
 seg_final:int = 6,
 fade_in:int = 3000,
 fade_out:int = 3000):

 inicial = AudioSegment.from_wav(f'{seg_inicial}_times.wav')
 inicial = inicial.fade_in(fade_in)
 inicial.export(f'{seg_inicial}_times.wav',format='wav')

 final = AudioSegment.from_wav(f'{seg_final}_times.wav')
 final = final.fade_out(fade_out)
 final.export(f'{seg_final}_times.wav',format='wav')

def mejoras_audio(number_segments:int=7,
 master_reference:str = 'audio.mp4'):
 "Convert audio from Mono to Stereo and then apply automatic mastering"

 for number in range(number_segments):
 song = AudioSegment.from_wav(f'{number}_times.wav')
 #convert to stereo
 song = song.set_channels(2)
 song.export(f'{number}_stereo.wav',format='wav')

 mg.process(
 # The track you want to master
 target=f"{number}_stereo.wav",
 # Some "wet" reference track
 reference=master_reference,
 # Where and how to save your results
 results=[
 mg.pcm16(f"{number}_master.wav")
 ],
 )
 os.remove(f"{number}_stereo.wav")
 os.remove(f"{number}_times.wav")

Audio for the Same Word

Generating audio for the same word

# List of generator variables
word = "learning" #@param {type:"string"}
prompt = "lofi programming with python" #@param {type:"string"}
palabra = (f"{word} "*7).split()

Generate audio files with a variable list and a fixed prompt, with a set duration of 10 seconds, which is the limit allowed by the free version of Google Colab.

list_get_music(palabra,prompt,10)

Display the generated audio files

list_display_audio(palabra)

Increase the duration of the generated audio files by multiplying by 3 to get 30 seconds

n_times(len(palabra), n_times=3)

Apply fade in and fade out to the generated audio files to create smooth transitions between segments.

crear_fades(0,6,3000,3000)

Download a reference audio file for automatic mastering


from pytube import YouTube as YT
import threading as th
video = YT('https://www.youtube.com/watch?v=NMKsn5puEsQ', use_oauth=True, allow_oauth_cache=False)
video.streams.get_by_itag(140).download()

Automatically master the audio files using the Python library Matchering

mejoras_audio(len(palabra), 'Bookoo Bread Co Instrumental - Scallops Hotel.mp4')

Create Images for the Video

This code block is necessary to create the images that will go in the video:

It defines a variable img_model_id containing the path to the pretrained model called “runwayml/stable-diffusion-v1-5”. This model is designed to work with images and is based on stable stochastic diffusion, which allows generating high-quality images with a wide variety of styles and themes.
It creates an img_pipe object using the StableDiffusionPipeline class. The from_pretrained function loads the pretrained model specified by img_model_id and sets the data type to torch.float16. By using the mixed-precision data type float16, memory consumption can be reduced and the performance of the image generation process can be improved.
Next, the img_pipe object is moved to the GPU using the to("cuda") method. This allows the model to run on the GPU, taking advantage of its parallel processing capability and accelerating image generation tasks.
Finally, efficient memory attention is enabled for the transformers within the model using the enable_xformers_memory_efficient_attention() method. This optimization allows reducing GPU memory consumption during inference, which can be useful for generating high-resolution images or running the model on devices with limited resources.

img_model_id = "runwayml/stable-diffusion-v1-5"
img_pipe = StableDiffusionPipeline.from_pretrained(img_model_id, torch_dtype=torch.float16, revision="fp16")
img_pipe = img_pipe.to("cuda")
img_pipe.enable_xformers_memory_efficient_attention()

This code block defines a function called create_images that takes as arguments a list of strings (list_variables) and a text prompt (text_prompt). The function aims to generate images from the elements in the variable list using the previously loaded stable diffusion model.

The function performs the following actions:

It iterates over the variable list and their indices using enumerate(list_variables).
For each variable in the list, it creates a complete prompt by concatenating the variable and the provided text_prompt. Thus, the complete prompt includes additional information that guides the model to generate high-quality images, such as “artstation hall of fame gallery” and “editors choice”.
It uses the previously loaded img_pipe object to generate an image from the complete prompt. The img_pipe function returns an object containing a list of images, and it selects the first image from the list with images[0].
It saves the generated image to a PNG file using the variable’s index in the list as the filename (for example, “0.png”, “1.png”, etc.).

In summary, this function creates images using the stable diffusion model from a list of variables and a text prompt, and saves the generated images to PNG files.

def create_images(list_variables:list,
 text_prompt:str)->None:

 for number, str_var in enumerate(list_variables):
 prompt = f"{str_var} {text_prompt}"
 image = img_pipe(prompt + ", artstation hall of fame gallery, editors choice, #1 digital painting of all time, most beautiful image ever created, emotionally evocative, greatest art ever made, lifetime achievement magnum opus masterpiece, the most amazing breathtaking image with the deepest message ever painted, a thing of beauty beyond imagination or words").images[0]
 image.save(f"{number}.png")

Create images for the video

create_images(palabra*2, prompt)

Display the generated images

from IPython.display import Image as display_image
imagenes2 = [f"{x}.png" for x in range(7)]
for img in imagenes2:
 display(display_image(filename=img))

Create the Final Video

This code creates a 1900x1080 video from a series of images and audio segments, applying transition and resizing effects, and adding a black background and text to the final video.

It creates a base black image of 1900x1080 pixels and saves it as ‘black.jpg’. This image will be used as the background in the final video so that it has high resolution.
It imports the necessary libraries for video and audio processing, such as moviepy.editor, pathlib.Path, and moviepy.video.fx.resize.
The creacion_video_segmentos(numero_segmentos) function takes the numero_segmentos argument indicating the number of audio and image segments to use. The function creates a list of video clips, where each clip consists of an image and its corresponding audio segment. The list of video clips is returned at the end of the function.
The Final_concatenar_crossfade(lista_videos, custom_padding=1, nombre='lofi') function takes a list of video clips, a custom padding value, and a name for the final video. This function creates a final video through the following actions:

a. It applies a slide transition to each video clip in the list and concatenates the clips with negative padding, which creates a crossfade effect between the clips.

b. It resizes the resulting video to 1080x1080 pixels and places the previously created black background behind this video.

c. It adds text with the video name at a specific position.

d. It combines the video and text into a single video object and sets the duration of the final video.

e. It writes the final video to an MP4 file with the provided name, using 24 frames per second and the ‘aac’ audio codec.

#create base black image
from PIL import Image;
Image.new('RGB',
 (1900, 1080),
 color = (0,0,0)).save('black.jpg')

from moviepy.editor import *
from pathlib import Path
from moviepy.video.fx.resize import resize

def creacion_video_segmentos(numero_segmentos:int) -> list:
 video_clips = []

 #list images files
 for number in range(numero_segmentos):
 clip_audio = AudioFileClip(f"{number}_master.wav").set_start(0)
 clip_video = ImageClip(f"{number}.png", duration=clip_audio.duration)
 clip_video = clip_video.set_audio(clip_audio)
 video_clips.append(clip_video)
 return video_clips

def Final_concatenar_crossfade(lista_videos:list,
 custom_padding:int=1,
 nombre:str = 'lofi'):

 slided_clips = [CompositeVideoClip([clip.fx( transfx.slide_out, custom_padding, 'bottom')]) for clip in lista_videos]
 video_slides = concatenate( slided_clips, padding=-1)
 video_slides = resize(video_slides, width=1080, height=1080)
 black_image = ImageClip("black.jpg")
 final = CompositeVideoClip([black_image, video_slides.set_position("center")])
 final = final.set_duration(video_slides.duration)

 ## TODO Add the song name

 # create a TextClip object with the desired text
 text_clip = TextClip(nombre,
 font="Amiri-regular",
 fontsize=40,
 color='grey40')
 # set the position of the text clip
 text_clip = (text_clip
 .set_position((100, 900))
 .set_start(0)
 )
 final = CompositeVideoClip([final, text_clip])

 # FINAL ASSEMBLY
 final_video_audio = CompositeVideoClip([final, text_clip]).set_duration(video_slides.duration)

 final_video_audio.write_videofile(f"{nombre}.mp4",
 fps=24,
 threads = 2,
 audio_codec='aac',
 audio=True)

Code to create each of the audio segments

lista_videos = creacion_video_segmentos(7)

Final creation of the video by concatenating the previously generated segments

# Final Video Creation
concatenar_crossfade(lista_videos,1, word)

In this post, we have explored how to use stable diffusion models to generate music and images from a list of variables and a text prompt. We then used the MoviePy library to combine these images with corresponding audio segments and create a video with smooth transitions and resizing effects.

This process illustrates how it is possible to combine artificial intelligence techniques and video processing to create unique and personalized audiovisual products from individual elements such as images and audio segments.

Next Album

These generated videos only have static images; the next album will feature videos with dynamic images.
The Riffusion audio still has quality limitations; the next album will use a new audio generation tool such as https://github.com/facebookresearch/audiocraft
MoviePy is a tool for automating the creation of videos.

References

Sesquialtera in the Colombian Bambuco: Perception and Estimation of Beat and Meter – Extended version

Fri, 10 Dec 2021 07:45:27 -0500

Noticias Falsas: ¿Por qué son incontrolables? Podcast HumanaMente Ep.9

Wed, 14 Oct 2020 10:11:07 -0500

Sesquialtera in the Colombian Bambuco: Perception and Estimation of Beat and Meter

Mon, 28 Sep 2020 10:57:42 -0500

Step by Step in a Machine Learning Project

Thu, 09 Apr 2020 00:00:00 +0000

Last updated: 2/Feb/2025

I based this checklist on Appendix B (Machine Learning Project Checklist) from the book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems 2nd Edition by Aurélien Géron with some of my own steps added or found in other books or courses I have taken.

I really enjoyed this book; for me it is the most complete practical book on machine learning with Python that I have read. It has an excellent structure, very well-explained Python code, and many tips and suggestions for carrying out a machine learning project.

The book is accompanied by a repository with Jupyter Notebooks: https://github.com/ageron/handson-ml2

Table of Contents

This checklist can serve as a step-by-step guide for Machine Learning projects.

1. Frame the problem and look at the big picture

Define the objective in business terms.
How will your solution be used?
What are the current solutions (if any)?
How should this problem be framed (supervised / unsupervised, online / offline, etc.)?
How should the performance or effectiveness of the solution be measured?
Is the performance measure aligned with the business objective?
What would be the minimum performance or effectiveness needed to achieve the business objective?
What are comparable problems? Can experience or tools already created be reused?
Is human expertise on the problem available?
How can the problem be solved manually?
List the assumptions made so far.
Verify the assumptions if possible.

2. Get the data

Note: automate this process as much as possible so that you can easily obtain fresh data.

List the data you need and how much you need.
Find and document where the data can be obtained.
Check how much storage space the data will occupy.
Check legal limitations and obtain authorization to access the data if necessary.
Obtain data access authorizations.
Reserve sufficient storage space for the project.
Get the data.
Convert the data to a format that can be easily manipulated (without changing the data itself).
Ensure that sensitive information is deleted or protected (e.g., anonymize the data).
Verify the size and type of data (time series, data sample, geo-positioning, etc.).
Set aside a test dataset, put it aside, and never look at it.

3. Explore the data to gain insights. (EDA: Exploratory Data Analysis)

Note: try to get insights from a domain expert for these steps.

Create a copy of the data for exploration (sampling it down to a manageable size if necessary).
Create a Jupyter Notebook to keep a record of the data exploration.
Study each attribute and its characteristics (Univariate analysis):
- Name
- Data type (categorical, int / float, bounded / unbounded, text, structured, etc.)
- Percentage (%) of missing values.
- Noise and type of noise (stochastic, outliers, rounding errors, etc.)
- Are they possibly useful for the project?
- Distribution type (Gaussian, uniform, logarithmic, etc.)
For supervised learning projects, identify the target attribute(s).
Visualize the data.
Study correlations between attributes (Bivariate analysis).
Study how to solve the problem manually.
Identify transformations that might be applicable.
Identify additional data that could be useful.
Document what you have learned.

Data Exploration Libraries

Pandas Profiling https://docs.profiling.ydata.ai/latest/
DataPrep https://dataprep.ai/
Mito https://www.trymito.io/
Dtale https://github.com/man-group/dtale
SweetViz https://github.com/fbdesignpro/sweetviz
AutoViz https://github.com/AutoViML/AutoViz
Bitrook Data cleaning https://www.bitrook.com/
dabl Simple autoeda with plot https://dabl.github.io/
Klib https://klib.readthedocs.io/
https://github.com/ml-tooling/best-of-ml-python?tab=readme-ov-file#data-visualization

4. Prepare the data

To better expose data patterns and use them with Machine Learning algorithms.

Notes:

Work on copies of the data (keep the original dataset intact).
Write functions for all data transformations you perform, for five reasons:
- So you can easily prepare the data the next time you obtain a new dataset
- So you can apply these transformations in future projects
- To clean and prepare the test dataset
- To clean and prepare new data instances once your solution is live (production)
- So it is easy to test different data preparation approaches as hyperparameters

Data cleaning:
- Remove duplicate data records (reduce the number of data points)
- Fix or remove outliers (optional).
- Outliers can be separated from the dataset depending on the project problem (e.g., anomaly detection).
- Fill in missing values (e.g., with zero, mean, median …) or drop the rows (or columns).
Feature Selection (optional):
- Drop features that do not provide useful information for the project.
- Remove duplicate records (after dropping features, some records may become identical)
Feature Engineering, where appropriate:
- Discretize continuous features.
- Decompose features into parts (e.g., categorical, date/time, etc.).
- Add promising feature transformations, for example:
  - log(x)
  - sqrt(x)
  - x^2
  - etc
- Apply functions to the data to add new features.
Feature Scaling:
- standardize
- normalize

Feature Engineering Libraries

Feature-engine https://feature-engine.trainindata.com/en/latest/
featuretools https://featuretools.alteryx.com/en/stable/

5. Select a model

Notes:

If you have a large amount of data, you may want to sample the data to have smaller training sets, so you can train several different models in a reasonable time (keep in mind that this penalizes complex models such as large neural networks or Random Forest).
Once again, try to automate these steps as much as possible.
Use Experiment Tracking tools to have traceability of models and their performance (e.g. MLFlow)

Train many quick models using standard parameters from different categories (e.g., linear, Naive Bayes, SVM, Random Forest, neural networks, etc.).
Measure and compare their performance.
- For each model, use N-fold cross validation (Cross validation) and compute the mean and standard deviation of the performance measure across the N evaluations.
Analyze the most significant variables for each algorithm.
Analyze the types of errors the models make.
- What data would a human have used to avoid these errors?
Quickly perform feature selection and feature engineering.
Perform one or two more quick iterations of the above five steps.
Shortlist the top three to five most promising models, preferring models that make different types of errors (diversity of errors).

6. Fine-tune the models

Notes:

You should use as much data as possible for this step, especially as you move toward the final stages of fine-tuning the model.
As always, automate what you can.

Fine-tune the hyperparameters (hyperparameter tuning) using cross validation (cross validation).
- Treat data transformation choices as hyperparameters, especially when you are not sure about them (for example, should you replace missing values with zero or with the mean value? Or simply drop the rows?).
- Unless there are very few hyperparameter values to explore, prefer random search over grid search. If training is very long, you may prefer a Bayesian optimization approach (for example, using Gaussian processes, as described by Jasper Snoek, Hugo Larochelle, and Ryan Adams¹).
Try ensemble methods. Combining your best models will often perform better than running them individually (better performance when there is diversity of errors among the models).
Once you are confident in your final model, measure its performance on the test set (set aside at the beginning) to estimate the generalization error.

Do not modify your model after measuring the generalization error: you would simply start overfitting the test set.

Model Hyperparameter Tuning Libraries

Ray-Tune https://docs.ray.io/en/latest/tune/index.html
optuna https://optuna.readthedocs.io/en/stable/
hyperopt http://hyperopt.github.io/hyperopt/
https://github.com/ml-tooling/best-of-ml-python?tab=readme-ov-file#hyperparameter-optimization--automl

7. Model Interpretability

Interpret the model obtained and identify its errors

Which features are most important?
How much does each feature contribute to the prediction?
What are the consequences of bad predictions?
What type of errors does the model make?
How can the errors be monitored?
What causes the errors?
- Outliers?
- Imbalanced class?
- Data entry errors?
- etc

Model Interpretability Libraries

Shap https://github.com/slundberg/shap
interpret https://github.com/interpretml/interpret
explainerdashboard https://explainerdashboard.readthedocs.io/
PiML https://github.com/SelfExplainML/PiML-Toolbox
Yellowbrick https://www.scikit-yb.org/en/latest/
Alibi Explain https://docs.seldon.io/projects/alibi/en/stable/
https://github.com/ml-tooling/best-of-ml-python?tab=readme-ov-file#model-interpretability

8. Present the solution

Document what you have done.
Create a good presentation.
- Make sure to highlight the big picture of the project or problem first.
Explain why the solution found achieves the desired objective.
Don’t forget to present interesting points noticed along the way.
- Describe what worked and what did not.
- List the assumptions and limitations of the system.
Make sure key findings are communicated through compelling visualizations or easy-to-remember statements (e.g., “median income is the number one predictor of housing prices”).

9. Deploy, monitor and maintain the system

Prepare the solution for production (connect production data inputs, write unit tests, etc.).
Write monitoring code to check the real-time performance of the system at regular intervals and trigger alerts when it drops or fails.
- Be careful about slow degradation: models tend to “rot” as data evolves, the model gradually loses validity over time.
- Performance measurement may require human supervision (e.g., through a crowdsourcing service).
- Monitor the quality of input data (e.g., a malfunctioning sensor sending random values, or another team’s data output becoming stale). This is particularly important for online learning systems.
Retrain your models regularly with fresh data (automate as much as possible); this is called Continuous Training and Continuous Deployment (CT/CD).
The area of process automation is called MLOps.

MLOps, Monitoring and Testing Libraries

Experiment tracking Libraries

MlFLow https://mlflow.org/
Weights & Biases https://github.com/wandb/wandb
Dvc Experiment Management https://dvc.org/doc/user-guide/experiment-management
Metaflow https://github.com/Netflix/metaflow
https://github.com/ml-tooling/best-of-ml-python?tab=readme-ov-file#workflow--experiment-tracking

Data and Model Monitoring and Testing Libraries

GreatExpectations Data quality https://greatexpectations.io/
Deepchecks Test Suites for ML Models & Data https://deepchecks.com/
evidentlyAI model monitoring https://evidentlyai.com/

MLOps and Orchestration Libraries

Kedro Modular, reproducible and maintainable data science code https://kedro.org/
ZenMl MLOps framework used to create production-ready ML pipelines https://zenml.io/
Dvc Version Control System for Machine Learning Projects https://dvc.org/
MlFLow Platform for the machine learning lifecycle https://mlflow.org/
KubeFlow Machine Learning Toolkit for Kubernetes https://www.kubeflow.org/
MetaFlow Build and manage real-life data science projects https://metaflow.org/
MLRun Machine-learning applications to production https://www.mlrun.org/

Code Testing Libraries

Pytest Unit tests https://docs.pytest.org/
Coverage Unit test coverage https://coverage.readthedocs.io/

References

“Practical Bayesian Optimization of Machine Learning Algorithms,” J. Snoek, H. Larochelle, R. Adams (2012) ↩︎

World Coronavirus (COVID-19) Data Visualization with Plotly

Tue, 17 Mar 2020 00:00:00 +0000

I have seen several visualizations of COVID-19 data on social media and wanted to create them in Python to have the charts updated every day, and also to practice using plotly for interactive visualization.

The data available mainly covers the number of infected and deceased people; I also want to visualize data on recovered people and active cases.

You can interact with the charts using the mouse, and the charts will update daily with new data!

Data extracted from the 2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE

https://github.com/CSSEGISandData/COVID-19

Since March 10, 2023, the Johns Hopkins Coronavirus Resource Center has stopped collecting and reporting global COVID-19 data.

Updates Made (Click)

25/May/2020 add recovered people data
29/May/2020 update to plotly 0.48
25/Sep/2020 World map of confirmed cases by country with choropleth
24/Nov/2022 Top 10 country progression
21/Oct/2023 update charts and libraries

1. Python Packages and Data

Python Packages

!pip install chart_studio -q

import pandas as pd
import plotly.express as px
import numpy as np
import chart_studio

To upload interactive plotly charts to chart studio

#chart-studio api
username = '' # your username
api_key = '' # your api api_key
chart_studio.tools.set_credentials_file(username=username, api_key=api_key)
import chart_studio.plotly as py

#print pandas, px an numpy version
print('pandas version: ', pd.__version__)
print('numpy version: ', np.__version__)

pandas version: 2.1.1
numpy version: 1.26.0

Import Data

confirmed = pd.read_csv('https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv')
death = pd.read_csv('https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv')
recovered = pd.read_csv('https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_recovered_global.csv')

CSSEGISandData/COVID-19 Data

Data description in English

Province/State: China - province name; US/Canada/Australia/ - city name, state/province name; Others - name of the event (e.g., “Diamond Princess” cruise ship); other countries - blank.

Country/Region: country/region name conforming to WHO (will be updated).

Last Update: MM/DD/YYYY HH:mm (24 hour format, in UTC).

Confirmed: the number of confirmed cases. For Hubei Province: from Feb 13 (GMT +8), we report both clinically diagnosed and lab-confirmed cases. For lab-confirmed cases only (Before Feb 17), please refer to who_covid_19_situation_reports. For Italy, diagnosis standard might be changed since Feb 27 to “slow the growth of new case numbers.”

Deaths: the number of deaths.

Recovered: the number of recovered cases.

confirmed.iloc[:5,:8]

	Province/State	Country/Region	Lat	Long
0	NaN	Afghanistan	33.93911	67.709953
1	NaN	Albania	41.15330	20.168300
2	NaN	Algeria	28.03390	1.659600
3	NaN	Andorra	42.50630	1.521800
4	NaN	Angola	-11.20270	17.873900

General DataFrame Information

confirmed.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 289 entries, 0 to 288
Columns: 1147 entries, Province/State to 3/9/23
dtypes: float64(2), int64(1143), object(2)
memory usage: 2.5+ MB

Countries with multiple data entries per Province/State

print(confirmed
 .loc[confirmed['Country/Region'].duplicated(keep=False),
 'Country/Region']
 .drop_duplicates()
 .unique()
 )

['Australia' 'Canada' 'China' 'Denmark' 'France' 'Netherlands'
'New Zealand' 'United Kingdom']

Sum the data for each country

def sumar_datos_region(df: pd.DataFrame) -> pd.DataFrame:
 """
 Suma los datos de cada región en el dataframe dado.
 Args:
 df (pd.DataFrame): El dataframe del cual se sumarán los datos.
 Returns:
 pd.DataFrame: Un nuevo dataframe con los datos sumados para cada región.
 """
 df = (df.groupby(['Country/Region'])
 .sum()
 .reset_index())
 return df

# aplicar la función a los tres dataframes
confirmed, death, recovered = (sumar_datos_region(df)
 for df
 in (confirmed, death, recovered))

Set Country/Region as the index of the dataframes


confirmed, death, recovered = (df.set_index('Country/Region')
 for df
 in (confirmed, death, recovered))

Recovered Cases Data

Click to view code

total_recuperados_dia = recovered.loc[:,:'8/15/21'] .sum(axis="index")
px.line(x=total_recuperados_dia.index,
 y=total_recuperados_dia.values,
 title='Numero de casos recuperados por dia')

The Recovered data is only available up to August 4, 2021

World Data by Day

Click to view code

column_names = ["Fecha", "Confirmados", "Recuperados","Muertos"]
world = pd.DataFrame(columns = column_names)
world["Fecha"] = confirmed.columns
world["Confirmados"] = confirmed.sum(axis='rows').values
world["Recuperados"] = recovered.sum(axis='rows').values
world["Muertos"] = death.sum(axis='rows').values
world["Activos"] = world["Confirmados"] - world["Recuperados"] - world["Muertos"]

2. Covid-19 in the World

Animated Evolution of Active Cases by Country

The animated chart of the temporal evolution of active cases by country was created using the Pandas alive and Bar Chart Race libraries.

Click to view code

import pandas_alive
active_evol = active_group.set_index('date')
active_evol.index = pd.to_datetime(active_evol.index)
active_evol.plot_animated(filename='evolucion_casos_activos.mp4', n_bars=8,n_visible=8,
 title='Evolución en el tiempo de Casos Activos COVID-19 por pais \n https://joserzapata.github.io/',
 perpendicular_bar_func='mean', dpi=300,
 period_label={'x': .99, 'y': .25, 'ha': 'right', 'va': 'center'},
 period_fmt='%B %d, %Y',
 period_summary_func=lambda v: {'x': .99, 'y': .18,
 's': f'Total Activos: {v.nlargest(8).sum():,.0f}',
 'ha': 'right', 'size': 9, 'family': 'Courier New'})

Visualization with Plotly

World Values of Confirmed, Active, Recovered, and Deceased Cases

Click to view code

fig = px.bar(world.iloc[-1][["Confirmados","Muertos"]],
 x = ["Confirmados","Muertos"], color = ["Confirmados","Muertos"],
 y = world.iloc[-1][["Confirmados","Muertos"]].values,
 text = world.iloc[-1][["Confirmados","Muertos"]].values,
 color_discrete_sequence=["navy","coral"],
 height=500, width=600,
 title='Total casos COVID-19 en el mundo al {}'.format(world.iloc[-1]['Fecha']),
 labels={'value':'Número de casos', 'variable':'Tipo de caso'})
fig.update_traces(textposition='outside')#poner los valores de las barras fuera
fig.add_annotation(x= 'Muertos', y=world["Confirmados"].max(), text='https://joserzapata.github.io/', showarrow=False)
fig.layout.update(showlegend=False,
 yaxis = {"title": {"text": "Numero de Personas"}}, # Cambiar texto eje y
 xaxis = {"title": {"text": ""}} #Esconder nombre eje x
 )
# grabar grafica en chart-studio si se proporciona el api-key
if api_key: py.plot(fig, filename = 'total_casos_general', auto_open=False)
fig.show()

World Map of Confirmed Cases by Country

Hover the mouse over the map to see information for each country

Click to view code

conf_max = confirmed.iloc[:,-1].copy()
conf_max = conf_max.to_frame().reset_index()
conf_max = conf_max.rename(columns = {'3/9/23':'Confirmados'})

fig = px.choropleth(conf_max, locations="Country/Region", locationmode='country names',
 color=np.log10(conf_max["Confirmados"]),hover_name="Country/Region",
 hover_data = ["Confirmados"],
 projection="natural earth", width=900,
 color_continuous_scale = px.colors.sequential.Jet,
 title='Mapa de Confirmados COVID 19 por Pais - 3/9/23')
fig.add_annotation(x=0.5, y=0,text='https://joserzapata.github.io/', showarrow=False)
fig.update(layout_coloraxis_showscale=False)
# grabar grafica en chart-studio si se proporciona el api-key
if api_key: py.plot(fig, filename = 'mapa_confirmados_pais', auto_open=False)
fig.show()

Confirmed vs Deceased by Country

Click to view code

max_fecha = "3/9/23"
death_max = death.iloc[:,-1].copy()
death_max = death_max.to_frame().reset_index()
death_max = death_max.rename(columns = {max_fecha:'Muertos'})

maxi_y = death_max["Muertos"].max()
maxi_x = conf_max["Confirmados"].max()

full_melt_max = pd.merge(conf_max[['Country/Region','Confirmados']],
 death_max[['Country/Region','Muertos']],
 on='Country/Region', how='left')

fig = px.scatter(full_melt_max.sort_values('Muertos', ascending=False).iloc[:15, :],
 x='Confirmados', y='Muertos', color='Country/Region',
 size='Confirmados', height=500,width=900,
 text='Country/Region', log_x=True, log_y=True,
 title= f'Muertos vs Confirmados - {max_fecha} - (15 Paises)')
fig.add_annotation(x=0.5, y=1, xref="paper",yref="paper",
 text='https://joserzapata.github.io/', showarrow=False)
fig.update_traces(textposition='top center')
fig.layout.update(showlegend = False)

# grabar grafica en chart-studio si se proporciona el api-key
if api_key: py.plot(fig, filename = 'scatter_muertos_confirmados', auto_open=False)
fig.show()

World Progression Over Time of Confirmed and Deceased Cases

Click to view code

fecha_datos_completos = '8/4/21'
pos_final = len(world.set_index('Fecha').loc[:fecha_datos_completos,:])
world = world.iloc[:pos_final,:]
world_melt = world.melt(id_vars='Fecha',
 value_vars= list(world.columns)[1:],
 var_name=None)
fig = px.line(world_melt, x="Fecha", y= 'value',
 color='variable', color_discrete_sequence=["teal","green","coral", "navy"],
 title=f'Total de Casos en el tiempo de COVID 19 - {fecha_datos_completos}')
for n in list(world.columns)[1:]:
 fig.add_annotation(x=world.iloc[-1,0], y=world.loc[world.index[-1],n],
 text=n, xref="x",yref="y",
 showarrow=True, ax=-50, ay=-20)
# Indicador de numero total de confirmados
fig.add_indicator( title= {'text':'Confirmados', 'font':{'color':'teal'}},
 value = world['Confirmados'].iloc[-1],
 mode = "number+delta", delta = {"reference": world['Confirmados'
 ].iloc[-2], 'relative': True },domain = {'x': [0, 0.25], 'y': [0.15, .4]})
#Indicador numero total de Activos
fig.add_indicator(title={'text':'Activos', 'font':{'color':'navy'}},
 value = world['Activos'].iloc[-1],
 mode = "number+delta", delta = {"reference": world['Activos'
 ].iloc[-2], 'relative': True },domain = {'x': [0, 0.25], 'y': [0.6, .85]})
#Indicador numero total de Recuperados
fig.add_indicator(title={'text':'Recuperados', 'font':{'color':'green'}},
 value = world['Recuperados'].iloc[-1],
 mode = "number+delta", delta = {"reference": world['Recuperados'
 ].iloc[-2], 'relative': True },domain = {'x': [0.25, 0.50], 'y': [0.6, .85]})
#Indicador numero total de muertos
fig.add_indicator(title={'text':'Muertos', 'font':{'color':'coral'}},
 value = world['Muertos'].iloc[-1],
 mode = "number+delta", delta = {"reference": world['Muertos'
 ].iloc[-2], 'relative': True },domain = {'x': [0.25, 0.5], 'y': [0.15, .4]})
fig.add_annotation(x=400, y=world_melt['value'].max(),
 text='https://joserzapata.github.io/', showarrow=False)
fig.layout.update(showlegend = False,
 yaxis = {"title": {"text": "Numero de Personas"}}, # Cambiar texto eje y
 )
# grabar grafica en chart-studio si se proporciona el api-key
if api_key: py.plot(fig, filename = 'total_casos_serie', auto_open=False)
fig.show()

Total Confirmed COVID-19 Cases by Country (Top 10)

Click to view code

df1 = confirmed.copy()
fecha = confirmed.columns[-1] #obtener la fecha del ultimo dato
paises = df1.iloc[:,-1].copy() #obtener la serie sin el primer dato, fecha
paises.sort_values(ascending=False, inplace=True)
top = 10
#keep top countries
df1 = df1.loc[paises[:top].index.to_list(),:]
df1 = df1.T

if api_key:
 # se toman la serie de tiempo cada 7 dias, por que las graficas
 # grandes no se pueden subir a chart-studio con subscripcion gratuita
 df1 = df1.iloc[::-7].iloc[::-1]

fig = px.line(df1, x=df1.index, y=df1.columns, color='Country/Region',
 color_discrete_sequence=px.colors.qualitative.G10, width=900,
 hover_name='Country/Region',
 title=f'Total Casos Confirmados de COVID 19 por Pais (Top 10) - {world.iloc[-1,0]}')
# top paises mas infectados

mas_infectados=[]
for n in range(top):
 fig.add_annotation(x=fecha, y=paises.iloc[n], text=paises.index[n],
 showarrow=True, ax=+45, xref="x",yref="y")
 mas_infectados.append(paises.index[n])

fig.layout.update(showlegend=False,
 yaxis = {"title": {"text": "Numero de Personas"}}, # Cambiar texto eje y
 xaxis = {"title": {"text": "Fecha"}} #Esconder nombre eje x
 )
fig.add_annotation(x=200, y=df1.max().max(),
 text='https://joserzapata.github.io/', showarrow=False)
# grabar grafica en chart-studio si se proporciona el api-key
if api_key: py.plot(fig, filename = 'total_casos_no_china', auto_open=False)
fig.show()

Animated Map of the Temporal Evolution of COVID-19

Hover the mouse over the map to see information for each country. Press the play button to see the animation.

Click to view code

# confirmed data frame en formato wide
confirmed = confirmed.T
confirmed.reset_index(inplace=True)
confirmed.rename(columns={'index':'Fecha'}, inplace=True)
confirmed_melt = confirmed.melt(id_vars="Fecha").copy()
confirmed_melt.rename(columns = {'value':'Confirmados'}, inplace = True)

if api_key:
 # se toman la serie de tiempo cada 18 dias, por que las graficas
 # grandes no se pueden subir a chart-studio con subscripcion gratuita
 confirmed_melt = confirmed.iloc[::-30].iloc[::-1].melt(id_vars="Fecha").copy()
 confirmed_melt.rename(columns = {'value':'Confirmados'}, inplace = True)

confirmed_melt['Fecha'] = pd.to_datetime(confirmed_melt['Fecha'], format='%m/%d/%y')
confirmed_melt['size'] = confirmed_melt['Confirmados'].pow(0.3)
confirmed_melt.dropna(inplace=True) #eliminar filas con valores faltantes

fig = px.scatter_geo(confirmed_melt, locations="Country/Region", locationmode='country names',
 color="Confirmados", size='size', hover_name="Country/Region",
 range_color= [0, max(confirmed_melt['Confirmados'])+2],
 projection="natural earth", animation_frame="Fecha",
 title='Contagiados COVID 19 en el Tiempo')
fig.update(layout_coloraxis_showscale=False)
fig.add_annotation(x=0.5, y=-0.1,text='https://joserzapata.github.io/', showarrow=False)
# grabar grafica en chart-studio si se proporciona el api-key
if api_key: py.plot(fig, filename = 'mapa_evolucion_temporal', auto_open=False)
fig.show()

3. Covid-19 in Colombia

Number of COVID-19 Cases in Colombia (Through August 4, 2021)

Click to view code

column_names = ["Fecha", "Confirmados", "Recuperados","Muertos"]
colombia = pd.DataFrame(columns = column_names)
colombia["Fecha"] = confirmed["Fecha"].values
colombia["Confirmados"] = confirmed["Colombia"].values
colombia["Recuperados"] = recovered.loc["Colombia"].values
colombia["Muertos"] = death.loc["Colombia"].values
colombia["Activos"] = colombia["Confirmados"] - colombia["Recuperados"] - colombia["Muertos"]

fecha_datos_completos = '8/4/21'
pos_final = len(colombia.set_index('Fecha').loc[:fecha_datos_completos,:])
colombia = colombia.iloc[:pos_final,:]


df_melt3 = colombia.melt(id_vars='Fecha', value_vars= list(colombia.columns)[1:], var_name=None)
fig = px.line(df_melt3, x='Fecha' , y='value', color='variable',
 color_discrete_sequence=["teal","green","coral", "navy"],
 title=f'Corona virus (COVID 19) en Colombia - {colombia.iloc[-1,0]}')
# Indicador de numero total de confirmados
fig.add_indicator( title= {'text':'Confirmados', 'font':{'color':'teal'}},
 value = colombia['Confirmados'].iloc[-1],
 mode = "number+delta", delta = {"reference": colombia['Confirmados'
 ].iloc[-2], 'relative': True },domain = {'x': [0, 0.25], 'y': [0.15, .4]})
#Indicador numero total de Activos
fig.add_indicator(title={'text':'Activos', 'font':{'color':'navy'}},
 value = colombia['Activos'].iloc[-1],
 mode = "number+delta", delta = {"reference": colombia['Activos'
 ].iloc[-2], 'relative': True },domain = {'x': [0, 0.25], 'y': [0.6, .85]})
#Indicador numero total de Recuperados
fig.add_indicator(title={'text':'Recuperados', 'font':{'color':'green'}},
 value = colombia['Recuperados'].iloc[-1],
 mode = "number+delta", delta = {"reference": colombia['Recuperados'
 ].iloc[-2], 'relative': True },domain = {'x': [0.25, 0.50], 'y': [0.6, .85]})
#Indicador numero total de muertos
fig.add_indicator(title={'text':'Muertos', 'font':{'color':'coral'}},
 value = colombia['Muertos'].iloc[-1],
 mode = "number+delta", delta = {"reference": colombia['Muertos'
 ].iloc[-2], 'relative': True },domain = {'x': [0.25, 0.5], 'y': [0.15, .4]})
fig.add_annotation(x=140, y=df_melt3['value'].max(),
 text='https://joserzapata.github.io/', showarrow=False)
fig.layout.update(showlegend=False,
 yaxis = {"title": {"text": "Numero de Personas"}}, # Cambiar texto eje y
 xaxis = {"title": {"text": "Fecha"}})
# grabar grafica en chart-studio si se proporciona el api-key
if api_key: py.plot(fig, filename = 'Colombia_general', auto_open=False)
fig.show()

Chart Updates

The charts created with plotly were uploaded to chart-studio and embedded in the web page using the HTML iframe tag. Previously, the charts were updated every 24 hours using Github Actions, but since the data source stopped being updated, it is no longer necessary to update the charts.

Jupyter Notebook Source Code

References

Data sources, visualizations, and data analysis.

PySpark with Google Colab

Mon, 09 Mar 2020 23:00:43 -0500

PySpark on Google Colab — Automatic Setup

Installation March/2020
Automatic Installation
1. Java Installation
2. Spark Installation
3. PySpark Usage Example

Open in Google Colab:

Quick Installation March/2020

In general, to use PySpark in Colab as of March/2020, the following commands would be used in a Colab cell:

Install Java

!apt-get install openjdk-8-jdk-headless -qq > /dev/null

import os # operating system management library
os.system("wget -q https://www-us.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz")
os.system("tar xf /spark-2.4.5-bin-hadoop2.7.tgz")

Install PySpark

!pip install -q pyspark

# Environment Variables
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = f"/content/{ver_spark}-bin-hadoop2.7"

# Load PySpark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Test_spark").master("local[*]").getOrCreate()
spark

However, when a new version of Spark is released, the download links will need to be updated, since the 2.x.x versions are always removed when a new one comes out.

The best approach is to configure the setup automatically so that it downloads the version greater than 2.3.4 (the previous one) and less than Spark 3.0.0 (which was still in development at the time).

To do this, the following code detects the current version of Spark, downloads it, extracts it, and then installs Spark on Google Colab.

Automatic Installation

Java Installation

Google Colaboratory runs in a Linux environment, so Linux shell commands can be used preceded by the ‘!’ character.

!apt-get install openjdk-8-jdk-headless -qq > /dev/null

Spark Installation

Automatically get the latest version of Spark from

from bs4 import BeautifulSoup
import requests

#Get the spark versions from the web page
url = 'https://downloads.apache.org/spark/'
r = requests.get(url)
html_doc = r.text
soup = BeautifulSoup(html_doc)

# read the web page and get the available spark versions
link_files = []
for link in soup.find_all('a'):
 link_files.append(link.get('href'))
spark_link = [x for x in link_files if 'spark' in x]
print(spark_link)

[‘spark-2.3.4/’, ‘spark-2.4.5/’, ‘spark-3.0.0-preview2/’]

The version to use will be those above spark-2.3.4 and below spark-3.0.0.

Get the version and remove the trailing ‘/’ character

ver_spark = spark_link[1][:-1] # get the version and remove the trailing '/' character
print(ver_spark)

spark-2.4.5

import os # operating system management library
#automatically install the desired version of spark
link = "https://www-us.apache.org/dist/spark/"
os.system(f"wget -q {link}{ver_spark}/{ver_spark}-bin-hadoop2.7.tgz")
os.system(f"tar xf {ver_spark}-bin-hadoop2.7.tgz")

# install pyspark
!pip install -q pyspark

|████████████████████████████████| 217.8MB 63kB/s
|████████████████████████████████| 204kB 53.8MB/s
Building wheel for pyspark (setup.py) ... done

Set environment variables

os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = f"/content/{ver_spark}-bin-hadoop2.7"

Load PySpark into the system

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Test_spark").master("local[*]").getOrCreate()
spark

SparkSession - in-memory

SparkContext

Version: v2.4.5
Master: local[*]
AppName: pyspark-shell

PySpark Usage Example

Read a test file

archivo = './sample_data/california_housing_train.csv'
df_spark = spark.read.csv(archivo, inferSchema=True, header=True)

# print file type
print(type(df_spark))

<class 'pyspark.sql.dataframe.DataFrame'>

Number of records in the dataframe?

df_spark.count()

Dataframe schema

df_spark.printSchema()

root
|-- longitude: double (nullable = true)
|-- latitude: double (nullable = true)
|-- housing_median_age: double (nullable = true)
|-- total_rooms: double (nullable = true)
|-- total_bedrooms: double (nullable = true)
|-- population: double (nullable = true)
|-- households: double (nullable = true)
|-- median_income: double (nullable = true)
|-- median_house_value: double (nullable = true)

Dataframe column names?

df_spark.columns

['longitude',
'latitude',
'housing_median_age',
'total_rooms',
'total_bedrooms',
'population',
'households',
'median_income',
'median_house_value']

View the first 20 records of the dataframe

df_spark.show()

+---------+--------+------------------+-----------+--------------+----------+----------+-------------+------------------+
|longitude|latitude|housing_median_age|total_rooms|total_bedrooms|population|households|median_income|median_house_value|
+---------+--------+------------------+-----------+--------------+----------+----------+-------------+------------------+
| -114.31| 34.19| 15.0| 5612.0| 1283.0| 1015.0| 472.0| 1.4936| 66900.0|
| -114.47| 34.4| 19.0| 7650.0| 1901.0| 1129.0| 463.0| 1.82| 80100.0|
| -114.56| 33.69| 17.0| 720.0| 174.0| 333.0| 117.0| 1.6509| 85700.0|
| -114.57| 33.64| 14.0| 1501.0| 337.0| 515.0| 226.0| 3.1917| 73400.0|
| -114.57| 33.57| 20.0| 1454.0| 326.0| 624.0| 262.0| 1.925| 65500.0|
| -114.58| 33.63| 29.0| 1387.0| 236.0| 671.0| 239.0| 3.3438| 74000.0|
| -114.58| 33.61| 25.0| 2907.0| 680.0| 1841.0| 633.0| 2.6768| 82400.0|
| -114.59| 34.83| 41.0| 812.0| 168.0| 375.0| 158.0| 1.7083| 48500.0|
| -114.59| 33.61| 34.0| 4789.0| 1175.0| 3134.0| 1056.0| 2.1782| 58400.0|
| -114.6| 34.83| 46.0| 1497.0| 309.0| 787.0| 271.0| 2.1908| 48100.0|
| -114.6| 33.62| 16.0| 3741.0| 801.0| 2434.0| 824.0| 2.6797| 86500.0|
| -114.6| 33.6| 21.0| 1988.0| 483.0| 1182.0| 437.0| 1.625| 62000.0|
| -114.61| 34.84| 48.0| 1291.0| 248.0| 580.0| 211.0| 2.1571| 48600.0|
| -114.61| 34.83| 31.0| 2478.0| 464.0| 1346.0| 479.0| 3.212| 70400.0|
| -114.63| 32.76| 15.0| 1448.0| 378.0| 949.0| 300.0| 0.8585| 45000.0|
| -114.65| 34.89| 17.0| 2556.0| 587.0| 1005.0| 401.0| 1.6991| 69100.0|
| -114.65| 33.6| 28.0| 1678.0| 322.0| 666.0| 256.0| 2.9653| 94900.0|
| -114.65| 32.79| 21.0| 44.0| 33.0| 64.0| 27.0| 0.8571| 25000.0|
| -114.66| 32.74| 17.0| 1388.0| 386.0| 775.0| 320.0| 1.2049| 44000.0|
| -114.67| 33.92| 17.0| 97.0| 24.0| 29.0| 15.0| 1.2656| 27500.0|
+---------+--------+------------------+-----------+--------------+----------+----------+-------------+------------------+
only showing top 20 rows

Statistical Description of the Dataframe

df_spark.describe().toPandas().transpose()

	0	1	2	3	4
summary	count	mean	stddev	min	max
longitude	17000	-119.56210823529375	2.0051664084260357	-124.35	-114.31
latitude	17000	35.6252247058827	2.1373397946570867	32.54	41.95
housing_median_age	17000	28.58935294117647	12.586936981660406	1.0	52.0
total_rooms	17000	2643.664411764706	2179.947071452777	2.0	37937.0
total_bedrooms	17000	539.4108235294118	421.4994515798648	1.0	6445.0
population	17000	1429.5739411764705	1147.852959159527	3.0	35682.0
households	17000	501.2219411764706	384.5208408559016	1.0	6082.0
median_income	17000	3.883578100000021	1.9081565183791036	0.4999	15.0001
median_house_value	17000	207300.91235294117	115983.76438720895	14999.0	500001.0

Statistical description of a single column (‘median_house_value’)

df_spark.describe(['median_house_value']).show()

+-------+------------------+
|summary|median_house_value|
+-------+------------------+
| count| 17000|
| mean|207300.91235294117|
| stddev|115983.76438720895|
| min| 14999.0|
| max| 500001.0|
+-------+------------------+

This is how you can automatically install Spark on Google Colab and use it for free.

The free version only provides one CPU; if you want to increase processing capacity, you need to pay.

Social Media Behaviour with exclusive Facebook data

Fri, 14 Feb 2020 17:09:59 -0500

This Research Project combines exclusive Facebook data(Condor Dataset, Crowdtangle, Ad Library) and public data to quantitatively (Coordinated link sharing behaviour analysis) and qualitatively (Content Analysis) analyze the Social Media behavior to determine if there is coordinated inauthentic behavior. We use TeleSUR’s English language media operations and its partners as a study case.

This project in founded by the Social Media and Democracy Research Grants from the Social Science Research Council and access to Facebook data via Social Science One.

the Social Media and Democracy Research Grants support research that examines the impact of social media and related digital technologies on democracy and elections.

https://upb-ss1.github.io/

Comprensión de la música con Python, una mirada desde la Física y la Matemática

Fri, 07 Feb 2020 12:35:00 -0500

ACMUS - Advancing Computational Musicology: Semi-supervised and Unsupervised Segmentation and Annotation of Musical Collections

Fri, 01 Nov 2019 19:28:09 +0800

Aplicaciones de Audio con Machine Learning

Tue, 02 Oct 2018 18:00:00 -0500

Audio Production Tutorial with Reaper

Sat, 01 Sep 2018 09:10:53 -0500

Video tutorials on using Reaper for audio production, the videos were created to support the Audio course in the Digital Entertainment Design Engineering program at the Universidad Pontificia Bolivariana.

Download REAPER for Windows, macOS or Linux.

Tutorial playlist on YouTube

1. Introduction to Reaper and Editing

Introduction to the configuration and management of REAPER for audio production.

Audio files to Insert

Image to Insert

Video to Insert

2. Channel Management and Sends with Reaper

Complete Reaper session example

3. Automations and Envelopes in Reaper

Volume, Panning and Effects automation using envelopes in Reaper

Audio files for Automation Practice

Completed Reaper Automation Project

4. Mixing and Mastering in Reaper

Basic principles of audio mixing and mastering in Reaper

Audio files for mixing and mastering practice

Final Reaper session with mixing and mastering

Tips for the Undergraduate Thesis Presentation

Wed, 01 Aug 2018 09:10:53 -0500

This is a summary of some concrete ideas I have collected from my experience as a teacher, researcher, and supervisor of several undergraduate and graduate final projects, to help deliver a strong presentation of the thesis proposal and the final thesis.

Tips for Creating the Slides

Use large font
Little text, only write what is essential and important
If the graphics are too small to be seen (Do not use them)
If the text you are about to add is too small (Crop it until it looks large or remove it)
Use colors from the same color palette as the template
You deliver the presentation, not the slides
The evaluators should be focused on the presenter and only look at the slides when the presenter points to them or explains something on them as a visual aid
Do not read the slides
Be direct
Remember the Zen of Python
Enjoy the presentation! (This is the most important one)

Thesis Proposal Presentation

This can be used for any type of thesis project. The rule for the number of slides is 1 slide per 1 minute of presentation time.

Presentation time = 15 Minutes (Between 15-17 slides)

Slides

Title, student and supervisor information
Context
Problem statement (a single paragraph)
Justification
Conceptual Framework (1/2)
Conceptual Framework (2/2)
State of the Art (1/2)
State of the Art (2/2)
Legal Framework
General Objective (a single paragraph)
Specific Objectives (Include Deliverables) (1/2)
Specific Objectives (Include Deliverables) (2/2)
Methodology
Timeline
Budget
Bibliography

Evaluation Questions

Answer these questions to be clear about what the jury wants to know

Title:
- Is the title clear and consistent with the topic and problem stated?
Problem:
- Is the problem and the context in which it occurs clearly described?
- Are the causes of the problem clearly described?
- Is sufficient valid evidence presented for the existence of the problem and its causes?
Justification:
- Are the benefits and the impact of carrying out the project clearly described?
Conceptual Framework:
- Does the conceptual framework recognize the concepts related to the categories developed in the proposal?
- Is the conceptual framework sufficiently supported by bibliographic and documentary sources?
State of the Art:
- Does the state of the art reflect an understanding of the field, at least at the national level with reference to the international context?
- Is the state of the art sufficiently supported by bibliographic and documentary sources?
Legal Framework:
- Are the national and international regulations and/or standards related to the project’s topic examined?
General Objective:
- Does the objective include an infinitive verb and clearly present what, how, and why the project is carried out?
- Is the project title clearly related to the objective?
- Does the objective offer a solution to the problem referenced in the project?
Specific Objectives:
- Does each objective include an infinitive verb and clearly state what is to be accomplished?
- Does each deliverable from the specific objectives contribute significantly to achieving the general objective, and do all of them together accomplish the general objective?
- Are the objectives presented in chronological order?
Methodology:
- Is there a close relationship between the specific objectives and the methodology phases?
- Are the project phases and the activities needed to meet the objectives described?
- Are the techniques, methods, tools, and instruments to be used in each phase described clearly and coherently, adding rigor to the work?
- Are the project deliverable(s) the result of applied research or a case study?
Timeline:
- Is the proposed timeline clear, consistent with the methodology, and does it propose a duration of 6 months?

Final Thesis Presentation

The slides explaining the methodology used (13-18) are in this case an example using the CRISP methodology used in data science.

Presentation time = 30 Minutes (Maximum limit of 30 slides)

Slides

This is a tentative outline of the slides that can be produced; some points require more slides, such as the study and understanding of the data and data analysis (Use tables with large numbers and large charts).

Title, student and supervisor information
Context
Problem statement (a single paragraph)
Justification
Conceptual Framework (1/2)
Conceptual Framework (2/2)
Legal Framework
State of the Art (1/2)
State of the Art (2/2)
General Objective (a single paragraph)
Specific Objectives (1/2)
Specific Objectives (2/2)
Methodology (CRISP, only show the diagram and do not spend too much time on this)
a. Business/Question Understanding
b. Data study and understanding (What characteristics do the data have?)
c. Data analysis and feature selection (What did you find particularly noteworthy in the data?)
d. Modeling (Which models were used and why?)
e. Evaluation (Which evaluation metric was chosen and why?)
Final Results (Which model was selected and what are the most important features and why?)
Deliverables obtained (evidence)
Conclusions
References

Evaluation Questions

Answer these questions

General Objective:
- Was the objective achieved 100%?
- What difficulties were encountered?
Specific Objectives:
- How was each specific objective accomplished and what problems arose during development?
- What deliverables were obtained?
Methodology:
- How was each stage of the Methodology developed?
- What problems arose and how were they solved?
Deliverables:
- What deliverables were obtained from the project?
Conclusions:
- What conclusions are drawn from the analysis of the results?
- What lessons did the development of this project teach you?

References

http://retractionwatch.com/2016/11/30/18-tips-giving-horrible-presentation/
Even a Geek Can Speak https://www.amazon.com/-/es/Joey-Asher/dp/0978577604

Audio signal analysis with python

Fri, 09 Feb 2018 11:40:35 -0500

Tempo Estimation

Thu, 17 Nov 2016 18:14:05 -0500

Sistemas Automaticos para Extraccion de Información Musical (Generalidades y Aplicaciones)

Fri, 22 Jul 2016 14:00:00 -0500

Multi-Feature Beat Tracking

Thu, 03 Apr 2014 11:52:17 -0500

Essentia: an open source library for audio analysis

Fri, 21 Mar 2014 00:11:35 -0500

Multifeature Beat tracker

Sat, 04 Jan 2014 23:00:38 -0500

Essentia: An Audio Analysis Library for Music Information Retrieval

Sun, 03 Nov 2013 23:32:53 -0500

Essentia: an open-source library for sound and music analysis

Mon, 21 Oct 2013 00:11:35 -0500

Comparative evaluation and combination of automatic rhythm description systems

Fri, 13 Sep 2013 14:49:50 +0100

Using voice suppression algorithms to improve beat tracking in the presence of highly predominant vocals

Sun, 26 May 2013 12:42:45 -0500

Selective Sampling for Beat Tracking Evaluation

Sat, 03 Nov 2012 12:16:05 -0500

Assigning a confidence threshold on automatic beat annotation in large datasets

Mon, 08 Oct 2012 19:38:56 -0500

On the automatic identification of difficult examples for beat tracking: towards building new evaluation datasets

Sun, 25 Mar 2012 19:44:33 -0500

Comparative Evaluation and Combination of Audio Tempo Estimation Approaches

Fri, 22 Jul 2011 12:33:35 -0500

Efficient Detection of Exact Redundancies in Audio Signals

Wed, 01 Oct 2008 22:34:53 -0500

Control de Rango Dinamico en Audio con Logica Difusa

Sun, 25 Jul 2004 13:40:24 -0500

Jose R. Zapata | Jose Ricardo Zapata

Procesamiento de Lenguaje Natural (NLP)

Practical Data Science Project with Python

Project Purpose

The dataset

Contents

Practical Data Science Project with Python

Project Purpose

The dataset

Contents

Univariate Data Analysis

📚 Import libraries

💾 Load data

📊 Data description

General data information

General statistics of the data set

Numerical variables

Categorical variables

📈 Univariate Analysis

Target Variable

Numerical Variables

age

sibsp

parch

fare

Categorical Variables

pclass

Sex

embarked

String columns

📊 Analysis of Results and Conclusions

Dataset Overview

Target Variable: survived

Numerical Variables

Age

SibSp (Siblings/Spouses aboard)

Parch (Parents/Children aboard)

Fare

Categorical Variables

Pclass (Passenger Class)

Sex

Embarked (Port of Embarkation)

Key Takeaways

📖 References

Bivariate Data Analysis

📚 Import libraries

💾 Load data

📊 Data description

General data information

General statistics of the data set

Numerical variables

Categorical variables

📈 Bivariate Analysis

Target vs Numerical Variables

survived vs age

survived vs sibsp

survived vs parch

survived vs fare

Target vs Categorical Variables

survived vs pclass

survived vs sex

survived vs embarked

Numerical vs Numerical Variables

age vs sibsp

age vs parch

age vs fare

sibsp vs parch

sibsp vs fare

parch vs fare

Categorical vs Categorical Variables

pclass vs sex

pclass vs embarked

sex vs embarked

sex vs pclass

embarked vs pclass

embarked vs sex

Categorical vs Numerical Variables

pclass vs age

pclass vs sibsp

pclass vs parch

Target Variable: `survived`

General configuration — `settings.json`

Debugging configuration — `launch.json`

Recommended extensions — `extensions.json`