INFS4203/7203 Data Mining Tutorial Week 2 Semester 2, 2023
Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit
INFS4203/7203 Data Mining
Tutorial Week 2 - Introduction to Classification
Semester 2, 2023
Question 1/4: Imputation of missing feature values and feature value normalization
Consider a dataset in Table 1.
1. Perform missing value imputation by all-class and class-specific values!
2. Perform max-min feature normalization!
3. Perform z-score feature normalization!
4. (Extra) Does it make sense to apply normalization on categorical (nominal) features? Justify your opinions!
Table 1: A dataset of 6 data points. The right most column is the target classification label.
Index Age Home Distance Browser Caffeine Level Hair Style
|
1 |
17 |
100 |
Chrome |
High |
Long |
|
2 |
20 |
50 |
Chrome |
Medium |
Long |
|
3 |
null |
750 |
Firefox |
High |
Long |
|
4 |
27 |
1000 |
Chrome |
Low |
Short |
|
5 |
26 |
9500 |
Safari |
Low |
Short |
|
6 |
24 |
7000 |
Firefox |
null |
Short |
Question 2/4: Classifier evaluation: accuracy and F1-score on balanced datasets
Consider the following true (ground-truth) class labels y and predicted class labels y(ˆ)a and y(ˆ)b from classifiers A and B, respectively, for 6 data points (instances).
y = [1, 1, 1, 0, 0, 0],
y(ˆ)a = [0, 1, 0, 0, 0, 1],
y(ˆ)b = [0, 0, 1, 1, 1, 0].
1. Calculate the accuracy and F1-score of Classifier A!
2. Calculate the accuracy and F1-score of Classifier B!
3. Which classifier is the best based on their accuracy scores?
4. Which classifier is the best based on their F1-scores?
Question 3/4: Classifier evaluation: accuracy and F1-score on imbalanced datasets
1. Consider a cancer dataset, where 85 people have cancer and the remaining 201 do not. Clas- sifier α always predicts that someone has cancer. Classifier β always predicts that someone does not have cancer. Compare these two degenerated classifiers α and β in terms of their accuracy and F1-scores! Which classifier is better (more preferable) for this domain?
2. Consider a city of 1 million people, where there are 10 COVID-positive individuals. A clas- sifier predicts that there are 1000 COVID-positive individuals (including 10 individuals who are indeed COVID-positive), and the remaining 999,000 are COVID-negative. Analyze this classifier in terms of its accuracy and F1-score!
(Extra) Question 4/4: Classification pipeline using cross-validation (CV)
Re-arrange the following 20 steps into a correct order of a classification pipeline with 3-fold CV. There may be multiple correct orders out of 20! possible orders.
The nomenclature is as follows.
. c( · ): a trained classifier model with some arguments inside the brackets.
. a( · ): an accuracy score with some arguments inside the brackets.
. mm: max-min normalization technique or the corresponding result
. zs: z-score normalization technique or the corresponding result
. avg: average, the averaging operation or the corresponding result
1. Validate the trained classifier c(mm,1, 3) using fold-2 to obtain its accuracy a(mm,2).
2. Train a classifier with max-min normalization using fold-1 and fold-2. Call the resulting trained classifier as c(mm,1, 2).
3. Validate the trained classifier c(mm,1, 2) using fold-3 to obtain its accuracy a(mm,3).
4. Validate the trained classifier c(zs,1, 3) using fold-2 to obtain its accuracy a(zs,2).
5. Train a classifier with the best hyperparameter value on training set (900 data points). Call the resulting trained classifier as c(zs). The training accuracy a(zs,train) may be obtained.
6. Train a classifier with max-min normalization using fold-2 and fold-3. Call the resulting trained classifier as c(mm,2, 3).
7. Compare the average validation accuracy: a(zs,avg) versus a(mm,avg).
Say, a(zs,avg) > a(mm,avg). Thus, the best hyperparameter value is z-score normalization.
8. Validate the trained classifier c(mm,2, 3) using fold-1 to obtain its accuracy a(mm,1).
9. Split the dataset into training and testing sets. There 1000 data points in total, therefore 900 training data points and 100 testing data points.
10. Train a classifier with max-min normalization using fold-1 and fold-3. Call the resulting trained classifier as c(mm,1, 3).
11. Average the validation accuracy of 3 classifiers with max-min normalization. That is, a(mm,avg) = [a(mm,1) + a(mm,2) + a(mm,3)]/3.
12. Test the trained classifier c(zs) on testing set (100 data points) to obtain its the test accuracy
a(zs,test). This a(zs,test) indicates the (approximate) accuracy on unseen data.
13. Validate the trained classifier c(zs,1, 2) using fold-3 to obtain its accuracy a(zs,3).
14. Train a classifier with z-score normalization using fold-2 and fold-3. Call the resulting trained classifier as c(zs,2, 3).
15. Split the training set (900 data points) into 3 folds equally. Those 3 folds are called fold-1, fold-2 and fold-3. There are 300 data points per fold.
16. Validate the trained classifier c(zs,2, 3) using fold-1 to obtain its accuracy a(zs,1).
17. Determine a set of hyperparameter values to tune. Here, there is one hyperparameter, i.e. the normalization types, namely max-min versus z-score normalizations.
18. Train a classifier with z-score normalization using fold-1 and fold-3. Call the resulting trained classifier as c(zs,1, 3).
19. Train a classifier with z-score normalization using fold-1 and fold-2. Call the resulting trained classifier as c(zs,1, 2).
20. Average the validation accuracy of 3 classifiers with z-score normalization. That is, a(zs,avg) = [a(zs,1) + a(zs,2) + a(zs,3)]/3.
2023-09-04
Introduction to Classification