The following table shows statistics for different datasets:

(a) CNN Architecture (Basic: 10%)

Explain convolutional neural networks' architecture and the operational mechanism by performing the following tasks. You answer to this question should be no more than 2 pages.

1.   Describe CNN components in your own words: 1) the fully connected layer, 2) the convolutional layer, 3) the max pooling layer, 4) the activation function, and 5) the softmax function. What are the functions of these components?

2.   What is the over-fitting issue in model learning? Explain any technique that has been used in CNN training to avoid over-fitting.

3.   Explain the difference between three activation functions: ReLU, Leaky ReLU, and ELU.

4.   Read official documents of three loss functions: the L1 loss, the MSE loss, and the BCE loss. List

applications where those losses are used and state why they are used in those specific cases.

Show your understanding as much as possible in your own words in your report.

(b) Compare classification performance on different datasets (30%)

Train the CNN given in Fig. 1 using the training images of MNIST, then test the trained network on the testing images of MNIST. Compute and draw the accuracy performance curves (epoch-accuracy plot) on training and test datasets on the same figure. You can adopt proper preprocessing techniques and random network initialization to simplify your training.

1.   Plot the performance curves under three different yet representative hyper-parameter settings (optimizers, initialization of filter weights, learning rate, decay, etc.). Discuss your observations and the effect of different settings.

2.   Find the best parameter setting to achieve the highest accuracy on the test set. Then, plot the performance curves for the test and training sets under this setting. Your testing accuracy should be no less than 99%.

3.   Repeat 2 for Fashion-MNIST. Your best testing accuracy should be at least 90%.

4.   Repeat 2 for CIFAR- 10. Your best testing accuracy should be at least 65%.

5.   Compare your best performances on three datasets. How do they differ, and why do you think such a difference exists?

Note: for each setting, you need five runs. Report the [best test accuracy among five runs, mean test accuracy of 5 runs,  standard deviation of test accuracy among five runs] to evaluate the performance.

(c) Evaluation and Ablation Study (30%)

Note: you may use the best setting you found in Problem 1(b) on each dataset.

1.   (Confusion Matrix) Generate the normalized confusion matrix for the 10 classes on the testing set. What are the top three confused pairs of classes? Show one example for each of these three pairs. Do this for all the datasets in 1(b). Describe your observations and explain.

2.   (One-vs-Rest multiclass ROC) The final softmax function will yield a confidence score for each class; they can be used to compute the ROC curve in a one-vs-rest manner. For CIFAR- 10, plot the ROC curve for each class and discuss its difference against accuracy.  (See reference [7])

3.   The area under the ROC Curve (AUC) is another important metric to evaluate the classifier. It computes  the  area  under  the  ROC   curve,  which  gives  numerical  values  to   compare  the performance. Compute the AUC for CIFAR- 10, which is defined as the weighted sum of AUC values for all One-vs.-Rest multiclass ROC’s AUC.

(d) Classification with noisy data (15%)

Data in real-world applications could be noisy with wrong labels. Symmetric Label Noise (SLN) is the type of labeling noise where a% of the data with the rue label of class i is labeled as other classes j ≠ i with uniform probability. For example, in a 3-class classification problem, the normalized confusion matrix between the true label and the noisy label is close to the following format, where epsilon is the noise level (say, 40%):

Now you’d like to synthesize the Symmetric Label Noise on the training set of Fashion-MNIST and investigate the performance of neural networks under different noise levels.

1.   Implement the Symmetric Label Noise. Describe your method and show the normalized confusion matrix for E = 50%.

2.   Train LeNet-5 with the noisy training set and measure the testing accuracy. Try E = 0%, 20%, 40%, 60%, and 80%. Draw the curve of [testing accuracy vs. E]. Note that for each E, five runs are needed to calculate the mean and standard deviation of the testing accuracy, which are then used to draw your plot.