Submission

You will submit three files: report as .pdf file, and your code in two versions: .ipynb and .pdf. There are three different submission boxes in Canvas.

The code for this assignment should be written in Python in the Jupyter Notebook environment. A Jupyter Notebook template with the data loading code is provided. Your core implementation of the algorithms and hyperparameter searching should use the same suite of libraries that we have used in the tutorials, such as keras, sklearn, numpy, and pandas. Other libraries may be utilised for minor functionality such as plotting, however please specify any dependencies at the beginning of your code submission.

To generate a .pdf of your notebook for submission, please use File> Download as > PDF or Print Preview - > Save as PDF. Remember to submit both the .ipynb and the .pdf of your code.

Before you submit, if you work in a group, you will need to create a group in Canvas. Under the “People” page on Canvas, select the “A2 Group” tab. You and your group partner should choose one of the empty groups listed under this tab, and both join it. Groups have a maximum of 2 members. If you are completing the assignment individually, you don’t need to create a group.

The submission file should contain the SID number(s) and should be named like this:

Code:

•    a2-SID.ipynb (.pdf) for a student working individually, where SID is the student’s SID number

•    a2-SID1-SID2.ipynb (.pdf) for a group of 2 students, where SID1 and SID2 are the SIDs of the two students

Report:

•    a2-report-SID.pdf (individual) or a2-report-SID1-SID2.pdf (group)

Task

In this assignment, you will implement and compare several machine learning algorithms, including a Multi- layer Perceptron (MLP) and Convolutional Neural Network (CNN), on an image classification task. You will need to demonstrate your understanding of a full machine learning pipeline, including data exploration and pre-processing, model design, hyperparameter tuning, and interpreting results. Moreover, the assignment will require you to consolidate your knowledge from the course so far to effectively discuss the important differences between the algorithms.

Since chasing the best possible performance may require large models and specialised GPU hardware, the main objective of the assignment is not to achieve the highest accuracy possible, but rather to thoroughly justify your decisions and analyse your results.

1.        Code

Your code submission for the assignment should be an .ipynb Jupyter notebook in similar fashion to the tutorials, including well commented code and sufficient text to explain the cells.

Data loading, pre-processing, and exploration

The dataset for this assignment is Fashion-MNIST, so named because the images are in the same 28x28 greyscale format as the original MNIST dataset. Each image depicts an item of clothing or fashion accessory, with ten classes in total. Despite each image having the same format as MNIST, the task is significantly more difficult.

Figure 1: Samples from the Fashion-MNIST dataset

The code to load the training and test sets is provided in the .ipynb template. For further information on

the dataset, you may be interested in the following link:https://github.com/zalandoresearch/fashion-mnist

Algorithm design and setup

You will be required to design and implement three algorithms that we have covered in the course using the sklearn and/or keras libraries, in order to investigate their strengths and weaknesses.

1.   An appropriate algorithm of your choice from the first 6 weeks of the course

2.   A fully connected neural network

3.   A convolutional neural network

Due to runtime constraints, you will not be required to thoroughly tune over all possible neural network designs, however you should justify your design decisions in the report, and you may wish to conduct a few rough experiments to converge on a reasonable design. Remembering that the focus of the assignment is to  demonstrate  your  understanding  of  and  compare  the  algorithms,  not  to  achieve  state-of-the-art performance, keep your models appropriately small such that you can complete the hyperparameter tuning in a reasonable time on your hardware of choice.

You will need to choose an appropriate number of epochs to use for your search, or use a stopping criterion.

Hyperparameter tuning

You may use cross validation via sklearn to search over different parameter combinations, or you may implement another reasonable search approach if you find the runtime of CV prohibitive (for example, a holdout procedure with the validation set). Please ensure you search over the following:

1.   Algorithm of your choice from the first 6 weeks:

•    You will need to choose an appropriate set of hyperparameters to search over based on what was covered in the tutorials or other references

2.    Fully connected neural network:

•    At least three learning rate values

•    At least two appropriate activation functions for the hidden layers, as well as the case where no activation function is used

3.    Convolutional neural network:

•    At least three learning rate values

•    At least two kernel sizes

•    At least two strides

Keep a record of the runtimes and results with each parameter combination (if using sklearn, consult the documentation)  and  use  these  to  produce  appropriate  visualisations/tables  of  the  trends  in  your hyperparameter search to aid the discussion in your report.

In your submission, keep the parameter search separate from other cells, and submit the .ipynb with the output preserved, as markers will not be able to run this code (it will take too long).

Final models

After selecting the best set of hyperparameters for each model, include cells which train the models with the selected hyperparameters independently of the parameter search cell, and measure the performance of each model on the test set.

2.        Report

An approximate outline of the report requirements is provided below, but make sure to reference the marking criteria also for the mark distribution. You may write your report in MS Word or LaTeX, but please ensure that it is well-formatted and submit it as a pdf.

Introduction

State the aim of your study. Outline the importance of your study. Even though this particular dataset may not be the most practically useful, consider the importance of comparing algorithms and their suitability for a given task, as well as the importance of image classification more broadly.

Data

In this section, you should describe the dataset and pre-processing.

Data description and exploration. Describe the data, including all its important characteristics, such as the number  of  samples,  classes,  dimensions,  and  the  original  source  of  the  images.  Discuss  your  data exploration,  including  potential  difficulties  for  classification  such  as  classes  with  similar  features  and differences in pixel intensities between samples. Where relevant, you may wish to include some sample images to aid this discussion.

Pre-processing. Justify your choice of pre-processing either through your insights from the data exploration or with reference to other sources. Explain the effect/purpose of any pre-processing techniques you have applied. If you have not performed pre-processing or have intentionally omitted possible steps such as image centring, justify these decisions.

Methods

In this section, you should explain the classification methods you used.

Summary. Briefly describe the main ideas of the three algorithms - your chosen algorithm, MLP and CNN. Explain why you chose your algorithm.

Strengths and weaknesses. Describe the relative strengths and weaknesses of the algorithms from a theory perspective.   Consider  factors  such   as   performance,   overfitting,   runtime,   number  of   params   and interpretability. Explain the reasons; e.g. don’t simply state that CNNs perform better on images, but explain why this is the case.

Architecture and hyperparameters – State and explain the chosen architectures or other relevant design choices, the hyperparameters you aim to tune and outline your search method. Briefly explain what each hyperparameter controls and how changing it may affect the algorithm. For example, consider the effects of changing the learning rate, or changing the stride of a convolutional layer.

Results and discussion

In this section, you should present and discuss your results.

Begin with the hyperparameter tuning results. Include appropriate tables or graphs (not code output) to illustrate the trends (performance, runtime etc.) across different hyperparameter values. Discuss the trends and provide possible explanations for their observation. Did they align with your predictions?

Next, present a table showing the best hyperparameter combination for each algorithm, its performance on the test set (e.g. accuracy and other performance measures), and the training runtime. Analyse and discuss the results. Refer to the stated strengths and weaknesses of the classifiers in the Methods section; did the results agree with your expectations? What factors influenced the runtime (time per epoch, total number of epochs required etc.)?

Include anything you consider interesting and/or relevant. For example, you may look at which classes each algorithm confused.

Conclusion

Summarise your  main findings,  mention any  limitations,  and suggest future work. When  making your conclusions, consider not only accuracy, but also factors such as runtime and interpretability. Ensure your future work suggestions are concrete (eg. not in the spirit of “try more algorithms”) and justify why they would be appropriate.

Reflection

Write one or two paragraphs describing the most important thing that you have learned while completing the assignment.