By Severin Schirmer and Didrik Sten Ingebrigtsen
This project covers linear and neural models for both regression and classification problems. We use least-squares linear regression with ridge regularisation, as well as feed forward neural network (FFNN) with different activation functions to fit to terrain data. Then, we put a logistic and a FFNN against eachother in trying to classify in the MNIST dataset.
The project report is in the root folder, with the plots in the /plots folder, and the python scripts in the /python folder. Dataframes generated by the scripts are stored as .csv-files in /dataframes, but this is useful mainly for debugging, as the data is stored with fixed names per function used, and so is overwritten all the time.
- activations.py houses the action functions used by neural_model.py, as well as tuples of activation functions and their derivatives
- benchmarking_classification.py uses machine-learning library Keras to test our implementation of the neural network against something we know to work well
- data_handling.py splits data into training, validation and testing sets, either by selecting it evenly from a grid (used for terrain), or randomly (used for classification)
- helpers.py contains various tools useful for working with our dictionary-heavy data flow, and making models with various parameters
- ireland2.tif is our terrain data
- learning_rate.py has the class Learning_rate, which creates adaptive learning rate functions
- linear_models.py contains a class for OLS, as well as a regularised subclass, and beta functions for ridge and LASSO regularised regression
- main.py connects everything together, and calculates, makes plots and prints out results based on command line arguments. More info on how to use it below.
- metrics.py has functions for metrics like MSE and R^2, as well as a bigger function that makes a confusion matrix and prints out some accuracy metrics for a set of classification models and data
- mnist.py imports the mnist dataset, and uses data_handling.make_classification_dict to split it
- neural_model.py has the Network class and the layer classes
- plotting.py has a bloated function called side_by_side, that takes in a complicated datastructure of nested lists and dictionaries to plot out virtually anything using one of the plotter functions also found in the module. The module also has a function to plot the architecture of a neural network
- real_terrain.py imports terrain data, and uses data_handling.make_data_dict to split it
- sgd.py has a stochastic gradient descent function, as well as a series of functions to make it easier to perform SGD on more than one model at a time, and then plot them by interfacing with plotting.side_by_side
- test_best_models_mnist.py manual testing of the best two models from the mnist tuning. Moving outside the main.py allows for a faster run of the code.
- tune.py also uses sgd.sgd to perform SGD on a set of models, but is useful for comparing more models when it is desirable to see the correlation between two parameters and error, because it makes heatmaps
The module main.py takes in command line arguments, and does different things based on them.
| Command line argument | Purpose | Dataset | Model |
|---|---|---|---|
| all | Performs all the commands listed below | All | All |
| ols_reg | Makes a big plot that demonstrates SGD working, and the dependence on initial conditions | Terrain | OLS and ridge |
| momentum | Shows the effect of the momentum hyper parameter | Terrain | Ridge |
| beta_variance | Plots out the variance in beta parameter after training with SGD, for different models | Terrain | OLS and ridge |
| neural_reg_tune | Tests out different combinations of hyper parameters to find the best | Terrain | Neural |
| reg | Compares the best ridge model with the best neural models | Terrain | Neural and ridge |
| mnist_tune | Tests out different combinations of hyper parameters to find the best | MNIST | Neural |
| mnist | Compares the best logistic model with the best neural models | MNIST | Neural |
| softmax_issue | Demonstrates an issue we have had with softmax on the output layer | MNIST | Neural |