Neural Networks and Optimization
Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
CMSC 421 Assignment
Neural Networks and Optimization
General Instructions. Please submit TWO (2) files to ELMS:
(1) a PDF file that is the report of your experimental results and answers to the questions.
(2) a codebase submission in form of a zip file including only the code folders/files you modified and
the Questions folder. Please do not submit the Data Folder we provided. The code should contain
your implementations of the experiments and code for producing visualizations of the results.
Please read through this document before starting your implementation and experiments. Your score
will be mostly dependent on the completion of experiments, the effectiveness of the reported results,
visualizations, the consistency between the experimental results and analysis, and the clarity of the
report. Neatness and clarity count! Good visualization helps!
As you would need to use pytorch for the second half of the programming assignment Convolutional
Neural Networks - 15 Points, We have included links to some tutorials and documentations to help
you get started with pytorch:
• Official Pytorch Documentation
• Quickstart Guide
• Tensors
• Data Loading
• Building models in Pytorch
Implementation Details
For each problem, you’ll need to code both the training and application phases of the neural network.
During training, you’ll adjust the network’s weights and biases using gradient descent. Use a single
parameter, η, to control the step size during gradient descent. The updated weights and biases will be
calculated as the old values minus the gradient multiplied by the step size.
We will be providing code snippets and datasets for some parts of the assignment. You will be required
to read the comments in the code file and fill in the missing pieces in the code files to correctly execute
these files. Please ensure that you are read through all the code files we provide. These will be available
in the CMSC421 - Fall2023 GitHub repository.
1
Part 1: Programming Task - (50 Points)
Objective
The goal of this assignment is to build a neural network from scratch, focusing on implementing the
backpropagation algorithm. You’ll apply your neural network to simple, synthetic datasets to gain
hands-on experience in tuning network parameters.
Language and Libraries
Python is mandatory for this assignment. Use numpy for all linear algebra operations. Do not use
machine learning libraries like PyTorch or TensorFlow for Questions 1,2 & 3; only numpy, matplotlib,
and Python built-in libraries are permitted.
1 Simple Linear Regression Model - (10 Points)
1.1 Network Architecture
• The network consists of an input layer, a hidden layer with one unit, a bias layer, and an output
layer with one unit.
• The output is a linear combination of the input, represented as a1 = Xw0 + a0 + b1.
1.2 Loss Function
Use a regression loss for training, defined as
1
2
Xn
i=1
(yi − a1(xi))2
1.3 Implementation
Using the template_for_solitions file, write code to train this network and apply it to data on both
1D data as q1_a and on higher dimensional data as q1_b.
• Data Preparation: Use the q1_
function from the Data.generator module to generate
training and testing data. The data module has both a and b so use the appropriate function
call to fetch the right data for each experiment.
• Network Setup: Use the net_setup method in the Trainer class to initialize the network, loss
layer, and optimizer.
• Training: Use the train method in the Trainer class to train the network. Plot the training
loss over iterations.
• Testing: Use the test data to evaluate the model’s performance. Plot the actual vs. predicted
values and compute evaluation metrics.
Tests and Experiments
1.4 Hyperparameters
• The main hyperparameters are the step size (η) and the number of gradient descent iterations.
• You may also have implicit hyperparameters like weight and bias initialization.
Hyperparameter Tuning
Discuss the difficulty level in finding an appropriate set of hyperparameters.
2
2 A Shallow Network - (10 Points)
The goal of this assignment is to implement a fully connected neural network with a single hidden
layer and a ReLU (Rectified Linear Unit) activation function. The network should be flexible enough
to accommodate any number of units in the hidden layer and any size of input, while having just one
output unit.
2.1 Network Architecture
The network consists of an input layer, a hidden layer with one unit, a bias layer, and an output layer
with one unit.
• Input Layer: a01
, a02
, . . . , a0d
• Hidden Layer: z1j =
Pd
k=1 Xw1k a0k + b1j
• ReLU Activation: a1j = max(0, z1j
)
• Output Layer: a2 =
Pd
k=1 Xw2k a1k + b2
2.2 Loss Function
Continue to use a regression loss for training the network. You can continue to use a regression loss
in training the network defined as
Xn
i=1
1
2
(yi − a
1
1
(xi))2
2.3 Implementation
Using the template_for_solitions file, write code to train this network and apply it to data on both
1D data as q2_a.py and on higher dimensional data as q2_b.py.
• Data Preparation: Use the q2_ function from the Data.generator module to generate
training and testing data. The data module has both a and b so use the appropriate function
call to fetch the right data for each experiment.
• Network Setup: Use the net_setup method in the Trainer class to initialize the network, loss
layer, and optimizer.
• Training: Use the train method in the Trainer class to train the network. Plot the training
loss over iterations.
• Testing: Use the test data to evaluate the model’s performance. Plot the actual vs. predicted
values and compute evaluation metrics.
Tests and Experiments
2.4 Hyperparameters
You now have an additional hyperparameter: the number of hidden units.
Hyperparameter Tuning:
• Discuss the difficulty in finding an appropriate set of hyperparameters.
• Compare the difficulty level between solving the 1D problem and the higher-dimensional problem.