Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
CIV4100: Autonomous Vehicle Systems
Assignment 2
This summative assignment, consisting of two parts
Part 1 (25%) and Part 2 (15%),
is due by Friday Week 12 of the semester.
Overview
This assignment includes the development, integration and testing of the perception modules in
automated driving systems (ADS). The assignment makes use of the accumulated knowledge and
experiences learnt in the course, especially in Weeks 6, 7, 8 and 10.
Tasks: There are two parts of tasks in this assignment: (i) Developing a deep learning system
for perception of signboards; (ii) Testing the developed perception system.
Time: These tasks can be commenced at any time during the semester and should be
submitted in the Moodle DropBox before or by the end of Week 12.
Resources
Dataset: The dataset traffic_dataset.zip will be loaded automatically in the Jupyter code or can
also be obtained separately (see Appendix A1).
Reference: Convolutional Neural Network, and Google Colab
Submissions
You must submit the followings:
■ Your complete report using the Microsoft Word template provided (Appendix A3).
■ Your complete Jupyter (Python) scripts (Appendix A2).
■ The video recording of showing your codes, main results and comments (Appendix A4).
CIV4100: Autonomous Vehicle Systems Assignment 2
Page 2 of 14
Important Note
This assignment is an individual assignment.
Assignment Information
There are 2 Parts in this assignment:
● Part 1: Developing a deep learning system for perception of signboards
● Part 2: Testing the developed perception system
The supplement information is provided in the Appendix:
● A1: Information on the dataset used.
● A2: The Jupyter code template for all tasks.
● A3: The report template for all tasks.
● A4: Note on the video preparation
● A5: Instructions to complete the assignment.
CIV4100: Autonomous Vehicle Systems Assignment 2
Page 3 of 14
PART 1: DEVELOPING A DEEP LEARNING SYSTEM FOR PERCEPTION OF SIGNBOARDS
In this Part, you will develop a working Convolutional Neural Networks (CNN)-based system for the
perception of Automated Driving System (ADS).
Refer to the instructions (Appendix A5), the code template (Appendix A2) and the report template
(Appendix A3) to complete this task.
Task 1.1: Data preparation for the CNN (5% marks)
Data preparation is the first step to build a machine learning or deep learning model. In data
preparation, you need to collect, combine, and organize data so that it can be effectively used to
train the models. In this task of assignment, you are required to perform data visualization and
image processing to obtain a new dataset as the input for the CNN system.
Task 1.2: Developing a CNN-based system (10% marks)
With a new training dataset prepared from Task 1.1, you are required to build and train a CNN-
based model underpinned by Keras and TensorFlow. The input is the dataset on traffic signs that
you have just prepared. Please follow the instructions when developing your model.
Note that if you are unable to perform Task 1.1, you can use the given processed dataset
(traffic_dataset_processed.zip) to continue the Assignment and perform this task. No marks will be
given to Tasks 1.1.
Task 1.3: Model assessment (10% marks)
You are required to evaluate the performance of your trained model on the testing dataset. First,
you need to record the testing accuracy, and then use your trained model in Task 1.2 to generate
predictions for some images in your test dataset and visualise the results.
You also need to discuss: (i) the performance of your CNN model; and (ii) the difference between
this model (in Step 7 of Task 1.2) and your mini VGG model (in Step 8 of Task 1.2).
Task 1.4: Developing report and recording video
You need to develop the report (using the template Appendix A3) that includes the outcomes of
Task 1.1, Task 1.2 and Task 1.3.
You are required to record a video (no more than 2 minutes) running through your code to
demonstrate its working and generate the results that you have included in the report and combine
it with the video in Task 2.4 (see Appendix A4).
CIV4100: Autonomous Vehicle Systems Assignment 2
Page 4 of 14
PART 2: TESTING THE PERCEPTION SYSTEM
After developing a deep learning system for perception of signboards, you are required to
undertake the following tasks.
Refer to the instructions (Appendix A5), the code template (Appendix A2) and the report template
(Appendix A3) to complete this task.
Task 2.1: Testing planning, execution and reporting with traceability matrix (5% marks)
Traceability matrix is a useful tool supporting good test planning as well as efficient test
management and control. You are required to create and maintain a traceability matrix (in a
tabular format) to map and trace requirements with test assets (including test condition, test cases,
and result) created in the whole testing process.
Task 2.2: Functional testing with known test oracle (5% marks)
You are required to undertake all activities for the dynamic testing, that is testing the system in
running for a given test case.
In software testing, a test oracle is a mechanism for determining whether a test has passed or
failed. With a known test oracle, you will be able to determine whether the system is faulty by
comparing the system actual result with the expected result. For example, if the test case is an
image containing the “Stop” sign board, the classification model should predict it with a “stop” label
(or some number in the class that is linked to this label). Otherwise, the model/system is faulty or
inaccurate.
In this task, you will be given 20 unique images as 20 source test cases, each with a known label
(that is, test oracle). Now you are required to create follow-up (new) test cases by applying some
transformation to these images. Their labels can be reused as the test oracle for the new test cases.
If the deep learning model/system predicts wrongly against these labels, we know that it is faulty
or inaccurate. You are required to create 3 different sets of test cases using 3 different
transformations:
● Rotating the image (by 20 degrees)
● Zooming (1.3 times) into the centre of images
● Adding noise (normal distribution with mean 0 and sigma 0.6)
Once done, all test cases should also be put into the traceability matrix created in Task 2.1 to map
with test conditions as well as requirements. You are required to schedule and execute all test
cases designed. All test execution results (pass and/or fail) should be analysed, summarised, and
reported. They should also be recorded in the traceability matrix created in Task 2.1.
Task 2.3: Functional testing without test oracle by Metamorphic Testing (5% marks)
You are required to adopt the Metamorphic Testing (MT) approach for testing the system. MT is
the method that alleviates the need for a test oracle in the software.
CIV4100: Autonomous Vehicle Systems Assignment 2
Page 5 of 14
You will be given 20 unique images as 20 source test cases (which are different from the source
ones in Task 2.2). The challenge is that you are not given the source labels. You are required to
create 20 follow-up (new) test cases by applying some transformations to these images. Since
we do not have the labels, we are unable to directly verify whether the outputs of 20 follow-up test
cases are correct. As a result, you will have 3 sets of Metamorphic Relations (MRs) as follows
● MR1: The ADS system should be robust against small misalignment of images.
● MR2: The ADS system should be robust against a slight cropping of images.
● MR3: The ADS system should be robust against noisy images.
Once done, all test cases should be put into the traceability matrix as what has been done in Task
2.3.
You are required to discuss the difference between methodology and outcomes of Task 2.2 and
Task 2.3.
Task 2.4: Developing report and recording video
You need to develop the report (using the template Appendix A3) that includes the outcomes of
Task 2.1, Task 2.2 and Task 2.3.
You are required to record a video (no more than 2 minutes) running through your code to
demonstrate its working and generate the results that you have included in the report and combine
it with the video in Task 1.4 (see Appendix A4).
CIV4100: Autonomous Vehicle Systems Assignment 2