301110 Applications of Big Data
Applications of Big Data
Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
301110 Applications of Big Data
Assignment
3 Task 1. Feature extraction (15 points)
Use the MapReduce model to convert all text data into matrices. Convert ratings
to vectors. ese will be used for classication in Task 2. Use TF-IDF to vectorise
the text les. See previous practical classes and lectures materials for TF-IDF.
One step further though is to represent each text le (review) as a very long
and sparse vector as the following. Assume wordslist is the nal list of distinct
words contained in all reviews and its length is N. en each review will be a
vector of length N, with each position associated with the word in wordlist and
the value being either 0, if the corresponding word is absent in the review, or the
word’s TF-IDF. For example, if wordlist = [‘word1’, ‘word2’, ‘word3’, ‘word4’] and
review 1 contains word1 and word4, then the vector representation of review 1
is [0.1, 0, 0, 0.4] assuming TF-IDF of word 1 and word 4 in review 1 is 0.1 and 0.4
respectively. Note that TF is calculated from one single document while IDF is
obtained from all documents in the collection.
3.1 Requirements:
3.1.1 Req. 1
A Map-reduce model is a must. Implement it using Hadoop streaming. All data
are available on SCEM HDFS. e recommendation is to work on the tiny version
of the data to make the code work. You may try your code on the full version.
However, the application to full version is not required.
3.1.2 Req. 2
Generate two matrices: training data, training targets, and two vectors:
test data, test targets. training data should have N rows and D columns
with each row corresponding to each review in the training set (N is the total
number of reviews in the training set and D is the total number of words). N
and D vary depending on which version of the data you use. training targets
should have N elements each of which is the rating of the review. test data
and test targets are similarly dened.
Notes:
1. If feature extraction is too dicult for you, you can use pre-computed
bag of words features included in this data set. Refer to the appendix and
README le for details. If pre-computed features are used, a 60% penalty
will be incurred for this task, i.e. the maximum marks you can get from
this task is 6 if you do so.
2
2. Using a map-reduce model to extract TF-IDF is mandatory. If not used, a
20% penalty for this task will be incurred. ere is no constraint on how to
form the training and test matrices and vectors. ere are many versions
of TF-IDF. ere is no preference for which version to use.
3. You can use data frame (using pandas package) instead of matrices and
vectors to store training and test data and targets.
Marking scheme for task 1:
• Text le reading (1pt): read the text les for TF-IDF extraction.
• Rating scores extraction (3pts): parse the name of text les to extract ratings.
• TF-IDF extraction (8pts): use MapReduce class to extract TF-IDF for each
text le.
• Forming matrices and target vectors (or data frames) (3pts): collect TF-IDFs
to form training and test data for task 2.
4 Task 2. Classication (15 points)
Construct a classication model for review sentiment prediction, meaning that:
given a customer movie review (taken from the test set), your program should be
able to predict whether it is positive or negative.
ere is no limitation on how many classiers and what specic model you
should use. You can simply pick one that works for you for this task, either
from those covered in lecture and practical class materials or any other classi-
ers from any python packages. A good starting point is the scikit-learn (i.e.
sklearn) package.
A few things you need to address in your python program are listed as requirements below.
4.1 Requirements:
4.1.1 Req. 1
Data pre-processing. In task 1, you extracted the ratings vectors for training
and test. ese are raw ratings. As we are interested in sentiment prediction,
i.e. to predict either the review is positive or negative, you need to convert all
ratings >5 as positive class and all ratings <=5 as negative class. Choose a
coding scheme, e.g. 0 for positive, 1 for negative.
3
4.1.2 Req. 2
Normalisation. Apply at least one normalisation scheme and compare the performance of the classier(s) with and without normalisation.
4.1.3 Req. 3
Training and model selection. Use cross validation to select the best parameters
for your classier. ere may be many parameters to tune in some classiers
(such as random forest classier — RFC). You can focus on the most important
one(s) such as max depth and n estimators in RFC. Refer to the scikit-learn
package documentation for details.
Hint: you can start with a small subset of the training set to test a few parameters
to get a feel of what range the parameters should be that make the model perform
well in terms of prediction accuracy. en turn on large scale cross validation on
the whole training set.
4.1.4 Req. 4
Test on test data. Aer model selection, apply the best model, i.e. the model with
the parameters that produce the best cross validation scores, to test data, make
a prediction for each review, and record prediction accuracy
. Note:
1. Always train your classier(s) ONLY on training data including cross validation. Aer model selection, apply the best model on test data to evaluate
the performance.
2. Good performance, i.e. higher accuracy on test data, is not essential for
this task. However, if your classier has accuracy lower than about 60%, it
usually means that there are some mistakes somewhere in your code. So
try to score as high an accuracy as possible.
3. You are encouraged to try many classiers. If the coding is right, this
should not be too dicult.
Marking scheme for task 2:
• Data pre-processing (1pts): convert ratings to positive and negative coding
scheme.
• Normalisation and comparison (3pts): apply normalisation and compare
performance dierence with and without it.