Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
BIOSTAT 274
Theoretical Part
1. (20 pt) This problem illustrates the estimator property in the shrinkage methods. Let Y be a single
observation, and regress Y on an intercept
Y “ 1 ¨ β ` .
(a) Using the formulation as shown in class, write down the optimization problem of general linear
model, ridge regression and LASSO in estimating β respectively.
(b) For fixed tuning parameter λ, solve for βˆ (general linear model), βˆRλ (ridge regression) and βˆ
L
λ
(LASSO) respectively.
Hint. For the LASSO problem, show that the objective function is convex even though not
everywhere differentiable, hence any local minima is also a global minima. Argue that it suffices
to solve for β ě 0 and β ă 0 respectively. Carefully discuss all possible situations (where the
minima locates and what’s the optimal value in both cases), and then pick the better one as βˆLλ .
(c) Represent βˆRλ and βˆ
L
λ by βˆ and create plots of them separately for λ “ 1, 5, 10. What can you
tell?
2. ([ISL] 6.5, 15 + 10 pt) It is well-known that ridge regression tends to give similar coefficient values
to correlated/collinear variables, whereas the LASSO may give quite different coefficient values to
correlated/collinear variables. We will now explore this property in a very simple setting. Suppose
that we have two observations px1, y1q and px2, y2q, where x1 ‰ 0, x1`x2 “ y1` y2 “ 0. Consider the
linear model yi artificially regressed on pxi, xiq without intercept:$’&’%y1 “ x1β1 ` x1β2 ` 1y2 “ x2β1 ` x2β2 ` 2
(a) Write out the ridge regression optimization problem in this setting.
(b) Argue that the ridge coefficient estimates satisfy βˆRλ,1 “ βˆRλ,2.
(c) Write out the LASSO optimization problem in this setting.
1
(d) (Optional) Argue that in this setting, the LASSO coefficients βˆLλ,1 and βˆ
L
λ,2 are not unique. De-
scribe these solutions.
Hint. Starting with an optimal coefficients pβˆLλ,1, βˆLλ,2q, indicate that you can find another one.
Use the relationship of a usual LASSO problem and its constrained version. Investigate into the
relationship between the contour of the objective function and the constraint set in the constrained
version.