Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
BIOSTAT 274
Theoretical Part 1. (20 pt) This problem illustrates the estimator property in the shrinkage methods. Let Y be a single observation, and regress Y on an intercept Y “ 1 ¨ β ` . (a) Using the formulation as shown in class, write down the optimization problem of general linear model, ridge regression and LASSO in estimating β respectively. (b) For fixed tuning parameter λ, solve for βˆ (general linear model), βˆRλ (ridge regression) and βˆ L λ (LASSO) respectively. Hint. For the LASSO problem, show that the objective function is convex even though not everywhere differentiable, hence any local minima is also a global minima. Argue that it suffices to solve for β ě 0 and β ă 0 respectively. Carefully discuss all possible situations (where the minima locates and what’s the optimal value in both cases), and then pick the better one as βˆLλ . (c) Represent βˆRλ and βˆ L λ by βˆ and create plots of them separately for λ “ 1, 5, 10. What can you tell? 2. ([ISL] 6.5, 15 + 10 pt) It is well-known that ridge regression tends to give similar coefficient values to correlated/collinear variables, whereas the LASSO may give quite different coefficient values to correlated/collinear variables. We will now explore this property in a very simple setting. Suppose that we have two observations px1, y1q and px2, y2q, where x1 ‰ 0, x1`x2 “ y1` y2 “ 0. Consider the linear model yi artificially regressed on pxi, xiq without intercept:$’&’%y1 “ x1β1 ` x1β2 ` 1y2 “ x2β1 ` x2β2 ` 2 (a) Write out the ridge regression optimization problem in this setting. (b) Argue that the ridge coefficient estimates satisfy βˆRλ,1 “ βˆRλ,2. (c) Write out the LASSO optimization problem in this setting. 1 (d) (Optional) Argue that in this setting, the LASSO coefficients βˆLλ,1 and βˆ L λ,2 are not unique. De- scribe these solutions. Hint. Starting with an optimal coefficients pβˆLλ,1, βˆLλ,2q, indicate that you can find another one. Use the relationship of a usual LASSO problem and its constrained version. Investigate into the relationship between the contour of the objective function and the constraint set in the constrained version.