Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
QTM 110
Please attempt all questions. Only one assignment needs to be turned in per group. Make sure that you
clearly mark which question you are answering (e.g. Question 1, Part A or 1.a.). Show work where
appropriate and explain your reasoning for any arguments you make. This assignment is due at 11:59
PM EST on Wednesday, November 10th – submit your answers as a PDF using the assignment link on
Canvas.
Question 1 (25 Pts.): In fiction and film, there is something known as the Bechdel test, named after
Allison Bechdel. It tests whether women are represented in a film by asking if there are least two
women who talk about something other than a man (PASS) or not (FAIL). We are interested in looking
at the effect of passing the Bechdel test on the U.S. domestic box office totals (how much revenue a film
takes in from people seeing it in movie theaters). But we’re worried about confounding by the film’s
budget (in 2013 dollars) – perhaps bigger-budget films are more or less likely to pass the Bechdel test,
while their budget also likely impacts their total domestic box office.
Below is R output from a crude and adjusted model for this question:
Crude Model
Adjusted
NOTE: -1.082e+07 = -1.082*107 = -$10,820,000 or -$10.82 million.
A few notes on regression output from R:
1. Regression output just translates our regression equation to computer interpretable formulas to
run linear regressions and produces estimates of coefficients that minimize the sum of squared
errors.
2. The top section – Call – shows the regression equation given to R
3. The residuals section gives the quantiles of the residuals or errors. Note that these are not
squared!
4. The coefficients section tells us about the estimates of various regression coefficients
a. Estimate gives us estimates for the regression coefficients that minimize the sum of
squared errors – this is the column that you want to pay attention to for this
assignment.
b. Std. Error tells us the standard error associated with regression coefficient. We didn’t
discuss these in class, but they have the same interpretation as standard errors for the
treatment effect from an experiment – if we repeated the data collection process a
bunch of times, we would expect that the standard deviation of the noise associated
with estimate would be equal to this value.
c. Assuming a null hypothesis that the regression coefficient is equal to zero, the t value
tells us the value of the t statistic used to generate a p-value for the regression
coefficient
d. Pr(>|t|) tells us the p-value associated with the estimate and standard error – if we
repeated the data collection procedure a large number of times, what is the probability
we would see an estimate of the coefficient as large as or larger than the one we
observed if the true coefficient was equal to zero.
5. Bechdel_testPASS is a dummy variable that encodes whether or not the observation (e.g. movie)
passed the Bechdel test – it is coded as 1 if it passed, 0 if it did not.
6. Budget_2013 is the budget of the movie in 2013 dollars (to account for inflation).
a. (3 pts.) Write regression equations that represent the crude/short and adjusted/long
models that were run above. Just use symbols, not numbers, at this stage. Make sure
that it is clear which variable each coefficient describes. Don’t forget about the error
terms!
b. (3 pts.) What are the crude and adjusted associations (e.g. coefficients) of passing the
Bechdel test on domestic box office totals? Interpret each one and be sure to properly
account for the multiple coefficients in the adjusted model.
c. (3 pts.) Interpret the coefficient for budget_2013 in the adjusted model above.
d. (5 pts.) What is the direction and magnitude of the confounding of the movie’s budget
on the relationship between passing the Bechdel test and the amount of money the
movie makes at the box office? (Hint: Compare the estimated effect of the treatment
from the short model to the estimate effect for the long model and work from there.
Remember that the short model is equivalent to the naïve ATE estimate!)
e. (5 pts.) Given this information, is there a positive or negative correlation between
passing the Bechdel test and the budget of a movie? How can you tell?
f. (6 pts.) Your movie executive friend looks at this analysis and says “See! There’s just no
demand for movies with strong, independent female leads. Even when we control for
the budget of the movie, there’s still a negative effect. This provides definitive evidence
that passing the Bechdel test causes people to not want to see a movie!” Argue that
this statement is not necessarily true using what you know about using regression to
control for confounding. Specifically, come up with an example of another variable that,
when adjusted for, will likely lead the regression coefficient on passing the Bechdel test
to be closer to zero.
Question 2 (25 Pts.): Jason Lyall (an excellent scholar and certifiable expert on all matters of
international conflict) wanted to assess whether bombing by military forces impacts insurgent activity,
either decreasing it by disrupting insurgent attacks or increasing in through a “backfire” effect from
civilian casualties. Lyall looked at this question in Chechnya during the Second Chechen War. He looked
at villages that were “randomly” shelled or not by Russian artillery, with the randomness coming from a
combination of drunk soldiers and Russian military doctrine that demanded indiscriminate shelling. He
compared insurgent activity in the shelled and matched unshelled villages 90 days before and 90 days
after the date of an attack by Russian artillery.
Below is an excerpt from Lyall’s study, which appeared in the Journal of Conflict Resolution in 2009.
a. (3 pts.) Fill in the table below based on the excerpt above.
Mean Insurgent Attacks Per 90 Days
Pre-Shelling Post-Shelling
Shelled Villages
Unshelled Villages
b. (3 pts.) Draw a line graph depicting the changes in the treated group and the changes in
the untreated group. Label each line, the axes, and the intervention time.
c. (3 pts.) How did insurgent activity in the shelled villages change after the shelling? In a
sentence or two, give a reason why this comparison may not give the effect of shellings
on insurgent attacks.
d. (3 pts.) Suppose instead that we compared the number of insurgent attacks in shelled
villages after the shellings to the number of attacks in the unshelled villages in the same
time period. In a sentence or two, give a reason why this comparison may not give the
effect of shellings on insurgent attacks.
e. (5 pts.) Calculate an appropriate difference-in-difference measure of the effect. Show
your work. Add the corresponding line to your graph in part b) and graphically
demonstrate where each part of the DiD estimator comes from.
f. (3 pts.) Why is your estimate from e) better than c) and d)? What does the DiD
estimator account for that the other two do not?
g. (5 pts.) Given the information in the table and the excerpt, provide two distinct
concerns we may have about the efficacy of this estimate. Obviously, we’re not experts
on conflict, but use reasoning and what you know about DiD estimators to point out a
few additional pieces of information we may want before claiming that we’ve gotten the
correct estimate.
Question 3 (27 Pts.):
League of Legends is a popular multiplayer online battle arena (MOBA) video game. Two teams of five
players control characters called “champions” and compete to complete objectives. Teams begin the
game on opposite corners of a sizeable map. To win the game, they must destroy a sequence of enemy
turrets, invade the enemy base, and destroy the enemy nexus.
This regression discontinuity design attempts to determine the causal effect of getting First Turret (being
the first team to destroy the opposing team’s turret) on the probability of winning. The graphs show the
time differential between first turret for the red team (positive if the red team gets their first turret first
and negative if the red team gets their first turret second) on the x-axis and the proportion of times that
the red team wins on the y-axis.1 Time differentials are binned into 100 bins that include an equal
number of matches and the proportion of times the red team wins is the proportion of victories within
1 Though this analysis looks at the red team’s turret gap, it also includes all the information about the blue team’s
gap – if the red team’s gap is 500 seconds, then the blue team’s gap is -500 seconds. This is just a convenient way
to organize the data to fit an RDD design.
that bin. The left graph shows this data for matches that took place without Patch 6.15 – a patch that is
supposed to reduce randomness in outcomes and increase the skill gap (the more skilled team is more
likely to win). The right graph shows the same analysis for matches that took place with Patch 6.15.
Correlation is not causation, so simply looking at the average proportion of games that you win after you
obtain First Turret does not give you a causal impact. For instance, one can easily argue that superior
teams will win the match and destroy the first turret in the process, which implies that other underlying
variables (e.g. skill) are strongly positively correlated to both and lead to biased estimators when failing
to account for it.
Note: For any of these questions, a picture can be used to communicate certain points. Feel free to
include pictures in your answers!
a. (2 pts.) In this research design, define the following:
i. Running Variable
ii. Outcome
iii. Discontinuity (where in the running variable does the exposure happen?)
iv. Treated vs. Untreated (what does it mean to be treated? What about
untreated?)
b. (5 pts.) Describe what the continuity assumption means for this RDD design. Specifically,
discuss the relationship between skill, time to capture the first turret, and the match’s
outcome. Using your knowledge about the problem, do we have any reasons to worry
about continuity violations?
c. (5 pts.) What causal quantity is identified for this regression discontinuity design? Eyeballing
the above graphs, what is the effect identified by the RDD estimator of getting first turret in
the pre-6.15 version of the game? What about post-6.15?
d. (5 pts.) Where do we expect that the estimates from c) will provide information for the true
treatment effect? Do we expect that the RDD estimator provides information about the
average treatment effect for all LoL teams? Why or why not? Draw a picture that supports
your claim.
e. (5 pts.) Your team plays in a LoL league that is determining whether to use the version of the
game that has Patch 6.15 - a patch that provides significant balance changes to the game
and weakens a team’s ability to buff turrets. Your team is right below the league average in
terms of skill (e.g. as arbitrarily close to the league average in skill as one team can get).
Your team can vote to approve the new patch or to remain in the pre-6.15 version of the
game. Given the results from the two regression discontinuity estimates above, what choice
should your team make? Why?
f. (5 pts.) The regression model in this analysis uses piecewise linear regressions with second
order polynomial terms. In a few sentences, explain how the RDD estimator is sensitive to
this choice and discuss whether or not this assumption makes sense. How might your
estimates change if we chose to use a simple best fit line that uses all of the data or (even
worse) averages from bins around the threshold?
Question 4 (8 Pts.): This example is modified from Georgia State University professor Andrew Heiss’s
RDD tutorial. Consider a hypothetical 1-on-1 tutoring program for a QTM course. Everyone takes an
entrance exam before the course begins, and you get placed into the 1-on-1 tutoring program if you
score below a 70; this is explained to the students taking the test. At the end of the course, all students
take an exit exam. We are curious about the effect of the tutoring program on exit exam scores.
Here are two sets of graphs of data you might see from this program. The first graph in each set is a
scatterplot of entrance vs. exit exam scores. The second is a histogram of entrance scores.
Set A
Set B
a. (4 pts.) Without consulting the completely made-up graphs, do you believe that there
could possibly be a continuity violation (via sorting/bunching/clumping) here? Why or
why not? Be sure to engage with the context of the study to make your argument.
b. (4 pts.) Which set of the above graphs (Set A or Set B) suggests a continuity violation?
Explain what you see in each graph in your chosen set that leads you to believe there’s a
continuity violation.
Question 5 (15 Pts.): We are interested in the impact of a tutoring program offered in the Emory main
campus library on student GPA. The problem is we have some unmeasured confounders including
inherent academic motivation that might impact both use of the tutoring program and GPA. We choose
to try and conduct an instrumental variable analysis for our question using residential proximity to the
library as our instrument.
Below is a graph for our problem that shows our assumed “causal flow”:
Use the above graph to answer the following problems, but feel free to draw attention to other
variables that may make our above graph (and corresponding IV assumptions) incorrect.
a. (3 pts.) Does the instrument satisfy the relevance condition? Why or why not?
b. (3 pts.) Does the instrument satisfy the exclusion restriction? Why or why not?
c. (3 pts.) Does the instrument satisfy the exchangeability condition? Why or why not?
Now consider the new graph below:
d. (3 pts.) What condition or restriction does the instrument “Proximity to Library” violate
now? Why?
e. (3 pts.) How could you alter your analysis so “Proximity to Library” still meets this
condition or restriction? (Hint: Think about another technique we discussed that could
get rid of the effect of a single known variable.)