Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

CHDV 30102 Introduction to Causal Inference

Second assignment: Propensity Score-Based Methods

(Due Monday February 12, 2024, 11:59 PM)

(Please submit your assignment as a .docx, .pdf, or .html file)

Research Question

In this assignment, you will apply the propensity score-based methods to quasi-experimental data in an evaluation of the effect of class size reduction in Grade 1 on math achievement by the end of Grade 1. You will use the nationally representative Early Childhood Longitudinal Study-Kindergarten cohort (ECLS-K) 1998 public-use data in the analysis. Please use the data file named “CHDV 30102_ECLSK98_class size.dta.”

Data

The treatment variable is Grade 1 class size (A4CLSIZE) which needs to be dichotomized. For this assignment, we consider a class of no more than 18 students (i.e., less than or equal to 18) to be relatively “small” and a class of 19 or more students to be “regular”. The outcome is Grade 1 math achievement score (C4R2MSCL). Please exclude from your analysis any student who has missing information in the treatment indicator or the outcome.

The observed pretreatment covariates include:

- student gender (you may create a three-category variable “gender3” by replacing “-9” with “3” for the missing category.),

- student race (you may create a six-category variable “race6” by combining:

o “3 hispanic, race specified” and “4 hispanic, race not specified” into one category named “Hispanic,”

o combining “6 native Hawaiian, other pacific islander” and “7 American Indian or Alaska native” into one category named “Indigenous or Native Americans,”

o combining “-9 not ascertained” and “8 more than one race, non-hispanic” into one category named “Other Races.”),

- reading score in fall kindergarten (C1RRSCAL),

- reading score in spring kindergarten (C2RRSCAL),

- math score in fall kindergarten (C1R2MSCL),

- math score in spring kindergarten (C2R2MSCL), and

- Grade 1 teacher’s teaching experience in years (B4YRSTC).

Note that “-1” and “-9” are missing values. As you did in assignment 1, please create a missing indicator for the missing cases and then use the sample mean to replace the missing values in a continuous covariate (i.e., mean imputation along with a missing indicator).

This assignment has eight sets of questions in total.

1. Descriptive analysis

What is the mean difference in Grade 1 math achievement (C4R2MSCL) between students attending small classes (i.e., the treated) and those attending regular classes (i.e., the control)? Also report the standard error and the hypothesis testing result. How large is the effect size (as before, using the standard deviation of the control group students’ math outcome as the scaling unit)?

2. Potential confounders

2a. What are some major pre-existing differences between students in small classes and those in regular classes? Show your evidence (you may use “CreateTableOne()” from the tableone package in R).

2b. Which group (i.e., the treated or the control) seems to be relatively advantaged, that is, would likely have a higher math score on average even if all students would have attended a small class or would have attended a regular class in Grade 1? Please provide your reasoning on the basis of empirical information.

3. Propensity score and common support: Use logistic regression to estimate every student’s propensity of attending a small class.

3a. Explain how you decide whether it is necessary to include a quadratic term or other nonlinear forms of a continuous covariate or an interaction between two covariates. Write down your logistic regression model. Report the estimated coefficients and their standard errors in a table. Save the logit propensity score in the same dataset.

3b. Compare the distribution of the logit propensity score between students in small classes and those in regular classes (a) by displaying the histograms for these two groups (you may use “teffects overlap” in Stata or “histbackback()” in the Hmisc package in R) and (b) by examining the between-group differences in the mean and variance of the logit propensity score (you may use “tabstat” in Stata or the “group_by() and summarize() functions in the dplyr package in R). To identify the region of common support (remember to allow for a caliper equal to 20% of a standard deviation of the logit propensity score), please report if there are any extreme cases in one treatment group that have no counterparts in the other group and therefore should be excluded from the analytic sample.

4. Propensity score matching

4a. Decide whether you plan to estimate the population average treatment effect on the treated (ATT) or the population average treatment effect (ATE) and explain why.

4b. Apply propensity score matching in Stata or R (Use “teffects psmatch” in Stata to conduct one-to-one matching with replacement which is the default option and takes into account that the propensity score is estimated in computing the standard error for the treatment effect estimator. R users may try the combination of “cobalt” and “MatchIt” packages or use “ps.match()”).

4c. Check balance within the matched pairs in the logit propensity score. Also check balance in the pretreatment covariates (You may use the “tebalance summarize” command in Stata; in R, you may use “bal.tab()” from the cobalt package or “ps.match()” followed by “ps.balance()”). In each case, compare the standardized difference (you may again use the standard deviation of the control group students’ math outcome as the scaling unit) and the variance ratio before and after matching. Summarize the results in a table.

4d. Use the matched sample to estimate the effect of class size type on Grade 1 math achievement. Report the standard error, the hypothesis testing result, and the effect size. What is your conclusion with regard to the class size effect?

5. Propensity score stratification for estimating the ATE

5a. Within the common support as identified in Question 3, divide the analytic sample evenly into five strata (i.e., quintiles) on the logit propensity score estimated in Question 3. Within each stratum, compare the distribution of the logit propensity scores between the treated and the control by tabulating the corresponding means and variances. Also include in the table the average within-stratum standardized difference and average variance ratio in the logit propensity score that have been averaged over all the strata.

5b. Explain with reasons whether you decide to further subdivide the sample. If so, generate another table for the logic propensity score showing the results after you have modified your stratification. (Note: for this exercise, there is no need to modify your stratification more than once.)

5c. Show the result of balance checking for each of the covariates by computing the average within-stratum standardized difference and the average variance ratio. (You may install the “pbalchk” package in Stata and display the results; in R, you may use “ps.makestrata()” followed by “ps.balance()” to generate the results of balance checking.)

5d. Compute the within-stratum mean difference in the Grade 1 math score between students attending small classes and those attending large classes. Tabulate the stratum-specific treatment effects (see Hong (2015) Chapter 3 Table 3.2 on page 65 for an example). (The above can be done manually in Stata or R; alternatively, you may use mmws.exe to carry out these steps and save the stratified data.) Do you observe any systematic pattern in the estimated effect of class size on Grade 1 math score across the strata? (You may graph the results as illustrated in slide 50 of the Week 4 handouts.)

5e. Estimate the average effect of class size on Grade 1 math score by regressing the outcome on the treatment indicator and the stratum indicators. Also report the standard error, the hypothesis testing result, and the effect size. What is your conclusion with regard to the class size effect?

6. Inverse-probability-of-treatment weighting (IPTW) for estimation the ATE

6a. Apply IPTW in Stata or R to estimate the ATE (e.g., use “teffects ipw” in Stata or “PSweight” in R).

6b. Check balance in the logit propensity score after weighting. This can be done by simply using the logit propensity score as the outcome in a weighted analysis. Also check balance in the pretreatment covariates. In each case, compare the standardized difference before and after weighting (again, you may use “tebalance summarize” in Stata or “bal.tab()” from the cobalt package in R). Summarize the results in a table.

6c. Estimate the population average effect of class size on Grade 1 math achievement, report the standard error, the hypothesis testing result, and the effect size. What is your conclusion with regard to the class size effect?

7. Marginal mean weighting through stratification: You may use the mmws.exe program to estimate the ATE.

7a. After logistic regression analysis, use the data within the common support with caliper as your analytic sample. Stratify the analytic sample on the logit propensity score. (You may use the same set of strata that you constructed in Question 5 for estimating the ATE.) Check balance in the logit propensity score and in each pretreatment covariate by computing the standardized difference and variance ratio before and after weighting (The standardized differences after weighting are reported in the “Balance Checking Tables” in the output of the stand-alone mmws software.). Summarize the results in a table as you did in answering the previous questions.

7b. Estimate the average effect of class size on Grade 1 math achievement. Also report the standard error, the hypothesis testing result, and the effect size. What is your conclusion with regard to the class size effect? How does the result compare with what you obtained in Question 5?

8. Identification assumption.

8a. What is the key identification assumption required for the above results to have causal validity? State your assumption both in symbols and in words. Explain each notation used when you express the identification assumption in symbols.

8b. Please think of one unmeasured covariate that might potentially confound the class size effect. Explain why the omission of such a covariate from the analysis might have potentially biased your result.

8c. What is the purpose of a sensitivity analysis in causal inference in general? In the context of the current study, under what conditions would you consider an analytic result to be sensitive to potential bias associated with an omitted confounder?