Purpose

Public preview: This page shows the analysis output, figures, diagnostics, and reporting guidance without exposing the R code. The paid version includes the complete editable R Markdown source, reusable functions, all analysis code, synthetic data, documentation, and exported results.

This workflow develops and evaluates models that predict on-time educational completion. It compares a transparent benchmark with lasso logistic regression, a decision tree, and a random forest. The workflow separates schools between development and test samples, fits every preprocessing decision on training data, checks calibration and subgroup performance, and distinguishes prediction from explanation and causation.

Learning outcomes

You will be able to define a prediction target and time origin, detect leakage, create a grouped development/test split, preprocess without borrowing test information, tune models within training data, compare performance on held-out schools, select thresholds from consequences, and document fairness and transportability limitations.

User settings

Simulate or import data

Define the prediction problem

The target is completion by the scheduled time. Predictions are framed as an early-warning aid, not an automated eligibility decision. All predictors must be available at the prediction date. post_year_gpa is deliberately included in the raw synthetic file as a leakage trap and is excluded because it occurs after the intended prediction time.

Data audit

Outcome and missingness

Prediction cohort
n schools events event_rate
3976 70 1552 0.39
Missing-predictor audit
variable missing_n percent
attendance 127 3.2
engagement 217 5.5

Leakage review

Predictor timing and leakage decisions
variable available_at_prediction decision
prior_gpa TRUE Retain
attendance TRUE Retain
credits_earned TRUE Retain
post_year_gpa FALSE Exclude: measured after prediction date
completed_on_time FALSE Outcome only

Grouped split and preprocessing

School-separated development and test samples
sample students schools event_rate
Development 2917 52 0.386
Held-out test 1059 18 0.403

No school appears in both samples. Numeric medians and categorical modes are learned from the development sample and then applied to the test sample. This simulates deployment to new schools and prevents preprocessing leakage.

Model development

Benchmark logistic regression

Lasso logistic regression

Nonzero lasso coefficients at lambda.1se
term coefficient
(Intercept) -6.525
prior_gpa 1.000
attendance 2.361
credits_earned 0.060
discipline_incidents -0.179
ses 0.152
engagement 0.263
special_educationYes -0.075
school_resources 0.134

Decision tree

Random forest

Held-out performance

Model comparison

Performance in held-out schools
model AUC Brier Accuracy Sensitivity Specificity PPV NPV
Benchmark logistic 0.763 0.191 0.709 0.527 0.832 0.680 0.723
Lasso logistic 0.766 0.191 0.711 0.489 0.861 0.704 0.714
Decision tree 0.695 0.206 0.697 0.412 0.889 0.715 0.691
Random forest 0.754 0.194 0.702 0.508 0.832 0.672 0.715

Calibration

Uncertainty for the leading model

Bootstrap interval for held-out AUC
model lower median upper
Lasso logistic 0.739 0.766 0.794

Thresholds and consequences

Performance across classification thresholds
threshold AUC Brier Accuracy Sensitivity Specificity PPV NPV
0.2 0.766 0.191 0.542 0.934 0.277 0.466 0.862
0.3 0.766 0.191 0.633 0.834 0.497 0.528 0.816
0.4 0.766 0.191 0.712 0.698 0.722 0.629 0.779
0.5 0.766 0.191 0.711 0.489 0.861 0.704 0.714
0.6 0.766 0.191 0.695 0.333 0.940 0.789 0.676
0.7 0.766 0.191 0.657 0.192 0.972 0.820 0.640
0.8 0.766 0.191 0.610 0.037 0.997 0.889 0.605

The default 0.50 threshold is not inherently fair or optimal. Select a threshold using the costs of missed support, unnecessary outreach, available capacity, and stakeholder input. Preserve probabilities when possible because hard categories discard information.

Subgroup performance and fairness

Held-out performance by multilingual status
multilingual AUC Brier Accuracy Sensitivity Specificity PPV NPV
No 0.754 0.197 0.695 0.480 0.850 0.698 0.694
Yes 0.789 0.179 0.744 0.512 0.881 0.717 0.753

Differences in performance identify questions for investigation; they do not prove algorithmic bias or fairness. Examine sample sizes, base rates, measurement quality, access patterns, and consequences. Protected attributes may be needed for auditing even when they are excluded from operational predictions.

Model interpretation

Variable importance can reflect signal, scale, correlation, and modeling choices. It does not show that changing a predictor will change the outcome. Use partial-dependence or local explanation tools only after checking whether the resulting summaries are stable and substantively plausible.

Dynamic reporting example

The strongest held-out discrimination in this synthetic example came from Lasso logistic, with an AUC of 0.766 and Brier score of 0.191. The bootstrap 95% interval for AUC was [0.739, 0.794]. These estimates evaluate transport to held-out synthetic schools; they do not establish performance in a new district, year, policy environment, or student population.

Reporting checklist

  • Define the target, prediction date, population, unit of analysis, and intended decision.
  • Identify every predictor’s availability at the prediction date and remove leakage.
  • Keep all preprocessing, feature selection, and tuning inside development data.
  • Respect school, cohort, classroom, or person grouping in resampling and testing.
  • Compare with a simple benchmark and report discrimination, calibration, and uncertainty.
  • Describe missing-data handling and test it under realistic deployment conditions.
  • Report threshold consequences and subgroup performance with sample sizes.
  • Preserve a final external or temporal validation sample when possible.
  • Distinguish prediction, explanation, causal inference, and intervention impact.

When this workflow is not enough

Use survival models when timing and censoring matter; multilevel prediction when school-specific deployment is central; time-aware validation for longitudinal updating; causal methods to estimate intervention effects; and prospective impact evaluation before integrating scores into high-stakes practice. Small samples, severe class imbalance, dataset shift, complex sampling, and automated decisions require additional expertise and governance.

Export results