Purpose

Public preview: This page shows the analysis output, figures, decision guidance, and reporting example. It does not include the underlying R code. The paid version includes the complete editable R Markdown source, all analysis code, synthetic data, documentation, and reusable output files.

This workflow demonstrates how to evaluate a multi-item psychology measure before creating scores or comparing groups. It joins item audits, reliability, exploratory factor analysis (EFA), confirmatory factor analysis (CFA), measurement invariance, validity evidence, score creation, and reporting in one reproducible report.

The synthetic example is the Doctoral Research Confidence Scale, a 12-item measure with three proposed dimensions:

  1. Research self-efficacy — confidence planning and conducting quantitative research.
  2. Methods support — access to clear methodological and advisor support.
  3. Reporting confidence — confidence interpreting and communicating results.
The included data are entirely synthetic. Psychometric evidence is sample- and use-dependent. Reliability, fit indices, or invariance tests do not prove that a measure is valid. Content evidence, response-process evidence, theory, sampling, and independent replication remain necessary.

Scope, audience, and prerequisites

This product is designed for doctoral students who are developing, adapting, or evaluating a Likert-type scale and need a defensible workflow for a dissertation methods or results chapter.

After completing the example, the user should be able to:

  • identify coding errors, reverse-keyed items, missing responses, and floor or ceiling effects;
  • distinguish internal-consistency estimates from evidence that a scale is unidimensional;
  • choose Pearson or polychoric correlations based on item measurement and distribution;
  • use parallel analysis and theory to select an EFA solution;
  • fit and diagnose an ordinal CFA with an appropriate estimator;
  • test configural, metric, scalar, partial scalar, and strict invariance;
  • assess convergent and discriminant evidence without treating one cutoff as definitive; and
  • create scale scores only after the measurement evidence supports the intended use.

Users should be comfortable with data frames, item coding, descriptive statistics, and basic regression concepts. This workflow assumes independent respondents and ordered Likert items. Complex samples, multilevel data, longitudinal invariance, bifactor models, item response theory, and very small samples require extensions.

Analysis map

Stage Main question Primary evidence
Item audit Are the responses coded and distributed as intended? Valid ranges, missingness, floor/ceiling rates, item-total patterns
Reliability Are scores sufficiently precise for the intended use? Alpha, omega, uncertainty intervals, item diagnostics
EFA How many common dimensions are plausible? Parallel analysis, pattern loadings, communalities, factor correlations
CFA Does the proposed structure reproduce the observed associations? Ordinal CFA fit, loadings, residuals, local misfit
Invariance Is the measurement structure comparable across groups? Configural through strict models, changes in fit, partial invariance
Validity and scoring Is score interpretation adequately supported? AVE, factor correlations, discriminant checks, documented scoring rule

User settings

Adapt these settings before changing analysis code. Confirm item wording, response options, reverse-keyed items, proposed subscales, and the grouping variable against the approved instrument and codebook.

Simulate and import the example

The data-generating process is documented in DATA GENERATION.md. The saved CSV contains only variables a customer would ordinarily possess; the latent simulation values are not included.

Item coding and data audit

Coding and missingness

item invalid_n missing_n missing_percent
i01 0 21 2.6
i02 0 18 2.2
i03 0 16 2.0
i04 0 14 1.8
i05 0 17 2.1
i06 0 13 1.6
i07 0 37 4.6
i08 0 16 2.0
i09 0 21 2.6
i10 0 15 1.9
i11 0 14 1.8
i12 0 23 2.9
stage n
Imported respondents 800
Complete item responses 604
Excluded from complete-case examples 196

The example uses complete item responses so every method analyzes the same respondents. This is a teaching choice, not a default recommendation. For real data, investigate the missingness process and consider full-information or multiple-imputation methods compatible with ordinal measurement models.

Floor, ceiling, and response use

item floor ceiling mean sd
i01 11.04 12.07 2.98 1.19
i02 12.53 13.43 2.94 1.23
i03 12.24 10.71 2.97 1.19
i04 12.34 12.34 2.98 1.22
i05 10.98 10.34 2.98 1.15
i06 6.61 5.21 2.97 0.94
i07 12.45 12.98 2.99 1.22
i08 10.59 13.52 3.00 1.19
i09 9.76 10.53 3.01 1.13
i10 14.39 15.67 3.01 1.27
i11 11.45 9.03 2.93 1.14
i12 11.20 10.30 2.92 1.17

Floor and ceiling percentages are descriptive prompts. Their importance depends on the construct, target population, response scale, and intended use.

Corrected item-total patterns

scale item corrected_item_total
Research self-efficacy i01 0.633
Research self-efficacy i02 0.607
Research self-efficacy i03 0.570
Research self-efficacy i04 0.567
Methods support i05 0.558
Methods support i06 0.538
Methods support i07 0.321
Methods support i08 0.521
Reporting confidence i09 0.607
Reporting confidence i10 0.497
Reporting confidence i11 0.530
Reporting confidence i12 0.507

A weak item-total correlation is a reason to review wording, scoring, content coverage, and subgroup behavior. Deleting an item solely to increase reliability can narrow the construct and capitalize on sample-specific noise.

Pearson or polychoric correlations?

Pearson correlations treat observed item scores as approximately continuous. Polychoric correlations estimate the association between latent continuous response propensities underlying ordered categories. Polychoric correlations are often preferable for Likert items with few categories, skewed distributions, or threshold effects, but they can be unstable in small samples or sparse tables.

pair pearson polychoric difference
i01 with i02 0.528 0.571 0.042
i05 with i06 0.513 0.564 0.051
i09 with i10 0.443 0.486 0.043
i01 with i09 0.365 0.392 0.027

This workflow uses polychoric correlations for ordinal reliability and EFA, and WLSMV with ordered indicators for CFA. With five or more reasonably symmetric categories and a large sample, robust continuous estimators may also be defensible; document the decision rather than treating one method as universally correct.

Reliability with uncertainty

scale coefficient estimate conf_low conf_high
Research self-efficacy Ordinal alpha 0.810 0.786 0.837
Research self-efficacy Ordinal omega 0.811 0.787 0.837
Methods support Ordinal alpha 0.731 0.687 0.759
Methods support Ordinal omega 0.744 0.710 0.771
Reporting confidence Ordinal alpha 0.772 0.740 0.806
Reporting confidence Ordinal omega 0.775 0.743 0.807

Internal consistency concerns score precision under a particular model and sample. Alpha does not establish unidimensionality, and omega is only as credible as the factor model used to compute it. Interpret the interval and the stakes of the intended decision rather than applying .70 mechanically.

Parallel analysis and exploratory factor analysis

The example uses a reproducible development split for EFA and retains the proposed three-factor structure for CFA. In an actual scale-development study, the factor count should combine theory, parallel analysis, interpretability, communalities, and replication.

Parallel analysis

component observed random_95
1 4.029 1.417
2 1.773 1.310
3 1.408 1.226
4 0.774 1.159
5 0.665 1.102
6 0.656 1.053
7 0.543 1.003
8 0.535 0.951
9 0.482 0.906
10 0.422 0.862
11 0.370 0.809
12 0.343 0.757

EFA pattern matrix

item MR1 MR2 MR3
i01 0.819 0.039 -0.033
i02 0.604 0.040 0.090
i03 0.638 0.064 -0.010
i04 0.699 -0.103 0.049
i05 -0.073 0.856 0.014
i06 0.119 0.670 -0.006
i07 0.001 0.460 -0.055
i08 0.079 0.640 0.033
i09 0.089 -0.005 0.756
i10 -0.001 -0.006 0.575
i11 -0.060 -0.041 0.690
i12 -0.043 0.104 0.617

The weak loading for i07 is intentional. Review its content contribution before considering removal. A clean loading pattern in one sample does not replace cross-validation or content review.

Confirmatory factor analysis

measure value
Scaled CFI 1.000
Scaled TLI 1.001
Scaled RMSEA 0.000
SRMR 0.034
factor item loading p_value
Research i01 0.760 0
Research i02 0.744 0
Research i03 0.726 0
Research i04 0.705 0
Support i05 0.773 0
Support i06 0.667 0
Support i07 0.364 0
Support i08 0.692 0
Reporting i09 0.804 0
Reporting i10 0.654 0
Reporting i11 0.655 0
Reporting i12 0.665 0

In the validation sample, the ordinal CFA produced CFI = 1.000, TLI = 1.001, RMSEA = 0.000, and SRMR = 0.034. These indices summarize different types of misfit and should be interpreted together with loadings, residuals, model complexity, and theory.

Standardized loading plot

Local fit and theory-led respecification

lhs op rhs mi epc sepc.all
Research =~ i06 6.626 -0.298 -0.222
i05 i08 5.767 -0.399 -0.399
i02 i06 5.052 -0.305 -0.305
Reporting =~ i02 4.246 -0.292 -0.195
i01 i02 3.684 0.261 0.261
Reporting =~ i06 2.978 -0.184 -0.137
Support =~ i02 2.840 -0.223 -0.149
i02 i11 2.643 -0.206 -0.206
i05 i06 2.226 0.225 0.225
Research =~ i11 2.207 -0.172 -0.130

Modification indices identify parameters that could improve fit if freed one at a time. They do not identify the true model. Add cross-loadings or correlated residuals only when item content, method effects, and prior evidence justify the change, and disclose all respecification.

Measurement invariance across program types

The invariance sequence uses the full complete-response sample to provide stable group estimates. For ordered indicators, scalar invariance constrains loadings and thresholds. Strict invariance additionally constrains residual variances.

model cfi tli rmsea srmr delta_cfi delta_rmsea
Configural 1.000 1.001 0.000 0.034 NA NA
Metric 0.993 0.992 0.025 0.042 -0.007 0.025
Scalar 0.986 0.986 0.032 0.042 -0.007 0.007
Partial scalar: i06 thresholds free 1.000 1.000 0.003 0.036 0.014 -0.029
Strict 0.957 0.961 0.055 0.045 -0.043 0.051

Changes in approximate fit are descriptive evidence, not universal pass/fail rules. Commonly cited values such as an absolute CFI decrease near .010 and RMSEA increase near .015 must be considered alongside sample size, number of groups and items, estimator, parameter behavior, and substantive consequences.

Invariance comparison plot

Partial scalar invariance

The synthetic data deliberately shift the response thresholds for i06 in the practice-focused group. The partial scalar model frees those thresholds while retaining the remaining equality constraints. Partial invariance should be proposed only after locating a substantively plausible source of noninvariance and confirming that enough invariant indicators remain to identify and interpret the factor.

The scalar model changed CFI by -0.007 relative to the metric model. After freeing the i06 thresholds, the partial scalar model produced CFI = 1.000 and RMSEA = 0.003. These results support using a carefully documented partial-invariance model for this synthetic example; they do not justify searching for whichever freed parameters produce acceptable fit in a real dataset.

Convergent and discriminant evidence

factor AVE
Reporting 0.473
Research 0.519
Support 0.451
factor Research Support Reporting
Research 1.000 0.488 0.585
Support 0.488 1.000 0.437
Reporting 0.585 0.437 1.000
factor AVE sqrt_AVE largest_absolute_factor_correlation fornell_larcker_supported
Reporting 0.473 0.688 0.585 TRUE
Research 0.519 0.721 0.585 TRUE
Support 0.451 0.672 0.488 TRUE

Average variance extracted (AVE) summarizes the variance captured by a factor relative to item variance. The Fornell–Larcker comparison asks whether the square root of AVE exceeds the factor’s largest absolute correlation with another factor. These summaries can identify concerns, but neither establishes construct validity. Strong content evidence and predicted relations with external variables are also needed.

Scale-score decision and export

factor AVE omega fornell_larcker_supported evidence_summary
Reporting 0.473 0.775 TRUE Questionable; review intended use and item evidence
Research 0.519 0.811 TRUE Supported for an illustrative mean score
Support 0.451 0.744 TRUE Questionable; review intended use and item evidence

Mean scores preserve the original 1–5 response metric and are easy to explain. Before using them, define the minimum number of completed items, document handling of missing responses, and confirm that equal item weighting is consistent with the scoring purpose. Do not compare latent or observed group means unless the chosen invariance model supports that comparison.

Dynamic results narrative

A psychometric evaluation was conducted using 800 synthetic respondents, of whom 604 had complete item responses. Item responses were within the planned 1<U+2013>5 range after reverse scoring. Corrected item<U+2013>total correlations identified i07 as the weakest item (r = 0.32), prompting content and subgroup review rather than automatic deletion. Ordinal omega estimates ranged from 0.74 to 0.81 across the three proposed subscales. Parallel analysis suggested 3 factors in the development sample. A three-factor oblimin EFA was examined because the proposed dimensions were expected to correlate. In the independent validation split, the ordinal three-factor CFA yielded CFI = 1.000, TLI = 1.001, RMSEA = 0.000, and SRMR = 0.034. Fit indices, standardized loadings, and local diagnostics were considered jointly. Across program types, the metric model produced CFI = 0.993. Constraining item thresholds in the scalar model changed CFI by -0.007. A planned partial scalar model freed the four thresholds of i06, the item whose response process was deliberately shifted in the synthetic data; this model yielded CFI = 1.000 and RMSEA = 0.003. The partial model illustrates how a theoretically defensible exception can be documented; it should not be treated as permission to free constraints until fit becomes acceptable.

Reporting checklist

  • State the construct, intended score use, target population, response scale, and proposed dimensional structure.
  • Describe item wording, reverse scoring, allowable values, missing-response handling, and exclusions.
  • Report the correlation type, extraction method, rotation, factor-retention evidence, and sample used for EFA.
  • Identify the CFA estimator, ordered indicators, identification approach, sample, and all model modifications.
  • Report multiple fit indices, standardized loadings, factor correlations, and any local misfit reviewed.
  • Define the grouping variable and report group sample sizes.
  • State which parameters were constrained at each invariance level.
  • Report changes in fit and the substantive rationale for every partially freed parameter.
  • Avoid group-mean comparisons unless the retained invariance model supports them.
  • Report alpha or omega with uncertainty and clarify the reliability model.
  • Document the final score formula, minimum completed items, and treatment of missing item responses.
  • Keep validity conclusions within the sample, measures, design, and intended use.

When this model is not enough

  • Very small samples or sparse response categories: Polychoric correlations and WLSMV models may become unstable. Consider collapsing categories only when substantively defensible, simplifying the model, or collecting more data.
  • Strong general factor plus specific domains: Compare theory-led higher-order or bifactor models, but examine reliability of specific factors and avoid using bifactor fit as an automatic solution.
  • Items nested in repeated waves or people nested in programs: Use longitudinal or multilevel measurement models.
  • Differential item functioning at the item-response level: Consider item response theory or MIMIC models.
  • Formative measures: Internal consistency and reflective factor models may be inappropriate.
  • Scale development: Add expert content review, cognitive interviewing, preregistered validation, criterion relations, and independent replication.
  • High-stakes classification: Evaluate conditional precision, decision consistency, fairness, sensitivity, and specificity; average reliability is insufficient.

Exports

file
cfa_fit_indices.csv
cfa_standardized_loadings.csv
convergent_discriminant_summary.csv
corrected_item_total_correlations.csv
efa_pattern_loadings.csv
item_coding_and_missingness.csv
item_floor_ceiling_summary.csv
measurement_invariance_comparison.csv
reliability_with_intervals.csv
scale_score_decision.csv
scored_synthetic_data.csv
synthetic_research_confidence.csv

References

  • Brown, T. A. (2015). Confirmatory Factor Analysis for Applied Research (2nd ed.). Guilford Press.
  • Flora, D. B. (2020). Your Coefficient Alpha Is Probably Wrong, but Which Coefficient Omega Is Right? Advances in Methods and Practices in Psychological Science, 3(4), 484–501.
  • Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90.
  • Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.
  • Svetina, D., Rutkowski, L., & Rutkowski, D. (2020). Multiple-group invariance with categorical outcomes using updated guidelines: An illustration using Mplus and the lavaan/semTools packages. Structural Equation Modeling, 27(1), 111–130.
Dissertation Stats Helper · Reproducible statistical guidance for doctoral research · Evaluate coding, dimensionality, reliability, invariance, and score use as connected decisions.