Assessing the assessment

Importance of methodology for measuring impact of immersion experiences

Kerri S. Warren, PhD*
Becky L. Spritz, PhD *
Andrew M. Staroscik, PhD*
*Roger Williams University

Post course questions

  • Did they learn?
  • Did they grow?
  • Was the class effective?

Collect Data

Survey: Likert-like questions

Look for trends in perception and learning

but is there a better way to dig into the rich dataset?

Disaggregate and visualize

Same mean, but not the same distribution
Chart shows this
Immersion Service Considerations

Interested in change over time.

Pre- and Post-surveys are common.

Want information on
  • Learning gains
  • Impact
  • Changes in perspective
Challenges
  • Measuring change
  • Finding statistical significance
Often looking at data like this

Q1 - Change!!

Q2 - No Change??

Often looking at data like this

Q1 - Change!!

Q2 - No Change??

Disaggregate and visualize to see patterns in data
Change
No change
Change or churn?
Visual patterns worth recognizing



Summary statistics for no change and churn can look similar


Difference can be seen using charts
Churn
The churn pattern can

  • Indicate confusion
  • Suggest varying reactions to challenging environment
  • Reveal opportunities for discussion topics for future sessions
No change
Is this change significant?
Visual statistics can help

  • Generate 19 randomized versions of data
  • Add the real data to the set to get a total of 20 data sets
  • Generate a chart for each set and shuffle them
  • Show panel of 20 charts to a naive audience
If an observer can identify the real data,
the pattern is significant at a p-value of 0.05
If an observer can identify the real data,
the pattern is significant at a p-value of 0.05
Real Data
Group moved towards SD with a p-vale <= 0.05
Q: I feel that social problems are not my concern.
Group moved towards SD with a p-value <= 0.05

Beauty of procedure is that it finds the major changes


Future direction - look at larger datasets

Final Thought

Use on-line tools to create data with continuous variables

Sliders:

Fin
Kerri S. Warren, PhD
kwarren@rwu.edu
Becky L. Spritz, PhD
bspritz@rwu.edu
Andrew M. Staroscik, PhD
ams@staroscik.com