Importance of methodology for measuring impact of immersion experiences

Kerri S. Warren, PhD^{*}

Becky L. Spritz, PhD ^{*}

Andrew M. Staroscik, PhD^{*}

- Did they learn?
- Did they grow?
- Was the class effective?

Survey: Likert-like questions

Look for trends in perception and learning

Disaggregate and visualize

Same mean, but not the same distribution

Chart shows this

Chart shows this

Interested in change over time.

Pre- and Post-surveys are common.

Want information on

- Learning gains
- Impact
- Changes in perspective

Challenges

- Measuring change
- Finding statistical significance

Often looking at data like this

Q1 - Change!!

Q2 - No Change??

Often looking at data like this

Q1 - Change!!

Q2 - No Change??

Disaggregate and visualize to see patterns in data

Change

No change

Change or churn?

Visual patterns worth recognizing

Summary statistics for no change and churn can look similar

Difference can be seen using charts

Churn

The churn pattern can

- Indicate confusion
- Suggest varying reactions to challenging environment
- Reveal opportunities for discussion topics for future sessions

No change

- Generate 19 randomized versions of data
- Add the real data to the set to get a total of 20 data sets
- Generate a chart for each set and shuffle them
- Show panel of 20 charts to a naive audience

If an observer can identify the real data,

the pattern is significant at a p-value of 0.05

the pattern is significant at a p-value of 0.05

If an observer can identify the real data,

the pattern is significant at a p-value of 0.05

the pattern is significant at a p-value of 0.05

Real Data

Group moved towards SD with a p-vale <= 0.05

Q: I feel that social problems are not my concern.

Group moved towards SD with a p-value <= 0.05

Beauty of procedure is that it finds the major changes

Future direction - look at larger datasets

Use on-line tools to create data with continuous variables

Sliders:

Kerri S. Warren, PhD

kwarren@rwu.edu

kwarren@rwu.edu

Becky L. Spritz, PhD

bspritz@rwu.edu

bspritz@rwu.edu

Andrew M. Staroscik, PhD

ams@staroscik.com

ams@staroscik.com