Learning Evaluation: Sample populations

A sample is a "set of elements drawn from and analyzed to estimate the characteristics of a population." (Russ-Eft & Preskill, 2001) Sample populations are used in evaluations when it is not feasible to involve the entire target audience due to: time or cost constraints, limited accessibility of participants, and situations in which validity or accuracy may be compromised by engaging the entire population.

Probability Sampling
- Participants must be randomly selected; each member of population has an equal chance of being selected
- Used when you want to generalize the evaluation's findings
– Ability to determine sample error

Simple Random – Randomly select participants from entire population; however, it sometimes difficult to obtain a complete and accurate list of population
Stratified Random – Divide the entire population into sub-populations called strata, and then using simple random sampling within each strata
Proportional Stratified – Purposive oversampling of some strata and under-sampling of others
Cluster – Sample a larger unit, then sample within the larger unit

Non-Probability Sampling
- Evaluator selects participants
- Potential for bias and threats to validity
– Findings cannot be generalized

Convenience – Select participants that are most accessible, or participants agree on their own accord to participate
Purposive – Select specific individuals to participate because of position, experience, knowledge, or attitudes
Snowball – Contact organizations within industry and ask for referrals; used when you don’t have a list of potential participants

Reference:
Russ-Eft, D., & Preskill, H.(2001). Evaluation In Organizations: A Systematic Approach To Enhancing Learning, Performance, and Change (1st ed.) Perseus Publishing.

Image: http://www.paduiblog.com/tags/war-against-dui/page/2/

Learning Evaluation: Summative Assessments

Summative evaluations represent the final evaluative judgment of the merit, worth, or value of a program. They are often conducted for the benefit of external audiences and/or decision makers to determine the efficacy of the evaluand. Summative evaluations are often conducted by external evaluators, due to the potential for a Conflict of Interest among internal evaluators.

According to Russ-Eft & Preskill, the 4 main purposes of conducting summative evaluations are:

Monitoring + Auditing - To determine if a program was administered in ethical or legal ways, and to gauge if resources and funding were used appropriately
Example: Were the project results worth the project costs?

Outcome Evaluation - Assessment of the result of a program, based on changes in knowledge, attitude, and practices of the participants.
Example: To what extent did the program meet its goals?

Impact Evaluation - A form of outcome evaluation that assesses the net effect of a program as compared to the absence of the program
Example: To what extent has the program improved employee productivity?

Performance Measurement - Documentation of results based on either program process, product, or services delivered.
Example: In what ways did the participants benefit from the program?

Reference:
Russ-Eft, Darlene, & Preskill, Hallie (2001). Evaluation In Organizations: A Systematic Approach To Enhancing Learning, Performance, and Change (1st ed.) Perseus Publishing.

Learning Evaluation: Formative Assessments

Formative assessments mimic internal quality control within organizations, and in this case refer to the initial assessments and testing of newlt developed eLearning programs. Formative assessment criteria largely falls into two main categories: instructional design and interface design. The goal of conducting formative assessments is to gauge learning effectiveness. Lockee, Moore, and Burton (2003) outline 6 formative evaluation needs as follows:

Design Review - Fellow designers evaluate Instructional/Visual Design
Expert Review - Industry experts evaluate relevancy of Content/Subject Matter
One-on-One Review - Member of target audience review
Small group Review - Audience component/module testing
Field Trials - Small groups testing program in distance setting
Ongoing Reviews - Reanalysis, Redesign, Redevelop as necessary

Formative assessments should evaluate the program's ability to meet learning objectives and accessibility standards while also providing for the special needs of learners. Assessments during the development stage are proactive not reactive. As quoted by Dr. Robert (Bob) Stakes, "When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative." We will talk about summative evaluation in a later post :)

Reference:
Lockee, B., Moore, M., and Burton, J. (2002) Measuring Success: Evaluation Strategies for Distance Education. EDUCAUSE Quarterly.