Learning Research: Methods of Data Collection

When developing or evaluating learning, there are many data collection methods to consider. In addition, there are many criteria you should consider when making data collection method decisions. The collection of multiple types of data will allow for more confidence and validity in your conclusions.



Surveys/ Questionnaires - Strongly designed questions allow researchers to analyze responses and summarize results *low response rates

Interviews - One-on-one or focus group interviews use standard or probing questions *high response rates

Observation - A participant or non-participant in the research activity immerses himself in observing behavior and reactions of others

Psychometrics - "Self-report questionnaires, objective tests, and normative, criterion-referenced and idiogrpahic tests" (Brewerton & Millward, 2002)

Company records - Attendance, participation, operations reports, performance reviews, reports, blogs etc.

Not all research methods are appropriate or feasible in all situations. Consider the following criteria when making your decision: credibility, practicality, timeliness, accuracy, objectivity, scope, availability, cost effectiveness.

Reference:
Brewerton, P., & Millward, L. (2002). Organizational Research Methods. Thousand Oaks, CA: Sage Publications.

Learning Management: ADDIE model

The ADDIE model for eLearning project management has five stages: Analyze, Design, Development, Implementation, and Evaluation.



Analyze: After determinging the scope, market and audience of your eLearning porgram, you must "clarify organizational and training program objectives" (Lynch & Roecker, 2007). This includes identifiying risks, opportunities, budget restrictions, timing, and personnel needed.

Design: Objectives are transformed into a program structure including the sequence, durantion, and pace of proposed learning modules. Learning methods are identified, and administrative requirements are defined. This is the storyboard and instructional design phase.

Development: Creation of infrastructure, content related communication packs, multimedia, reference guides, job aids, necessary additional resources, and assessment criteria.

Implementation: Install all developed materials to delivery channels, set up administrative databases, roll out program communication, schedule learning sessions, and implement training.

Evaluation: Collect all training and project evaluation data, review participant and project performance for report to stakeholders. You should be able to deliver a program and project evaluation report upon completion of the evaluation.

Reference:
Lynch, M. M., & Roecker, J. (2007.) Project managing eLearning: A handbook for successful design, delivery, and management. Routledge. Chapter 1.

Learning Design: Interactive Multimodal Environments

“The most effective learning environments are those that combine verbal and non-verbal representations of knowledge using mixed-modality presentations.” (Mayer & Moreno, 2007) Multimodal learning environments involve both the auditory and visual sensory modalities of the learner. Interactivity, the characteristic of multidirectional communication, has the ability to further increase learning in a multimodal environment. Along a continuum of interactivity, learning environments can range from highly interactive to non-interactive. Based on the knowledge construction view of learning, interactive multimodal learning environments “guide [the] learner to actively make sense of the instructional materials”, toward promoting “deep cognitive processing.” (Mayer & Moreno, 2007)

Mayer & Moreno list the 5 types of interactivity as:
Dialoguing- ask question/ receive answer or give answer/ receive feedback
Controlling- determine pace and/or order of presentation
Manipulating- control parameters, zoom in or out, move on screen objects
Searching- find new content material through query, range of options, and selection
Navigating- select content areas from various available information sources

Clark & Mayer list several principles for multimodal learning as:
Contiguity- Present text and graphics in an integrated fashion
Redundancy - Avoid on screen text and simultaneous narration
Coherence - Avoid content inessential to instructional goal
Personalization- Use conversational style
Segmenting - Break complex lessons into segments
Pre-training - Elliminate extraneous processing by providing some material in advance

Reference:
Moreno, R., & Mayer, R. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19, 309–326.

Clark, R., and Mayer, R. (2007). e-Learning and the Science of Instruction. 2nd edition. San Francisco: Pfeiffer.

Learning Development: Leveraging Mobile Technology

“Personal, portable, wirelessly networked technologies will become ubiquitous in the lives of learners”, (Looi, Seow, So, Chen, & Wong, 2010) and informal learning through mobile technology represents a potential fundamental shift in the way people learn. Using mobile technology, learners can actively participate in learning outside of formal learning environments.

“Use of these technologies facilitates communication, collaboration, sharing and learning in informal settings with their peers, friends, and family unbounded by time and location.” (Looi, Seow, So, Chen, & Wong, 2010) However, learners can also use mobile devices as a supplement to formal learning in the classroom by connecting the mobile device to an online learning portal created by his/her learning community. “An online portal provides a platform where students can move from the individual space on the mobile device to the public space to facilitate collaboration and sharing.” (Looi, Seow, So, Chen, & Wong, 2010)

Distributed cognition theory, proposed by Edwin Hutchins, determines that learning occurs through three processes: social group, events/experiences in time, and between the interaction of material and environmental structure. In the case of mobile technology, learning can occur by way of distributing cognition among social collaboration in public learning spaces via a platform in the “cloud”, and through experience and/or artifacts found individually via informal learning in both real and virtually augmented environments.

Some of the difficulties encountered in evaluating the potential for informal and technology-enhanced learning through mobile devices are: individual student’s access to mobile devices, compatibility and cost of software and operating system of devices, inability to evaluate all learning activities outside a controlled environment, and ethical consideration.

“The notions for place, time, and space for learning have changed” (Looi, Seow, So, Chen, & Wong, 2010). With that, we must try to harness the potential impact of informal learning experiences upon students. The cognitive distribution of learning objects across social platforms, individual experiences, real and virtual spaces, and time through mobile devices has the potential to change how we learn, and should be used to leverage both individual and formal learning opportunities.

Reference:
Looi, C.-K., Seow, P., Zhang, B., So, H.-J., Chen, W. and Wong, L.-H. (2010), Leveraging mobile technology for sustainable seamless learning: a research agenda. British Journal of Educational Technology, 41: 154–169.