In the past, too often were instructors blamed for poor teaching and learners for poor learning when, in fact, the materials were not sufficient to support the instructional effort. This is why formative evaluation to newly developed materials, to selected and adapted materials, to instructor delivered instruction, and to combinations of these 3 presentation modes, is nowadays a must in order to ensure that instruction is properly implemented and managed! 

Formative evaluation is the collection of data and information during the development of instruction that can be used to improve the effectiveness of the instruction (Dick and Carey, p. 277; based on Cronbach, 1975 and Scriven 1967).

=> Formative evaluation is the process designers use to obtain data that can be used to revise their instruction to make it more efficient and effective (gathering information from learners in order to revise the materials before proceeding with the design process).

The emphasis in formative evaluation is on the

1. collection and analysis of data
2. revision of the instruction

[Two related activities that share many of the same principles as formative evaluation:  usability testing and rapid prototyping]

Three basic phases of formative evaluation: (p.279)

1. One-to-one or clinical evaluation: the designer works with individual learners to obtain data to revise the materials
2. Small-group evaluation: a group of 8 to 20 learners (who represent the target population) study the materials on their own and are tested to collect the required data
3. Field trial: often 30 learners are sufficient. The emphasis here is on the testing of the procedures

The three phases of formative evaluation are typically preceded by the review of instruction by interested specialists who are not directly involved in the instructional development project, but have relevant expertise.

Designing Formative Evaluations (FEs): (p.279)

The purpose for the formative evaluation is to pinpoint specific errors in the materials in order to correct them.

The best anchor or framework for the design of the FE is the instructional strategy: create a matrix that lists the components of the instructional strategy on one side and the major areas of questions about the instruction on the other.

The 5 areas of questions that would be appropriate for all materials:

1. Type of learning: are the materials appropriate for the type of learning outcome? (ask expert)
2. Content: do the materials include adequate instruction on the subordinate skills, and are these skills sequenced and clustered logically? (ask expert)
3. Clarity: are the materials clear and readily understood by representative members of the target group? (ask target learners)
4. Motivation: do learners find the materials relevant to their needs and interests? (ask target learners)
5. Management: can the materials be managed effectively in the manner they are mediated? (ask expert & target learners)

In designing instrumentation for gathering information from learners, you must consider:

1. the phase: one-to-one, small-group, field trial
2. the setting: learning or performance context
3. the nature of the information being gathered

The types of data the we need to collect:

1. data collected on entry behavior tests, pretests, posttests, and performance context
2. learners' comments
3. data collected on attitude questionnaires and/or debriefing comments (reveals learners' overall reactions to instruction)
4. time required
5. SME's reactions
6. manager or supervisor's reactions

Relevant Link:

Example of a Formative Evaluation

Posted by Nada at 12:01 AM EDT

Updated: 05/05/09 1:17 AM EDT

Share This Post

Post Comment | Permalink

Summative evaluations -- which link us back to the needs assessment-- are conducted to document the strengths and weaknesses of instructional materials, to verify their effectiveness with target learners, and to make decisions about whether to maintain currently used materials or adopt materials that have the potential for meeting an organization's defined instructional needs. (pp.340-341)

Summative evaluations answer the questions:

• "Did the intervention, including the instruction, solve the problem that led to the need for the instruction in the first place?" (p.340)
• "Did it fill the identified gap?" (Dr. Watkins)

1. Expert Judgment Phase: to determine the potential of candidate instruction for meeting the needs of an organization.
   • Overall decisions: do the materials have the potential for meeting this organization's needs?
   • Specific decisions: (p.341)
     1. Congruence analysis: are the needs and goals of the organization congruent with those in the instruction? > information summary form
     2. Content analysis: are the materials complete, accurate, and current?  > product checklist or rating scale
     3. Design analysis: are the principles of learning, instruction, and motivation clearly evident in the materials?  > product checklist or rating scale
     4. Utility & Feasibility analysis: are the materials convenient, durable, cost-effective, and satisfactory for current users?  > information summary form, product checklist, or rating scale
2. Field Trial Phase: to document the effectiveness of instruction with target learners in the intended setting.
   • Overall decisions: are the materials effective with target learners in the prescribed setting?
   • Specific decisions: (p.341)
     1. Outcomes analysis: > criterion-reference tests, attitude questionnaires, interviews, observations, company records
        • Impact on learners: are their achievement and motivation satisfactory after instruction?
        • Impact on job: are learners able to transfer information/skills/attitudes to the job setting or to units of related instruction?
        • Impact on organization: are learners' changed behaviors (performance, attitudes) making positive differences in the achievement of the organization's mission and goals?
     2. Management analysis: > questionnaires, interviews, observations, company records
        • Are instructor and manager attitudes satisfactory?
        • Are recommended implementation procedures feasible?
        • Are costs related to time, personnel, equipment, and resources reasonable?

Formative Evaluation Vs. Summative Evaluation:

• Formative Evaluation: to collect data in order to revise instruction
• Summative Evaluation: to collect data in order to document the strengths and weaknesses of instruction

Posted by Nada at 12:01 AM EDT

Updated: 05/05/09 12:02 AM EDT

Share This Post

Post Comment | Permalink

It goes without saying that distance education is gaining tremendous grounds and is very likely to become The Future of Education. Hence, it needs to be taken very seriously; researchers should team up to find the best ways to develop excellent distance education course materials. The ISD process is in itself an excellent system for the regular classroom. So, one would think that it would be a great idea to adapt it to distance education, just as Young & Young (2002) did. Indeed, their ISD-Web Conversion Process is very comprehensive and can easily be followed for the development of distance education course materials.

The approach to distance education/training that I would follow, based on Young & Young’s article (outlined below), is a combination of online self-study materials that would be very well presented and organized (with media rich simulations), virtual classrooms that “enable live audio AND video-based interactions”, and computer-mediated learning communities “in which learners meet asynchronously in online groups to debate, test, and exchange ideas, as well as receive expert advice.” Besides, since learners in general are so fascinated by Facebook and Twitter, we can make use of one of those two services to promote human interaction. In addition to all the aforementioned, I strongly recommend that the instructor require two class meetings per semester: one towards the beginning and another towards the end. This way, the “human” presence will not be sacrificed.

In order to go into more details, here is how, based on Young & Young’s article, I see the ID process being used for the development of distance education course materials: after re-analyzing the audience, the learning context, the existing materials, and the delivery system, we need to re-design that delivery system, the interface (the overall structure of the Web environment which fosters learners’ information processing—making it user-friendly), the instructional strategies (by cultivating a new view of learning, one that emphasizes how and why, as opposed to what—incorporating a lot of practice and feedback), and assessment (online scoring and security). Afterwards, we need to re-develop materials and make them very interesting and captivating, implement technical deployment and integration and, finally, formatively evaluate to ensure an error-free environment (there should be no errors or shortcomings in how the material is displayed or accessed) and incorporate online evaluations.

As for the instructor/teacher’s role in the design of distance education materials, I firmly believe that it should not be ignored or belittled. Just as no system can replace a mother, no matter how good or less good (not to say bad) she is to her children, no system should replace a teacher or instructor. The human interaction between an instructor and his/her students is invaluable, whether the instructor is good or less good. It is true that some learners have negative experiences with their instructors; it is also true that some children have negative experiences with their mother. Do we remove their mother from their lives?   I am a believer in the instructor’s positive impact on the learners who learn so much more from him/her than just the content of the subject matter. Why should we accept to lose that? So, in order for distance education (based on an adapted version of ISD) to be successful, it needs to give an appropriate role to the teacher/instructor so that the instructor and the students will still have this bond that is developed in regular classrooms.

Nada S.A.

On using the ISD Process for the Development of Distance Education Course Materials:

Based on the article by Andrea & James Young [2002] titled "Converting existing training products for the Web: a new look at the old ISD process" :

1. Online self-study materials
2. Virtual classrooms
3. Computer-mediated learning communities
4. Knowledge bases and decision-support tool

E-learning

• is time-independent
• has a more reference-oriented (less didactic) approach to training
• has a different concept of how, where, and when learning takes place
• has a different delivery medium: the Web

ISD-Web Conversion Process:

1.  Re-Analyze:
   • audience

     • Entry-level knowledge and skills (prerequisites)

     • Computer and Internet literacy (Internet access? Same bandwidth?) 

     • Cultural diversity

     • Spoken language(s)

     • New learning application on the job (is it the same for new and original audience members?)

   • learning context

     • Place: where learners will access the training (the office, home, a computer lab)

     • Time: when learners will access the training (during the workday, at night, and/or on weekends)

     • Frequency: will learners use the training for short periods to solve work-related problems, and/or engage with the training for prolonged periods of intense study time?

     • Materials: hardware and software needed to complete the training (do any plug-ins or software applications need to be installed before users can receive the training?)

     • Internet speed: at what speed will users connect to access the training materials?

   • existing materials: not all existing materials can or should be converted for the web. Many interpersonal and problem solving skills associated with management development are difficult to achieve without some face-to-face interaction.

     Decisions about whether or not to convert, and what blend of e-learning solutions to select are influenced by several factors:

     • Types of learning outcomes

     • Budget and time constraints

     • Capabilities of the delivery system (which is supposed to support the delivery method).

   • delivery system: we need to know exactly what the system is or is not capable of before we begin to re-design training for the Web.
2. Re-Design:
   • delivery system:
     • If you have decided to purchase a commercial delivery system, the features of the system you select will drive many of the delivery system design issues > your task is to choose which features you wish to enable.
     • If you have decided to develop a delivery system from scratch or perhaps customize an existing one > you are faced with a rather large design task. > what attributes and functionality can you afford given your timeline and budget?
   • interface:
     It refers to the overall structure of the Web environment, including patterns of navigation and organization of related groups of information and the graphical presentation and formatting conventions used on individual Web pages. The interface design fosters learners’ information processing.
   • instructional strategies:

     The instructional strategies that worked in the original training will not transfer to the new e-learning environment because modes of interaction, degree of learner control, access to information and learning resources, methods of organizing content, conceptions of time, and technology differ so profoundly in the new environment.
     When making decisions about how to redesign instructional strategies for e-learning, it is helpful to cultivate a new view of learning; one that emphasizes how and why, as opposed to what.

   • assessment:

     When designing assessments to be used in an e-learning environment, there are three primary issues we must address:

     • How to measure learning outcomes online

     • How to score the results of online assessments and provide learner feedback

     • How to maintain a secure online assessment environment

3. Re-Develop Materials:
   • acquire and re-purpose existing materials:

     Issues to address in the revised copyright agreement include the following:

     • The size of the new audience

     • The use of a password-protected site to limit access to the materials

     • The layout and resolution of the materials in digital format

   • assemble courseware: Reduce file size and use templates
   • enable learning management
4. Implement:
   • technical deployment: it involves installing the delivery system platform and courseware on servers, configuring the delivery system platform (for example, adding users and courses to the system), addressing firewall and network access issues, and, in some cases, configuring end-user computers to prepare for e-learning.
   • integration: it refers to the human or soft side of implementation.
5. Formatively Evaluate:
   • quality assurance: an extensive review of the functionality of the course and delivery system is necessary to ensure an error-free environment (there should be no errors or shortcomings in how the material is displayed or accessed). In the software development world, this is referred to as quality assurance (QA).
   • test usability: additional data concerning the usability of the course delivery system should be collected (usability testing).
   • take evaluation online: by using an online form to gather input from learners, you can expect to increase access to geographically dispersed participants.

This retooled ISD process differs from the traditional ISD process in three respects.

1. First, it envisions the learning as a dynamic and highly personalized event; one that can take place anytime, anywhere, and in any way.
2. Second, it assumes the use of existing materials and thus has been tailored to meet the challenges of a conversion effort.
3. Third, it presupposes the Web as the primary delivery media and attempts to take advantage of its unique capabilities to enable superior learning, and ultimately, superior performance on the job.

Relevant Article:

"The Future of Distance Learning: Defining and Sustaining Useful Results" by Kaufman, Watkins, & Guerra (2001)

Posted by Nada at 12:01 AM EDT

Updated: 05/05/09 12:03 AM EDT

Share This Post

Post Comment | Permalink

If you are interested in seeing how ISD can be applied:

- Click HERE for a Short Story Unit (by Nada S.A.)

- Click HERE for a Running Training Program (by Carol L.P.)

- Click HERE for a Study Strategy Plan for Reading and Summarizing
  a 200-Page Novel in German (by Susanne H.W.)

• Instructional Analysis
• Goal Analysis + Learning & Context Analyses
• Objectives, Assessment Instruments and procedure, Instructional Strategy
  

Posted by Nada at 12:01 AM EDT

Updated: 05/06/09 9:46 AM EDT

Share This Post

Post Comment | Permalink