Program Evaluation in Instructional Design and Technology

Course Description

EME4083 is programme evaluation — determining whether an instructional programme, project or intervention actually worked.

Florida's statewide description is specific about the emphasis: students "will develop skills used in selecting the appropriate model for conducting various types of evaluations. A series of models will be reviewed and aligned with evaluation purposes and questions. Applying the appropriate evaluation model is critical to ensuring that interventions, programs, and projects are successful. Development of a comprehensive evaluation plan" aligns a model with data collection strategies for a specific evaluation purpose.

⚠⚠ Notice what the course is organised around: model SELECTION, not a single method. That is deliberate and it is the course's main intellectual content. Evaluation questions differ — did people like it, did they learn, did behaviour change, did the organisation benefit, should we continue, how do we improve it — and different questions require genuinely different designs. Choosing the wrong model produces a competent answer to a question nobody asked.

The other half of the course is a distinction students often meet for the first time here. Formative evaluation improves something while it is being built; summative evaluation judges it once it is finished. ⚠ They serve different audiences and different decisions, and confusing them is a common and expensive error — formative findings delivered summatively read as criticism, and summative findings delivered too late change nothing.

⚠ This course closes the loop the sequence opened. EME3624 asked what the need was; this one asks whether it was met — and it uses substantially the same data-collection methods at the other end of the project.

Two Florida public institutions carry this number — the University of North Florida and the University of West Florida — at three credits, with identical titles.

Learning Outcomes

Required Outcomes

• Explain the purpose and role of evaluation in instructional design and performance improvement.
• Distinguish formative from summative evaluation, and identify which a given situation requires.
• Describe the principal evaluation models and their underlying assumptions.
• ⚠ Select an appropriate model for a given evaluation purpose and set of questions, and justify the choice — the course's central skill.
• Formulate clear, answerable evaluation questions from a stakeholder's actual decision needs.
• Apply Kirkpatrick's four levels — reaction, learning, behaviour, results — and explain the well-known limits of the framework.
• Design data collection strategies aligned to the evaluation model and questions: surveys, tests, observation, interviews, extant data and performance metrics.
• Develop a comprehensive evaluation plan specifying questions, model, methods, sources, timeline and reporting.
• Analyse and interpret evaluation data, and distinguish what the data supports from what it does not.
• Report findings to stakeholders usefully — including findings that a programme did not work.

Optional Outcomes

• Apply return-on-investment methods and explain their assumptions and limitations.
• Apply utilisation-focused or participatory evaluation approaches.
• Address evaluation ethics — independence, confidentiality, conflicts of interest and reporting honestly under pressure.
• Apply transfer-of-training measurement, which is where most evaluation actually fails.
• Conduct a usability evaluation of instructional materials.

Major Topics

Required Topics

• Purposes of evaluation — improvement, accountability, decision support, and knowledge generation.
• Formative and summative evaluation — audiences, timing and use.
• Evaluation models — Kirkpatrick; Phillips' ROI extension; CIPP (context, input, process, product); goal-free evaluation; utilisation-focused evaluation; responsive and participatory approaches.
• Model selection — matching model to purpose, question, audience and constraint.
• Evaluation questions — deriving them from stakeholder decisions rather than from convenience.
• Evaluation design — comparison, pre- and post-measurement, control, and the practical limits on each in organisational settings.
• Data collection — instruments, sources, sampling, and alignment to the chosen model.
• Measuring learning and behaviour change — assessment design, and the difficulty of measuring transfer to the job.
• Analysis and interpretation — what the data supports, confounds, and alternative explanations.
• Reporting and use — writing for stakeholders, and the conditions under which findings actually get acted on.
• Evaluation ethics and independence.

Optional Topics

• Return on investment and cost-benefit analysis in training.
• Transfer of training and the factors that determine it.
• Learning analytics and data from learning platforms.
• Usability evaluation of instructional materials.
• Programme evaluation in grant-funded and public-sector contexts.
• Evaluation capacity building within an organisation.

Resources & Tools

• Evaluating Training Programs: The Four Levels (Kirkpatrick) — the framework everyone in the industry names, and worth reading directly rather than second-hand.
• Program Evaluation: Alternative Approaches and Practical Guidelines (Fitzpatrick, Sanders and Worthen) — the standard comprehensive text on models, and best aligned to this course's model-selection emphasis.
• Utilization-Focused Evaluation (Michael Quinn Patton) — the strongest argument that an evaluation nobody uses has failed regardless of its rigour.
• Evaluating Training Programs and ROI material from Jack Phillips for the financial-return extension.
• American Evaluation Association — professional body; its Guiding Principles for Evaluators are short, freely available and directly relevant to the ethics component.
• ATD (Association for Talent Development) for industry practice and benchmarking data on how organisations actually evaluate.
• Survey and analysis tools — Qualtrics, Microsoft Forms, spreadsheets; statistical software where the course goes further.
• ⚠ A real programme to evaluate. As with needs assessment, the most valuable version of this course involves an actual client — and an evaluation plan for a real programme is strong portfolio material.

Career Pathways

• Instructional designer and training and development specialist (SOC 13-1151) — evaluation is a specified stage of the design process and a routine professional expectation.
• Training and development manager (SOC 11-3131) — ⚠ managers are the ones asked to justify a training budget, and evaluation evidence is what that justification is made of.
• Programme evaluator — a profession in its own right, in consulting, government, non-profit and foundation settings.
• Performance consultant; organisational development and human resources analytics roles.
• Institutional research and assessment in higher education — closely related methods, and a stable employment sector.
• Quality improvement roles in healthcare and other regulated sectors.
• ⚠ Grant-funded and public-sector work, where evaluation is frequently a contractual requirement rather than an option — relevant to the Orlando simulation and training cluster (Naval Air Warfare Center Training Systems Division, Army PEO STRI, the UCF Institute for Simulation and Training and their contractors), where training effectiveness evaluation is a specified deliverable.
• Other Florida employers: the theme parks and hospitality groups, major healthcare systems, school districts and state agencies.

Special Information

Offering Notes

Institution	Its title	Credits	Contact hours
University of West Florida (SUS)	Program Evaluation in Instructional Design and Technology	3	not published
University of North Florida (SUS)	Program Evaluation in Instructional Design and Technology	3	not published

✅ Both carriers use the statewide title unchanged and award three credits, and UWF's description matches the statewide description almost word for word. The 45 contact hours recorded here is Florida's convention for a three-credit lecture course. UWF places the course in the School of Education, Department of Instructional Design and Technology.

⚠ UNF's catalogue was not reachable when this guide was written — its online catalogue is client-rendered and its archived catalogue PDFs are blocked to automated retrieval. The matching titles make divergence unlikely, but UNF students should treat their syllabus as governing.

⚠⚠ Kirkpatrick is what industry uses — and knowing its limits is part of the professional skill

Worth setting out because a student will be asked about these four levels in an interview, and because the framework's weaknesses are as important as its structure.

The four levels — reaction, learning, behaviour, results — are close to universal vocabulary in corporate training, and fluency with them is professionally expected.

⚠ What actually happens in organisations is that evaluation stops at level one. The reaction survey at the end of a course — the "smile sheet" — is cheap, immediate and almost entirely uninformative about whether anyone learned anything or changed any behaviour. The evidence that satisfaction ratings predict learning is weak, and this is among the better-established findings in the field.

⚠⚠ Levels three and four are where evaluation would be valuable and where it is hardest. Measuring whether behaviour changed on the job requires access, time and cooperation; attributing organisational results to a training intervention runs into every confound imaginable. The professional skill is designing something feasible that still answers a real question — not proposing a perfect design nobody will fund.

⚠ The framework has also been criticised substantively — for implying a causal chain between levels that the evidence does not support, and for encouraging evaluation to be bolted on afterwards rather than designed in. A student who can discuss those criticisms as well as recite the levels is demonstrably better prepared than one who cannot.

⚠⚠ An evaluation nobody uses has failed

The argument the utilisation-focused tradition makes, and it is worth taking seriously because it changes how the work is done.

A methodologically impeccable evaluation that arrives after the decision, answers a question nobody asked, or is written so that no stakeholder reads it, has not succeeded. ⚠ Use is a design criterion, not an afterthought — which means identifying who will act on the findings and what decision they face before designing the study.

⚠⚠ The practical corollary is uncomfortable and worth naming: evaluation findings are frequently unwelcome. Someone commissioned the programme, defended its budget and may be evaluated on its success. An evaluation concluding it did not work is a political document as well as a technical one. Delivering it honestly and usefully — with evidence, with appropriate confidence, and framed around what to do next — is the professional skill, and it is why the American Evaluation Association's guiding principles address independence and integrity explicitly.

Position in the curriculum

The evaluation stage of the sequence — the final E in ADDIE — taken after EME3351 (foundations), EME3624 (needs assessment) and typically alongside or after EME4674 (Development).

⚠⚠ The pairing with EME3624 is the one to notice. Needs assessment and evaluation use substantially the same data-collection and analysis methods at opposite ends of a project, and they ask complementary questions — what is the gap and did we close it. A student who sees them as one methodological skill applied twice understands the field considerably better than one who treats them as two separate methods courses.

It feeds the capstone EME4684, where an evaluation component is common. No prerequisite is recorded statewide or by UWF, though the sequence's logic assumes the earlier courses.

Transfer

⚠ A 4000-level course carrying upper-division credit, which a bachelor's degree requires and a lower-division course cannot supply.

Florida's statewide record classifies it as transferable to an institution offering the same course, with no Gordon Rule designation and no general-education category — programme coursework. It is marked available for dual enrolment with elective high-school credit; ⚠ that marking is near-universal across the EME prefix, so it carries no information about this course.

✅ Matching titles and credits make this a straightforward transfer. Keep your syllabus.

AI Integration

⚠⚠ Evaluation is the stage of instructional design where AI is most genuinely useful and where the temptation to misuse it is most serious, because the output is a claim about whether something worked.

The legitimate uses are substantial. Drafting survey instruments and interview protocols, suggesting themes across open-ended responses you collected, summarising data, restructuring a report for a particular audience, and drafting alternative framings for a difficult finding — all of these are real work done faster. Qualitative analysis support in particular is a genuine improvement, provided the tool is checking your coding rather than replacing it.

⚠⚠⚠ The misuse is specific and it is worth naming bluntly: an evaluation finding is a factual claim about a real programme, and a generated one is a fabrication. A model asked to evaluate a training programme will produce a fluent, professionally structured report describing the results such programmes typically have. It will look right. It will not be about the programme, because the tool has no data from it. Presenting that to a client or a sponsor is not a shortcut — it is inventing research findings that an organisation will act on.

⚠ There is a subtler version worth watching for, since this course teaches model selection. Asked which evaluation model to use, a model will recommend Kirkpatrick, because Kirkpatrick dominates the written record. The course's whole point is that model choice should follow from the evaluation question, the audience and the decision at stake — and a recommendation generated from base rates rather than from your situation is exactly the default-to-the-familiar reasoning the course exists to replace.

There is also a forward-looking point that belongs here specifically. ⚠⚠ Organisations are increasingly buying AI-driven training tools — adaptive platforms, generated content, automated coaching — and someone has to evaluate whether they work. That is this course's skill applied to a new class of intervention, and the evaluation questions are the ordinary ones: did learning happen, did behaviour change, compared to what, and how do we know. Claims about AI-driven learning tools are currently made with far more confidence than evidence, and a graduate who can design an honest evaluation of one is unusually valuable.