Assignment #3 Evaluation Worksheet

Planning a Program Evaluation: Worksheet

Steps in Program Evaluation

Manage the evaluation § Human Subjects Protection § Timeline § Responsibilities § Budget		1. Engage stakeholders
		2. Focus
		§ Describe program-logic model § Define purpose § Determine use/users § Determine key questions § Select indicators § Determine design	3. Collect data
			§ Identify sources § Select method(s) § Pilot test § Set schedule § Determine sample	4. Analyze & interpret
				§ Process data § Analyze § Interpret data § What did you learn? § What are the limitations?	5. Use
					§ Share findings and lessons learned § Use in decision making § Determine next steps
Standards of evaluation: · Utility     · Feasibility     · Propriety     ·Accuracy

           Engage Stakeholders

Who should be involved?

Mount Royal Collegiate, Classroom teacher, Paul Martin Education Program Rep., Aboriginal students, program sponsors

How might they be engaged?

Hearing about interest level and success of students involved and seeing students pursue business opportunities in the future.

           Focus the Evaluation

What are you going to evaluate?  Describe program (logic model).

The Martin Aboriginal Education Program is a course designed to improve the graduation rates of Aboriginal students in Canada by introducing them to entrepreneurship and business opportunities.  They will be able to learn business strategies and methods to give them an advantage in the work force after high school.  If students wish to pursue a business after high school they will be granted $500 to assist with their startup costs. I will evaluate if the information they are receiving will prepare them to seek business opportunities in the future as well as if they are actually planning on seeking business opportunities after high school. In addition, I will evaluate if the funding they receive as a program is adequate and allocated appropriately.

What is the purpose of the evaluation?

The purpose of the evaluation will be to determine if the program is providing the necessary skills to Aboriginal youth who are interested in beginning their own business in the future through a review of their available resources and engagement level.

Who will use the evaluation? How will they use it?

Who/users	How will they use the information?
Classroom Teacher	To adapt instruction to accommodate student learning and success
Martin Aboriginal Education Program	To review the coordination of funding and available resources to the classroom teacher
Program Sponsors	To review the funding and practical knowledge provided to students involved in the program

What questions will the evaluation seek to answer?

Is the funding model for the program adequate?

Are enough resources available to the students and classroom teacher?

Is the program actually seeing Aboriginal students pursuing business careers in the future?

What information do you need to answer the questions?

What I wish to know	Indicators – How will I know it?
If the funding provided to Mount Royal students is adequate.	Reviewing what is being done in the program and how funding is allocated to the students. Need to determine if there are opportunities being missed due to insufficient funds.
Are students gaining valuable information from the class	Interviewing or surveying students
Are teachers seeing an interest level from the students	Interviewing or surveying the teachers
Are students choosing to pursue business ventures in the future?	Reviewing statistics on the number of students, that were part of the program, seeking business opportunities after high school

When is the evaluation needed?

By the end of the school year to allow for adjustments of the program for the 2014 school year.

What evaluation design will you use?

For this evaluation I will be using Scriven’s approach of a blended formative and goal-oriented evaluation.  This will allow for on-going information to help adapt the program in the future but at the same time evaluating if the program is achieving its intended goals.

           Collect the information

What sources of information will you use?

Existing information:	Martin Aboriginal Education Program website, program brochures, course outlines and resources
People:	Teacher, students, school administrators, MAEP administrators, business mentors
Pictorial records and observations:	Surveys, resource documents, and viewing lessons.

What data collection method(s) will you use?

x	Survey	x	Document review
x	Interview	x	Testimonials
x	Observation		Expert panel
	Group techniques		Simulated problems or situations
	Case study		Journal, log, diary
	Tests		Unobtrusive measures
	Photos, videos		Other (list)

Instrumentation: What is needed to record the information?

Online survey, voice recorder, program documents and resource list, and rooms for interviews.

When will you collect data for each method you’ve chosen?

Method	Before program	During program	Immediately after	Later
Survey	x	x		x
Interview	x	x	x
Observation		x
Document Review	x	x
Testimonials		x		x

Will a sample be used? 

No
Yes	x	If yes, describe the procedure you will use.
A sample will be used for interview purposes only.  The class size only consists of 10 students at this point so information should be able to be gathered from each individual student.

Pilot testing: when, where, how? 

This program is already in a pilot testing phase so the evaluation will just add to the review.

           Analyze and Interpret

How will the data be analyzed?

Data analysis methods:	Surveys will be compiled into a spreadsheet for analyzing. Interviews, observations, and testimonials will be audio recorded and listened to afterward to record information.
Who responsible:	The evaluator will be responsible for gathering and analyzing the data, as there will only be a smaller amount of data to collect from the group.

How will the information be interpreted—by whom?

Information will be interpreted by the evaluator with some collaboration with the classroom teacher.

What did you learn? What are the limitations?

Limitations will be getting unbiased and honest responses from the students as they may feel their answers could impact their mark in the class.  Due to the program being early in its stages it may be difficult to get fully informed answers from staff and students.  There may also be trouble getting information from parents due to the lower involvement of parents with this school.

               Use the Information

How will the evaluation be communicated and shared?

To whom	When/where/how to present
MAEP	In the summer through a written report, statistics, and graphs to allow for any alterations to the program for the 2014 school year
Classroom teacher	At the end of the school year through a collection of statistics and graphs indicating areas of success and improvement
Program sponsors	In the summer through a written report, statistics, and graphs to allow for any alterations to the program for the 2014 school year.
School Administrators	At the end of the school year through a collection of statistics and graphs indicating areas of success and improvement.

Next steps?

Gathering and analyzing data for the groups listed above in order to help improve and adjust parts of the program or provide details on the successes of the program to community members who don’t have the opportunity to regularly hear about the dynamics of the program.

           Manage the evaluation                   Standards

x	Human subject’s protection	x	Utility
	Management chart		Feasibility
	Timeline		Propriety
x	Responsibilities	x	Accuracy
x	Budget