Program Evaluation: Formal Interview and Likert Survey

Guiding Questions for the Overall Evaluation Process

1. Which skills do you wish to gain after the completion of this program?
2. How often do you apply critical thinking when making decisions?
3. What are your current oratory skills? Are you a good speaker? Do you like making speeches in public?
4. Which type of work is more effective for you: individual or in the team?
5. Are you able to cooperate with other people to meet common objectives?
6. Do you know much about working with people of other cultures? What should you pay attention to most of all?
7. Do you think informational technologies can be harmful to the environment?
8. What is hardware infrastructure? Would you like to know more about it?

Data Collection Strategies

There are a number of data collection strategies that can be applied in the case of this program. However, to make the results of the program more accurate, these data collection strategies were limited to several most important ones. There are hardly any universal approaches to data collection this is why those which are the most applicable to particular research can be regarded as the most useful and beneficial ones.

With respect to this program, two main data collection strategies were used. They are observational data and rating scales. Observational data were collected by means of informal and formal interviews. The data collected by these methods can then be arranged into graphs and tables, which would facilitate their interpretation. As far as the rating scales are concerned, the Likert survey was used to collect quantitative data. Basing on the results of this data collection method, the Likert scale can be drawn; this scale will help to evaluate the results of the program and introduce corresponding changes into it if the program will be carried out the next time.

Data Collection Tools

Two main data collection tools have been used in this program. They are interviews (formal and informal) and Likert surveys. Each of these methods ensured the researchers with valuable and reliable data helpful in the program evaluation.

Method 1: Interviews

Two different types of interviews have been used as data collection tools. With regards to the informal interview, the following purposes of this tool can be outlined:

• To find out the attitude of the participants towards the program;
• To discover any influence of the program on the participants (The interview has been shaped in a way that helped to define whether any of the participants’ skills have improved under the influence of the program).

Guiding questions for the informal interviews are as follows:

1. What is your attitude towards program planning for adults?
2. Do you think it is possible to develop new skills in the personnel?
3. How would you evaluate the facilities of this University?
4. Do you think it has enough potential for implementing such a program?
5. Have you personally noticed any positive effects of this program so far?

The implementation strategy for the informal interviews includes the following steps:

• Designing of questionnaires
• Assigning the interviewers to definite participants
• Collecting the data

The analysis of the informal interview’s findings will be performed by arranging the obtained data into the tables and shaping descriptive statistics from these data. The results will then be added to the final report and discussed by the staff members.

The purpose of the formal interview is to discover whether the participants have any judgments about the program, that is whether they consider it beneficial or non-beneficial and which changes they would introduce into the program if they could. The formal interview has the following guiding questions:

1. Were your communication skills developed before participating in this program?
2. Did the activities within this employee training program managers facilitate your communication with other people?
3. Have your computer acquisition skills been developed before entering the program?
4. Did you learn anything new about informational and computer technologies after joining this program?
5. Did you know anything about program management before the program?
6. Are there any difficulties that you currently have with project management? If so, do you believe you can easily cope with them after finishing this program?
7. What was more convenient for you before: working alone or working in a team?
8. Have you learned to collaborate with your colleagues and work in teams in the course of this program?
9. Did you exhibit any leadership skills before entering this program?
10. For the time of the program implementation, did you manage at least once to influence the decision of your whole group?

The implementation of the program will take place in the same steps as that of the formal interviews. After the implementation, the statistics allowing to compare the skills of the participants before and after the program will be compiled. It then will be attached to the final report and presented at the staff meeting.

Method 2: Likert survey

The Likert survey is one of the best methods to collect qualitative data. The purpose of this method is to evaluate the success of the program basing on the participants’ rating the following statements (guiding statements); data collection instruments may consist of slightly modified statements:

1. This program has given me much new knowledge
2. I am going to use this new knowledge in future
3. There are certain changes I would introduce into the program
4. I think this program is successful
5. I am sure that other participants benefited greatly from this program

The implementation of the program will take place in accordance with the following steps:

• Designing the statements
• Collecting all the participants and asking them to rate the statements
• Collecting the data from the participants

The results will then be arranged into the Likert scale and analyzed. The analysis will then be reported to the staff members.

Synopsis of the Program for Evaluation

The program under consideration is an employee training program developed to improve the quality of work and increase the efficiency of the working process. The aim of the program is to identify the factors that influence the university’s enrollment and marketing management team, as well as to address the needs of the employees with respect to the students’ online learning. Heads of the University, the educators, and the learners are the stakeholders of the training process; they all are expected to benefit from the program through the improvement of their knowledge and qualification. The objectives of this program focus on preparing the new university enrollment employees for the adjustment to the higher education environment. In frames of this program, the employees will be developing their communication skills, learning to manage changes, work in a team, and collaborate. After the implementation of this employee training program, the employees are expected to acquire project management skills, use critical thinking in evaluating the information, learn to apply different information technologies, and improve technical understanding of computing systems (Hauer & Daniels, 2008). It is necessary to conduct the evaluation of this program in order to determine whether it meets predetermined outcomes and learning objectives. To do this, a list of questions that will guide the evaluation will be developed together with three program evaluation documents designed; these documents will use informal and formal methods for gathering evaluation data two of which will use quantitative measures, namely formal interview and Likert survey.

List of Questions Guiding the Evaluation

Closed Questions

1. How old are you?
   1. 16-25
   2. 26-35
   3. 36-45
   4. 46-55
   5. 56-65
   6. 65 and older
3. How long have you been working for this organization?
   1. Less than a year
   2. 1-2 years
   3. 2-5 years
   4. 5-9 years
   5. 10 years and more

Open-Ended Questions

1. What position do you occupy in this organization?
2. What was the highest rank you have ever occupied?
3. Have you ever been dismissed due to the lack of communication, leadership, or any other skills?
4. How good are you with modern technologies (information and computer ones)?
5. What do you think about this program?
6. What do you expect from this program?

Program Evaluation Documents

Informal Interview

This type of program evaluation document will use an informal method of data collection for the program evaluation. This method will be an unstructured informal interview with the program participants. The unstructured interviews allow shaping the approximate questions for the structured ones (Kumar, 2005). The key purpose for this interview is to identify whether the employees consider it necessary to carry out higher education program planning for adults and how they evaluate the potential which the university has with regards to such programs (Kilgore, 2003). Another purpose of this activity is to register possible changes which the program in question produced on the participants. The interviews will not have a particular structure and will be conducted in accordance with the following guiding questions:

1. What is your attitude towards program planning for adults?
2. Do you think it is possible to develop new skills in the personnel?
3. How would you evaluate the facilities of this University?
4. Do you think it has enough potential for implementing such a program?
5. Have you personally noticed any positive effects of this program so far?

Several participants of the employee training program will be asked these questions. Their responses will be coded as “positive”, “negative” or “other”, which will facilitate the coding process (Babbie, 2009). The codes will be developed prior to the evaluation on the basis of the existing program theory (Wholey, Hatry, & Newcomer, 2004). The implementation of this activity will take place in three main stages. At the first stage, the guiding questions will be compiled. At the second stage, several interviewers will be assigned with a definite number of interviewees (total – 10 interviewees, 3-4 per each interviewer). And at the third stage, the data will be collected in the course of the interviews. Furthermore, the data will have to be analyzed. The responses will be arranged into a table, which will allow shaping descriptive statistics from them. The statistics will then be added to the results section in the evaluation report (Royse, Thyer, & Padgett, 2009) which will be handed out at the staff meeting and evaluated in detail by the staff members.

Formal Interviews

This activity will use formal methods for collecting the data, as well as quantitative measures which will present certain numeric data. Quantitative data are more reliable than qualitative ones; they can be obtained by means of carefully structured formal interviews in the course of which the group of respondents will be answering a specific set of questions (Wells, 2007). The key purpose of this instrument is to make judgments about the program, for instance, to identify whether the program is good, bad, below or above the expectations, and, possibly, to develop certain recommendations which the program managers will be advised to follow (Owen & Rogers, 1999). Guiding questions for this instrument will include (but are not limited to) the following:

1. Were your communication skills developed before participating in this program?
2. Did the activities within this employee training program managers to facilitate your communication with other people?
3. Have your computer acquisition skills been developed before entering the program?
4. Did you learn anything new about informational and computer technologies after joining this program?
5. Did you know anything about program management before the program?
6. Are there any difficulties that you currently have with project management? If so, do you believe you can easily cope with them after finishing this program?
7. What was more convenient for you before: working alone or working in a team?
8. Have you learned to collaborate with your colleagues and work in teams in the course of this program?
9. Did you exhibit any leadership skills before entering this program?
10. For the time of the program implementation, did you manage at least once to influence the decision of your whole group?

The implementation of this activity will take less time than the informal unstructured interviews because the questions will be asked in the same sequence for each of the participants and the answers to these questions will be limited (Myers, 1999). The interviews will take place face-to-face with the responses of each interviewee carefully recorded and then evaluated. The obtained data will then be shaped into statistics (contrasting the participants’ skills before entering the program and after taking part in it), which will allow not only discovering how many people benefited from the program but tracing positive impacts of the program on each interviewee. The statistics will then be presented at the staff meeting together with other findings in a separate section of the evaluation report.

Likert Survey

This instrument will also use the formal methods of data collection which will help to obtain quantitative data. The Likert survey is one of the best formal methods to use because it allows evaluating the success of the program providing additional quantitative data. This activity is formal because it will take place according to a definite scheme and an agreed number of respondents will take part in it. The Likert survey is less reliable than a face-to-face interview, but it is time-saving for it allows obtaining the data from all the participants at once. The key purpose of this activity is to measure the success of the program. The survey will have the following guiding statements:

1. This program has given me much new knowledge
2. I am going to use this new knowledge in future
3. There are certain changes I would introduce into the program
4. I think this program is successful
5. I am sure that other participants benefited greatly from this program

All the participants will be gathered in one room and will take part in the survey simultaneously. They will be asked to evaluate the statements by indicating whether they “agree”, “strongly agree”, “slightly agree” “disagree”, “slightly disagree”, or “strongly disagree’ with a particular statement (Spaulding, 2008). In this way, the Likert survey will help to measure the participant’s attitude towards the program and the rate of its success. The obtained data will be added to the evaluation report and presented at the staff meeting.

Reference List

Babbie, E.R. (2009). The Practice of social research. London: Cengage Learning.

Brinkerhoff, R.O. (2006). Telling training’s story: evaluation made simple, credible, and effective. San Francisco: Berrett-Koehler Publishers.

Hauer, A. and Daniels, M. (2008). “A Learning Theory Perspective on Running Open Group Projects (OEGPs).” Australian Computer Society. Web.

Khosrow-Pour, M. (2006). Emerging trends and challenges in information technology management. New York: Idea Group Inc.

Kilgore, D. (2003). Planning programs for adults. New Directions for Student Services, 102, 81-88.

Kumar, R. (2005). Research methodology: a step-by-step guide. London: SAGE.

Myers, A.M. (1999). Program evaluation for exercise leaders. Champaign: Human Kinetics.

Owen, J.M. & Rogers, P.J. (1999). Program evaluation: Forms and approaches. London: SAGE.

Royse, D., Thyer, B.A., & Padgett, D.K. (2009). Program Evaluation: An Introduction. London: Cengage Learning.

Spaulding, D.T. (2008). Program evaluation in practice: Core concepts and examples for discussion and analysis. New York: John Wiley and Sons.

Wells, M.K. (2007). Grantwriting beyond the basics book 3: Successful program evaluation. Portland: Continuing Education Press.

Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (2004). Handbook of practical program evaluation. New York: John Wiley and Sons.