External Evaluator Finds Unexpected Gains in Basic Skills

Denver coordinator Deb Mumford and her staff didn't really need an external evaluator to tell them their summer program was a success. The change in student attitude alone told them they'd done well. The Career Choices curriculum, unlike other materials participants had used, proved popular because it addressed real issues, prompted response, and made for positive debates. Even the speech of these at-risk youth showed dramatic improvement.

Still, it had to be satisfying to read Dr. Charles Branch's report which found nothing about the program itself to criticize except the quality of the lunches, something that both students and instructors found not to their liking.

Dr. Branch recommended the following: program expansion to include more participants at more sites; identification of participants earlier in the process; more cooperative planning between community-based organizations and schools; a longer academic enrichment program; full and early funding; and establishment of a year-long pilot program using the "real world" summer experience as a model.

"The most unexpected positive result was the significant gain made in basic skills in such a short time," he reported. "While the significant improvement can be attributed to the interdisciplinary approach, most of the credit goes to highly competent and dedicated staff and volunteers." During the six days he spent observing the staff, administrators and volunteers working with the participants, Dr. Branch observed the positive influences on the participants. Interviews of participants, staff and administrators indicate that a true sense of community, mutual respect, personal growth and responsibility were developed.

Dr. Branch added: "The program did provide 90 hours of quality instruction for the youth which produced significant improvement in academic basic skills as well as positive attitudinal changes towards learning. The vast majority of participants did connect academic learning to practical life skills and the majority of participants' self-concepts as learners did improve."

Pre- and post-WRAT tests showed significant positive gain in reading scores among seven of eight groups, with a positive gain in the eighth, as well. In math scores, five of eight groups scored significantly higher in post tests, and all eight showed positive gains. In addition, since participants were required to keep a journal throughout the program, a selection of writing samples showed that writing skills improved as they moved through the program. Attitudes toward self, school, teachers, and society significantly improved over time.

The results are especially gratifying because, as coordinator Mumford said, "We had kids from two different sides of town, two different ethnic backgrounds, two different gang mentalities." A majority had been suspended from school at least once in the past two years, mostly for fighting. In spite of this, she reports, "We had not one fight." There were heated debates, Mumford says - positive ones about the issues being considered in class. The two groups worked together, however, and even competed in spelling bees and other classroom activities.

The Denver program included 149 youth, 14 to 17 years old, and ten instructors. The academic component of the program ran for 90 hours over three weeks, and included the use of computers and math worksheets, in addition to the Career Choices curriculum. Beyond spending time in the classroom, participants toured area businesses and took part in a variety of outside activities to reinforce the messages of the text.

Staff ran the program like a business, with participants expected to clock in and out. Just being there wasn't enough - they were expected to participate and perform as they would on a job. Those who did not behave in a business-like manner had their pay docked. Those who completed the program received five hours of elective credit besides their salary.

At the outset, goals for the academic enrichment program included not only improvement in basic skills, but "an increase in students' application of academic learning to practical life skills and enhanced self-esteem and motivation." The post-program evaluation determined that the data gathered throughout the three-week academic enrichment program clearly indicated the three goals were attained.

Of course, this kind of success cannot be attributed solely to a program - any program. The youth involved were highly pleased with their instructors. "We had teachers who wanted to be there," Mumford says. Instructor responses to the evaluator's question on that topic included the following: "I'm a caring person," "I want everyone to be successful, and treat them that way," "I find kids' strengths and build on them," "I do respect them," and "teachers here are dedicated to at-risk kids and are patient."

The efforts of this dedicated group paid off. Dr. Branch's report states that seventy-one percent of the participants felt what they were learning in the program would help them in school next year. Eighty-three percent felt it would help them get a job and 90 percent felt it helped them to get along with others unlike them. Sixty-nine percent felt the program helped them to like themselves more, 83 percent reported the program helped them be more responsible and 81 percent felt the program helped them become more independent.

Sixty-four percent reported they would or probably would be in the program even if they were not being paid, and 93 percent of the 42 subjects interviewed would recommend the Academic Enrichment program to friends.

Beyond the numbers, there were individual success stories as well. One 14-year-old mother, too young for other programs, became intrigued with computers and began to envision a new future for herself and her daughter. Two young men assigned to the program by their probation officer decided to get their GED.

When asked to what they attributed the success of the program, instructors suggested the following points:

1. The program emphasized cooperative learning, with more discussion and group work.
2. It avoided rote learning and "busy work," focusing instead on real world, relevant subjects - themselves and their future careers.
3. "Most of the students had such a good experience with this program that I don't think they think of it as school. Therefore their expectations were different."
4. Instructors who care about their students and treat them with respect.
5. "It gives them an opportunity to express themselves and voice their opinions."

Of course, there are things they would have done differently as well. If you are setting up a similar program, consider the following advice:

1. Identify participants as early as possible. Try to begin the program immediately after the end of the regular school year.
2. In addition to instructors, think about using peer counselors who are more immediately identifiable as role models.
3. Allow adequate prep time and training for teachers. While the curriculum is easy to use, training facilitates enthusiasm and team work.
4. Integrated teams of three instructors would be ideal.
5. Keep groups small - aim for a 10:1 student-teacher ratio.
6. Instructors who care about the youth in their program are essential to its success. This should be the deciding factor in hiring, rather than seniority or teaching background. An enthusiastic, empathetic art or physical education teacher is a better candidate than a cynical or disinterested career educator or vocational counselor.
7. If you are targeting young people with low level basic skills, you will probably require a 1:1 student-teacher ratio. Think about bringing in education students from an area college.
8. "Expand the program to other areas of the city. I would like to see a program in every community." According to Mumford, the Denver program was a rewarding one for everyone involved.