Solving the Mystery of Program Evaluation: Start with the End in Mind

Sergakis_smPresenters, Georgianna Sergakis and Jill Clutter, from Ohio State University, discussed the importance of using a logic model for program evaluation. The breakout session began with both presenters sharing about their backgrounds and moving into an interactive activity with the attendees. Georgianna, a respiratory therapist, stated that she trains new trainees’ at Ohio State University who are becoming respiratory therapists. Most of her work and knowledge is on Tobacco and the phenomenon of e-cigarettes.

Jill is a professor at Ohio State University at the School of Health Science and works closely with the Rehabilitation Division as well. She teaches graduate students and explains that science is in everything we do and everywhere.

The presenters led the group into a pole activity where attendees were asked few questions about program evaluation. Then, attendees answer the questions and they were able to visually see what others thought about program evaluation. This sparked a conversation among attendees and the presenters.

During the breakout session, Jill explained the importance of the logic model and explains the benefits of using this tool. She stresses that the logic model is a tool used to clarify an initiative and to graphically depict a plan for future program evaluation. The logic model serves as a foundation for planning, communicating and evaluation. Although there are many different types of logic models, it is important to understand that one size does not fit all. In addition, she stressed how it is very important to start with the end in mind when sorting out a plan for program evaluation.

Presenters discussed about evidence-based practices such as the literature of CDC Framework for Program Evaluation. This framework explains the importance of working backwards to connect all the gaps and find the missing pieces that may need additional evaluation to have a successful program. By using a logic model, it visually provides a big picture of evaluation. Recognizes what has been done and what still needs improvement for future evaluation.

The presenters shared with the attendees three different types of evaluation. The first was context evaluation. This particular evaluation method looks closely at the issues of the program relationships and reevaluates the needs assessment. Second, implementation evaluation is the implementation of the program, what activities were executed and to what extent, and the quality of the overall program. Lastly, outcome evaluation initiates a base to discuss the overall progress and what still remains unresolved for future evaluation.

The breakout session concludes by attendees engaging in a active program evaluation activity.


Effectively Communicating Risk

Trudeau_smChristopher Trudeau, J.D., Professor, Thomas M. Cooley Law School

Risk means something different to each of us: Lawyers think of risk as how to prevent the hospital from getting sued. Providers wonder, how do we communicate the info to patients so they can make good decisions?

What is risk communication? Helping people understand the nature and seriousness of a certain action so they can make an informed decision.

When someone is selling us something, they’re very clear and direct, but when they need to give us information, it can be very complicated.

Key components of risk communication:

  1. What we communicate
  2. How much patients want to know
  3. When we communicate it (ex. After you’ve prepped for a colonoscopy is not the time to ask for consent — The patient doesn’t care at this point.)
  4. How we communicate it.

Communicating risk is a process, not just documents to sign. We have to keep in mind both these factors:

  • What is legally required for consent?
  • How much info does the patient want?

Legal disclosure requirements differ for clinical procedures and research trials. There are a lot more legal requirements for research trials.

  • The requirements for clinical procedures are state-based and grow out of the common law (lawsuits, etc.).
  • The requirements for research studies were developed in response to bad acts by researchers and are federally controlled.

For procedural informed consent, we use the “reasonable physician-standard” and “reasonable-patient standard” as legal guidelines. But communicating risk goes way beyond the legal requirements.

Written communication does not substitute for in-person conversations that incorporate teach-back. So writing good consent forms isn’t enough. The process of informed consent has to be developed.

Design the consent process first, before you design the consent form. Example:

  1. Patient and provider discuss.
  2. Patient gets handout and consent form.
  3. Patient preps for the procedure.
  4. Patient goes to appointment.
  5. Patient and provider further discuss consent form, using teach-back (before anesthesia is given).

Designing Forms:

It’s important to create hierarchies of information: Prioritize the information so that it emphasizes the key risks to patients. Prioritize risk in the document by creating risk hierarchies.

  • The most important info needs to go first, such as the fact that you need someone to drive you home after a colonoscopy.
  • People decide for themselves how much attention to pay to a document.
  • Documents are not novels; they are not meant to be read cover-to-cover. You may look at it when you get it, but when you consult it later, you’re only going to look for the information you’re after at that time. (You’re not going to read it through again.)
  • People actively interpret AS THEY READ. They don’t wait till the end to get the big picture, so organization is KEY.

Structure all documents as if everyone has limited health literacy.

  1. Consider intended users (focus group? User testing afterwards?)
  2. Write down what your intended users would want to know.
  3. Rank the info from most important to least important.
  4. Then write down any other info you have to disclose by law.

How do you lobby the document owner for clearer documents? Get the owner to focus on the content, rather than the words. They will be more likely to buy in.

How to convey numbers?

Every medical decision requires some statistical literacy. Aim for communicating with a minimal need for statistical literacy. Minimal literacy focuses on the main concepts, such as absolute risks.

Tips on numbers:

  1. Focus on what you want the patient to do. The patient needs to get the gist. Include a call to action and relate the message to everyday life. Then you can reinforce with data or visuals that relate directly.
  2. Start with the most important info and group logically.
  3. Less is more, but underscore the gist.
  4. Round numbers are easier to understand and remember.
  5. Keep denominators consistent, eg. 1 in 10, 2 in 10.
  6. How you frame the question influences the patient’s decision.
  7. Talk about absolute risk rather than relative risk. (Relative risk: If you don’t take this medication, your risk of stroke will double. Absolute risk: Your risk of stroke will increase from 5% to 10%.)
  8. Use visual aids and cues. Icon arrays help people  personalize/humanize numbers. You can create an icon array at:
  9. Think about questions patients SHOULD ask: risk of death? Time frame? How big is the risk? Do the studies apply to me and my demographic? What are the benefits and harms of treatment?


Chris showed a rewritten consent form that has places for the provider to fill in the patient’s answers to teach-back questions. This forces the provider to use teach-back. This is also helpful as legal evidence of understanding. And if the patient has to initial statements, he or she is more likely to read them.

Health Literacy Online: How to Simplify the User Experience

Scrimgeour_smXanthi Scrimgeour and Stacy Robison of Communicate Health

Online Health information should incorporate:

  • Plain language
  • Health Education Theory
  • Usability
  • User-centered design

90% of adults are online in the U.S.

Those who don’t have broadband are accessing internet on their phones.

72% of internet users looked online for health info in 2013.

Half of people looking for health info are looking for someone else.

69% said their most recent search had an impact on their health or the way they care for someone else.

More often it’s a poorly designed website, rather than user’s poor health literacy that get in the way.

Voice features can help with dexterity and communication challenges.

Alerts and auto reminders can help compensate for short-term memory loss. (second edition) See the class handout.

What we know about users with limited literacy skills:

  • Web users with limited literacy skills are on line and willing to look for info there.
  • They are prone to skipping and focussing on the center of the screen.
  • They are easily overwhelmed and have a limited working memory.
  • They struggle with search functions. Spelling is an issue, ads get in the way, not knowing what to look for, not knowing what sites are reputable. They will only click on 1 or 2 results.
  • Low-literacy users are more likely to be smartphone dependent. 1 in 5 adults rely on their phones for internet access. Visually impaired people often prefer the smaller screen of a smartphone.

Strategies for clear online communication:

  1. Write actionable content. (Information, not documents)
  2. Answer users’ questions
  3. Help users complete a task
  4. Help users make a decision
  5. You have about 15 seconds to grab the user’s attention.
  6. Put the most important information first.
  7. Provide specific action steps (bite, snack, meal – user can drill down as far as they want).
  8. Limit paragraph size, use bullets and short lists
  9. People will often skip paragraphs of 3 lines or more.
  10. Chunking information is important for people with limited working memory.
  11. It’s critical to design for small screens! Website needs to be responsive to variable screen sizes.
  12. Place key info and buttons either near enter or center bottom. Many people cover lower right corner with hand.
  13. Limited lit users like radio buttons. Make as much of the choice active as possible (more than just the circle).
  14. Make your site accessible (works with screen readers and other assistive technologies, navigate with keyboard only; strong color contrast, etc.).
  15. Create a linear information path: Give meaningful labels, such as “track meals ->” rather than “next”
  16. Label information clearly.
  17. Include a simple search function. Some users just hit the search button without entering any text. Allow for common misspellings, using auto-complete, most actionable content should come up first, avoid scrolling, clear page titles, brief plain language descriptions of URL.
  18. Limited literacy users like to browse more than they like to search. Putting the alphabet across the top is helpful.

Engage users with interactive content:

  1. Let them share and print info; give them info about themselves, e.g., calculate the user’s BMI index, etc.
  2. You can use multi-media online.
  3. Super simple quizzes work well.
  4. Online information can be tailored by collecting just a little information such as age and sex.
  5. Make the content printer-friendly.
  6. Test your site with users.
  7. Usability testing with 5 participants will reveal 85% of usability problems.


10 Attributes of an Organization that supports Health Literacy

Frempong_smJanet Ohene-Frempong in 2014 was invited to work with the Oregon Health Authority on their initiative, Developing Equity Leadership through Training and Action: DELTA. According to Janet, Oregon is a leader in coordinated care.

Goals of the Initiative were:

  1. Reduce healthcare costs and improve equity
  2. Improve the effectiveness of communication as one of many ways to reach those goals.

DELTA has a leadership program for 25 members per year from many fields; e.g., health, education , social service, government.

Part of the DELTA program is to institutionalize best practices in organizations.

  • Oregon is including health literacy issues in the development of equity leadership as well as in their attempt to reduce the cost of providing health care to the most vulnerable.
  • Marginal literacy and effective communication issues impact not only health, but education, social services, law enforcement and even  art and entertainment. Supporting effective communication is supporting those who provide information, service and /or care across literacy levels, languages, cultures, etc.
  • Institutionalizing best practices includes expanding the use of IOM’s 10 Attributes of a Health literate organization for use outside the healthcare industry to indirectly improve health. (10 Attributes is 2012 document. It was a seminal document. We need institutional support, and this document codifies it.)

“A Health Literate organization is health-literacy supportive,” says Janet.

Janet drafted a prescriptive checklist for the DELTA program based on the 10 attributes.

  1. Educational, legal/law enforcement, and social service systems affect health and need to be part of the health literacy solution. Literature in these fields supports a connection between health literacy and issues of importance to these sectors.
  2. If health, educational, legal/law enforcement and social service systems can adopt most of the 10 attributes, they may be able to make a substantial contribution to improve population health.
  3. You must identify barriers that stand in the way: Time and money are the most often cited.
  4. There are many resources on the link between health communication and health equity. (See the citations in Janet’s talk handout.)
  5. The communication skills required to support health, as well as health literacy, apply to those in other fields as well. It takes a community to move people towards health. We can extract principles of health communication and extend them to the entire community.

Health Literacy Insights for Public Health Practice: 3 Case Studies in Disaster Management & Relief Efforts


Health literacy research and practice over these last two decades are enabling us to enrich the clinical encounter, transform health care settings and, hopefully, remove barriers to care and services. It is time however, to adopt and adapt these insights for public health practice. This presentation focuses on three case studies [anthrax,  a hurricane, and a tsunami] related to disaster management illustrating the critical value of health literacy inquiry and application of findings.


Rima Rudd, Sc.D

Senior Lecturer on Health Literacy, Education, and Policy, Harvard School of Public Health

We have to think outside the box to help with:

Preparedness, community health and safety and disaster management

Case 1 Anthrax event of 2001:

Anthrax-laced letters sent to people in communications. We needed to communicate the nature of the substance, risk, health implications, treatment, preventive action. There were problems in the clearance system, rumors, misspeaks, and leaks.

A postcard was sent to all households by the Postmaster General for immediate precautionary action. The language used was uncommon and calls to action were vague and abstract. The design was not great either. The case study identified these needs:

  1. Match level of communication with the average person.
  2. Use plain language
  3. Develop a communication plan
  4. Apply formative research and rigorous piloting. – We know about this and we don’t do it.
  5. Need for ‘first responders’ – Call for help from health literacy experts as part of the first responder team.

Case 2 Hurricane Katrina:

National disaster compounded by political neglect.

They needed basic information.

The available materials were not suitable for the general public. (From HHS, CDC, etc.) They paid no attention to health literacy.

Len and Ceci Doak coordinated a volunteer effort with CDC to prepare accessible information with a 6-24 hour turnaround from submission of originals, and pilot test where possible.

The first responders improved the clarity of the information.

Lessons learned:

  • We must demand rigor in our work.
  • Health materials far exceed the reading skills of most members of the public.
  • Assessment tools can help identify needs.

Case 3 Fukushima City Disaster:

This was a multi-level disaster. Many people left the city, including families with children, but most people were still there. The Public Health Nurses were left to deal with the population. The nurses had difficult scientific info they could not adequately translate. The result was loss of trust and worsening of the situation. The nurses needed to get a glossary of terms and improved health communication skills. Skills were improved; systemic barriers were uncovered for the nurses to engage in the work they need to do.

Lessons learned:

  • Communication clarity influences trust.
  • Scientific info must be translated for health professionals, and further translated for the public.
  • People’s professional abilities were developed and they responded positively to health literacy skill developemt.
  • But training professionals without changing systemic inhibitors places undue stress on professionals.
  • You must pay attention to the institutions that shape peoples (e.g. professionals’) abilities to perform.


  • There’s a mismatch in demands of health info and literacy skills.
  • We must develop materials with respect and vigor.
  • We can learn from assessment tools, but need to develop new ones.
  • Literacy is based on interactions.
  • Health literacy involves multilingual skills on multiple levels.
  • Health Lit requires a supportive political/ normative environment.

Mechanisms for Change

  1. Dissemination of info and discussion
  2. Regulations for text production including strategies for times of chaos. Materials must be pilot-tested as part of development process.
  3. Required training for scientists on how to share information, as well as health communicators and educators.

We need training programs to assess health information and prepare health information. Not just written materials.

We need tools for calibrating math demands, assessing risk communication, assessing entry forms and documents for evaluations and research.

We need studies for communicating to the next level of the public: science, risk, reporting medical and scientific findings, displaying findings, reporting test and study results.

WE HL experts need to work together with other organizations such as volunteer orgs, first respondrs, CDC, FEMA, etc.

“The word is as important as any tool.”

Opening Panel: Deeper Dive: Signposts on the Road to Health Literacy’s Future


Rima Rudd, Sc.D (Rima) – Senior Lecturer on Health Literacy, Education, and Policy, Harvard School of Public Health

Cliff Coleman, M.D., MPH (Cliff)- Assistant Professor of Family Medicine, Oregon Health & Science University School of Medicine

Susan Reid, M.A. (Susan) – Director, Health Literacy New Zealand

Marin P. Allen, Ph.D. (Marin) – Deputy Associate Director for Communications and Public Liaison and Director of Public Information , National Institutes of Health

Michael Villaire, MSLM – CEO, Institute for Healthcare Advancement (Moderator)(Michael)

The panel discussed the current level of health literacy penetration in the U.S. healthcare system and identified some objectives and best practices to strive for in the next 10 years. Following is a summary of questions posed by the Moderator (Michael) and answers from the panel, with some key points bolded.

Question: Where do you want to be in 2026?

Rima: Literacy has become an important variable in health research. We need to go much further into looking at the skills of the communicator, difficulty of materials and context, etc. How do we transform the system to reduce disparities?

Cliff: We have done a disservice to patients by ignoring our responsibility in patient understanding. Rather than putting onus on patients and caregivers, we need to turn the spotlight on ourselves.

The State of training for practitioners: There’s a sizable minority of programs training health professionals – less than half. There’s a giant lack of faculty skill in this area. Academic institutions are not as aware as they should be. And the schools teaching health literacy are not teaching it enough. In a 4-year curriculum, they’re teaching about 3 hours on how to communicate with patients. Out of 362 attendees at this conference, there are only 7 physicians. Doctors don’t know what they don’t know: they don’t know what HL is. And they think they are already good communicators. The ones who think they’re the best communicators are rated lowest by patients.

Rima: Doctors who take a year to get an MPH are growing in numbers. There is lots of enthusiasm among young doctors.

Cliff: There are still physicians who think HL does not apply to their populations.

Marin: HL for the individual saves lives, for the physician it saves time; for the system it saves money. We need to equate HL with these facts. We needed to prove the connection. In the future we need to bring the knowledge we already have into the field and think globally, bringing the whole world in. (There are more than 300 languages spoken in homes in U.S.) The work needs to be more comprehensive and more universal.

Michael: HL field needs to work much more closely with the reading field, as Ceci Doak has said for years.

Marin: We need to bring the knowledge of how adult learners learn to the field.

Susan: I come to HL from 24 years in Adult Literacy. We focus on a strength-based approach (rather than a deficit approach). We need to draw on our adult literacy background in relation to listening, speaking, numeracy, and writing. We need to collaborate and make sure all disciplines come through: We stand on and draw on a large range of fields. How do we use that, rather than reinventing the wheel, is the question.

Michael: We need to keep respect in the forefront. We need to bridge the communication gap with those of other cultures, backgrounds, etc.

Question: What are the areas where we haven’t gone and should go?

Rima: HL practice has not sufficiently ventured into public health. Our work has stayed within the institutions of health care. But we have to spend more time in the community, in homes, in workplaces, on key issues such as air and water quality. We should engage with people working in environmental health and others. We may need to be more qualitative and creative in our approaches. We need to get outside, breathe the air, and comment on it.

Michael: We should do as President Obama did: He drank the water in Flint, MI.

Susan: We need to hold a mirror up to organizations to get them to look at what they’re doing. Health care needs to give up its culture in order to work with the community, rather than just “doing things to” the community. We need to shift power.

The health care system in New Zealand is very different from the U.S. It’s much simpler.

Marin: After Fukushima, the Japanese government lost the public trust in 3 days. They have been investigating what happened. Together with a researcher in Boston, they got citizens in Japan to measure radiation levels. It may not have been ideal, but it engaged a lot of people and made data available. There is so much to be done with community engagement. “Active listening” is important to moving forward.

Cliff: We need to get out of our “silo,” get out of institutions and engage with the public. How do we describe what we do? It takes a long time. How do we get the public to understand and get excited about it and exert pressure from the outside? How do we teach the public to get excited?

Marin: Let’s get young people excited about the field.

Michael: We can collaborate with groups that work to make science understandable to the general public.

Marin: Alan Alda is using theatre games in his Stonybrook program to teach science.

Susan: Health professionals overestimate their effect, especially on chronic conditions. We have to acknowledge people doing a good job of taking care of their health, rather than just pointing out what they’re doing wrong.

Michael: Think about ways you can partner with other groups in your community.

Question: What obstacles and challenges do we face?

Rima: One obstacle was lack of scholarship, not learning from my colleagues. I had faulty assumptions about how to get there. It’s not a foot race, but rather walking with colleagues. Another one is lack of patience. A third is entrenched interests: We need to look at policy and policy change, and regulation of materials, testing of materials before releasing to public. We need to engage the policy sector to bring about mandates and regulations.

Michael: When we encounter an entrenched interest, we have to engage them in figuring out what we are trying to achieve with the barriers we have put in place, and how we can do better to achieve the overall goal.

Cliff: We have some catch-up to do in providing an evidence base for our work. Health care professionals won’t listen if there is no data. We need to show that HL changes outcomes for patients.

Institutionally, there’s no incentive for clinicians to get things right the first time. If the patient doesn’t get it the first time and has to come back for a second visit, the physician is rewarded for this.

Marin: There’s no funding to support policies. We need to look for opportunities for policies to have definition, funding, or at least some activities to support them.

Michael: What gets done first is what we get paid to do. There is funding for research into health literacy. There need to be incentives for the practice.

Susan: The code of patient rights in New Zealand says that the patient has the right to understand risk, but we’re still very bad at it.

Marin: We’re not successfully communicating risk. We have to find a real way to explain risk without scientific terminology. We need to show risk over time, pre-conditions: Understanding and explaining risk is a challenge.

Rima: Baric, an English health educator writing 30-40 years ago, said that we have the illusion that the notion of risk has the same power as the feeling of pain. Understanding risk is not necessarily linked to action. What we’re not doing is making the understanding of risk actionable: What can they do with the information?

Cliff: I need to be able to know when a nurse, physician or therapist is competent in clear communication. So that 10 years from now we can have measurable standards in our degree programs.

Rima: What can I do now to help us get where we want to be in 20 years? I want to work with young people, to pass on the mantle, to run with it, expand it, and change it. I want to work together, co-publish, work with young people.

Marin: 1 in 6 people in America have a communication disorder of some kind. We need to think of these people who need to have the health literacy models work for them. It’s more than translation or handing someone a form.

Susan: In the future, I want to share the generosity of my HL colleagues with the community – YOU, and with communities, as well as others in HL field.

From the audience: Janet Ohene-Frempong: We need to support providers in communicating clearly.

Cliff: We need to incentivize the practitioners to practice their HL skills with patients.

Marin: We need to figure out what incentives work.

Rima: We need a common language for scientists, practioners, and the general public.