Solving the Mystery of Program Evaluation: Start with the End in Mind

Sergakis_smPresenters, Georgianna Sergakis and Jill Clutter, from Ohio State University, discussed the importance of using a logic model for program evaluation. The breakout session began with both presenters sharing about their backgrounds and moving into an interactive activity with the attendees. Georgianna, a respiratory therapist, stated that she trains new trainees’ at Ohio State University who are becoming respiratory therapists. Most of her work and knowledge is on Tobacco and the phenomenon of e-cigarettes.

Jill is a professor at Ohio State University at the School of Health Science and works closely with the Rehabilitation Division as well. She teaches graduate students and explains that science is in everything we do and everywhere.

The presenters led the group into a pole activity where attendees were asked few questions about program evaluation. Then, attendees answer the questions and they were able to visually see what others thought about program evaluation. This sparked a conversation among attendees and the presenters.

During the breakout session, Jill explained the importance of the logic model and explains the benefits of using this tool. She stresses that the logic model is a tool used to clarify an initiative and to graphically depict a plan for future program evaluation. The logic model serves as a foundation for planning, communicating and evaluation. Although there are many different types of logic models, it is important to understand that one size does not fit all. In addition, she stressed how it is very important to start with the end in mind when sorting out a plan for program evaluation.

Presenters discussed about evidence-based practices such as the literature of CDC Framework for Program Evaluation. This framework explains the importance of working backwards to connect all the gaps and find the missing pieces that may need additional evaluation to have a successful program. By using a logic model, it visually provides a big picture of evaluation. Recognizes what has been done and what still needs improvement for future evaluation.

The presenters shared with the attendees three different types of evaluation. The first was context evaluation. This particular evaluation method looks closely at the issues of the program relationships and reevaluates the needs assessment. Second, implementation evaluation is the implementation of the program, what activities were executed and to what extent, and the quality of the overall program. Lastly, outcome evaluation initiates a base to discuss the overall progress and what still remains unresolved for future evaluation.

The breakout session concludes by attendees engaging in a active program evaluation activity.


Effectively Communicating Risk

Trudeau_smChristopher Trudeau, J.D., Professor, Thomas M. Cooley Law School

Risk means something different to each of us: Lawyers think of risk as how to prevent the hospital from getting sued. Providers wonder, how do we communicate the info to patients so they can make good decisions?

What is risk communication? Helping people understand the nature and seriousness of a certain action so they can make an informed decision.

When someone is selling us something, they’re very clear and direct, but when they need to give us information, it can be very complicated.

Key components of risk communication:

  1. What we communicate
  2. How much patients want to know
  3. When we communicate it (ex. After you’ve prepped for a colonoscopy is not the time to ask for consent — The patient doesn’t care at this point.)
  4. How we communicate it.

Communicating risk is a process, not just documents to sign. We have to keep in mind both these factors:

  • What is legally required for consent?
  • How much info does the patient want?

Legal disclosure requirements differ for clinical procedures and research trials. There are a lot more legal requirements for research trials.

  • The requirements for clinical procedures are state-based and grow out of the common law (lawsuits, etc.).
  • The requirements for research studies were developed in response to bad acts by researchers and are federally controlled.

For procedural informed consent, we use the “reasonable physician-standard” and “reasonable-patient standard” as legal guidelines. But communicating risk goes way beyond the legal requirements.

Written communication does not substitute for in-person conversations that incorporate teach-back. So writing good consent forms isn’t enough. The process of informed consent has to be developed.

Design the consent process first, before you design the consent form. Example:

  1. Patient and provider discuss.
  2. Patient gets handout and consent form.
  3. Patient preps for the procedure.
  4. Patient goes to appointment.
  5. Patient and provider further discuss consent form, using teach-back (before anesthesia is given).

Designing Forms:

It’s important to create hierarchies of information: Prioritize the information so that it emphasizes the key risks to patients. Prioritize risk in the document by creating risk hierarchies.

  • The most important info needs to go first, such as the fact that you need someone to drive you home after a colonoscopy.
  • People decide for themselves how much attention to pay to a document.
  • Documents are not novels; they are not meant to be read cover-to-cover. You may look at it when you get it, but when you consult it later, you’re only going to look for the information you’re after at that time. (You’re not going to read it through again.)
  • People actively interpret AS THEY READ. They don’t wait till the end to get the big picture, so organization is KEY.

Structure all documents as if everyone has limited health literacy.

  1. Consider intended users (focus group? User testing afterwards?)
  2. Write down what your intended users would want to know.
  3. Rank the info from most important to least important.
  4. Then write down any other info you have to disclose by law.

How do you lobby the document owner for clearer documents? Get the owner to focus on the content, rather than the words. They will be more likely to buy in.

How to convey numbers?

Every medical decision requires some statistical literacy. Aim for communicating with a minimal need for statistical literacy. Minimal literacy focuses on the main concepts, such as absolute risks.

Tips on numbers:

  1. Focus on what you want the patient to do. The patient needs to get the gist. Include a call to action and relate the message to everyday life. Then you can reinforce with data or visuals that relate directly.
  2. Start with the most important info and group logically.
  3. Less is more, but underscore the gist.
  4. Round numbers are easier to understand and remember.
  5. Keep denominators consistent, eg. 1 in 10, 2 in 10.
  6. How you frame the question influences the patient’s decision.
  7. Talk about absolute risk rather than relative risk. (Relative risk: If you don’t take this medication, your risk of stroke will double. Absolute risk: Your risk of stroke will increase from 5% to 10%.)
  8. Use visual aids and cues. Icon arrays help people  personalize/humanize numbers. You can create an icon array at:
  9. Think about questions patients SHOULD ask: risk of death? Time frame? How big is the risk? Do the studies apply to me and my demographic? What are the benefits and harms of treatment?


Chris showed a rewritten consent form that has places for the provider to fill in the patient’s answers to teach-back questions. This forces the provider to use teach-back. This is also helpful as legal evidence of understanding. And if the patient has to initial statements, he or she is more likely to read them.

Health Literacy Online: How to Simplify the User Experience

Scrimgeour_smXanthi Scrimgeour and Stacy Robison of Communicate Health

Online Health information should incorporate:

  • Plain language
  • Health Education Theory
  • Usability
  • User-centered design

90% of adults are online in the U.S.

Those who don’t have broadband are accessing internet on their phones.

72% of internet users looked online for health info in 2013.

Half of people looking for health info are looking for someone else.

69% said their most recent search had an impact on their health or the way they care for someone else.

More often it’s a poorly designed website, rather than user’s poor health literacy that get in the way.

Voice features can help with dexterity and communication challenges.

Alerts and auto reminders can help compensate for short-term memory loss. (second edition) See the class handout.

What we know about users with limited literacy skills:

  • Web users with limited literacy skills are on line and willing to look for info there.
  • They are prone to skipping and focussing on the center of the screen.
  • They are easily overwhelmed and have a limited working memory.
  • They struggle with search functions. Spelling is an issue, ads get in the way, not knowing what to look for, not knowing what sites are reputable. They will only click on 1 or 2 results.
  • Low-literacy users are more likely to be smartphone dependent. 1 in 5 adults rely on their phones for internet access. Visually impaired people often prefer the smaller screen of a smartphone.

Strategies for clear online communication:

  1. Write actionable content. (Information, not documents)
  2. Answer users’ questions
  3. Help users complete a task
  4. Help users make a decision
  5. You have about 15 seconds to grab the user’s attention.
  6. Put the most important information first.
  7. Provide specific action steps (bite, snack, meal – user can drill down as far as they want).
  8. Limit paragraph size, use bullets and short lists
  9. People will often skip paragraphs of 3 lines or more.
  10. Chunking information is important for people with limited working memory.
  11. It’s critical to design for small screens! Website needs to be responsive to variable screen sizes.
  12. Place key info and buttons either near enter or center bottom. Many people cover lower right corner with hand.
  13. Limited lit users like radio buttons. Make as much of the choice active as possible (more than just the circle).
  14. Make your site accessible (works with screen readers and other assistive technologies, navigate with keyboard only; strong color contrast, etc.).
  15. Create a linear information path: Give meaningful labels, such as “track meals ->” rather than “next”
  16. Label information clearly.
  17. Include a simple search function. Some users just hit the search button without entering any text. Allow for common misspellings, using auto-complete, most actionable content should come up first, avoid scrolling, clear page titles, brief plain language descriptions of URL.
  18. Limited literacy users like to browse more than they like to search. Putting the alphabet across the top is helpful.

Engage users with interactive content:

  1. Let them share and print info; give them info about themselves, e.g., calculate the user’s BMI index, etc.
  2. You can use multi-media online.
  3. Super simple quizzes work well.
  4. Online information can be tailored by collecting just a little information such as age and sex.
  5. Make the content printer-friendly.
  6. Test your site with users.
  7. Usability testing with 5 participants will reveal 85% of usability problems.