Info

social, health, political imagery through the lens of G J Huba PhD © 2012-2021

Posts tagged program evaluation

Choose another tag?

There are many problems that can plague a person with dementia. Some of these are easily detected but others may be “hidden” because of the nature of the major symptoms of the disease or “hidden” because the person with dementia (or caregiver or in some cases family members) is trying to hide some of the problems from outside observers.

For instance physical, psychological, or financial abuse will be hidden by the abuser and perhaps the person with dementia. Memory loss may make it difficult for the person with dementia to accurately report accidents.

It is important that healthcare providers, caregivers, and family members be trained to identify the hidden problems.

To some degree or another, it is likely that most persons with dementia have some of these hidden problems. For instance, I bump against things all day long, usually because I am rushing around or not paying attention because I am trying to multitask. When asked by a family member or friend where the bruise came from, I have to try to reconstruct where the accident must have happened by thinking through a lot of alternatives for a bruise half-way between my ankle and knee.

Click image to expand.

some-hidden-problems-among-persons-with-dementia

Governments and other public entities are increasing their use of web sites as the primary publication outlet for medical, human services, and research information.

The transition to electronic publication saves money as well as other resources and at the same time is much more environmentally-friendly. At least a few forests in the world owe their lives to the decision of some of the largest paper users in the world to move to electronic publishing.

Electronic publishing offers a special advantage not generally available in traditional publishing on paper. On the Internet it costs no more to include colors, simple and complex images, and images that expand to show greater detail. And it is much less expensive for publications to present, in addition to their traditional text, graphics maximized facilitate creative thinking, memory retention, “big picture thinking,” and explanations that may be easier for individuals using other languages and from other cultures to understand.

Not everyone in the world does their primary thinking using words. Many — including me — find visual information more valuable, easier to assimilate, and more supportive of creative insights.

How often do you see a #MindMap, #ConceptMap, #FlowDiagram, or other visual representation on a government web site? While there are plenty of pie diagrams and line charts, such representations of data are quite limited and do NOT incorporate informed interpretation of information. Also, while there are plenty of pictures on government web sites, these images do NOT incorporate informed interpretation of information and they may give a quite biased view of data.

I do not recall ever seeing a #MindMap, #ConceptMap, or #FlowDiagram on the (otherwise extremely useful and high quality) web sites of the US Social Security Agency, the abstracts in the PubMed medical and scientific information databases, and the US government’s explanations of research and social programs, diseases and social conditions, and social service eligibility forms.

World-wide thinking is increasingly visual. Official information should be presented using both the traditional text-based methods currently employed AND newer, very effective methods of visual thinking. The brain is not limited to a single form of thinking and in fact research shows clearly that some of us (including me) handle visual data far more effectively and perform some of our best work using visual thinking techniques. Research also suggests that as the brain changes through disease processes such as Alzheimer’s disease and other more rare neurodegenerative conditions, as verbal centers suffer damage, visual centers may assume increasing importance.

While I strongly prefer #MindMaps as the method of presenting visual information, I could accept #ConceptMaps, #FlowDiagrams, and other visual thinking representations as at least a first start.

Of the mind mapping methods, I strongly believe that the Buzan-style organic mind maps including color-coding, size-coding, radiant information structures, and methods designed to optimize memory retention, memory retrieval, creativity, and cross-cultural communication are the most effective. A recent addition to mind mapping has been Huba’s method of mind modeling that adds all of the components shown in the figure below.

Click image to expand.

IMPROVING GOVERNMENT INFORMATIONAL WEB SITES

Comedy and tragedy theatrical masks

Remember the “gold standard” research paradigm for determining if a medical treatment works: the DOUBLE BLIND, RANDOM ASSIGNMENT EXPERIMENT?

The design has historically been considered the best way to “prove” that new medical interventions work, especially if the experiment is replicated a number of times by different research teams. By the double blind (neither the treating medical team nor the patient know whether the patient is taking a placebo or active medication) design, investigators expect to negate the placebo effects caused by patient or medical staff beliefs that the “blue pill” is working.

A key part of virtually all double-blind research designs is the assumption that all patient expectations and reports are independent. This assumption is made because of the statistical requirements necessary to determine whether a drug has had a “significantly larger effect” as compared to a placebo. Making this assumption has been a “standard research design” feature since long before I was born more than 60 years ago.

2013

Google the name of a new drug in clinical trials. You will find many (hundreds, thousands) of posts on blogs, bulletin boards for people with the conditions being treated with the experimental drug, and social media, especially Twitter and Facebook. Early in most clinical trials participants start to post and question one another about their presumed active treatment or placebo status and whether those who guess they are in the experimental condition think the drug is working or not. Since the treatments are of interest to many people world-wide who are not being treated with effective pharmaceuticals, the interest is much greater than just among those in the study.

Google the name of a new drug being suggested for the treatment of a rare or orphan disease that has had no effective treatments to date and you will find this phenomenon particularly prevalent for both patients and caregivers. Hope springs eternal (which it SHOULD) but it also can effect the research design. Obviously data that are “self reported” from patient or caregiver questionnaires can be affected by Internet “the guy in Wyoming says” or the caregiver of “the woman in Florida.”

OK you say, but medical laboratory tests and clinical observations will not be affected because these indices cannot be changed by patient belief they are in the experimental or placebo conditions. Hhmmm, Sam in Seattle just posted that he thinks that he in the experimental condition and that his “saved my life” treatment works especially well if you walk 90 minutes a day or take a specific diet supplement or have a berry-and-cream diet. Mary in Maine blogs the observation that her treatment is not working so she must be in the placebo condition and becomes very depressed and subsequently makes a lot of changes in her lifestyle, often forgetting to take the other medications she reported using daily before the placebo or experimental assignment was made.

Do we have research designs for the amount of research participant visible (blogs, tweets, bulletin boards) and invisible (email, phone) communication going on during a clinical trial? No. Does this communication make a difference in what the statistical tests of efficacy will report? Probably. And can we ever track the invisible communications going on by email? Note that patients who do not wish to disclose their medical status will be more likely to use “private” email than the public blog and bulletin board methods.

Want an example. Google davunetide. This was supposed to be a miracle drug for the very rare neurodegenerative condition PSP. The company (Allon) that developed the drug received huge tax incentives in the USA to potentially market an effective drug for a neglected condition. The company, of course, was well aware that after getting huge tax incentives to develop the pharmaceutical, if the drug were to prove effective in reducing cognitive problems (as was thought), it would then be used with the much more common (and lucrative from the standpoint of Big Pharma) neurodegenerative disorders (Alzheimer’s, Parkinson’s) and schizophrenia.

Patients scrambled to get into the trial because an experimental medication was better than no medication (as was assumed, although not necessarily true) and the odds were 50/50 of getting the active pills.

Patients and caregivers communicated for more than a year, with the conversations involving patients from around the world. In my opinion, the communications probably increased the placebo effect, although I have no data nor statistical tests of “prove” this and it is pure conjecture on my part.

The trial failed miserably. Interestingly, within a few weeks after announcing the results, the senior investigators who developed and tested the treatment had left the employ of Allon. Immediately after the release of the results, clinical trial participants (the caregivers more than the patients) started trading stories on the Internet.

Time for getting our thinking hats on. I worked on methodological problems like this for 30+ years, and I have no solution, nor do I think this problem is going to be solved by any individual. Teams of #medical, #behavioral, #communication, and #statistical professionals need to be formed if we want to be able to accurately assess the effects of a new medication.

Click on the image to expand.

Clinical Trial Double-Blind Treatment Evaluation in the Era of the Internet

This mind map was originally prepared in MindManager in the late 2001 for an evaluation of an initiative to increase the capacity of US Nursing Schools to meet the need for graduate-trained gerontological/geriatric nurses. I spent 10 minutes running the original file through iThoughtsX to add some color hues.

This was an evaluation of an important initiative funded by the John A Hartford Foundation (Building Academic Geriatric Nursing Capacity).

Click on the image to expand.

BAGNC

2006 versions of mind maps from an evaluation of 21 projects funded to demonstrate various models of elder abuse services by the Archstone Foundation.

The program evaluation was managed and explained using a large mind map created in the version of Mind Manager current at that time. We also had many detailed mind maps, used internally, of the hundreds of indicator variables collected and coded.

Consistent with my current thinking, I would now categorize these maps as outline maps, not mind maps. These are really outlines that have undergone cosmetic surgery, not true actively-developed mind maps.

Click on images to expand.

A MM9

A MM8

A MM7

A MM6

A MM5

A MM4

A MM3

A MM2

A MM 1

Coding for Archstone Elder Abuse & Neglect Initiative

As a simple exercise, the set up for the mind map above was imported to the iThoughtsX computer program released in mid September 2013.  Simple color coding in iThoughtsX makes the map above much more useful.

Coding System for Archstone Foundation Progress Reports. Elder Abuse & Neglect Initiative (2)

I’ve asked myself which of my skills were the most important in a senior role providing consulting and executive leadership of a small (<30 employees) consulting firm. We specialized in evaluating healthcare and socialcare programs for high need, disenfranchised groups who are often excluded from health services. I do think, however, that these are general skills which fit about every content area for a senior consultant expected to make significant creative contributions to the client organization.

It is important to know that the skills and technique specified are designed to optimize the value of the work and not to maximize profit to the consultant. There are other processes one might use to maximize profit.

After my mind map, I included some notes I made about the topic as I was thinking it through.

The five mind map diagrams start at the most abstract level and then each unveils part of the map showing the full detail of that section. The final map is fully unveiled.

I’ve gotten the reaction “you must be lying to me, it can’t be that simple” when I have provided conclusions and advice to others. All the while my business partner was sitting in the meeting alternating between having a panic attack because I was giving away the company secrets and falling off the chair laughing because the recipient of the information did not know to enough to realize that it was very hard to not over-think, cut through the distractions, and get to the bottom line when nobody (including you) knows the answer to the questions asked.

It’s simple. Just build the confidence that you can always fill a blank page with a great answer and implementation plan, build a safety net to avoid a major mistake, and use successive approximation in a resource-limited environment. It took me a long time to figure all of this out. It’s hard but do-able.

Live long and prosper (going where no-one has dared to go before).

Please click on the image to zoom.

#########

senior creative consulting 2 skills and 1 technique

#########

 

Appendix: Notes (from iPad)

two skills

blank paper

safety net

feedback circle

Irv Oii is known to many international news organizations and researchers as a star data journalist. Being a home worker (although home may be the UK, Ohio, the Middle East, Central Africa, Hong Kong, or Antartica) and a fairly reclusive person, nobody seems to have met Irv. Some speculate that he might be a Jewish Asian-American. Others believe Irv is short for Irvelina, a Russian immigrant physician who went to Ohio (or was it Ojai, California) when the Soviet science programs collapsed and turned into the lower funded Russian collaborative efforts with the EU and USA. The collapse of the Soviet Union resulted in the closing of her laboratory in Minsk. Some even think Irv Oii is an acronym.

Irv is thus an enigma and no pictures of her/him seem to exist. An artist’s conception (mine) based on the writings and consultations of Irv Oii on healthcare breakthroughs is shown below. My belief is that a portrait of Irv should hang over the desk of every data journalist and researcher.

Please click the image to zoom.

Irv Oii

Click on mind map to expand.

academia and healthcare big data

Splatter10

9

Please click on the graphics to zoom.

it's program evaluation, not research, dummy

 

3

evaluationmap

researchmap

There are a number of things that can be done to cut the cost of healthcare while, at the same time, freeing doctors and others to do their jobs better. These improvements cost almost nothing to implement [if all of the constituencies and politicians do not compete to be King Kong].

Visiting legislator who stumbled across this web page? Here’s your chance to act like a grown-up and represent the people of the world, not drug companies nor major research universities nor individual “researcher” egos and retirement funds.

Click on images to expand.[almost free] strategies to improve healthcare

Cartoon Rabbit - Giving A Thumbs Up

image

image

image

Sketchnote Example: My Predictions of Changes in the Field of Psychology Over The Next 20 Years

image

image

Click on the images to zoom.

The fictional detectives would have been great program evaluators. All looked at all types of data. Miss Marple was a model of pleasantry who could work her way into an organization or group and see it as it was without changing anything by observing. Holmes and Watson — whether in the original books and movies, the Ironman version of the movies, their current BBC incarnation in 21st Century London, or their CBS incarnation in 21st Century Manhattan with Dr John Watson now Dr Joan Watson (for the better) — use Holmes’ razor sharp mind and Watson’s intuitiveness and questioning. Sam Spade, wise cracks, an iron fist, and underlying sensitivity.

Program evaluation is not about conducting research, randomly assigning participants to conditions, or using quasi-experimental designs. Program evaluation is about understanding why programs produce certain outcomes, intended or not, positive or not, unique or not. To truly understand a program quantitative and qualitative data needs to be collected with great attention to the sensibilities, needs, risks, and potential confidentiality breaches of data of program participants, program staff, program administration, funders, and other stakeholders.

I love program evaluation. Every program is unique and at the same time representative of certain classes of human service organizations.

Be a detective. Look carefully and understand the beauty of a well-running program and how to help staff improve a program that is not working as well as it could.

evaluation detective work

The Program Evaluator's Tool Kit

Measure, Find Relationships, Communicate

image image

image image
Ask the Big Questions and Study How the Answers Relate to One Anotherimage

Decode Events and Naturally Occurring Data

image image

Examine in Detail

image

Listen
 image

Find Patterns
image

This is the first of a series of posts I am making about program-organizational (and individual) evaluation. Much of what I will discuss is not in the mainstream of traditional program evaluation methodology.

My approach is different. It works.

In this first section the point is — obviously — that evaluation is iterative and nonlinear. This led to my first model that EVALUATION IS DETECTIVE WORK several decades ago. [Perhaps that explains my current obsession with all versions of Sherlock Holmes, whether in the original, present London, present New York, or by Iron Man.] At any rate, it seems ELEMENTARY to me that instead of thinking of program evaluation as a linear research experiment with a fixed design (a metaphor that works at best imperfectly), it is more important to treat evaluation as detective work where good rules of evidence must be followed and the evaluator is at fault if all outcomes are not found.

My initial development of the Detective Model in 1992 came from my observation that in much traditional program evaluation the evaluator applies a flawed “research” experimental model and the insensitivity of this approach means that a program looks worse than it is because the evaluation methodology is in error. Who pays for this problem? The program, of course, since the evaluator walks away saying that the “program sucks” and not that the evaluator screwed up. In the Detective Model, applied iteratively and nonlinearly, the evaluator and the program are partners, and it is clear what the responsibilities and level of success each has.

Seems ELEMENTARY to me.

As usual click on the image to zoom.

Evaluation is an Iterative, Non-linear Process