Posts tagged Program Evaluation
Choose another tag?
A Way to Kickstart the Development of Effective Treatments for Rare Diseases without Taking Needed Resources from Research on Diseases that Affect Many
The only way I see to develop effective medical treatments and care models for many of the thousands of rare diseases is to pool the RESEARCH resources that individual countries are spending and the data countries are collecting about individual rare diseases and put those research resources under international control for prioritizing research agenda and ensuring public access to ALL results and research data.
Yes, I know the USA (probably the largest resource contributor) Congress will go in front of the television cameras and say that the failure of the United Nations and the disproportionate contributions to a pooled resource fund will ensure failure. They will point to the failure of the world to effectively coordinate collaborative research on HIV/AIDS and point to politics, homophobia, disrespect, and the hatred of American politics by certain national and fundamentalist groups and say we would be wasting our money by letting Africans and Arabs and the Russians and Chinese and Indians and Asians and South Americans collaborate with the USA on research and ensuring that research leads to effective treatments for at least some rare diseases.
Enough already. Let’s rise to the occasion of solving resource limitations in studying rare diseases and get an effective mechanism in place for expanding the impact of admittedly small research efforts by individual countries through international cooperation. I trust the governments of the world to collaborate, contribute as they can, and help us start to get some of these diseases treatable. Disease knows no boundaries.
In the last century we collectively developed very advanced medical research techniques. In this century we need to use these methods to solve all of the medical problems possible by putting aside the nonsense politics and nationalism and individual egos and predatory profits and focus on solving many medical issues and ensuring access to effective treatment world wide.
Here’s a way to start. Any yes, this is a test of our humanity and commitment to universal human rights of which medical treatment is but one. But let’s start somewhere that should be relatively easy to agree on (and let a few hundred angry politicians in the USA know that the world considers them bratty children and cannot tolerate their obstructionist and oppositional behavior).
Click on the image to expand. And let’s start the process of collaboration.
- Rare Disease Treatments On The Rise: Will Big Pharma’s Focus On Orphan Drugs Benefit Us All? (medicaldaily.com)
- More Than 450 Innovative Medicines in Development for Rare Diseases (hispanicbusiness.com)
The design has historically been considered the best way to “prove” that new medical interventions work, especially if the experiment is replicated a number of times by different research teams. By the double blind (neither the treating medical team nor the patient know whether the patient is taking a placebo or active medication) design, investigators expect to negate the placebo effects caused by patient or medical staff beliefs that the “blue pill” is working.
A key part of virtually all double-blind research designs is the assumption that all patient expectations and reports are independent. This assumption is made because of the statistical requirements necessary to determine whether a drug has had a “significantly larger effect” as compared to a placebo. Making this assumption has been a “standard research design” feature since long before I was born more than 60 years ago.
Google the name of a new drug in clinical trials. You will find many (hundreds, thousands) of posts on blogs, bulletin boards for people with the conditions being treated with the experimental drug, and social media, especially Twitter and Facebook. Early in most clinical trials participants start to post and question one another about their presumed active treatment or placebo status and whether those who guess they are in the experimental condition think the drug is working or not. Since the treatments are of interest to many people world-wide who are not being treated with effective pharmaceuticals, the interest is much greater than just among those in the study.
Google the name of a new drug being suggested for the treatment of a rare or orphan disease that has had no effective treatments to date and you will find this phenomenon particularly prevalent for both patients and caregivers. Hope springs eternal (which it SHOULD) but it also can effect the research design. Obviously data that are “self reported” from patient or caregiver questionnaires can be affected by Internet “the guy in Wyoming says” or the caregiver of “the woman in Florida.”
OK you say, but medical laboratory tests and clinical observations will not be affected because these indices cannot be changed by patient belief they are in the experimental or placebo conditions. Hhmmm, Sam in Seattle just posted that he thinks that he in the experimental condition and that his “saved my life” treatment works especially well if you walk 90 minutes a day or take a specific diet supplement or have a berry-and-cream diet. Mary in Maine blogs the observation that her treatment is not working so she must be in the placebo condition and becomes very depressed and subsequently makes a lot of changes in her lifestyle, often forgetting to take the other medications she reported using daily before the placebo or experimental assignment was made.
Do we have research designs for the amount of research participant visible (blogs, tweets, bulletin boards) and invisible (email, phone) communication going on during a clinical trial? No. Does this communication make a difference in what the statistical tests of efficacy will report? Probably. And can we ever track the invisible communications going on by email? Note that patients who do not wish to disclose their medical status will be more likely to use “private” email than the public blog and bulletin board methods.
Want an example. Google davunetide. This was supposed to be a miracle drug for the very rare neurodegenerative condition PSP. The company (Allon) that developed the drug received huge tax incentives in the USA to potentially market an effective drug for a neglected condition. The company, of course, was well aware that after getting huge tax incentives to develop the pharmaceutical, if the drug were to prove effective in reducing cognitive problems (as was thought), it would then be used with the much more common (and lucrative from the standpoint of Big Pharma) neurodegenerative disorders (Alzheimer’s, Parkinson’s) and schizophrenia.
Patients scrambled to get into the trial because an experimental medication was better than no medication (as was assumed, although not necessarily true) and the odds were 50/50 of getting the active pills.
Patients and caregivers communicated for more than a year, with the conversations involving patients from around the world. In my opinion, the communications probably increased the placebo effect, although I have no data nor statistical tests of “prove” this and it is pure conjecture on my part.
The trial failed miserably. Interestingly, within a few weeks after announcing the results, the senior investigators who developed and tested the treatment had left the employ of Allon. Immediately after the release of the results, clinical trial participants (the caregivers more than the patients) started trading stories on the Internet.
Time for getting our thinking hats on. I worked on methodological problems like this for 30+ years, and I have no solution, nor do I think this problem is going to be solved by any individual. Teams of #medical, #behavioral, #communication, and #statistical professionals need to be formed if we want to be able to accurately assess the effects of a new medication.
Click on the image to expand.
An Early #MindMap (actually #OutlineMap) of Program Evaluation Goals in a Project with >100 Nursing School Geriatric Nursing Programs
This mind map was originally prepared in MindManager in the late 2001 for an evaluation of an initiative to increase the capacity of US Nursing Schools to meet the need for graduate-trained gerontological/geriatric nurses. I spent 10 minutes running the original file through iThoughtsX to add some color hues.
This was an evaluation of an important initiative funded by the John A Hartford Foundation (Building Academic Geriatric Nursing Capacity).
Click on the image to expand.
An Early Example of Developing a #MindMap Vision Statement/Plan for Public Services for High Need Women and Children in Los Angeles 2020
This is a #MindMap (or more properly an #OutlineMap) of a year 2020 Vision Statement/Plan for one of the largest Los Angeles County comprehensive socialcare agencies for women and their children. The Vision Statement/Plan was developed with management over a six-month period.
Click on the map to enlarge it.
An Early Example of the Use of #MindMaps to Manage a Complex #Evaluation of 21 Elder Abuse Service Projects
2006 versions of mind maps from an evaluation of 21 projects funded to demonstrate various models of elder abuse services by the Archstone Foundation.
The program evaluation was managed and explained using a large mind map created in the version of Mind Manager current at that time. We also had many detailed mind maps, used internally, of the hundreds of indicator variables collected and coded.
Consistent with my current thinking, I would now categorize these maps as outline maps, not mind maps. These are really outlines that have undergone cosmetic surgery, not true actively-developed mind maps.
Click on images to expand.
As a simple exercise, the set up for the mind map above was imported to the iThoughtsX computer program released in mid September 2013. Simple color coding in iThoughtsX makes the map above much more useful.
I corrected a huge mistake in my thinking about mind maps during 2010.
I had started using the program Mindjet MindManager for mind maps at the time version 2 of the program was released. Over almost 20 years I used occasionally used MindManager, alternating periods of a few days of intensive use with months of ignoring mind mapping.
I hardly considered organic mind mapping in the early days because: a) I cannot draw clearly or even print clearly even though Tony #Buzan says everyone can; b) I am a “tech guy or nerd” and damn it, why would I hand draw something if a computer program was available to turn my brilliant thoughts and words into pictures.
Secondarily, how could I possible use wavy lines with labels in all kinds of orientations and colors best-reserved for a child’s coloring book or a circus? I worked with groups of federal/state health policy makers, physicians, psychologists, social workers, nurses, counselors, grant funders, politicians, and public advocacy groups. Colors that looked like they came from a crayon box and drawings that looked like they were drawn by a second grader would be seen as childish, silly, not useful, and (most importantly) disrespectful by a group of senior professionals in the health/social care areas.
I bought every upgrade of MindManager over 20 years. Those upgrades were pretty expensive for a small consulting firm charging public sector fees less than half of those of private-sector companies.
I had strong misgivings about the MindManager mind maps I presented in meetings about HIV/AIDS services, research designs, elder abuse, optimally training geriatric nursing leaders, statistical analyses, and the many related topics I worked on during my career. Nonetheless I kept presenting the maps and using them in written reports.
I came to the conclusion that the method of mind mapping was primarily a way of presenting outlines in a somewhat novel way that introduced a lot of “white space” into diagrams typically plagued with too many words on a boring and ignored PowerPoint slide. Business executives liked the MindManager approach since it was in their comfort zone (outline in a picture).
I was becoming a Bleeping Idiot for continuing to use MindManager style Outline Mapping.
I read about the iMindMap program in a variety of tweets from individuals I followed on Twitter and started trying the program and then reading much of the collected writings of Buzan; I watched some of the YouTube videos derived from his telecasts.
I thought organic mind mapping was kind of cool. It interested me at first because it would lead to presentations that were far more interesting than the ones with PowerPoint I suffered through 100 times a year (and gave myself to large audiences at least 50 times a year).
A couple of months later I decided that I would give an entire presentation (and the final report) using iMindMap 5 maps to a group at the US Health Resources and Services Administration, the major US government agency for financing public healthcare clinics and programs (and especially those targeted to HIV/AIDS services).
The project was to develop a framework for teaching program managers of US-funded, locally-administered African projects on increasing the number of nurses trained in and providing clinical services for treating HIV/AIDS. The topic was about program evaluation theory and implementation. Program evaluation can be a very technical area dominated by methodologists who speak “numbers” not concepts, acronyms, and is often perceived as excruciating by its participants.
The meeting was with two senior federal grant administrators and USA-funded program managers and service providers, half from the Columbia University (USA) and half from Africa who were part of a six African-nation collaborative team.
I developed a dozen pretty large mind maps on evaluation goals and results, ways to conduct the evaluation and why, how to improve services using the results, respecting clients, and other issues including ethics and reporting results to the funders. The general topics were ones I had discussed with hundreds of groups in the prior 20 years.
All of the mind maps were developed in iMindMap using circus colors, curves, cartoony clip art provided in the program, font coding, and a nonlinear organization. I wanted to animate the presentation by jumping around the map “automatically.” This was before mind mapping programs in general (and iMindMap specifically) included presentation animations. At the suggestion an expert on visual thinking, Roy Grubb (a Twitter buddy from Hong Kong — @roygrubb), I used the program Prezi to animate the jumps around the map into to what could be a presenter-guided talk or a self-running kiosk video.
To say that the presentation was well received by the audience of program managers, senior policy makers, and medical professionals from the USA and various African nations) would be a gross understatement. The presentation was praised, a couple of physicians said this was the first time they really understood what evaluation was, and perhaps more concretely, the participants insisted on having the one-hour presentation evolve into a two-hour greatly interactive and animated group problem solving session that pissed off the US State Department because the participants arrived to their meeting at State an hour late. The evaluation for the next five years of an extremely large funding program in Africa on HIV/AIDS treatment capacity was altered. A subsequent program evaluation project for the African project was funded to our company.
I was just presenting the same-old/same-old conclusions I had evolved over two decades. But the information after I reformatted it into a #Buzan style mind map using the iMindMap program forced me to re-think the overall system of evaluation I believed in so as to prepare a liberating and valuable experience for the audience. The new mind maps were nonlinear THEORETICAL MODELS accessible to individuals with training neither in program evaluation nor mind mapping.
By contrast, the old way i would have presented the same information in MindManager or as bullets in PowerPoint was as nothing more than a formatted outline (or what I now call an Outline Map) and my thinking and that of the participants would not have gone in such creative directions.
I was pleased to find out that one of the meeting participants had been trained in a workshop by Mr. Buzan and that she felt that the presentation mind maps were the most Buzan-like she had seen since the training.
The hundreds of mind maps I have made for this blog have reinforced the conclusion I reached from that HRSA meeting on HIV/AIDS that computer-assisted, Buzan-style organic mind maps and visual thinking methods are far superior to the “traditional” linear methods that are forced by some computer programs that do not encourage Buzan-style thinking and mapping.
Bright colors, contrasting fonts, curvy lines, cartoon graphics, one word per branch, nonlinear organization …
I joined the Circus.
Let’s be honest, there is not enough empirical, hard scientific evidence that mind map based learning programs are as effective as there should be. In fact there is FAR less evidence to support efficacy claims about mind mapping than there should be. This has to be fixed.
Before the mind mappers start cursing me out, put this into context — I strongly support mind mapping and think it should be used far more than it is But I cannot find specific studies that strongly support efficacy.
Don’t flip the channel yet … I am now going to give you the most valuable free consulting I have ever provided anyone.
A few studies give people some training into “who knows what” mind mapping and see if they remember or “learn who knows what” better. Creativity is not measured, communication is not measured, long-term efficacy is not measured, training clinical practice efficacy is not measured, and many other aspects of cognitive enhancement claimed are not measured. Still, I believe that mind mapping is useful for most of these things and mind mapping works.
Now “prove” it.
Here is the biggest reason why mind mapping has not been shown to work in anything approaching a “definitive” scientific study or unbiased evaluation — too many things are called “mind mapping” are all lumped together.
A strong research (evaluation) design includes the following factors.
a) Different things called mind mapping are compared. As I see it, there is are three major things called “mind mapping.” The first is Buzan-style organic mind mapping. My bias is to say that this will work best in most (but not all) applications, but I would like to see hard data that my observations are correct. The second style of mind mapping is that embraced by those who use Mindjet aka Mind Manager and comparable programs. Such a style seems to be preferred among business types, and I used Mindjet (formerly known as Mind Manager) for about 15-20 years with many different types of health- and social-care professionals. Then there are dozens of other methods and diagrams called “mind maps,” most of which probably could be called spider maps. I would clump all of these methods together although I do recognize that the category is very heterogeneous.
Addition to original post: Separating these three categories will almost certainly show that the three clusters of methods are not equally effective for all applications. Combining them together dilutes the effects of the first and second methods because the third is probably comprised of a number of less than effective methods.
b) The effects of mind mapping need to be maximized. That is, the participants learning mind mapping or being taught to read existing mind maps need to be trained by experts (and I mean real top-of-the-food-chain mind mapping instructors) in one of the three types of “mind mapping.” The instructor needs to be a “real pro” at this, not a teacher or consultant who has had minimal formal training in mind mapping. Random assignment of participants (subjects) to one of the three mind mapping conditions needs to be made.
c) A lot of before and after variables need to be measured like memory, creativity, ability to learn new materials, ability to increase upon prior knowledge, sophistication of information processing, and all of the other things people claim about mind mapping.
d) Then the data need to be analyzed for enhancements (or not) from mind mapping according in each of the three three dominant models. That is, there needs to be a study of the interactions of learning one of the three mind mapping models from an expert, type of application, and type of effects.
Show me a dozen studies that support mind mapping (with random assignment, large samples, and conducted by a neutral investigator in this highly competitive commercial area) and I will tell everyone it has been proven that mind mapping works for these 10 applications and not these 5 others and what the best kind of mind mind mapping is for achieving certain goals.
Show me even better and more complex studies and I will jump with glee that my own observations have been confirmed.
Or, if it doesn’t work, accept the fact that this is voodoo, a management-education-training fad, or just plain commercial exploitation. (I don’t believe it is any of these things but I also cannot say YET that science unequivocally understands mind mapping.)
You wanna make the big claims, get independent parties to test them in an unbiased way that meets the most rigid scientific-educational standards. The odds are you will be happy you did as will potential users and educator-trainers.
[If you are an education, psychology, neuroscience, or healthcare student there are a lot of good PhD dissertations to be written in this area.]
A few of my examples of using mind maps from around this blog/website.
Click images to zoom.
Between 2008 and 2010 (when I discovered Twitter), I published a blog. Here is one of my favorite posts from that time. It still applies today as much as back then.
Please click on the text below to zoom in.
Irv Oii is known to many international news organizations and researchers as a star data journalist. Being a home worker (although home may be the UK, Ohio, the Middle East, Central Africa, Hong Kong, or Antartica) and a fairly reclusive person, nobody seems to have met Irv. Some speculate that he might be a Jewish Asian-American. Others believe Irv is short for Irvelina, a Russian immigrant physician who went to Ohio (or was it Ojai, California) when the Soviet science programs collapsed and turned into the lower funded Russian collaborative efforts with the EU and USA. The collapse of the Soviet Union resulted in the closing of her laboratory in Minsk. Some even think Irv Oii is an acronym.
Irv is thus an enigma and no pictures of her/him seem to exist. An artist’s conception (mine) based on the writings and consultations of Irv Oii on healthcare breakthroughs is shown below. My belief is that a portrait of Irv should hang over the desk of every data journalist and researcher.
Please click the image to zoom.
Click on mind map to expand.
Please click on the graphics to zoom.
There are a number of things that can be done to cut the cost of healthcare while, at the same time, freeing doctors and others to do their jobs better. These improvements cost almost nothing to implement [if all of the constituencies and politicians do not compete to be King Kong].
Visiting legislator who stumbled across this web page? Here’s your chance to act like a grown-up and represent the people of the world, not drug companies nor major research universities nor individual “researcher” egos and retirement funds.
Click on the images to zoom.
The fictional detectives would have been great program evaluators. All looked at all types of data. Miss Marple was a model of pleasantry who could work her way into an organization or group and see it as it was without changing anything by observing. Holmes and Watson — whether in the original books and movies, the Ironman version of the movies, their current BBC incarnation in 21st Century London, or their CBS incarnation in 21st Century Manhattan with Dr John Watson now Dr Joan Watson (for the better) — use Holmes’ razor sharp mind and Watson’s intuitiveness and questioning. Sam Spade, wise cracks, an iron fist, and underlying sensitivity.
Program evaluation is not about conducting research, randomly assigning participants to conditions, or using quasi-experimental designs. Program evaluation is about understanding why programs produce certain outcomes, intended or not, positive or not, unique or not. To truly understand a program quantitative and qualitative data needs to be collected with great attention to the sensibilities, needs, risks, and potential confidentiality breaches of data of program participants, program staff, program administration, funders, and other stakeholders.
I love program evaluation. Every program is unique and at the same time representative of certain classes of human service organizations.
Be a detective. Look carefully and understand the beauty of a well-running program and how to help staff improve a program that is not working as well as it could.