Info

social, health, political imagery through the lens of G J Huba PhD © 2012-2021

Posts tagged evaluation

Choose another tag?

The majority of the posts on this blog are about using visual thinking methods — of which I think that by far the best is #Buzan-style organic mind mapping — to understand, explain, evaluate, and communicate about healthcare. A lot of my own thinking has focused on using visual thinking techniques to potentially improve the quality of life of those with cognitive impairment and dementia.

Tony Buzan and Chris Griffiths and their colleagues and staff at ThinkBuzan have done a very comprehensive job at getting many of Buzan’s ideas embedded into a general purpose computer program (iMindMap) which provides a general visual thinking environment, of which mind mapping is a special part. There are many computer assisted mind mapping programs, but I have concluded that iMindMap is by far the best for creative visual thinking and communication, in no small part because it fully incorporates Buzan’s theory and theoretical implementation.

Like scientists and management consultants and educators and healthcare providers and patients and patient caregivers and students and many others, illustrators struggle with how to best use visual representations to support better thinking and communications.

Which brings up this beautifully conceived and executed little book that I have found to be mind expanding and liberating in how to develop and use a series of illustration techniques and “tricks” to look at things differently when trying to make creative breakthroughs.

Whitney Sherman is the author of the book “Playing with Sketches” which provides 50 exercises which collectively will change the way you think about creating images to understand and communicate ideas.While Ms. Sherman wrote the book for designers and artists, the techniques will be just as useful for visual thinkers in science, education, medicine, industry, and other fields. The beauty of Ms Sherman’s exercises is that in showing you fairly simple ways to make hugely informative and well designed images, the tools will themselves suggest many applications to visual thinkers of all types.

And, I have found that Ms. Sherman’s techniques can be used by the severely artistically challenged (of which I am one); the techniques are ones for Visual THINKERS, not necessarily artists and designers.

I have mentioned this book before in much less detail, but in the months I have used the methods, I have found that they WORK very well to facilitate creative visual thinking. For me they have promoted a breakthrough in how I see the visual thinking canvas.

Get the book, try some of the techniques (pick a random one here and there to start), discover that great artistic talent or aptitude is not required, and see how the techniques fit the information you study in search for better healthcare or disease prevention or decision making or facilitating creative group processes.

In partnership with Tony Buzan’s techniques for organic #mindmapping and Mike Rohde’s framework for #sketchnoting, the techniques codified by Whitney Sherman provide very powerful visual thinking tools.

Ms. Sherman’s website is http://www.whitneysherman.com. She tweets at @Whitney_Sherman. The book is available from major online book sellers.

2014-10-26_21-24-51

 

 

I will be posting some examples of using the sketching techniques of Ms. Sherman to developing assistance and communication techniques for those with cognitive impairment or early-mid stages of dementia.

2014_10_27_08_51_03

 

2014_10_27_08_51_05

 

At the end of looking at one of those 30-page tiny font CVs from academics or the pro-forma 2-page resumes in industry, have you ever thought, “Wow, I know what this person is like.” The 1% of you who said “yes” probably didn’t understand the question.

There are lots of alternatives. Here is mine. And, yeah, I wore the John Lennon eye glasses in the 1960s. Wore a few peace symbols too.

Personally I think you learn more about the person from looking at the picture.

Click to expand.

george huba the summary may 2014

Remember the “gold standard” research paradigm for determining if a medical treatment works: the DOUBLE BLIND, RANDOM ASSIGNMENT EXPERIMENT?

The design has historically been considered the best way to “prove” that new medical interventions work, especially if the experiment is replicated a number of times by different research teams. By the double blind (neither the treating medical team nor the patient know whether the patient is taking a placebo or active medication) design, investigators expect to negate the placebo effects caused by patient or medical staff beliefs that the “blue pill” is working.

A key part of virtually all double-blind research designs is the assumption that all patient expectations and reports are independent. This assumption is made because of the statistical requirements necessary to determine whether a drug has had a “significantly larger effect” as compared to a placebo. Making this assumption has been a “standard research design” feature since long before I was born more than 60 years ago.

2013

Google the name of a new drug in clinical trials. You will find many (hundreds, thousands) of posts on blogs, bulletin boards for people with the conditions being treated with the experimental drug, and social media, especially Twitter and Facebook. Early in most clinical trials participants start to post and question one another about their presumed active treatment or placebo status and whether those who guess they are in the experimental condition think the drug is working or not. Since the treatments are of interest to many people world-wide who are not being treated with effective pharmaceuticals, the interest is much greater than just among those in the study.

Google the name of a new drug being suggested for the treatment of a rare or orphan disease that has had no effective treatments to date and you will find this phenomenon particularly prevalent for both patients and caregivers. Hope springs eternal (which it SHOULD) but it also can effect the research design. Obviously data that are “self reported” from patient or caregiver questionnaires can be affected by Internet “the guy in Wyoming says” or the caregiver of “the woman in Florida.”

OK you say, but medical laboratory tests and clinical observations will not be affected because these indices cannot be changed by patient belief they are in the experimental or placebo conditions. Hhmmm, Sam in Seattle just posted that he thinks that he in the experimental condition and that his “saved my life” treatment works especially well if you walk 90 minutes a day or take a specific diet supplement or have a berry-and-cream diet. Mary in Maine blogs the observation that her treatment is not working so she must be in the placebo condition and becomes very depressed and subsequently makes a lot of changes in her lifestyle, often forgetting to take the other medications she reported using daily before the placebo or experimental assignment was made.

Do we have research designs for the amount of research participant visible (blogs, tweets, bulletin boards) and invisible (email, phone) communication going on during a clinical trial? No. Does this communication make a difference in what the statistical tests of efficacy will report? Probably. And can we ever track the invisible communications going on by email? Note that patients who do not wish to disclose their medical status will be more likely to use “private” email than the public blog and bulletin board methods.

Want an example. Google davunetide. This was supposed to be a miracle drug for the very rare neurodegenerative condition PSP. The company (Allon) that developed the drug received huge tax incentives in the USA to potentially market an effective drug for a neglected condition. The company, of course, was well aware that after getting huge tax incentives to develop the pharmaceutical, if the drug were to prove effective in reducing cognitive problems (as was thought), it would then be used with the much more common (and lucrative from the standpoint of Big Pharma) neurodegenerative disorders (Alzheimer’s, Parkinson’s) and schizophrenia.

Patients scrambled to get into the trial because an experimental medication was better than no medication (as was assumed, although not necessarily true) and the odds were 50/50 of getting the active pills.

Patients and caregivers communicated for more than a year, with the conversations involving patients from around the world. In my opinion, the communications probably increased the placebo effect, although I have no data nor statistical tests of “prove” this and it is pure conjecture on my part.

The trial failed miserably. Interestingly, within a few weeks after announcing the results, the senior investigators who developed and tested the treatment had left the employ of Allon. Immediately after the release of the results, clinical trial participants (the caregivers more than the patients) started trading stories on the Internet.

Time for getting our thinking hats on. I worked on methodological problems like this for 30+ years, and I have no solution, nor do I think this problem is going to be solved by any individual. Teams of #medical, #behavioral, #communication, and #statistical professionals need to be formed if we want to be able to accurately assess the effects of a new medication.

Click on the image to expand.

Clinical Trial Double-Blind Treatment Evaluation in the Era of the Internet

2006 versions of mind maps from an evaluation of 21 projects funded to demonstrate various models of elder abuse services by the Archstone Foundation.

The program evaluation was managed and explained using a large mind map created in the version of Mind Manager current at that time. We also had many detailed mind maps, used internally, of the hundreds of indicator variables collected and coded.

Consistent with my current thinking, I would now categorize these maps as outline maps, not mind maps. These are really outlines that have undergone cosmetic surgery, not true actively-developed mind maps.

Click on images to expand.

A MM9

A MM8

A MM7

A MM6

A MM5

A MM4

A MM3

A MM2

A MM 1

Coding for Archstone Elder Abuse & Neglect Initiative

As a simple exercise, the set up for the mind map above was imported to the iThoughtsX computer program released in mid September 2013.  Simple color coding in iThoughtsX makes the map above much more useful.

Coding System for Archstone Foundation Progress Reports. Elder Abuse & Neglect Initiative (2)

Can Big Data/Data Science avoid the train wreck of Big Pharma? I believe that the Big Data disaster will make the Big Pharma issues seem small in comparison.

But the issues will be about the same. A lot of the Big Pharma execs have become quite skilled at “beating the system” using “undocumented science” and many will move to Big Data and employ all of their very “best” moves and tricks. Big Data/Data Science has the potential to hurt the average individual even more than the greediness of Big Pharma.

Big Pharma

Big Pharma Train Wreck

Big DataBig Data Train Wreck

Help!!!!!

HubaMap™ by g j huba phd

I guess it’s just me … I search Google for sites with “psychology mind maps” and I get lotsa pages returned. Of course very FEW of these pages let you know where the ideas, recommendations, and organization comes from. That makes me pretty pissed off.

I have a simple rule for evaluating psycho-pop, psycho-babble, psycho-art, and psycho-schmaltz: if the author (artist, developer) cannot prove to me that the information came from a credible source and is being communicated by a credible source, I assume it is psycho-fantasy and just walk (actually run) away.

Here’s a few things to ask about before you go ahead and change your job, spouse, running shoes, or haircut because somebody gives you some magic MBTI letters, a number on a test published in a self-magazine, or advice that must be right because it appears in a pretty mind map.

I love great psychology content conveyed in an easy to understand manner. I hope I produce some. Most do not produce anything except profits. Know what you are buying (and staking your life on) when you get information from a book, TV, the Internet, text, or a graphic.

Please click on the diagram to zoom in.

Don't Believe a Psychology (Self Help) Mind Map Unless it Tells You

Irv Oii is known to many international news organizations and researchers as a star data journalist. Being a home worker (although home may be the UK, Ohio, the Middle East, Central Africa, Hong Kong, or Antartica) and a fairly reclusive person, nobody seems to have met Irv. Some speculate that he might be a Jewish Asian-American. Others believe Irv is short for Irvelina, a Russian immigrant physician who went to Ohio (or was it Ojai, California) when the Soviet science programs collapsed and turned into the lower funded Russian collaborative efforts with the EU and USA. The collapse of the Soviet Union resulted in the closing of her laboratory in Minsk. Some even think Irv Oii is an acronym.

Irv is thus an enigma and no pictures of her/him seem to exist. An artist’s conception (mine) based on the writings and consultations of Irv Oii on healthcare breakthroughs is shown below. My belief is that a portrait of Irv should hang over the desk of every data journalist and researcher.

Please click the image to zoom.

Irv Oii

Click on mind map to expand.

academia and healthcare big data

Splatter10

9

Please click on the graphics to zoom.

it's program evaluation, not research, dummy

 

3

evaluationmap

researchmap

Big data this, big data that. Wow. At the end we will have better ways to sell underwear, automobiles, and “next day” pills (although in the latter case politics and religion might actually trump Amazon and Google). Blind empiricism. Every time you click a key on the Internet it goes into some big database.

“Little data” — lovingly crafted to test theories and collected and analyzed with great care by highly trained professionals — has built our theories of personality, social interactions, the cosmos, and the behavioral economics of  buying or saving.

Big data drives marketing. Little data drives the future through generalizable theory.

Click on the figure below to zoom.

in praise of little data

There are a number of things that can be done to cut the cost of healthcare while, at the same time, freeing doctors and others to do their jobs better. These improvements cost almost nothing to implement [if all of the constituencies and politicians do not compete to be King Kong].

Visiting legislator who stumbled across this web page? Here’s your chance to act like a grown-up and represent the people of the world, not drug companies nor major research universities nor individual “researcher” egos and retirement funds.

Click on images to expand.[almost free] strategies to improve healthcare

Cartoon Rabbit - Giving A Thumbs Up

image

Click on the images to zoom.

The fictional detectives would have been great program evaluators. All looked at all types of data. Miss Marple was a model of pleasantry who could work her way into an organization or group and see it as it was without changing anything by observing. Holmes and Watson — whether in the original books and movies, the Ironman version of the movies, their current BBC incarnation in 21st Century London, or their CBS incarnation in 21st Century Manhattan with Dr John Watson now Dr Joan Watson (for the better) — use Holmes’ razor sharp mind and Watson’s intuitiveness and questioning. Sam Spade, wise cracks, an iron fist, and underlying sensitivity.

Program evaluation is not about conducting research, randomly assigning participants to conditions, or using quasi-experimental designs. Program evaluation is about understanding why programs produce certain outcomes, intended or not, positive or not, unique or not. To truly understand a program quantitative and qualitative data needs to be collected with great attention to the sensibilities, needs, risks, and potential confidentiality breaches of data of program participants, program staff, program administration, funders, and other stakeholders.

I love program evaluation. Every program is unique and at the same time representative of certain classes of human service organizations.

Be a detective. Look carefully and understand the beauty of a well-running program and how to help staff improve a program that is not working as well as it could.

evaluation detective work

The Program Evaluator's Tool Kit

Measure, Find Relationships, Communicate

image image

image image
Ask the Big Questions and Study How the Answers Relate to One Anotherimage

Decode Events and Naturally Occurring Data

image image

Examine in Detail

image

Listen
 image

Find Patterns
image

This is the first of a series of posts I am making about program-organizational (and individual) evaluation. Much of what I will discuss is not in the mainstream of traditional program evaluation methodology.

My approach is different. It works.

In this first section the point is — obviously — that evaluation is iterative and nonlinear. This led to my first model that EVALUATION IS DETECTIVE WORK several decades ago. [Perhaps that explains my current obsession with all versions of Sherlock Holmes, whether in the original, present London, present New York, or by Iron Man.] At any rate, it seems ELEMENTARY to me that instead of thinking of program evaluation as a linear research experiment with a fixed design (a metaphor that works at best imperfectly), it is more important to treat evaluation as detective work where good rules of evidence must be followed and the evaluator is at fault if all outcomes are not found.

My initial development of the Detective Model in 1992 came from my observation that in much traditional program evaluation the evaluator applies a flawed “research” experimental model and the insensitivity of this approach means that a program looks worse than it is because the evaluation methodology is in error. Who pays for this problem? The program, of course, since the evaluator walks away saying that the “program sucks” and not that the evaluator screwed up. In the Detective Model, applied iteratively and nonlinearly, the evaluator and the program are partners, and it is clear what the responsibilities and level of success each has.

Seems ELEMENTARY to me.

As usual click on the image to zoom.

Evaluation is an Iterative, Non-linear Process

Content is Queen. The ultimate point of any mind map is to use and present information clearly in a way that communicates conclusions  that are valid, reliable, and important.

Some examples. Are all of those mind maps floating around showing psychological variables and purporting to illustrate major findings and theories actually using valid information? (Guessing what all people feel like or how they learn and thinking it must be valid since, after all, you are a human, is probably not an indication that you are using highly valid data.) What is the expertise of the individuals who generated the information portrayed in the mind map? Was the information based on empirical studies, well-established theory, the musings of a pop psychology writer, what your Mom taught you, what your best friend thinks, what you saw in a movie? Did you (as a student or casual reader) just read a popular psychology book and accept what that person wrote on how you can be more rich, famous, happy, socially connected, sexy,and thin?

Much attention in mind mapping goes into the “artistic presentation” aspects of the maps, the colors, the rules, the images. And yes, prettier, neater, more original, and more creative maps are probably better received than those that use none of the great tools of visual thinking. But the reality is that the clothing does not make the person nor does the artistry of the map make the content more valid or reliable or important.

The first mind map below shows some of my thoughts and suggestions about how mind maps should be reviewed by experts in the content areas being addressed if the map will be used for purposes other than personal learning or process documentation or as art. That is, if the point of the map is to present facts, then the purported facts really need to be checked by someone who is an expert in the content area. In most cases, I have no problem with authors being responsible for their own work so long as they clearly state their own expertise levels and where the data for the mind maps originated. I have a big problem with someone who is not a trained mental health professional telling the world how to diagnose depression or ADHD. If the author of the map is not an acknowledged expert presenting her or his own work, then the source and limits of the information in the mind map need to be stated, and in some cases, independently evaluated.

Evaluating Mind Maps with Expert Content

The second mind map is actually just the first one produced in iMindMap exported into the alternative computer program MindNode Pro. Is the first map prettier than the second? Sure seems so to me. Is the first map more valid? No. It contains identical information. Does the first map communicate better than the second? Sure seems so to me.

Keep in mind that the goal of most mind mapping is to present valid, reliable, and important information in way that is easily understood, easily remembered, and easily communicated. Using this criterion the first map is probably significantly better.

iMindMap5 Map

The third mind map is identical in content to the two maps just considered but was generated using default options in the program XMIND. The style of the mind map is similar to that of another program (Mindjet AKA MindManager), and is that many argue is the best for presenting information to those in business.

XM Evaluating Mind Maps with %22Expert Content%22

Hopefully by the time you read this, you will have looked carefully at the actual content of the mind map in one or more of the variations. Content is Queen; it is all about the ideas. In the process of mapping, we need to incorporate references to the source of the information displayed. Pretty is good and memorable, but is not more important than the information presented. Content is Queen, although she does look better in a nice dress or business suit.

Keyword Boardtopics and sub-topics: evaluating mind maps with “expert content” criteria information accurate source stated authoritative recognized cited by others opinion? state adult learning multi-channel non-hierarchical non-linear iterative approximations successive small steps link existing knowledge experience emotions cultural memory consensus neuroscience “catchy” style serious disease disaster war human toll horror funny often many topics “lighter” facts graphic usually images fonts colors this opinion mine g j huba phd @drhubaevaluator © 2012 all rights reserved based professional judgment experience 15 years healthcare professionals researchers physicians nurses psychologists social workers others administrators no science citations but read dr seuss really early lexical mind mapper organic style tony buzan thinking flexible suggestions discussion @biggerplate quick notes iteration 1 imindmap mac written on limited to content purportedly expert reproducible empirical “textbook” peer review? content content content content most important meaningful valid reliable educational goals objectives audience mind maps uniqueness used color fonts non-linearity “artistic” memorable by established experts content visual thinkers other concerns mission critical data good empirical public never present as perfect examples medical safety criminal justice financial mental health reproducibility mind map logic data logic education logic expert knowledge conclusions