|
Critical Evaluations by Dwight L Adams
1-Are you doing quantitative (deductive) research or qualitative (descriptive) research?
2- How many subjects do you have?
3- Sometimes you are doing a specialized study; you may be evaluating a specific program for its effectiveness
4- With all papers, you will want to outline the "who, what, when, where, how, and why" for your discussion
5- Most controversies involve more than just "2 sides to every question."
6- Major political persuasions in America
7- Deductive Vs Inductive Reasoning
8- The Eleven Stages of a Quantitative Research Project
9- The written outcome of one type of Qualitative Research
10- Program Evaluations
11- A Social Impact Assessment is a part of the larger Environmental Impact Assessment process
12-"Variations on Experimental Designs."
13- A scientific journal editor's viewpoint of how a good paper should be reviewed and what, according to the scientific
community, it should contain
14-One of the most frequent errors made in research is either ecological fallacy or reductionism
15-Example of the Best Research Design given the Research Purpose
16- The research sample
17-Extraneous Variables and Definition or the possible problem
18- Helpful Gerontological Research Notes
19-Studies based on samples of the population can yield more accurate results than studies that examine every individual
in the population.
20- Statistics that allow you to analyze what data you have gathered
21- How to get specific statistical output
22- SPSS and how it can help you in analyzing your data.
23- The final part of your text covers the ethical requirement to publish your findings so that others can benefit, replicate,
and build further in the inquiry for truth
When a student is called upon in a paper or project to "critically evaluate" the evidence, the nature of the
assignment helps determine how to evaluate it.
1-Are you doing quantitative (deductive) research or qualitative (descriptive) research?
The most common assignment is one dealing with quantitative
information. Less common, but equally important is the qualitative research project. This area includes case studies.
2-How many subjects do you have? Case studies, as mentioned above, lend themselves to descriptive (qualitative) research
as does projects delving into areas that are not studied in the mainstream research projects. Do you find moderate to large
amounts of research on this topic in the literature? Then you should go about your evaluation as deductive or quantitative.
3-Sometimes you are doing a specialized study; you may be evaluating a specific program for its effectiveness. Example:
A POLICY is the explicit or implicit standing plan that an
organization, system or government uses as a context for making its decisions. The plan is the totality of the principles,
positions, political platforms, social norms and guidelines that are relevant to the people it serves. In effect, "policy
rules our lives." It includes laws, rules, regulations and guidelines, and reflects the
norms and values of our society. This encompasses PUBLIC
POLICY, which includes all policy that affects citizens.
SOCIAL POLICY is defined as those activities and principles of society, which guide the way society intervenes in and
regulates relationships between individuals, groups, communities and social institutions. Social policy is the result of society's
values and customs, and determines the distribution of resources and level of well being of its people. It helps guide immediate
and future decisions, especially with regards to allocation of resources that are both valuable, and in short supply.
Social policy involves government plans and programs
(education, crime prevention, health care, etc.) which are good for the citizens, and help decide who gets what, when
they get it, and for how long. There is a necessity to make choices because of limited funds.
Social policy is determined by bureaucracies, who take the laws passed by Congress, and transform them into workable policies
that affect the people. When a group of policies are all aimed at a specific target group (example: Social Security, senior
centers, etc., provided for the elderly), these are termed "GENERIC" PUBLIC POLICY. Taxes collected by the government
to fund these programs are often referred to as GENERAL REVENUES.
Social policy does not spring only from new and exciting ideas. It has its basis in the past; in what career government
workers are accustomed to and suggest to newly elected officials. Some believe that this bureaucracy is so powerful that it,
not the elected officials, really runs the legislative agenda and therefore, maps out the direction of policies into the future.
The policy analyst must address several key questions:
1- Is the proposed policy workable? How well is the current
policy working?
2- Is the proposed policy desirable? Is the current policy still needful?
3- What, if any, modifications should be made in the proposed or current policy?
4- Does the proposed or current policy represent a wise use of resources?
5- What barriers, if any, are there to the full the implementation of the proposed policy? Are there barriers to the current
policy? Can they be removed?
6- Are there alternative policies that would be preferable to the proposed or current policy?
7- How feasible is the implementation of the proposed policy? Is the current policy being implemented as or better than
originally planned?
4- With all papers, you will want to outline the "who, what, when, where, how, and why" for your discussion.
An example:
OLDER AMERICANS ACT - OAA "Cornerstone of aid to the elderly in America"
Where: Federal Government to Local AAAs
When: 1965
What: Community services for older Americans
Who: Older Americans (60+)
Why: Services, advocacy, politics
How: General revenues; "contributions" (the law prohibits
means testing or charging of fees for services for older Americans, so the elderly are encouraged to "contribute".
Many contribute because they want to pay for what they get.)
5- Most controversies involve more than just "2 sides to every question." Consider the following:
DECISION-MAKING THEORIES:
1. Traditional Model - public spirited citizens form planning groups, make rational decisions and propose fair plans.
2. Power Pyramid - a few prominent, powerful business leaders influence politicians and impose decisions on those lower
in the social structure.
3. Yale Polyarchic Power - different issues each have different leadership patterns.
4. Qualified Diffused Influence Model - influence is spread among many different interest groups.
ECONOMICS OF FUNDING POLICY:
1. Keynesian Economics - the government regulates the flow of money, with the increase in government subsidies resulting
in increased production.
2. Supply-Side Economics - (Reagan's trickle-down theory) believes that tax dollars are better used to help spur economic
growth by helping businesses grow, with the concept that economic growth will create more jobs, income
and goods.
3. Socialism - believes that social welfare programs support businesses and serve as a form of social control. They blame
Capitalism as the cause of need for social welfare.
4. Traditionalist - believe that God's laws should be the laws of state; there should be no separation of church and state.
Under this policy, there is little tolerance for welfare programs.
5. Libertarians - believes in very little or no government regulation; wants more responsibility by the people.
6. Self-Reliance - prefers low technology, and local solutions. All production is for internal use
---------------------------------
++Major political persuasions in America: MATCHING: (make the BEST match!)
a. conservative
b. liberal
c. radical
d. neo conservative
e. neo liberal
f. Keynesian
g. supply-side
h. socialism
i. tradionalist
j. libertarians
k. self-reliance
l. public choice
m. OAA
n. OSDI
o. GDP
____ 1. trickle down theory
____ 2. transfer to private sector
____ 3. unjust society
____ 4. historically Republican
____ 5. moral majority
____ 6. historically Democrat
____ 7. government stabilizes money
____ 8. the system is broken p. devolution
--------------------------------------------------
7-Deductive Vs Inductive Reasoning
When doing science, the choice of using inductive or deductive reasoning on a given project is basic to the planning of
the research. Inductive reasoning, which often is called "qualitative or descriptive research," is used when science
embarks on a phenomenon that is new or one that has had little or no previous work done on it. Simply put, there is not enough
in the literature to begin doing deductive research; there is not enough knowledge to begin to ask the pertinent questions
the researcher wants to ask.
A major methodology in inductive research is field work or
observation research (which may be participant observation or non-participant observation and overt observation or cover
observation). The idea is to get into the field and find out what is going on, without taking in pre-conceived notions of
what will be found. This method has been used often in anthropology and, by trial and error, many strengths and weaknesses
have been uncovered.
Because most textbooks on social research cover this methodology only lightly, this guide will emphasize it through both
readings and a project. It can be a valuable tool in your future.
Deductive logic is the most often used in sociology research projects. To better understand this methodology, the following
funnel is presented:
------------
General Sociological Imagination or Curiosity
Over all Paradigms (ex: nature or nurture)
Chosen theoretical perspective
Hypotheses which follow
Defined Questions
Instrument
Findings
Report
---------
At the top of the funnel is your imagination or curiosity. C. Wright Mills talks about this concept. A scientist needs
to brainstorm an idea to see it in the panorama of other
phenomenon and to tease out the real area of interest.
Next comes the paradigm or set of assumptions to be used. Yes, science has to start by assuming, and, yes, if those assumptions
are wrong, the whole project is in trouble. This is, indeed, showing the bias of the researcher when he chooses the paradigm.
One choice often made in social research is "nature or nurture?"
The psychologist, sociobiologist, botanist, and medical doctor are more likely to choose "nature" as the paradigm
to assume. In this, they are assuming that such physical characteristics as genetic make-up, out-side invasion of bacteria,
brain structure and function or other such tangible attributes are prominent in out-comes.
Those who choose "nurture" are more likely sociologists, family clinicians, criminologists, philosophers and
teachers. They assume that the conditions surrounding the phenomenon had the greater influence on outcomes; change the environment
and you change the outcomes.
Theories next flow from these paradigms. One theory flowing
from the nurture paradigm involves crime. It suggests that the "relative deprivation" of lower SES people in
striving for the American Dream help create the conditions conducive to crime; they may even "compel" some to crime.
It is obvious that the theory chosen needs to conform to the paradigm assumptions. If a researcher chose nurture, but then
used a nature theory, the research project is doomed to mess up.
Once a specific theoretical perspective is chosen, the
hypotheses for this particular study can be decided. Not only should the hypotheses be in harmony with the theory, but
it should also be stated as a "null hypotheses" in compliance with Popper's Proposition. If you believe there is
a difference in the income potential between males and females (his hypothesis), but you must try to disprove yourself, so
the null hypotheses states: "There is no difference between the income potential of college trained males and college
trained females working at the same job."
Please note that the hypothesis needs to be quite specific. The one just given needs to be further specified to make a
good study with reliable findings; the next step helps this to happen.
The actual research questions need to harmonize with the
hypothesis and be very specific (to help insure that we have reliability and validity). Doing a project only to find that
you asked the wrong questions is poor science.
The instrument we use, say a survey questionnaire, therefore must comply with our exacting research questions. The type
of scales used needs to reflect the type of data needed to really test the phenomenon. Likert scales, for an example, have
strengths and weaknesses; use this type of scale where it will best suit the specific question being asked. Carefully choose
open-ended questions for some types of inquiry and closed-ended, forced response questions for other types.
The findings, or the "bottom line" is what all this effort expended has been about. But now the analysis of
those findings is as critical as any of the other steps. Many researchers fall short at this point, perhaps after the long
trail feeling that this will take care of itself. But it will not. Ethical, pain-staking analysis is needed, not "lying
with statistics." It is also a tenant of science
that the research is not completed without the proper
presentation of findings to others---for this is how science is done: "brick upon brick."
8-The Eleven Stages of a Quantitative Research Project
1) Define a research topic
2) Intensify knowledge about the topic
3) Clarify concepts and their measurements
4) Select a data collection method
5) Consider the purpose, value, and ethics of the study
6) Operationalize concepts and design the data collection
instruments
7) Select a sample
8) Collect the data
9) Process the data
10) Analyze the data
11) Write up the results.
9-The written outcome of one type of Qualitative Research
What to look for when you read a field study:
1- Why did the researcher choose the particular setting?
2- Does the researcher begin with a refined set of hypotheses or a set of vague orienting questions?
3- What are different means of gaining access to a research site?
4- What is the researcher's degree of involvement or immersion in the setting?
5- What constitutes the data (the observations which the
researcher takes away from the field)?
6- How does the author begin to develop generalizations from the data?
7- How elaborate was the preplanning to determine how the
study would be carried out?
8- Can you anticipate ethical problems in doing this kind of research?
10- For most students, a good real-world job is the reason for attending college, and, more especially, the reason for
suffering through tough classes. Most tough classes seem to be labeled "required" and usually seem to be the least
interesting. But we have a good paying real-world job opportunity in this lecture:
Program Evaluations. Like many of the topics I choose for
lecture, evaluative research in the real world has some troubles for you to watch out for.
It is usually the case that evaluators are employed too late to save the company, program, or campaign. This may mean
you, as the messenger, deliver a deathblow of reality instead of the hoped for miracle cure. This may not greatly help your
popularity standings!
You may work for a governmental agency, like the State
Legislative Auditor, and have great pay and benefits, but lack in job security. This may not be popular with your spouse!
You may find that the people you are evaluating despise being evaluated. This may mean little or no cooperation! But you may
find the job fun and some-what "James Bondish" as you uncover "truth, justice, and the American Way."
There are, of course, varying levels of evaluative research. In one type of study, you measure the actual behavior of
the unit against its stated goals and objectives. This is easy to imagine with regards to the Driver's License Department
or some other government agency. You have probably spent agonizing hours waiting in lines and wading through red tape yet
you read on the
wall that these are public servants dedicated to serving you--the public. You might get a certain enjoyment out of work
that evaluates and changes these types of programs.
Yet another use of evaluative research, called structural
evaluation, does not concern itself with goals and objectives but in comparing units with other known programs. This type
of study employs such devices as organizational charts of differing programs to suggest changes. It works at comparisons of
the actual performances against the norms in like-programs.
Cost-benefit evaluations use an economy approach as they
compare increasing costs with possible increasing productivity, as well as decreasing costs and still trying to increase
productivity. Changes may or may not pay off as intended, and this type of research tries to predict future changes as well
as evaluate former changes with respect to efficiency.
Process evaluation, often employed in manufacturing, appeals to concerns about improvement of the various stages of production.
Its focus is to clearly describe what is going on (in contrast to what is thought to be going on) and how it can be improved
upon. This type of evaluative research is employed heavily in car manufacturing in Japan where a different management style
allows the floor worker to be part of the evaluation of his team's work.
Outcome evaluation is used to focus on benefits that need to be realized by the consumer of goods or services to keep
them satisfied. One important example of this research is the business community's viewpoint of the students graduating from
the University of Utah. It was found that the local business people who do the hiring do not perceive the U students are graduating
with real-world understanding---especially in research! One of the results of this study was to expand the undergraduates'
experience in doing research, so that the prospective employers were placated.
Impact evaluative research is broader than outcome evaluation. It involves the entire community's viewpoint; it considers
the opinions and attitudes of even those not directly benefited by the program under study. This is the world of PR and is
a major area that you may decide to work in.
Whatever level of evaluative research that is chosen, the
personal attributes of the researcher impacts these type of
studies more than regular research projects. Working with
people in an "auditor" way, in a way that is designed to
"criticize" their work or program, is usually a delicate matter.
The information obtained can be very important to an
organization, yet they shy away from employing an evaluator or from listening to the evaluation for some of the following
common complaints:
1) "the evaluation is of poor quality" (usually when the findings are not favorable to the employers mind)
2) "we were not involved enough or the out come would differ"
3) "your recommendations are arguable, too vague to use, more research must be needed"
4) "your language in the presentation was too difficult for all of us to understand, let alone to implement"
5) "you were in an unreal vacuum--you missed the political
and/or environmental concerns"
6) "you lack credibility"
7) "you cooperated too much with special interests"
8) "your evaluation comes too late to be of value"
To help improve the value of your evaluative study, try to:
1) Involve personnel that would do the actual implementation of your suggestions. Be careful that this does not unduly
bias your work--you are still the researcher, they should not taint findings
2) Present your findings to outsiders as well, especially those that would have an interest in the outcomes and influence
implementation
3) Get very timely information
4) Clearly admit the limitations of the study early on to reduce the number of possible objections
11-A Social Impact Assessment is a part of the larger
Environmental Impact Assessment process. Herein lies yet
another career possibility. There are 5 types of social impact that are studied:
1) Economic: changes in business activity, jobs, employment, personal income, and in the economic "base" of
the community.
2) Demographic: changes in population (not just local: regional) and in population characteristics (gender ratio, age
differential)
3) Fiscal: changes in public costs (ex: school districts tax base)
4) Community Service: changes in demand, distribution and
quality of public services
5) Social: changes in community organizations, perceptions,
lifestyles and life satisfaction. Especially changes in specific groups such as the elderly, minorities, or other sub
groups.
As law mandates these assessments, jobs in these areas can be found. Yet the same regulations require the study to be
completed in a given time frame, so use of existing data is
necessary (you don't have time to do a lot of planning of new research). This secondary data can be located as given in
the next table:
Data Source Information that is Available
U.S. Census Demographic data (age, gender, education, ethnic, migration patterns, SES distribution)
Bureau of Labor Employment, unemployment, income levels,
Statistics types of employment in the area
National Center for Vital stats on birth, deaths, disease and Health Statistics other health variables
FBI Crime data and rates by category
State Dept Birth & death data, morbidity rates, hospital
number of Health beds, nursing home beds, numbers of Drs, etc
State Employment Numbers employed by industry, projected
Agency levels of employment, skill shortages, jobs available
State Industrial Labor availability by city, type, and
Commission location of firms, housing availability
State Highway Conditions and miles of highways, streets
Dept and capital & maintenance costs
State Police or Number and type of vehicles, types of crimes
Dept Public Safety & numbers of police by city & judicial costs
State Recreation Number & type of parks, campgrounds, location
Dept and rates for parks, lakes, hunting, etc.
State Human Numbers on assistance, size of staff &
Services counselors, numbers on abuse, adoptions, etc
Survey research and observation research are also used in these assessments. As shown in your text, each of these methods
have strengths and each have weaknesses. Your expertise is in bringing the study together in a harmonious way using "triangulation"
of these various methods.
Qualitative designs help to describe existing behaviors,
attitudes, and identify needs of sub groups. As mentioned earlier in the course, many studies begin qualitative (to find
out the basics) and later develop into quantitative (after you have an idea of what questions to ask).
One key to qualitative research should be obvious: you must go where and when the phenomenon occurs. The meaning of the
behavior observed, within the context it is found, is the treasured data of this type of study. The general types of observation
are as follows:
1) structured observation: the purpose is to document specific behaviors as they occur. The length, frequency, and conditions
under which these behaviors occur are recorded for later analysis
2) participant observation: all behavior occurring is of interest as you surround yourself in the actual field experience.
This is time consuming but very rich in detail--you may discover things that can not come out in any other research design.
This type of fieldwork is usually overt (the research subjects know that you are there and have asked permission to do
research on them). But it can use some types of covert methods. With covert research, the subjects may see you in person,
but your intent of doing research is unknown to them.
Another method is to be around the subjects, but not be noticed by them. This might be in research done as an "ancillary
person" (such as a waiter who presumably does not listen in on the dinner conversation).
At some point in participant observation, you probably will want to include ethnographic interviews in which you question
"why?" as to the beliefs and behaviors of the subjects. This will allow you to ask what the subjects are feeling
or thinking in addition to what they are doing. Don't be shocked if their behavior does not align with their beliefs!
Qualitative designs are rich with detail and strong in internal validity as long as the observer bias is under reasonable
control. The main weakness is that you learn a lot about the particular setting and particular subjects, but they are not
necessarily generalizable to other situations. Ecological fallacy can be fallen into easily in this design.
12-"Variations on Experimental Designs."
It is now important to differentiate between "pure research" and assessments intended to effect outcomes. You
may find yourself sometimes a counselor (trying to intervene to affect beneficial change) and sometimes a scientist (working
hard NOT to have your presence influence, in any way, the outcomes).
This concept has been stated well in Techniques and Guidelines for Social Work Practice 1988, by Bradford W. Sheafor,
Charles R. Horejsi, and Gloria A. Horejsi on page 225:
"Assessment tools and instruments should not be confused
with research instruments. Instruments designed for research purposes may or may not be of use to the social work practitioner.
A good research tool is not necessarily a good practice tool.
"Some research instruments are too complex or too time consuming to administer in direct service situations. Also,
many research instruments yield little information that can be used in planning intervention. The instruments that the practitioner
finds most useful are those that are easily understood by the client and simple to administer. Moreover, the worker wants
an instrument that can identify specific problems and guide the intervention."
Their book then lists some assessment tools that they
recommend to the practitioner. Among them are:
1-The ABC Model and the Behavior Matrix, which has the stated purpose: "To achieve greater precision in observation
and the analysis of behavioral interactions." This is highly associated with participant observation research and Robert
F. Bales methods used in small group analysis.
2-- Eco-mapping: "To graphically depict family and interactional data as an aid in the social assessment process."
This concept is very closely associated with a qualitative research method called "Network Analysis."
3- Life History Grid: "To graphically depict the year-by-year unfolding of significant events in a client's life
and/or development of significant problems through time." Again, this resembles the qualitative research method called
"Life Event History."
4- Life Cycle Matrix: "To graphically depict the developmental stages of all persons in a household." Using
Life Event History with Network Analysis" approximates this tool.
The list they present goes on quite a bit further and there is a research method (usually qualitative) to match nearly
all of the clinical tools. Why is that? Pure research finds concepts that practitioners find useful in their clinics ---they
take the original research instrument, strip it down to the most useful parts, and then start applying it (making corrections
as they go) to real-life situations. The important point for you to understand is that, like the authors above stated; there
is an important difference in the proper use of research instruments and assessment tools.
|
