Cooperative Evaluation

What is the role of an evaluator? Said differently, what is the goal of program evaluation? On complex projects and programs, certainly the role is not to sit, white-coated, judging programmers and curricula with the final word on if things “worked” or not. To judge without knowledge of best practices, or keep recommendations private is to betray the very expertise that validated evaluation as a necessary component of any new policy or program.

Proponents of strict, cold, and static evaluation argue that providing continuous feedback to programmers will affect the results of the program.  This, they say, threaten the internal validity of the evaluation.

To this argument I say “so be it”.  Final judgment will always rest with the evaluation team, and, if properly insulated from outside influences, can remain fair and scientific.  Giving updates and feedback to programmers can, in this way, be framed as both detached and cooperative, discerning and helpful.

What if the application of the program is improved due to the newly acquired information? Does this evince a breach of protocol by the evaluator? No, it does not. While a feedback loop between external evaluators and programmers is acceptable, a short circuit is not.  This is to say that the direct involvement by the evaluator in program application should not be condoned.  For example, coaching the answers of respondents in a written survey, or otherwise moving into a programming role would clearly affect results.

So how can a cooperative evaluation framework thrive in a climate of complex programming, funding renewal applications, and friendships forged along the way? To answer this question a distinction must be drawn between academic and managerial recommendations.  The former should be encouraged, while the later should be reserved for programmers alone.  An academic recommendation is based on scientific findings, formalized and submitted to programmers.  Differently, managerial recommendations could involve friendly reminders, disciplinary warnings, and any changes to programming based on efficiency or economic concerns; they are not necessarily based on research or best practices.

While the evaluator should not ‘care’ if a program succeeds or not, they should be encouraged to help programmers at each turn.  This may come in the simple form of providing ongoing feedback as data because available and the necessary analyses are completed.  This is to say, no value should be placed by evaluators in the success or failure of a program, only if data was collected in a timely fashion, analyzed properly, and submitted with prudence.

To work outside of the cooperative evaluation framework invites hostile feelings on both sides of such studies, breeding resentment.  Evaluation has become so ingrained in social policy and programming, why should it be seen as an obstacle of hurdle, rather than a team of collaborators with specialized knowledge? The time has come to work together to achieve best practices.

Sociological and Socio-legal Research In The Public Domain

It can reasonably be assumed that science holds a place in public policy.  Indeed, sound laws, programs, and initiatives should be based on some amount of theory, fact, or empirical research.  This is not to undercut the other influences that are often considered, namely economic concerns.  The purpose of this post is to strengthen the connection between researchers, or the producers of knowledge, and those that aim to apply this knowledge for the public good.

Despite the fine balance often sought between practical and scientific influences, the introduction of science and academic research into policy creation is, by definition, problematic.

When one talks about academic research, they are drawing a metaphorical line in the sand between us, the academy, and them, general public who are ostensibly not capable of understanding or engaging with our material.  Our material.  This sense of ownership is what allows the bastardization of quality research in preparation for its introduction to the political economy.  Moreover, if politicians do not understand or care to learn the basic methodological principles on which academic research is based, they are free to interpret findings how they see fit, often to suit their own ends.  This, in turn, gets relayed to the public through policy and election rhetoric, among other ways.

So how does excellent research get disseminated for practical application without marring the values on which it is based? Perhaps public education could help in this regard. While unlikely, this sole solution would have a perceptible effect; providing the public with the appropriate material may allow us to be more discerning regarding how we consume media, and more critical as we head to the voting booths.

Perhaps the answer also lies the research itself.  From all fields, a paradigmatic shift could guard the scientific principles which are held so dear, while at the same time infiltrating the decisional matrix of policy makers.  Currently, universities, colleges and professional schools are branded as places to receive knowledge.  They are centres where, if one so desires, they can go to better themselves and become an expert in a variety of fields.  This individualization of education is not helpful; it only appeals to the sense of self-worth of each student, without mention of a collective good, or pushing the boundaries of what is known of the world around us.

Should universities be rebranded? Perhaps they should come to be known first as centres of learning (writ large), places not only where students come to learn to contribute to the university’s body of knowledge, but where government officials come for information on policies.

The job of researchers and students will remain constant; produce quality research they can proudly stand behind. A collective effort to make this research available and useful would slowly fade the divide between academics and policy makers, fostering cooperation and, most importantly, creating a sound foundation upon which new policy can be created.

Paid Duty: Refocusing the Debate

Many uniformed police officers take advantage of the Toronto Police Service’s paid duty program, guarding construction sites, LCBO’s, and other private events and businesses. Critics of this program argue that these officers are public employees and should not also be privately employed.  Moreover, the air of legitimacy given to police officers is inappropriately applied to private endeavours when it comes to paid duty.

Supporters of paid duty argue that more officers on the street, whether on active or static patrol, benefit the public; more officers means a greater chance for the apprehension of criminals.  More reaching supporters argue that static patrol deters would-be criminals.  To support these arguments, proponents of paid duty point to cases where officers guarding construction sites have observed and subsequently apprehended shoplifters, speeding drivers and the like, simply by being in the right place at the right time.

However, to use such examples devolves the discussion on paid duty into the presentation of anecdotal evidence from supporters, rather than a philosophical argument for why paid duty is a good thing.  On the other side of the debate, critics often use misguided fiscal arguments that seem to emanate from veiled jealousy.  Instead of these irrelevant and post-hoc examples of why paid duty does a (dis)service to the public, the debate should be refocused and seen through the lens of the security privatization and the fading borders of public and private space.

The question should not be “is paid duty working?” or “what are the benefits of paid duty?” but instead “should sworn officers be allowed to guard private spaces?”.  After resolving this fundamental question, the debate can then move into the logistical and practical realm, where results-based conversations are more appropriate.  Without theoretical basis, both critics and supports cannot lay their arguments on a solid foundation.  Said differently, no matter how many crimes officers on paid duty stop, if their sheer presence in these spaces cannot be justified, the result is irrelevant.  The ends do not justify the means.

Peltzman goes to Starbucks

A random thought about starbucks. Why do they call their drinks “short”, “tall”, “grande”, “venti” and “trenta”? Why not extra small, small, medium, large and extra large?

Perhaps it has to do with the bell curve, in a round-a-bout way. Ask someone to pick a number between 1 and 10, and they will, on average, pick 5. Ask them to pick between 1 and 20, 10 will be the mean. What does this have to do with starbucks drinks? With little value placed on the current sizing monikors, customers are likely to land right in the middle – grande.

Now think about the value of the second set of options, from small to extra large. The relative value is more obvious than the difference between a venti and a trenta. So if you went to McDonalds and ordered an extra-large drink, you know that you are selecting the biggest beverage possible – you must make the concious decision that indeed, you desire the most amount of sugar that is possibly allotted by the food establishment you entered. Is the stigma attached to venti or trenta the same? I would argue that it’s not.

It’s a genius play by Starbucks. They’ve added in a new order option, theoretically raising the mean order size (think of the 1-10/1-20 thought experiment above). To counteract the possible stigma, and reverse of the above increase in mean order size, they’ve opted to make the names of their beverage sizes less obvious. This encourages customers to place orders relative to the number of drink options, and not the names attached to them.

Another thought experiment. Lets talk about likert scales. You could have a balanced 3 point scale – disagree, neutral, and agree. Or an asymetrical 4 point scale; disagree, neutral, agree, strongly agree. On a series of statements, what do you think the average answer would be? I would guess, since there are more options in the second set, answers would be more positive.

Furthermore, though 3 of the possible answers are identical in each set, the underlying meaning of, for example, neutral, is only obtained through the relative placement of other possible answers.

Can you think of any other examples of the above principle?

ChatBout Poll #1

Ubiquitous Repetitious Application

 

In Exit Through The Gift Shop, street artists explain the rise of an alternative art movement in London and LA.  In brief, a documentarian-turned-artist begins plastering colourful depictions of recognizable celebrities around town.  Spray-painted Maralyn Monroe’s and the like.  When asked to comment on this new artist, Banksy, a mainstay of the London scene, described his sentiments in terms of repetition.  To paraphrase, he notes that Andy Warhol painted iconic people over and over until the concept of celebrity was torn down.  In contrast, the former filmmaker, now known as Mr. Brainwash, painted repeating images until his art was meaningless in itself.

 

I’m not an artist, I’m a criminologist.Can the importance of a theory be diminished through repetition? Surely most people would agree that the meaning of a theory could become warped as it passed from person to person and context to context, but what of the importance?Take, for example, Michel Foucault. His work on total institutions and panopticism are, by any account, extremely influential.  They cut to the core not only of academic movements at the time of writing, but ways of thinking that are evident in today’s political climate.  However, a distinction must be made between a proper application of the theory, and superfluous inclusion.The proper application of a theory could conceivably go as follows.  One does some reading on the blurring of public and private space. With this background knowledge, they proceed to observe their surroundings, and test these situations against the theory and readings they have already done. In this way, modern day examples sometimes, and sometimes do not, line up with theories from a time past.Superfluous application of theory perverts this process.  All too often a theory is blanketed over a widely defined phenomenon, resulting in some tenuous applications.  Stated simply, theory is applied to circumstance.  This stands in contrast to circumstance being tested against theory.  In the former, the resulting discrepancies are either evidence of a theory’s shortfalls, or, more dangerously, an opportunity to extend theory beyond the confines in which they typically reside.Extending an existing theory weakens the basis on which it was formed.  For a concrete example, consider Foucault’s panopticism.  This theory posits a certain stance on control and surveillance that often gets applied to systems like CCTV.  While there are many similarities, the differences between the two are important as well.Some of the problem comes from the pedestal on which writers like Foucault are placed.  Universities devote entire classes to them.  Academics brag they are a Foucaultian, and tout his writings as having the ability to explain everything.  This type of language discourages young minds from questioning Foucault and criticizing the application of such theory in modern society.  After all, it’s hard enough understanding The History of Sexuality, let alone critique it, right? One might be apt to think, if Foucault was that influential, maybe I’m just not looking hard enough to find the correct modern applications.Enter: stagnation.Inquiring minds should familiarize themselves with these works, but not learn to lean on them like a crutch.  Foucault and others are not necessary bibliographic entries, but can be helpful in explaining the work around us…sometimes. To operate from a different standpoint is to diminish the value of theory.  I repeat; it diminishes the value of theory.  Not one theory in particular, but theory as an explanation for a specific process, phenomenon or observation.  If theory becomes a blanket logic for everything, its power is lost.