Rituals of verification: consultants, second-rate evidence, and policy

There is growing consternation in Canada as the extent of major consultants’ involvement in federal policy-making is revealed. Firms such as McKinsey seem to have taken over the public service.

This is symptomatic of wider changes that have been described by Michael Power as rituals of verification, i.e. an audit culture that feeds upon itself, mistaking performative risk management (in this case seeking the reassurance and plausibility of big-name consultants, irrespective of their competence and adequacy to the task) for substantive outcomes.

Anecdotes from my own experience

In my experience, consultants have by-and-large taken over from publically funded academic researchers, replacing them as the principal source of in-depth background evidence for policy-making in the field urban and regional development. They provide rapid, but second-rate, evidence for ‘evidence-based’ policies.

The 1990s and 2000s

From 1998 to 2013 I worked at the INRS, a research institute whose principal role was to perform analysis and research for municipal, provincial and federal government bodies.

Between 1998 and about 2010 I undertook many contracts, providing original background research and analysis that civil servants (and their political masters) used as input for decision-making.

Typically a civil servant, with experiencre and knowledge of urban and regional development (the area I performed most of my consulting in), would approach me with a question. This would usually require a state-of-the art literature review and some empirical research focussed on the specific topic or region under study. The civil servant would, again typically, approach 2 or 3 Canadian or Québec researchers close to the topic, receive submissions (i.e. proposals with detailed budgets – usually two or three pages), and allocate the contract.

Because this was original background research, deliverables would be due within 6 to 12 months, sometimes even 24 months for major projects.

Late 2000s and 2010s

In the late 2000s things began to change. The senior civil servants with whom I had worked retired, and a generation of younger civil-servants, trained in management schools, were recruited. These new civil-servants knew little or nothing about regional and urban development, but were specialists in management and audit culture (or rituals of verification as Michael Powers eloquently puts it).

New procurement rules were introduced. No longer could a civil servant discuss the research they required with me. All contracts were advertised on an anonymous centralised platform, which required constant scanning to identify contracts.

Submissions became complex and onerous : it became necessary to have an assistant, able to produce tenders that contained the correct buzzwords and all required business information about the contracting consultant. To get a single contract, one needed to apply for ten – and then be prepared to work on whatever contract one happened to obtain…

Deadlines were shortened – reports became required in 3 months rather than 6 to 12.

In short, it became very difficult for academic researchers such as myself, despite my verifiable expertise in the fields of regional and urban development, to be considered for government contracts.

Therefore, the production of background research (at least in my field) shifted from academic consultants towards major consulting firms, able to navigate complex submission requirements, deliver reports in three months, and satisfy increasingly labyrinthine accounting norms.

Do major consultants do a good job?

No. Not when it comes to in-depth background research (and I write as someone who performed consulting research in the early 1990s).

They are able to compile existing research, and maybe give advice on policy implementation, but they simply do not have the time to perform proper original background work. Billable hours do not allow for thinking, analyzing original data, or analyzing interviews in-depth.

On at least two occasions I have been privy to reports I had been interested in working on (these reports had been prepared by major consultants).

• In both cases these ‘reports’ were extended powerpoints.
• In both cases my work was extensively cited (after all, these powerpoints were in my field of expertise).
• In both cases the powerpoints cited my older research, not necessarily invalid, but not up-to-date.
• In both cases there was almost no original empirical research (in one there was a quick and dirty survey) or critical thinking.

What do consultants deliver to our governments?

From my (limited and anecdotal) experience, governments get what they pay for.

Results are required very fast, so consultants’ research is shoddy, not original, but delivered on time.

Powerpoints are shiny, buzzword-heavy, present re-heated results, all at a cost far exceeding what university professors would charge (the bulk of whose salary is covered by the university).

Consultants tend to say what their client wants to hear: questioning a client is bad for repeat business.

Rituals of verification: why second-rate research has triumphed

As more and more civil servants are recruited from business schools, the ethos of the civil service has shifted from ‘public service’ to ‘efficiency’ and ‘metrics’. There has been a hollowing out of expertise : given the prevailing ignorance of many civil-servants in charge of giving out contracts, using major consultants is an exercise in risk reduction. McKinsey, and other large consultants, provide a seal of authority that no minor professor could: the civil-servant will not be criticised for turning to such authority (even if they personally suspect the consultants’ competence).

Layer upon layer of audit, verification, and performative risk management are piled upon decisions, drowning desirable outcomes under a sea of appearances. This process, described by Michael Power in his 2008 book Organized Uncertainty, explains why it is objectively safer for civil-servants to knowingly procure bad research from large and reputable consulting firms, than risk obtaining original research from unknown (to their bosses) specialists employed in Canadian universities.

The consequence? ‘Evidence-based’ policies are premised on second-rate evidence.