Anthony and I are developing some conceptual equipment around ‘biopreparedness’ in relation to Syn-BERC. One area where we are stumped relates directly to the recent posts on the National Response Framework, and I thought would make an interesting VSS discussion. The issue is about what kinds of thinking go into the design of scenario-based plans such as the NRF. As Dale noted, current DHS plans are oriented towards fifteen scenarios. The question is: what criteria are employed to develop the specific scenarios used in exercises? Dale wondered whether the number of scenarios (and apparently distinct response-assemblages for each scenario) meant a shift away from all-hazards planning. But assuming that all-hazards, or generic, planning is still the goal, how does one develop specific ‘scenario-events’ in order to test generic preparedness? I think the answer might be that, in the language of the document, “The focus of the scenarios is on response capabilities and needs, not threat-based prevention activities.” That is, specific events can be used to test generic capabilities, not responses to that event itself. Still, there is a fundamental question of which specific events should be used that remains a question of criteria. Second question: given that the development of scenario plans, and their exercise in practice, involves many distinct and often incomensurable expert-groups (public health, international security, bioscience, etc), how are truth-claims adjudicated (i.e. this type of response is ‘realistic’, etc.)? Any thoughts would be much appreciated–including in particular “ethnographic” insights from discussions with actual scenario planners.
COLLABORATORY: VITAL SYSTEMS SECURITY
The Vital Systems Security collaboration examines how, today, security is being constituted as an object of knowledge, intervention, and political reflection. It proposes that the security of vital systems such as energy, transportation, communication and health is one norm in relationship to which security is being reproblematized. A central goal of the collaboration is to examine these issues through collective, conceptually driven inquiry that addresses rapidly developing contemporary problems.
This is something Andy will probably have more to say about than I do. But I would say that you are probably misreading the “fifteen scenarios” and their function in preparedness planning. If you look at the documents carefully — in particular at the original preparedness plan from a few years ago — you will see that these fifteen scenarios are supposed to be the basis of simulations and exercises that simply test the *range* of capacities that are necessary for “preparedness” in the United States. There is not supposed to be any “predictive” meaning to them. So these are examples of using “enactments” to test generic response capacities, or system resilience. The point is not at all that they are only preparing for those fifteen events.
A similar point can be made about enactment in insurance. The point is not to predict the event at all, but to understand what the range of possible losses are, and to try to figure out whether the system of insurance can handle the likely worst case (what is called, if I remember, probable maximum loss in catastrophe insurance discussions).
There is much more to say about this, but that is my (initial) two cents. I am sure Andy has much to add about the “scenario planners” themselves.
I completely agree that the scenarios are not meant to be predictive. However, I think there is still a question of how one imagines an event capable of testing the “complete” range of capacities. And what kind of expertise is authorized to claim that a certain scenario–or range of scenarios–can provide such a test of “preparedness”. And on another level, there is a question of the details incorporated into a scenario. For example, in a summary of TOPOFF that I read, public health participants complained that the paramters of the scenario didn’t provide an opportunity for contact-tracing, and therefore it was ‘unrealistic’.
TOPOFF summary at:
[http://www.upmc-biosecurity.org/website/resources/publications/2001_orig-articles/2001-02-01-observationsfromtopoff.html]
Right. But the question was about all-hazards? So the issue is that you need some kind of tool to test an “all-hazards” capacity. So the standard for figuring out whether these are the “right” scenarios seems, as you suggest, to be whether they test the right capacities. Maybe we are saying the same thing.
In principle, scenarios are used to test the preparedness of an organization to deal with the event in question. Canonically, scenarios are developed by a diverse (multi-functional) team of experts who try to weight realism and probability, and weighting major and minor variables, assumed static and dynamic forces, and so on.
Canonically, these scenarios are then used to “wind tunnel” the ability of an organization to respond to the scenario. In the “wind tunnel” metaphor, the scenario is the “wind” and the organization is the “wing” or “car” being tested in the “wind tunnel.” A well-designed scenario will include minor elements that suggest implications for apparently “secondary” organizational factors.
The wind tunnel metaphor is also meant to suggest the structured and “labortatory” quality of the test one conducts — the scenario is meant as grist for a thought-experiment. The point of contrast here is with “war-gaming,” which typically is designed to test the interactions between different dynamic actors. A scenario of course can be a structuring condition for a war game.
It strikes me that in the case of disaster preparedness, wargaming more than windtunnelling is probably the more appropriate use of a scenario — wargaming under a scenaric condition would be a way to “prove” the incommensurability of different organizational elements within assemblages being proposed in an overhasty fashion.
I am not so sure about this. So in disaster response, the things being tested are: do first responders communicate with other emergency personnel? are clear command and control relationships set up? etc. In the “all-hazards” version of it, the point really isn’t the specificity of the event, but rather, as Lyle says, to test capacities. Nils: wouldn’t you say the same thing is often true in your world? Ie, you think about alternative scenarios as much to identify key inflexibilities in an organization, not necessarily because you think that the organization has to respond to the specific event being simulated. No?
To return to Lyle’s question: part of my response was connected to the fact that actually the “fifteen” scenarios have been there all along (at least since the original preparedness plan) and I think may have even been there in the initial planning for a DHS-like thing in the late 1990s (again, Andy has looked at this more recently than I).
Working with public health officials in NY who are planning, coordinating, and conducting such exercises, I have never heard a discussion about the “realistic” qualities of the scenarios. It seems not to be an immediate concern for them. They choose according to specific criteria what kind of exercises they are conducting. The scenarios are of course adapted so that they fit the specifics of the locations. There is much more discussion about HOW to conduct a successful exercise. It also seems that the success of the exercise is not determined in relation to the realistic qualities of the scenario enacted.
I would also say that the exercises are not primarily about “testing capabilities”. This assumes that the capabilities already exist …
if there is more discussion around how to conduct the exercise, is there any insight into how the criteria are agreed upon (and by who, through what kinds of claim)?
if the criteria are not necessarily chosen relative to an already existing set of capabilities / situation and are specific enough for the “kind of exercise” being run, then how are criteria for the scenario agreed upon that will be general enough to be useful as a preparedness tool?
An importantcriteria for the practitioners seems to be that these exercises need to be designed as a “learning experience”. A good exercise should run smoothly, but not too smoothly (whatever the particular scenario). There should be a sense of accomplishment and purpose among participants, but also a sense that “much more needs to be done”. It seems that what counts more than realism is the effect of such exercises. They are, among other things, about people.
I should probably add that I have more experience with “minor” exercises that are conducted locally, on the community level, in hospitals, schools, nursing homes, etc. Things may look very different when you’re looking at things like TOPOFF. The latter are probably much more “politicized” and generate particular kinds of debates among particular kinds of publics (all worth to attend to in terms of their “function”). In terms of numbers, however, the “minor” exercises are not minor at all. The focus is, as Stephen already said, on capabilities, but these capabilities do not always already exist before the exercise. Finally, of course you want to train something like “decon” in a hospital but certainly not in a nursing home or a school. So there are possible capabilities which need to be realized and there are virtual capabilities which need to be actualized. The former are specific and the latter are generic.
Let me second a few points that have been made so far. First, that there are important differences between different kinds of exercises, in scale as well as in the organization that is running it. These have different tasks, and different ways of assessing success. One might compare, for example, TOPOFF (run by DHS) with smaller scale coast guard exercises run by contractors such as SRA International, with local public health preparedness exercises run by CDC in partnership with RAND. In the latter case, I have definitely heard a lot of people saying that the exercises are about creating “lines of communication” between people who normally don’t talk to one another – eg. the fire dept chief and the public health director. Another task is to generate knowledge about needs: so one discovers that one needs a plan for distributing vaccines to the elderly. Or that there is a plan, but no one knows what it is. I think the question of authorization of claims is an interesting one: how would one know if one is prepared or not? An initial task is to agree about shared standards of preparedness, which is a process now taking place.
As for the question of how one decides which event to simulate, again, I there are differences – but here is a very pragmatic articulation – excerpted from a recent interview I did with a think tank expert who has been leading efforts to develop preparedness metrics for local public health jurisdictions.
Q: Can you say a little bit about how scenario-based exercises were brought in as a as a method for doing assessment and training in public health preparedness?
A: When we started this work, which was 2002, no one just had any idea what preparedness was, how to asses it, how to measure it, how to practice it, how to do any of these things. And, you know, we started out by looking at all the instruments that were out there and trying to figure out, oh what are the essential components? You know, how to kind of be the mini experts to do this. Because as you say [these events] almost never happen. They’re rare events. We had to think of some way to sort of say, ‘well what if?’ and simulate that, and so we decided, you know, based on exercises that are done in the military or exercises that are done sometimes in business for strategic planning. To try to adapt that approach, really to public health. Now since that time… we thought a lot about, and are real proponents of what we now call embedded exercises. Which, just to make, you know, the terminology, to make others happy. And, you know, those are really…, in daily life, in routine public health practice things happen that don’t rise to an incident of national significance, but give you an opportunity to test your plan. So when this clown interviewed me for Anderson Cooper the other night, the first thing I said to him about XDR TB is, nature has just given us a wonderful opportunity to test how well we would perform these functions on a pandemic. Now of course he didn’t use that quote, but that’s okay, that’s what I think. And those kinds of events happen a lot, spinach with e. coli, salmonella peanut butter… a car and truck exploding, a tornado wiping out a hospital in Georgia, all those things are, they are real events.
Q: Aside from actual events, when you’re deciding which simulated events to use, how do you decide which events to model?
A: Well, what we have done is we have said, what aspects of preparedness do we want to test?
Q: OK.
A: And, you know, I don’t believe you can test them all at once. Especially if you want to come out with something that you can measure and figure out if you can approve, and I think this is a huge flaw in the every stage of the pandemic plan is tested once a year, or lets have one of these TOPOFF type of events, which I personally think are ludicrous. I mean, I think that they’re expensive and disruptive and you don’t maximize learning, you can’t maximize learning from those. And the other thing that happens is a lot of people just say, oh you’re right we can’t do all this and they shut down. So, but anyway, so we figure out what is it that we want the participants to learn about or exercise. And then we design a scenario around that, now we did early on, kind of map, in terms of diseases or whatever, we sort of said, ‘okay, here are all the different kinds of things that you have to do in a public health emergency.’ You know, you have to do [survey of all the techs] and blah blah blah, you might have to do isolation, quarantine, vaccination, prophylaxis… and we did this huge matrix and then we said, so, and this was pretty early on, what diseases and disease-based scenarios tested different functions, and what we saw was that almost all of them are tested by any disease. A distinction you would make between vaccination and prophylaxis and a distinction you would make in terms of simple isolation to quarantine and stuff depending on how contagious and how fast. And whether there’s counter measures. But other than a radio-logic event, there’s a huge core, at least of what public health does that gets tested by any disease. So that the issue’s not really what disease do you build it around – you want to build it around something that’s plausible and the people, and that that could be something that could epidemiologically happen.
Great stuff. My two cents on top of all the others is as follows: First, I don’t think anyone ever suggested that because the NRF focuses on 15 scenarios, that those are the only ‘types’ of situations being exercised per se. It does strike me that, at some level, the logic being espoused at DHS and in parts of a wider disaster management/preparedness arena is that these 15 scenarios pretty much cover a) the range of (categories of) disaster and crisis that the US both is concerned about and needs to be prepared for; and b) the range of response capability that experts and officials want to find out about and improve. There is no question, as Andy’s data and other comments illustrate, that this is not, generally speaking, a disease-specific or threat-specific enterprise. We’ve known that for ages. But I would point out that, as several exercises and not a few commentators have pointed out (myself included), the argument that exercising for anthrax (or whatever) produces knowledge that is wholly relevant or applicable across the disease spectrum is dangerous thinking itself. Despite a heavy pitch job by senior officials to say that it would feed into a larger preparedness effort (at the time: bioterrorism preparedness), the Smallpox Vaccination Program generated only a limited variety and quantity of useful knowledge for BT preparedness more generally (it did, however, illustrate, many tips on what not to do in designing and implementing a vaccination program).. Because smallpox, like every disease, spreads, hurts, and kills individuals and populations differently by virtue of a whole host of biological (and other) characteristics, and because there is a very effective vaccine currently available to treat every individual in the United States for it, and because of a hundred other reasons, the knowledge gleaned from a smallpox exercise must be thoroughly examined, thought about, critiqued and scrutinized before a planner can say “aha we know that our weakness lies here in responding to an infectious disease outbreak in our county.” So again, to say the same thing too many times now, Lyle’s original point is still valid: it is an empirical question the extent to which and how these scenarios produce all-hazards knowledge. The key word here in the sentence from a critical social science standpoint is “how.” Under what conditions? With what authority? With what techniques?
Do I have time for a few more points? I’ll give it a go. I have always found TOPOFF a fascinating series of events. Debate amongst professionals in the biosecurity/preparedness arenas has been pretty vigorous, as many know. My read of TOPOFF is that it has served not only as a mechanism to generate an understanding of weaknesses and strengths in capacity and capability, but also as a very successful catalyst for political interest and activity. In other words, it has served a number of functions, both technical and political. And this is perhaps underappreciated in our current discussions so far of exercises and scenarios. One might hypothesize, just to push discussion a bit, that the larger national exercises are more effective, or are in fact designed precisely to engage the political process (and interest groups, the executive, legislators, etc.) in ways that local exercises generally do not. To the extent that preparedness is an object of politics (by this I do not mean a product of it), scenarios and exercises, especially the larger scale ones, seem to be an integral component or driver. The debates of TOPOFF are illuminating in this regard, esp. around the generally catastrophic assumptions utilized. What do extreme assumptions *do*? What are their effects in producing new technical problems, but also in the realm of the political? What do they mobilize, and how?
Great comment. So what kinds of experts tend to underline the difficulties of applying knowledge across a wider spectrum? It might be helpful if we could try to differentiate different types of biosecurity/preparedness expertise. This will get us back to Lyle’s original question.
I completely agree with what Dale says about TOPOFF. These exercises are designed to create certain kinds of publics.
Finally: where is the place of anthropology here? It seems that we are not inclined to value generic knowledge out of professional habit. What works in Indonesia does not work in China, etc. What works for anthrax does not work for smallpox, etc. This is a familiar critical strategy in anthropology. It is an important one, of course, and very valuable but I would suggest that we need to push ourselves a bit. On the one hand we can analyze analogical thinking as a social practice. What kind of analogies do our informants make and what kind of analogies do they not make? But what of our own analogies? I.e. what kind of analogies do we want to make? And to what purpose? Foucault has never written anything about bioterrorism or preparedness and he is nonetheless very helpful…
I agree this sentiment, Carlo. Sometimes things that work in Indonesia *do* work in China and *do* work in the United States (capitalism, industrial mass production, etc. seem to do pretty well “cross culturally”). Might we not be equally interested in the conditions of possibility for that “mobility” as much as we are interested in those cases where things aren’t mobile? Maybe “critique” and “inquiry” means charting these mobilities and their limits rather than presuming that the critical (or anthropological) position is on one side or the other?
Exactly!
Agreement! Everyone mark the date on your calendar. It might not happen for another 12 years.