I’m currently working on a paper that I’m presenting at the Philosophy of Science Association biennial meeting in Chicago next month. While I’m making slow progress on the paper, I’ve discovered a couple of examples of practical reasoning about evidence that you might find interesting.
The paper itself deals with a (pretty hoary) philosophical problem known as the reference class problem. Briefly, this describes a difficulty about inferring the probability of individual events by relating those individual events to a group of similar events, or a reference class. That’s definitely in philosophese, so perhaps a nice example would make things clearer. My favourite owes to Connor Cummings (who wrote an excellent BSc dissertation on reference classes), and is about house fires. Say that I want to estimate the probability of my house burning down during the next year. Which statistics should I look at? Well, I could look at those statistics that describe the number of of houses that burn down each year in the UK as a whole. Or, perhaps, I could look at statistics that deal with houses that are built from bricks? Alternatively, perhaps I should look for figures that describe the chances of houses with blue front doors being consumed by flames? Or (even) statistics for brick-built houses in London that have blue front doors?
Each of these statistics are likely to provide different probability estimates. This means that – depending on our choice of reference class – we will come up with very different estimates of the probability of my house burning down next year. This is just the kind of thing that might make an insurance agent very unhappy. Worse, though, is to come: given that each might give different estimates of our individual probability, which should we prefer? None of them are straightforwardly wrong, because they all describe groups that are in some respects similar to my actual house. Assuming that we could generate reliable statistics for each one, choice between them seems to be a matter of subjective preference. In other words, there doesn’t seem to be an objectively correct choice of reference class.
This is the reference class problem, and it has engaged philosophers of science for at least 65 years (Reichenbach 1949). My aim while putting together my PSA paper, though, is not to try and formulate some novel solution to the problem, but instead to talk about some of the solutions to this problem that have been employed in scientific practice. I was very interested to learn that a recent piece of guidance from NICE’s had suggested that different prescription practices should be adopted for hypertension sufferers of different ages, and from different ethnic backgrounds. The usual first-line treatment for high blood pressure in people under 55 would be an ACEI or ARB:
1.6.6 Offer people aged under 55 years step 1 antihypertensive treatment with an angiotensin-converting enzyme (ACE) inhibitor or a low-cost angiotensin-II receptor blocker (ARB)…(NICE 2011: 17)
However, prescription practices should vary because of both age and ethnicity:
1.6.8 Offer step 1 antihypertensive treatment with a calcium-channel blocker (CCB) to people aged over 55 years and to black people of African or Caribbean family origin of any age. If a CCB is not suitable, for example because of oedema or intolerance, or if there is evidence of heart failure or a high risk of heart failure, offer a thiazide-like diuretic. (NICE 2011: 17)
I’d like to suggest that this difference in recommended prescribing practices is some interesting reference class work on the part of NICE. However, it seems hard to align this kind of thinking with the more philosophical approaches to the reference class problem that I know. Here, I’m largely thinking of Salmon’s (1971) suggestion that we should prefer homogeneous reference classes of one kind or another. However, we know that neither age nor ethnicity form homogeneous reference classes. Yet (as far as NICE is concerned) these groups are intended to behave like homogeneous reference classes, in that a) they are intended to give unequivocal guidance as to the reference class membership of an individual and b) in that membership of one of these reference classes changes individual probability estimates. So what are the grounds for this clinical guidance confidently picking out these groups?
While looking for possible solutions to this difficulty, which I’ll have to leave hanging for the time being, I ran into some very interesting work on the reference class problem in the law. That I had no idea that the reference class problem was something that lawyers argued about is probably more an indicator of my ignorance of the law than anything else, but I was surprised to find several different ways of resolving (or, at least, giving ways of arguing about) reference class difficulties in legal practice. One excellent introduction is the paper by Cheng (2009) in the Columbia Law Review. This also contains a brilliant example of the reference class problem as applied to international drug smuggling, which alone is worth reading the paper for. Anyway, the substances of Cheng’s argument is that inference based on reference classes is structurally very similar to regression analysis. This means, I think, that the reference class problem can be regarded as a special case of the model selection problem. In turn, this means that we can employ established techniques, developed to deal with the problem of model selection, to pick between different reference classes in a principled way. While the details of these techniques – the main one discussed in Cheng’s paper is Akaike’s Information Criterion (AIC) – are not something that I’m not terribly familiar with, this approach does appear to offer the advantage of providing practitioners (legal, in this case) the advantage of at least being able to pick between different reference classes in a consistent manner. I wonder if something similar might be developed for the medical context…
Cheng, EK. 2009. A Practical Solution to the Reference Class Problem. Columbia Law Review. 109(8): 2081-2105.
Hájek, A. 2007. The reference class problem is your problem too. Synthese, 156(3): 563-585.
NICE (2011). CG127: Hypertension: clinical management of primary hypertension in adults. National Institute for Health and Clinical Excellence, London. Available from: http://www.nice.org.uk/guidance/cg127/resources/guidance-hypertension-pdf
Reichenbach, H. 1949. A Theory of Probability. Berkeley University Press
Salmon, W. 1971. Statistical Explanation. In Salmon, W. (Ed.), Statistical Explanation and Statistical Relevance. University of Pittsburgh Press