Monday, August 17, 2015

Cherry picking theories

Cherry picking data is not just done by scientific "denialists" but also some "respected" theorists who are seeking support for their scientific theory.

I recently realised that some experimentalists cherry pick theories to describe their experimental data. I heard a talk by a theorist who reported having several disturbing conversations along the following lines.

Experimentalist: We fitted our data to your theory.

Theorist: But the theory is not valid or relevant in the parameter regime of your experiment.

Experimentalist: We don't care. The theory fits the data.

5 comments:

  1. This is not cherry picking - this is "fitting apples into the shape of cherries".

    To me cherry picking implies choosing a dataset (model) because it fits the model (dataset) //among models (datasets) with equal attributes//. I.e. there is no reason to make the choice other than that it fits.

    If the model (dataset) is not applicable to the dataset (model), it's plain wrong.

    While cherry picking is bad (because it does not falsify the datasets or models that are not chosen! - and falsification is the only way to narrow down towards a "final" choice), choosing a model (dataset) that is demonstrably incompatible because of boundary conditions, is Wrong, as in "you Lie and that is Wrong". Capitalization on purpose!

    ReplyDelete
  2. Addition:
    "no reason to make the choice other than that it fits".
    This implies that there is a chance it is the correct choice. (So you're betting your reputation on probability.)

    The thing you describe has a chance equal to zero. So you're betting on a horse that is guaranteed to loose!

    ReplyDelete
  3. However, lacking any alternatives and with honest caveats, I think there is deserved interest in the success of a theory beyond its apparent domain of applicability. The independent electron model is an obvious example of this, and required the breakthrough machinery of Fermi liquid theory to be fully understood.

    ReplyDelete
    Replies
    1. Tim,

      Thanks for the helpful comment.

      You are giving the most generous interpretation of what some do.
      In the problematic cases I am thinking of experimentalists do not acknowledge
      -the existence of alternative theories
      -clearly state that the theory they are using is beyond the regime of the experiment.

      In the worst case they say "because the theory fits the data, we have proven the theory is correct".

      Delete
  4. Tim, I would say that depends on how the domain of applicability was defined.
    If a model (equation) is solely valid because it was derived using a boundary condition ("parameter a has to be larger than 0", and if it is not, the resulting equations are incorrect), then applying this equation to an experimental system where parameter a is demonstrably negative is mathematically wrong.
    Stretching a domain, and decreasing the quantitative agreement can have its benefits indeed.

    ReplyDelete