How to test rationality? The case of argument evaluation

Books and Papers
41

With grouping the 4 ordinal categories collapse into 2.

The 4 categories and using the median create scenarios like 1.5, which can be problematic. If I were to interpret 1.5, it’s between “weak” and “very weak” and the choices given to you are 1, 2, 3, 4, but you know for certain 1.5 is weak.

Statistics is complicated. Somebody already brought up \beta weighting. I would’ve preferred mode as it wouldn’t have “correct answers” like 1.5.

Such objections are bound to arise and I’m almost certain CART designers have appropriate responses.

42

but I told you mode could and gave you an example. You could still have 2.5 with mode and 8 ratings. The problem isn’t the median.

43

Possibly, but wouldn’t it be nice if there weren’t scores like 1.5 or 2.5?

44

No as I wouldn’t be able to infer that the experts rated differently.

But more seriously, yes indeed but for that it’s not mode or median, it’s having 7 or 9 or any odd number of experts.

45

That’s correct, but the mode is a kind of rating. It tells you the most popular category.

Good of you to mention the oddness/evenness of the board of experts. We can always fix that, but will that suffice?

46

If I understand your “mode grouping” correctly then yes just having an odd number of ratings is enough for the median or your mode to return a whole number.

47

Yes, the Devil is in the details. You’ll have to work it out. I’m a bit tired. :tired_face:

I discussed this with MS Copilot. A good conversation. Allegedly psychologists were baffled by why intelligent people do stupid things.

1 Like
48

It doesn’t support the idea that the scores were taken from the median raw scores of 1-5. First they did a regression analysis for the experts’ evaluation of the arguments, then they use beta weights to assign the score 1-5. The median here is not the median of 1-5.

49

How do you read “for the purposes of regression analyses, we used the median” and understand that the median is obtained from a regression analysis?

You know you can read the paper right? from the paper:

As an example, on the above item, the median of the experts’
ratings of the rebuttal was 1.5 (between weak and very weak). The
mean rating that the participants gave the item was 1.93 (weak),
although the participants’ mean prior belief score indicated a
neutral opinion (2.64).

There is the appendix A where it’s explicitly stated as being the median of experts’ rating. Unless you think ‘median’ doesn’t mean ‘median’.

50

I give up. I don’t know what to think about the scoring. Sometimes we just have to sit back and enjoy the smell of coffee.