I suspect Almaviva is talking about the biases inherent to publication / talking about results.
Consider, by analogy, the event "rolling a six sided die and getting a 6 and announcing that fact to the world".
"What is the probability that random events could produce a result that large?": one in six, per die roll. The question excludes the whole "announce it to the world" filter.
"What is the probability that these results [getting a six and announcing it] occurred by chance, rather than being a signal?": We have no idea. If the person announced "I'm going to roll one die and announce the results, regardless of the outcome", then it's one in six. If they kept rolling dice until they got a six, then the probability is 1. If they rolled 3 dice, then the probability is 91/216.
The point is that the scientific method has all sorts of biases (publication bias, confirmation bias, etc.) and p-values are rarely "probability that the result is wrong".
Consider, by analogy, the event "rolling a six sided die and getting a 6 and announcing that fact to the world".
"What is the probability that random events could produce a result that large?": one in six, per die roll. The question excludes the whole "announce it to the world" filter.
"What is the probability that these results [getting a six and announcing it] occurred by chance, rather than being a signal?": We have no idea. If the person announced "I'm going to roll one die and announce the results, regardless of the outcome", then it's one in six. If they kept rolling dice until they got a six, then the probability is 1. If they rolled 3 dice, then the probability is 91/216.
The point is that the scientific method has all sorts of biases (publication bias, confirmation bias, etc.) and p-values are rarely "probability that the result is wrong".