I am having a little bit of trouble understanding the concept of p-values. On the one hand, we are saying that the null hypothesis is a hypothesis we set, and the alternative hypothesis is something we would like to prove.
In that case, how does it make sense to say that the p-value is the probability of the alternative hypothesis being true given that the null hypothesis is true if from the above definition it seems that both are mutually exclusive?
you are correct that the null hypothesis $H_0$ must be set (fixed, chosen) before sampling. You should write it down early in the process. Only then do you get a sample, and look at your random sample. Then, the p-value of your test is the probability of seeing what you see, under the assumption that $H_0$ is true.
For example, I have a bag of 100 marbles. Each one is either red or green. Write down:
$$H_0:\,\,\text{exactly half of the marbles are red.}$$
$$H_a:\,\,\text{more than half of the marbles are red.}$$
Then sample with replacement. (with replacement so the calculation is easier.) Why not $n=3$. Resulting sample: RED, RED, RED. This is what we see, what we observe. The probability of seeing this (if $H_0$ is true and if the sampling is really random) is $0.125.$ So the p-value of this test for this sample is the number $0.125$.
Note $0.125$ is not $P(\text{alternative hypothesis is true})$, and it is not so clear that $P(\text{alternative hypothesis is true})$ has any meaning.
But $0.125$ is a number that summarizes the evidence from our sample. In fact, the smaller the number, the more we are open to the idea that the alternative is true. In other words, a small p-value makes us doubt the null hypothesis.
By the way, in my bag are 100 red marbles. So you see how $0.125$ is a hypothetical probability.
p.s. I do not think having the null and alternative hypotheses simultaneously possible is a good idea. I cannot at the moment think of any example where they fail to be mutually exclusive.
p.p.s. I cheated you not a little bit in the above. The truth is, the p-value of your test and sample is the probability of seeing what you see, OR of seeing something equally surprising, OR of seeing something even more surprising. And how to understand 'more surprising'? This means possible samples which are more consistent with $H_a$ than our actual sample, when assuming a world in which $H_0$ is really true. This is tougher to understand.
To test yourself, use this idea to find the p-value for the sample: RED, RED, GREEN. (now using a new bag #2 of 100 mystery marbles, each either red or green.)
Another self-test: convince yourself that $0.125$ is still correct above, using this real, more complicated definition of p-value.
"how does it make sense to say that the p-value is the probability of the alternative hypothesis being true given that the null hypothesis is true"
That's not what a $p$-value is!
The $p$-value is the probability of observing a test statistic value which is at least as extreme as the one you observed, assuming that the null hypothesis is true.
Statistics is all about assessing hypotheses based on the probabilities (or probability densities, but we'll just talk about probabilities herein for simplicity) they assign to observed outcomes. The p-value is the probability, conditional on the null hypothesis, that the results would be at least as "surprising", assuming the null hypothesis, as they in fact are. A very low p-value means what's observed is very far removed from what the null hypothesis considers a plausible outcome. By contrast, a high p-value provides no such challenge to our accepting the null hypothesis.
For this to work, the null hypothesis has to predict probabilities. Typically, we take an alternative hypothesis that doesn't do this, because it gives an inequality concerning a parameter, rather than an equation. We assess such a hypothesis by taking the null hypothesis as the default, and if its p-value is too low we may find the alternative hypothesis more convincing. Or we may not: if our null hypothesis says half the marbles are red and the alternative hypothesis says most are red, data in which few if any marbles are read makes both hypotheses look unconvincing.
But since an alternative hypothesis is only meant to be accepted with the right kind of evidence against the null hypothesis, it ought to be inconsistent with the null hypothesis: we're rejecting that, to accept the alternative.
