UK · A-Levels

Introduction to hypothesis testing

This topic sets out the logic of significance testing: the null and alternative hypotheses, one- and two-tailed tests, the significance level, the critical region and the p-value. It carries out a hypothesis test for a binomial proportion, and analyses the two kinds of error a test can make and the actual significance level of a discrete test.

5 sections~13 min reading time3 competencies

T·111111 / 18

Exam profile

AO1 · Set up and carry out a hypothesis test with a stated significance levelAO2 · Interpret the significance level, critical region and p-valueAO3 · Conclude a test in context and discuss the risk of error

Operators:testfinddeterminestateconcludecomment

basic level

The AS foundation expects a one-tailed binomial hypothesis test, the hypotheses, significance level and conclusion.

higher level

The full A-Level expects two-tailed tests, the p-value approach, critical regions, actual significance levels and the two error types.

Contents · 5 sections

Introduction to hypothesis testing

5 sections10 key takeaways4 formulas10 mistake warnings

The logic of a hypothesis test#

~3 min reading timeStandardPearson Edexcel 9ST0, Topic 11 (Paper 2)

The hypothesis-testing procedure

Fig. 1The fixed sequence of a hypothesis test, ending always in a contextual conclusion.

A hypothesis test weighs evidence against a specific claim about a population. It begins by assuming the claim — the null hypothesis

H_0

— is true, then asks how surprising the observed data would be under that assumption. If the data would be very unlikely when

H_0

is true, that counts as evidence against

H_0

in favour of the alternative

H_1

; if not, there is insufficient evidence to reject

H_0

The reasoning is deliberately asymmetric, like a trial in which the null is presumed until the evidence is strong enough to overturn it. We never prove

H_0

true; we either reject it or fail to reject it. This is why the correct conclusion is 'there is (in)sufficient evidence to reject

H_0

', never 'accept

H_0

' — the test can only fail to find evidence against the null, not confirm it.

The procedure follows fixed steps, shown in the flow below: state

H_0

and

H_1

; choose a significance level; find the critical region (or the test statistic and its p-value); compare the data with it; decide to reject or not; and state the conclusion in context. Following the steps in order, and never skipping the contextual conclusion, is what full-mark answers do.

Every step has a purpose. The hypotheses fix what is being tested and in which direction; the significance level fixes how much evidence is required; the critical region or p-value measures the evidence; and the contextual conclusion translates the statistical decision back into the language of the problem. Understanding the role of each step prevents the mechanical, context-free answers that lose marks.

Worked example

Framing a test

A manufacturer claims at most 10% of its items are faulty. A retailer suspects the rate is higher. Set up the hypotheses and describe the logic of the test.

Null hypothesis
$H_0$ : $p = 0.10$ (the claimed faulty rate).
Alternative
$H_1$ : $p > 0.10$ (the retailer's suspicion), a one-tailed test.
Logic
Assume $p = 0.10$ ; if the observed number of faults would be very unlikely under this assumption, reject $H_0$ in favour of $H_1$ .

Result: $H_0$ : $p = 0.10$ against $H_1$ : $p > 0.10$ ; reject $H_0$ only if the data are too surprising under $p = 0.10$ .

Typical mistakes

Writing 'accept $H_0$ ' instead of 'insufficient evidence to reject $H_0$ '.
Omitting the conclusion in context, leaving only a statistical decision.

Active revision

Explain, in your own words, why a hypothesis test can never prove the null hypothesis true.

Practise matching questions

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Hypotheses, tails and significance levels#

~2 min reading timeStandardPearson Edexcel 9ST0, Topic 11 (Paper 2)

The null hypothesis

H_0

specifies a particular value of the parameter (for example

p = 0.5

), and the alternative

H_1

specifies the departure of interest. If

H_1

points in one direction (

p > 0.5

p < 0.5

) the test is one-tailed; if it merely says the value is different (

p \neq 0.5

) the test is two-tailed. The wording of the question — 'has increased', 'has changed', 'is biased' — fixes the tail, and choosing it correctly is essential.

The significance level

\alpha

is the probability threshold below which the data are judged too surprising to attribute to chance, commonly 5%, 1% or 10%. It is fixed BEFORE seeing the data, so the standard of evidence cannot be adjusted to suit the result. A smaller

\alpha

demands stronger evidence to reject

H_0

and so reduces the chance of a false rejection.

For a two-tailed test the significance level is split between the two tails, so a 5% two-tailed test uses

2.5\%

in each tail. This matters because the same data can be significant one-tailed but not two-tailed: a two-tailed test is more conservative, reflecting that it looks for a departure in either direction and so demands more in the direction observed.

The direction of

H_1

must be justified by the context, not chosen after seeing the data. Testing 'has the proportion increased?' one-tailed because the sample happened to be high, when the original question was 'has it changed?', is a form of cheating that inflates the true error rate. Setting the hypotheses and the tail from the wording, before any calculation, is the disciplined approach the examination rewards.

Hypotheses and tails

\text{one-tailed: } H_1: p > p_0 \;\text{or}\; p < p_0; \qquad \text{two-tailed: } H_1: p \neq p_0

A two-tailed test splits $\alpha$ between the tails ( $2.5\%$ each at the 5% level).

Worked example

Setting up a two-tailed test

A die is suspected of being biased towards or against sixes. It is tested at the 5% level. State the hypotheses and the tail allocation.

Parameter
Let $p = P(\text{six})$ ; a fair die has $p = \tfrac{1}{6}$ .
Hypotheses
$H_0$ : $p = \tfrac{1}{6}$ against $H_1$ : $p \neq \tfrac{1}{6}$ (bias in either direction), so two-tailed.
Tail allocation
At the 5% level, each tail carries $2.5\%$ .

Result: $H_0$ : $p = \tfrac{1}{6}$ against $H_1$ : $p \neq \tfrac{1}{6}$ , two-tailed with $2.5\%$ in each tail.

Typical mistakes

Selecting the tail after seeing the data rather than from the question.
Using the full $\alpha$ in one tail of a two-tailed test instead of $\alpha/2$ .

Active revision

A coin is tested for bias (in either direction) at the 5% level. State $H_0$ , $H_1$ and the probability allocated to each tail.

Practise matching questions

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Critical regions and the p-value approach#

~2 min reading timeAdvancedPearson Edexcel 9ST0, Topic 11 (Paper 2)

There are two equivalent ways to make the decision. The critical-region method finds, before seeing the data, the set of extreme outcomes whose total probability under

H_0

is at most

\alpha

; if the observed value falls in this region,

H_0

is rejected. The p-value method computes, after seeing the data, the probability under

H_0

of a result at least as extreme as the one observed, and rejects

H_0

if this p-value is at most

\alpha

The two methods always agree, because the observed value lies in the critical region exactly when its p-value is at most

\alpha

. The p-value is often preferred because it reports HOW surprising the data are, not merely whether they cross the threshold: a p-value of 0.001 is far stronger evidence against

H_0

than a p-value of 0.049, though both reject at the 5% level.

A critical p-value must be interpreted precisely. It is the probability of the observed (or more extreme) data GIVEN that

H_0

is true — it is NOT the probability that

H_0

is true. A small p-value means the data are surprising under

H_0

, which is evidence against it, but it says nothing directly about the probability of the hypothesis itself.

For a discrete test the critical region is built from cumulative probabilities and rarely has probability exactly

\alpha

, a subtlety pursued in the binomial test next. In every case the decision rule is the same: reject

H_0

when the evidence (the p-value) is at most the pre-set significance level, and otherwise do not reject.

The p-value decision rule

\text{p-value} = P(\text{data at least as extreme} \mid H_0); \quad \text{reject } H_0 \iff \text{p-value} \leq \alpha

The p-value is a conditional probability given $H_0$ , not the probability that $H_0$ is true.

Worked example

Deciding from a p-value

A one-tailed test at the 5% level gives a p-value of 0.032. State the decision and interpret the value.

Compare with alpha
$0.032 \leq 0.05$ , so the data fall in the critical region.
Decision
Reject $H_0$ at the 5% level.
Interpret
If $H_0$ were true, data this extreme would occur with probability 0.032 — surprising enough to reject $H_0$ , though not overwhelming.

Result: Reject $H_0$ : the p-value 0.032 is below 0.05, meaning such data are unlikely (probability 0.032) under $H_0$ .

Typical mistakes

Interpreting the p-value as the probability that $H_0$ is true.
Comparing the p-value with the wrong threshold (for example $\alpha$ rather than $\alpha/2$ in a two-tailed test).

Active revision

In a one-tailed test at the 5% level the p-value is 0.032. State the decision and interpret the p-value.

Practise matching questions

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

A test for a binomial proportion#

~3 min reading timeAdvancedPearson Edexcel 9ST0, Topic 11 (Paper 2)

Critical region on the number of heads

Fig. 2Critical region X >= 15 for testing H0: p = 0.5 against H1: p > 0.5 with n = 20 at the 5% level.

The archetypal test concerns a binomial proportion. To test whether a coin is biased towards heads, take

H_0

p = 0.5

against

H_1

p > 0.5

, model the number of heads

X

n

tosses as

B(n, 0.5)

under

H_0

, and reject

H_0

for values of

X

in the upper tail. The critical region is the smallest set of large values whose total probability under

H_0

does not exceed

\alpha

For

n = 20

tosses at the 5% level, the cumulative binomial gives

P(X \geq 15) = 1 - P(X \leq 14) = 1 - 0.9793 = 0.0207

, which is below 5%, whereas

P(X \geq 14) = 0.0577

exceeds 5%. The critical region is therefore

X \geq 15

, shown on the number line below, and the critical value is 15.

Because the binomial is discrete, the critical region's probability (0.0207) is below the nominal 5% rather than exactly equal to it; this true probability is the ACTUAL significance level of the test. The gap arises because

X

jumps in whole steps, so no region has probability exactly 0.05 — a feature of every discrete test.

The decision then compares the observed count with the region: 15 or more heads falls in the critical region and rejects

H_0

, while 14 or fewer does not. The conclusion must be contextual — 'there is sufficient evidence at the 5% level that the coin is biased towards heads' — and should acknowledge that the actual significance level is 2.07%, not 5%.

Critical region for the binomial test

\text{Under } H_0,\; X \sim B(20, 0.5); \quad P(X \geq 15) = 0.0207 < 0.05 \leq P(X \geq 14) = 0.0577

The critical region is $X \geq 15$ ; its actual probability 0.0207 is the actual significance level.

Worked example

A one-tailed binomial test

A coin tossed 20 times shows 16 heads. Test at the 5% level whether it is biased towards heads, using $P(X \leq 14) = 0.9793$ under $H_0$ .

Hypotheses
$H_0$ : $p = 0.5$ against $H_1$ : $p > 0.5$ ; under $H_0$ , $X \sim B(20, 0.5)$ .
Critical region
$P(X \geq 15) = 1 - 0.9793 = 0.0207 < 0.05$ , and $P(X \geq 14) = 0.0577 > 0.05$ , so the region is $X \geq 15$ .
Compare
The observed 16 lies in the region $X \geq 15$ .
$16 \geq 15 \Rightarrow \text{reject } H_0$
Conclude
Reject $H_0$ : there is sufficient evidence at the 5% level (actual level 2.07%) that the coin is biased towards heads.

Result: Since 16 lies in the critical region $X \geq 15$ , reject $H_0$ ; the coin appears biased towards heads (actual significance level 2.07%).

Typical mistakes

Using $P(X \geq c) \leq \alpha$ but taking $c$ one step too small so the region exceeds $\alpha$ .
Reporting the nominal level (5%) as the actual significance level for a discrete test.

Active revision

Test at the 5% level whether a coin tossed 20 times, landing heads 16 times, is biased towards heads, given $P(X \leq 14) = 0.9793$ .

Practise matching questions

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Type I and Type II errors#

~3 min reading timeAdvancedPearson Edexcel 9ST0, Topic 11 (Paper 2)

A test can be wrong in two distinct ways. A Type I error is rejecting

H_0

when it is actually true — a false alarm — and a Type II error is failing to reject

H_0

when it is actually false — a missed detection. The two are not symmetric, and which is worse depends on the context: a false alarm on a smoke detector is a nuisance, but a missed fire is a disaster.

The probability of a Type I error is controlled directly: for a discrete test it is the actual significance level, the probability under

H_0

of landing in the critical region. For the binomial test above,

P(\text{Type I}) = P(X \geq 15 \mid p = 0.5) = 0.0207

. Lowering the significance level reduces this probability but makes rejection harder, which increases the chance of a Type II error.

The probability of a Type II error, usually denoted

\beta

, is the probability of NOT landing in the critical region when a specific alternative value of the parameter is true. It can only be computed against a stated alternative, because 'the null is false' is not specific enough. Its complement

1 - \beta

is the power of the test, developed fully in a later topic.

There is an unavoidable trade-off: for a fixed sample size, reducing one error probability increases the other. The only way to reduce both at once is to gather more data. Recognising this trade-off, and choosing a significance level that reflects the relative costs of the two errors, is the mature judgement that hypothesis testing demands.

The two error probabilities

P(\text{Type I}) = P(\text{reject } H_0 \mid H_0 \text{ true}) = \alpha_{\text{actual}}, \qquad P(\text{Type II}) = \beta

$\alpha$ is controlled by the significance level; $\beta$ needs a specific alternative; power is $1 - \beta$ .

Worked example

Probability of a Type I error

For the test $H_0$ : $p = 0.5$ against $H_1$ : $p > 0.5$ with $n = 20$ and critical region $X \geq 15$ , state the probability of a Type I error and describe a Type II error.

Type I probability
$P(\text{Type I}) = P(X \geq 15 \mid p = 0.5) = 0.0207$ .
Meaning
It is the chance of wrongly declaring the coin biased when it is in fact fair.
Type II error
A Type II error here is failing to detect a genuinely biased coin, i.e. observing $X \leq 14$ when in truth $p > 0.5$ .

Result: $P(\text{Type I}) = 0.0207$ ; a Type II error is failing to reject $H_0$ (observing $X \leq 14$ ) when the coin is really biased.

Typical mistakes

Swapping the two error types (false alarm versus missed detection).
Trying to compute a Type II error probability without a specific alternative value.

Active revision

For the binomial test with critical region $X \geq 15$ under $H_0$ : $p = 0.5$ ( $n = 20$ ), state the probability of a Type I error and describe a Type II error in context.

Practise matching questions

Active recall

Recall the key points — then reveal.

Sources: GCE AS and A level subject content (Statistics) (Department for Education / Ofqual)

Full version via the depth control — same place, same anchors

Plan a review

References & sources

Sources

Pearson Edexcel

Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification

Department for Education / Ofqual

GCE AS and A level subject content (Statistics)

Introduction to hypothesis testing

5 sections~13 min reading time3 competencies

Introduction to hypothesis testing

Key points

Framing a test

Key points

Setting up a two-tailed test

Key points

Deciding from a p-value

Key points

A one-tailed binomial test

Key points

Probability of a Type I error

Sources

Introduction to hypothesis testing

Key points

Framing a test

Key points

Setting up a two-tailed test

Key points

Deciding from a p-value

Key points

A one-tailed binomial test

Key points

Probability of a Type I error

Sources