How many sends do I need for a real subject line test?

Enough that a handful of replies cannot flip the result: usually several thousand per variant for reply rate. If you send 500 a week, run sequential tests and treat them as evidence of direction, not proof. The alternative is to trust judgement and iterate slowly, which is honest at that scale.

Should I use emojis in cold email subject lines?

Generally no. They raise spam scores, render inconsistently across clients, and read as marketing to a recipient who has never heard of you. There are exceptions in consumer niches, but they are exceptions and should be tested rather than assumed.

Does personalisation in the subject help?

A relevant specific detail helps; a first name inserted into an otherwise generic line does not, and can read as automated. The useful version is a reference to something real about their business, which costs research and shows in the reply rate.

What about A/B testing tools built into a sending platform?

They handle the split and the timing, which is useful. They do not solve sample size, and most report opens first because opens are easy. Check whether the tool reports replies per variant before trusting its winner.

Testing Cold Email Subject Lines Without Fooling Yourself

The short answer

Test one variable at a time on a large enough sample, and judge replies rather than opens. Below about a thousand sends per variant, differences are noise. Change the line, keep the body and list segment identical, and hold the test for at least a week before deciding.

Subject line testing is where small senders most often mislead themselves. Two lines differ by three opens across 40 sends, one is declared the winner, and the next campaign uses it against a different list in a different week. The result is a rotating set of beliefs with no evidence behind any of them.

Why open rate is the wrong target

Opens are counted by a tracking pixel, and the pixel does not load when images are blocked, when a client pre-fetches, or when a privacy feature on the recipient's side proxies the content. Apple's Mail Privacy Protection made this worse by loading images automatically for many users, which registers an open that nobody performed.

Replies and positive replies are harder to fake and closer to what the campaign is for. A line that gets fewer opens but more replies is the better line. If you must use opens, treat them as a rough signal alongside clicks and replies, never alone.

What a valid test looks like

One variable per test. If the subject changes and the body changes, you have learned nothing about either. Split the list randomly rather than by recency, because a newer segment behaves differently regardless of subject, and keep the sending time and mailbox pool the same across variants.

Sample size is the part people skip. To detect a difference of a few percentage points in reply rate you need thousands of sends per variant, which most senders do not have. Below that, run sequential tests over weeks rather than a split test, and look for consistent direction rather than a single winning week.

One change at a time: subject, or body, or sender name, never two.
Random split of the same segment, sent in the same window.
Same mailboxes for both variants, or the pool becomes the variable.
Judge replies and meetings, not opens alone.
Decide the sample size before you start, not after you see the numbers.

What actually moves a cold subject line

Specificity about the recipient's situation beats cleverness. A line that names the thing they are working on reads as relevant; a line that could be sent to anyone reads as bulk. Short lines survive mobile truncation, and lowercase or plain phrasing avoids the look of a marketing blast.

Avoid the patterns that trigger spam scoring: all caps, exclamation marks, currency symbols, and phrases that promise an outcome. Those are not just deliverability risks, they are also the lines recipients have learned to ignore.

Worked example: the line that won on opens and lost on replies

A two-person agency tested two subject lines on 900 sends. The first, a question about the recipient's current provider, got 41 percent opens. The second, a short statement of what they do, got 29 percent. The agency switched to the question everywhere.

Replies told the opposite story. The question line attracted curiosity opens and few replies, because it read like a survey. The statement line got fewer opens but a reply rate nearly twice as high, from people who already had the problem it described. Over the following month, the statement line produced more calls.

Neither result was statistically strong at that volume, and the agency was careful to describe it as a direction. But the exercise changed what they measured, and the next three months of sends were planned around replies rather than opens.

Test design, sample, and what the result can support

Design	Minimum sample per variant	What it can tell you	Risk
A/B split, same segment	Several thousand sends	A real difference in reply rate	Slow to reach at low volume
Sequential weekly test	A few hundred per week	A direction worth continuing	Seasonality and list drift
Open-rate comparison only	Any	Almost nothing reliable	Privacy features inflate opens
Two changes at once	Any	Nothing attributable	You cannot tell which change worked
Subject plus new list	Any	Nothing attributable	List and subject are confounded

A test that cannot reach its sample size is not useless, it is just weaker: call it a direction, not a winner, and keep the line that consistently does better over months.

Before you call a winner

1Write down the hypothesis and the one variable you are changing.
2Randomise the split, same segment, same sending window.
3Use the same mailboxes for both variants.
4Record replies, positive replies and meetings, not just opens.
5Hold the result for at least a week before changing the winner.

Where this connects on this site

Before testing copy, make sure the mail arrives, with SMTP accounts tested before delivery
For more on campaign structure, browse the writing on cold email
For the sending side, see campaign tooling and terms

What this will not do

No subject line fixes a list that does not want the offer.
Testing cannot reach significance at volumes most small senders have, so results are directional at best.
Open rate is unreliable as a metric because of image proxying and privacy features.
A winning line can stop working when the list or the offer changes, and a test does not transfer automatically.

Questions this page gets asked

How many sends do I need for a real subject line test?: Enough that a handful of replies cannot flip the result: usually several thousand per variant for reply rate. If you send 500 a week, run sequential tests and treat them as evidence of direction, not proof. The alternative is to trust judgement and iterate slowly, which is honest at that scale.
Should I use emojis in cold email subject lines?: Generally no. They raise spam scores, render inconsistently across clients, and read as marketing to a recipient who has never heard of you. There are exceptions in consumer niches, but they are exceptions and should be tested rather than assumed.
Does personalisation in the subject help?: A relevant specific detail helps; a first name inserted into an otherwise generic line does not, and can read as automated. The useful version is a reference to something real about their business, which costs research and shows in the reply rate.
What about A/B testing tools built into a sending platform?: They handle the split and the timing, which is useful. They do not solve sample size, and most report opens first because opens are easy. Check whether the tool reports replies per variant before trusting its winner.

Last reviewed 2026-10-27. Written for readers in the United States and elsewhere; rules about access, email and consent differ by country, and nothing on this page is legal advice.