Mrthdrb

I have 2 samples, one with sample size of 30,000 customers and the other with 150,000. I have to perform a 2 sample t test(on conversion rates of the 2 groups). My question is, will t test in this case be biased towards the smaller sample? If yes, what is the correct approach to perform a test?

Maybe a couple of examples will help to illustrate some of the issues.

Suppose the two populations are $X sim mathsfNorm(mu = 500, sigma =30)$
and $Y sim mathsfNorm(mu = 501, sigma = 20.)$

If both sample sizes are $150,000,$ then there is sufficient power to detect
the small difference in means.

set.seed(422)
x = rnorm(150000, 500, 30)
y = rnorm(150000, 501, 20)
t.test(x, y)

 Welch Two Sample t-test

data: x and y
t = -10.983, df = 261530, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.2042715 -0.8395487
sample estimates:
mean of x mean of y 
 499.9804 501.0023 

If we use only the first 30,000 values in the first sample, results are
very nearly the same for most practical purposes.

t.test(x[1:30000], y)

 Welch Two Sample t-test

data: x[1:30000] and y
t = -6.3728, df = 35463, p-value = 1.879e-10
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.5126269 -0.8010336
sample estimates:
mean of x mean of y 
 499.8455 501.0023 

Here is a boxplot of the data used in the second t test (the wider box indicates a larger sample):

Issues of minimal concern:

• Even though labeled as 'Welch t tests', sample sizes are sufficiently large
  that these are essentially t tests. Unless the data are very far from normal,
  we would still detect the small difference in means.




• The power of the test is heavily dependent on the smaller sample size. But
  power is not a concern here.

Issues warranting attention:

• With such large samples
  in the real world (not the simulation world),
  one is entitled to wonder whether data are truly simple random samples from
  their respective populations. Could smaller, more carefully collected samples provide better information?




• Although we did not do a formal test to confirm that variances differ, it seems clear from the boxplot that they do. In the Welch test,
  it is OK for variances to differ. But would different variances have important practical implications?




• Although the null hypothesis that the two population means are equal is soundly rejected with minuscule P-values, it is important to realize that "statistically significant" differences (by whatever definition) are not necessarily differences of practical importance or interest. For what purpose are you
  taking the effort of check whether means are different? And what do the results
  of the t test actually contribute to that purpose?

I can hardly imagine any worthwhile effect size that requires such a large sample size to be decently powered. There's no "bias" of having unequal sample sizes$^1$. The only disadvantage is that the power of the test tends to be somewhat limited by the smaller group. For even very small effects, 30,000 observations may confer quite a powerful test.

$^1$ except if you inappropriately use the "equal variance" assumption, in which case the "pooled variance" estimate is more heavily weighted toward the larger group (not toward the smaller as you suggested).

I agree with the most that was said so far but I do not completely agree with the satement from @AdamO that "There's no "bias" of having unequal sample sizes".

Unfortunately, we don't know what the purpose of your study is. But let's assume you are interested in gender differences in regard to salary. We know that in population there should be about 50% male and 50% female and hence if you had drawn random samples and if there was MAR (missing at random) we would expect both groups having approximately same sample sizes. To put it differently, if the ratio between sample sizes of both groups is very different than their ratio in the population this can indicate that either the samples are not random (what could cause a bias) or that the missings are not random (what could cause a bias, too). Talking about the gender example I would be surprised if someone would report such big differences in samples sizes (for example: Did more women refuse to answer questions about their salary? Are the women who responded to the question representative or did only those with a high salary answer the question? And so on... Non-random missings would obviously cause a bias here and make the results misleading).

Thus the question that I would ask myself is why the groups have unequal sample sizes. If there is an reasonable answer like "There are more people without heart failure than people with heart failure" the data might be alright. But if you would expect equal sample sizes based on what you know about the groups in the population there might be some bias because the samples/ the missings seem to be not random.