Now suppose $X_{1},\ldots ,X_{n}$ are i.i.d. random variables from some distribution with mean $\mu$ and variance ${\sigma}^{2}$, both unknown. How can we construct a confidence interval for $\mu$?
In case of normal distribution, recall that the $(1-\alpha)$-CI that we gave was $$ \left[\bar{X}_{n}-\frac{s_{n} }{\sqrt{n} }t_{n-1}\left(\frac{\alpha}{2}\right),\bar{X}_{n}+\frac{s_{n} }{\sqrt{n} }t_{n-1}\left(\frac{\alpha}{2}\right)\right] or \left[\bar{X}_{n}-\frac{s_{n} }{\sqrt{n} }z_{\alpha/2},\bar{X}_{n}+\frac{s_{n} }{\sqrt{n} }z_{\alpha/2}\right] $$ Is this a valid confidence interval in general? The answer is ''No'' for both. If $X_{i}$ are from some general distribution then the distributions of $\sqrt{n}(\bar{X}_{n}-\mu)/s_{n}$ and $\sqrt{n}(\bar{X}_{n}-\mu)/{\sigma}$ are very complicated to find. Even if $X_{i}$ come from binomial or exponential family, these distributions will depend on the parameters in a complex way (in particular, the distributions are not free from the parameters, which is important in constructing confidence intervals).
But suppose $n$ is large. Then the sample variance is close to population variance and hence $s_{n}\approx {\sigma} $. Further, by CLT, we know that $\sqrt{n}(\bar{X}_{n}-\mu)/{\sigma}$ has approximately $N(0,1)$ distribution. Hence, we see that $$ \mathbf{P}\left\{-z_{\alpha/2}\le \frac{\sqrt{n}(\bar{X}_{n}-\mu)}{s_{n} } \le z_{\alpha/2}\right\} \approx \Phi(z_{\alpha/2})-\Phi(-z_{\alpha/2}) =1-\alpha. $$ Consequently, we may say that $$ \mathbf{P}\left\{ \bar{X}_{n}-\frac{s_{n} }{\sqrt{n} }z_{\alpha/2} \le \mu \le \bar{X}_{n}+\frac{s_{n} }{\sqrt{n} }z_{\alpha/2}\right\} \approx 1-\alpha. $$ Thus, $\left[\bar{X}_{n}-\frac{s_{n} }{\sqrt{n} }z_{\alpha/2}, \bar{X}_{n}+\frac{s_{n} }{\sqrt{n} }z_{\alpha/2} \right]$ is an approximate $(1-\alpha)$-confidence interval. Further, when $n$ is large, the difference between $V_{n}=\frac{1}{n-1}\sum_{i=1}^{n}(X_{i}-\bar{X}_{n})^{2}$ and $V_{n}:=\frac{1}{n}\sum_{i=1}^{n}(X_{i}-\bar{X}_{n})^{2}$ is small (indeed, $s_{n}^{2}=(n/(n-1))V_{n}$). Hence it is also okay to use $\left[\bar{X}_{n}-\frac{\sqrt{V_{n} }}{\sqrt{n} }z_{\alpha/2}, \bar{X}_{n}+\frac{\sqrt{V_{n} }}{\sqrt{n} }z_{\alpha/2} \right]$ as an approximate $(1-\alpha)$-confidence interval.