If $G$ is a compact topological group, how to show that a finite index subgroup of $G$ is open?

This is false, so don't feel bad you cannot prove it.

One of the standard non-finite counterexamples is below, it's mostly to illustrate that the problem is more systematic than just the trivial, finite ones. In particular, it starts with a Hausdorff group rather than what some might consider cheating by endowing the original group with a non-Hausdorff topology. From this it constructs a non-closed--but still finite-index--subgroup, so that the quotient space has a non-Hausdorff topology (which is equivalent to the original subgroup being non-closed).

Consider the finite group $A=\Bbb Z/2\Bbb Z$, endowed with the discrete topology. (Note that there are four topologies on a $A$, and that only two of them (the discrete one and the coarsest one) are turning it into a topological group, and only one of them (the discrete one) turns it into a compact topological group.)

Construct the group

$$G=A^\Bbb Z=\prod_{n\in\Bbb Z} A$$

This is compact by Tychonoff's theorem, and manifestly a group. Now from here we let $\tau$ be the topology generated by an ultrafilter, $\mathcal{F}$, containing the Fréchet filter. Let $H\le G$ be defined by the fact that Let $\pi_n(g)=g_n$ be the $n^{th}$ coordinate map, and

$$h\in H\iff |\{n\in\Bbb Z : \pi_n(h)=1\}|\in\{0,\infty\}.$$

$H$ is also dense in $G$ because the Fréchet filter is cofinite in the powerset of $\Bbb Z$, but clearly also $H\ne G$. Then distinguish two special elements, $\mathbf{0},\mathbf{1}\in G$, defined by the rules

$$\begin{cases}\pi_n(\mathbf{0}) =0 & n\in\Bbb Z \\ \pi_n(\mathbf{1})=1 & n\in\Bbb Z\end{cases}$$

Now let $g\in G$ be arbitrary and define sets $S_0, S_1$ by the rule

$$S_i = \{n\in\Bbb Z : g_i=n\}$$

So that $\Bbb Z=S_0\coprod S_1$ since $g_i\in\{0,1\}$. Then we see one of the $S_i\in\mathcal{F}$ by maximality. So $g\equiv x\mod H$ for some $x\in\{\mathbf{0},\mathbf{1}\}$, i.e. $[G:H]\le 2$. Since $H\ne G$, clearly $[G:H]=2$, but $H$ is dense, hence cannot be closed, and since closed + finite index = open, it must be that $H$ is not open, despite being finite index.

The key problem: we chose a non-Hausdorff topology on the quotient, $G/H$, i.e. we took only the trivial topology $\tau=\{\varnothing, G/H\}$. It is classical that $G/H$ Hausdorff iff $H$ is closed (provided $G$ is Hausdorff), so this was the natural approach to producing a non-closed subgroup of finite index. The algebra still allows two cosets, but the topology doesn't allow you to force them apart.


Edit: An even easier counterexample is just $G=\Bbb Z/2\Bbb Z$ with the trivial topology. Then $\{e\}\le G$ is not open, but clearly has finite index.


Edit 2: Since the op is using the convention that "compact" means "Hausdorff + finite subcovers" I moved the first edit to the bottom.


As I gave a wrong answer I will try to amend myself. Note (P) the following property for a topological group : all finite index subgroups are open. Recall that a profinite group is a topological group isomorphic (in the category of topological groups) to a projective limit of finite groups endowed with discrete topologies. Since a product is but a projective limit, Adam's answer shows in particular that (P) is in general not satisfied for profinite groups. I will give an everyday life counter-example.

Let $L$ be a Galois extension of $K$, but not necessarily a finite extension, and $G$ be the Galois group $Gal(L/K)$. Put the dicrete topology on $L$, the product topology on $L^L$, and the induced topology on $G$. Let $A$ be the set of finite subextensions of $L/K$. For $\sigma\in G$ and $E\in A$, set $U_{\sigma}(E) := $ the subset of $G$ consisting of elements $\tau$ having same restriction on $E$ that $\sigma$. One can show that $U_{\sigma}(E)$ is a filter basis of $\sigma$ for this topology, and that the restrictions $G \rightarrow Gal(L'/K)$ are continuous for any subextension $L'/ K$ that is Galois, but not necessarily finite. Moreover, one can show that $G$ is compact and totally disconnected. Actually, if $(L_i)_i$ is a filtered familly of galois subextensions such that $L = \cup_i L_i$, you have $G \simeq \varprojlim_i Gal(L_i / K)$. (As you can choose all $L_i$'s to be finite over $K$, you got in particular that $G$ is profinite.)

From now we will considerer the extension $\overline{\mathbf{Q}} / \mathbf{Q}$ where $\overline{\mathbf{Q}}$ is the algebraic closure of $\mathbf{Q}$ in $\mathbf{C}$. Let $E = \mathbf{Q}\left[ \{\sqrt{-1}\}\cup\{\sqrt{p}\;|\;p\textrm{ prime}\} \right]$ and $$G := Gal(E/\mathbf{Q}) \simeq \varprojlim Gal\left(\mathbf{Q}\left[\sqrt{-1}, \sqrt{2}, \ldots, \sqrt{p}\right] / \mathbf{Q}\right) $$ which is a subgroup of $Gal(\overline{\mathbf{Q}} / \mathbf{Q})$ and let $H$ be the subgroup of $G$ consisting in elements permuting only finitely many elements among $\sqrt{-1}$ and all the $\sqrt{p}$ for $p$-prime. The subgroup $H$ is dense in $G$ as by definition of $Gal(\overline{\mathbf{Q}} / \mathbf{Q})$'s topology and $G$'s induced topology every open of $G$ contains obviously an element of $H$. Now $H$ is normal and the group $G / H$ is an $\mathbf{F_2}$-vector space in an obvious manner. Now fix a $d\in\mathbf{N}^{*}$ and let $E$ be a sub-$\mathbf{F_2}$-vector space of $G / H$ of codimension $d$. (Possible by Zorn's lemma.) The inverse image of $E$ in $G$ is a subgroup $K$ of $G$ containing $H$ and of index $Card(\mathbf{F_2})^d = 2^d$. The subgroup $K$ is not open because were it open he would be closed which would be a contradiction with the density of $H$ in $G$. Now $K$ is not open, and its inverse image in $Gal(\overline{\mathbf{Q}} / \mathbf{Q})$ is not open and is of finite index. Indeed, if the inverse image were open, its fixed field would be a nontrivial extension $F$ of $\mathbf{Q})$ contained in $E$ but then $F$ would be fixed by $K$, which is dense... The group $Gal(\overline{\mathbf{Q}} / \mathbf{Q})$ has not the property (P).

Remark 1. I wrote Were it open he would be closed. For a subgroup $H$ of finite index of a topological group (not necessarily quasi-compact nor compact), being closed or being open is equivalent. Indeed. Make $H$ act of $G$ by left translations and, as $H$ is of finite index, let $g_1,\ldots,g_n$ be a set whose classes modulo $H$ partition $G$ in classes, and suppose that $g_1$'s class is $H$. You can therefore write the disjoint union $G = H \cup \left( \cup_{i=2}^n g_i H\right)$. Note that the $g\mapsto x g$ are homeomorphisms so that a $xH$ is closed (resp. open) if and only if $H$ is. Now, if $H$ is closed, the $g_i H$'s are also, so that the finite union of closed $\cup_{i=2}^n g_i H$ also is, and its complement in $G$, which is but $H$, is open. The same argument applies mutatis mutandis to show that if $G$ is open, it is closed. Note that is this case, our proof would have worked also if there wouldn't have been finitely many $g_i$'s, so that in fact, any open subgroup of a topological group is closed.

Remark 2. The group of $p$-adic integers $\mathbf{Z}_p$ is a profinite group in which all subgroups of finite index are closed and open. You can show this by finding explicitely all these subgroups, but how could you show it "conceptually" ?