Center-commutator duality
Rewritten to ease the "start-up" cost.
First, your first question:
The fact that any subgroup of $G$ that contains $[G,G]$ is normal in $G$ follows by the isomorphism theorems: if $H$ contains $N\triangleleft G$, then $H/N$ is a subgroup of $G/N$, and $H\triangleleft G$ if and only if $(H/N)\triangleleft (G/N)$. In the instant case, $N=[G,G]$ and $G/N$ is abelian, hence $H/[G,G]$ is necessarily normal in $G/[G,G]$ (since every subgroup of an abelian group is abelian), so $H$ is normal in $G$, as claimed.
Your second question is actually much more interesting (well, to me). The short answer is that the center and the commutator subgroup are not dual constructions in the categorical sense, but they are dual constructions in a different sense (which is related to a special kind of category that turns up a lot in General Algebra).
First, let me give "universal" definitions of both subgroups, so we can see how the two are directly connected to one another.
The commutator and the center.
As you note, the commutator subgroup of $G$ is the smallest normal subgroup $N$ of $G$ such that $G/N$ is abelian.
One way to characterize abelian groups is via identities: equations in the language of group theory that may or may not be satisfied by a group. For example, the identity "$xy=yx$" is not satisfied by every group (meaning, given a group $G$, it need not be true that for all $x,y\in G$ you have $xy=yx$, though it may be true for some choices of $x,y\in G$). However, those groups that do satisfy this identity are precisely the abelian groups.
Any identity in group theory can be rewritten in the form $\mathbf{w}=1$, where $\mathbf{w}$ is a word in the language of group theory (a product of variables and their inverses); just move everything to the same side. For abelian groups, we have $(xy)(yx)^{-1}=1$, or $xyx^{-1}y^{-1}=1$. You may recognize the left hand side as none other than the commutator $[x,y]$ of $x$ and $y$. Call it $\mathbf{w}(x,y)$; that is, $\mathbf{w}(x,y) = [x,y] = xyx^{-1}y^{-1}$.
So another way of defining the commutator subgroup of $G$ is to use the word $\mathbf{w}(x,y)$ that characterizes the class of all abelian groups (as "the groups for which every possible evaluation of that word equals the identity"): the commutator subgroup of $G$ is precisely the subgroup generated by all values that $\mathbf{w}(x,y)$ takes as $x$ and $y$ range over the elements of the group. Because we define the subgroup as "the subgroup generated by all values that the word $\mathbf{w}(x,y)$ takes", we say that this is a verbal subgroup.
Here is an interesting connection between the center and the commutator subgroup: you can think of the word $\mathbf{w}(x,y)$ as giving a function (not a homomorphism) $G\times G\to G$, that maps $(x,y)$ to $\mathbf{w}(x,y)$. Even though it is not a homomorphism, we may ask: does it factor through something? Is there a normal subgroup $N$ of $G$ such that the map $G\times G\to G$ factors through $(G/N)\times (G/N)$?
In other words: what is the set of all elements $z$ of $G$ such that, for all $x,y\in G$, $[xz,y] = [zx,y] = [x,zy] = [x,yz] = [x,y]$?
I'll spill the beans: it's the center. To see this, note that if $z$ has this property, then $[z,y] = [1,y] = 1$ for all $y\in G$, so necessarily $z\in Z(G)$. Conversely, if $z\in Z(G)$, then $$[xz,y] = xzy(xz)^{-1}y^{-1} = xzyz^{-1}x^{-1}y^{-1} = xzz^{-1}yx^{-1}y^{-1}=xyx^{-1}y^{-1}=[x,y]$$ and similarly for $[zx,y]$, $[x,zy]$, and $[x,yz]$.
In this context, the center is refered to as the marginal subgroup of $G$ relative to the word $\mathbf{w}(x,y)$. The term was coined by Phillip Hall; the idea is that these are elements that can be "pushed to the margins" when computing the values of $\mathbf{w}(x,y)$.
So here's a sense in which the commutator subgroup and the center are "dual": the commutator is the subgroup generated by all values of $\mathbf{w}(x,y)$, and the center is the subgroup of all elements that don't affect the values of $\mathbf{w}(x,y)$ when you introduce them as factors.
However, there are several ways in which the two constructions are not truly "dual" in a categorical sense:
The commutator subgroup is associated to the left adjoint of the forgetful functor from $\mathcal{A}b\mathcal{G}roup$ to $\mathcal{G}roup$. The left adjoint maps a group $G$ to $G/[G,G]$; that is, there is a natural bijection between the homomorphism from $G$ to an abelian group $A$, and the (abelian group) homomorphisms between $G/[G,G]$ and $A$. The group $G/[G,G]$ is left universal among all abelian groups into which $G$ maps, etc. All the things that are related to the fact that we have a left adjoint and that $\mathcal{A}b\mathcal{G}roup$ is a reflective subcategory of $\mathcal{G}roup$, as Qiaochu mentioned.
However, the center is not associated to a right adjoint.
The commutator subgroup is the equalizer of all maps from $G$ into abelian groups; the center, however, is not the coequalizer of all map from abelian groups into $G$.
While $[G,G]$ is always the smallest normal subgroup of $G$ such that the quotient is abelian, it is not true in general that $Z(G)$ is the largest normal subgroup of $G$ that is abelian.
The set of all marginal elements is already, directly, a subgroup; we don't need to talk about the "subgroup generated by all marginal elements of $G$". But the set of all values of $\mathbf{w}(x,y)$ is not necessarily a subgroup, we need to talk about the "subgroup they generate" to get the corresponding verbal subgroup.
Nonetheless, the construction of verbal and marginal subgroups does establish some "dual relation" between the two. In fact, that relation fits into a much larger picture of verbal and marginal subgroups (which also explains the "duality" between the upper and lower central series of a group, which is what Keith Conrad's notes point out).
Verbal and marginal subgroups in general
The commutator subgroup of $G$ is an example of a verbal subgroup of $G$, and the center is the corresponding marginal subgroup.
Varieties. The key to these notions is the idea of a variety of groups. A variety of groups is any collection of groups which is closed under subgroups, homomorphic images, and arbitrary direct products. That is, if $\mathfrak{V}$ is the class, then you require that if $H\lt G$ and $G\in\mathfrak{V}$, then $H\in\mathfrak{V}$. That if $\varphi\colon G\to K$ is an onto group homomorphism and $G\in\mathfrak{V}$, then $K\in\mathfrak{V}$; and that if $\{G_i\}_{i\in I}$ is an arbitrary family of groups with $G_i\in\mathfrak{V}$ for all $i\in I$, then $\prod_{i\in I}G_i\in\mathfrak{V}$. (Note that since the empty product is the trivial group, a variety of groups is always nonempty.)
Examples of varieties are: the class of all groups; the class of all abelian groups; the class of all groups of exponent $n$ for a given $n$; the class of all nilpotent groups of class at most $c$ (for a given $c$); the class of all solvable groups of solvability length at most $s$ (for a given, fixed $s$).
Examples of collections that are not varieties are: divisible groups (subgroup of divisible need not be divisible); finite groups (arbitrary products of finite groups need not be finite); solvable groups (arbitrary products of solvable groups need not be solvable); torsionfree groups (homomorphic image of torsionfree groups need not be torsionfree).
Verbal subgroups. Given any variety of groups $\mathfrak{V}$, and a group $G$, there is a smallest normal subgroup of $G$, called $\mathfrak{V}(G)$, with the property that $G/\mathfrak{V}(G)\in\mathfrak{V}$. To see this, note that the family of all such normal subgroups is nonempty, since it contains $G$ itself; and given any family $\{N_i\}_{i\in I}$ of normal subgroups of $G$ such that $G/N_i\in\mathfrak{V}$ for all $i\in I$, then mapping $G$ to $\prod G/N_i \in\mathfrak{V}$ in the obvious way shows that $G/\cap N_i$ is a subgroup of a group in $\mathfrak{V}$, hence is itself in $\mathfrak{V}$. So the class of such subgroups has a smallest element (the intersection of all subgroups in the class). This subgroup is called the $\mathfrak{V}$-verbal subgroup of $G$.
(In the case where $\mathfrak{V}=\mathfrak{A}$, the variety of all abelian groups, the $\mathfrak{A}$-verbal subgroup is precisely the commutator subgroup).
The reason it's called a "verbal subgroup" is that to every variety there corresponds a set of "identities" that every group in $\mathfrak{V}$ satisfies, and such that every group that satisfies the identities is in $\mathfrak{V}$; these identities can be coded as words in the free group of countable rank (they are none other than the elements of $\mathfrak{V}(\mathbf{F}_{\infty})$, where $\mathbf{F}_{\infty}$ is the free group of countable rank), and the $\mathfrak{V}$-verbal subgroup corresponds to the subgroup of $G$ given by all the possible evaluations of these words in $G$ (equivalently, it is the subgroup of $G$ generated by all the images of $\mathfrak{V}(\mathbf{F}_{\infty})$ under all possible homomorphism $\mathbf{F}_{\infty}\to G$). In general you don't need all the elements of $\mathfrak{V}(\mathbf{F}_{\infty})$, you only a set of elements that generates this subgroup as a fully invariant subgroup of $\mathbf{F}_{\infty}$ (see below for the definition of "fully invariant subgroup").
In the case of abelian groups, for instance, the identity that defines abelian groups is $xy=yx$ (every group in which every possible evaluation makes $xy=yx$ true is abelian, and in an abelian group every possible evaluation makes $xy=yx$ true), which can be "coded" as the single word $xyx^{-1}y^{-1}$. This word completely defines the variety of abelian groups in the sense that a group is abelian if and only if every evaluation of $xyx^{-1}y^{-1}$ in the group is trivial. (Though abelian groups satisfy many other identities that are not true for arbitrary groups, they can be shown to be consequences of the fact that $[x,y]=1$ for all $x$ and $y$). The verbal subgroup corresponding to the class of all abelian groups is then precisely the subgroup generated by all the possible evaluations of this word, namely the commutator subgroup.
Verbal subgroups are very, very nice: they sit at the top of a hierarchy of subgroups, that is ordered in terms of how well they behave with respect to homomorphisms:
- At the bottom we have the plain, "vanilla" subgroups.
- Then we have the normal subgroup: a subgroup $H$ of $G$ is normal if and only if for every $\varphi\in\mathrm{Inn}(G)$, $\varphi(H)\subseteq H$. ($\mathrm{Inn}(G)$ are the inner automorphisms of $G$, the automorphisms of the form $x\mapsto g^{-1}xg$, for some fixed $g\in G$).
- Then we have the characteristic subgroups: a subgroup $H$ of $G$ is characteristic if and only if for every $\varphi\in\mathrm{Aut}(G)$, we have $\varphi(H)\subseteq H$.
- Then we have the fully invariant subgroups: a subgroup $H$ of $G$ is fully invariant if and only if for every $\varphi\in\mathrm{End}(G)$, we have $\varphi(H)\subseteq H$.
- Verbal subgroups, however, have an even stronger property: for every group homomorphism $\varphi\colon G\to K$, between any two groupos, $\varphi(\mathfrak{V}(G))\subseteq\mathfrak{V}(K)$. That is, the image of a verbal subgroup under any homomorphism is contained in the corresponding verbal subgroup of the image. In particular, verbal subgroups are fully invariant (hence characteristic, hence normal).
(It so happens that a subgroup of a free group is fully invariant if and only if it is verbal, which is why when we are looking for a set of generators for $\mathfrak{V}(\mathbf{F}_{\infty})$ it is enough to find one that generates it as a fully invariant subgroup).
So verbal subgroups are absolutely the bee's knees of subgroup (at least, relative to homomorphisms). They are also related to the subcategory $\mathfrak{V}$ of $\mathcal{G}roup$ in exactly the same way as the commutator subgroup is related to $\mathcal{A}b\mathcal{G}roup$: the map $G\to G/\mathfrak{V}(G)$ is a functor from $\mathcal{G}roup$ to $\mathfrak{V}$, and it is the left adjoint of the forgetful functor from $\mathfrak{V}$ to $\mathcal{G}roup$.
Marginal subgroups are related to verbal subgroups. Given any element $\mathbf{w}\in\mathbf{F}_{\infty}$, $\mathbf{w}$ is a word in finitely many letters, say $\mathbf{w}=\mathbf{w}(\mathbf{x}_1,\ldots,\mathbf{x}_n)$. Given a group $G$, you can look at the subgroup generated by all the values of the word $\mathbf{w}$, $$\mathbf{w}(G) = \Bigl\langle \mathbf{w}(g_1,\ldots,g_n)\Bigm| g_1,\ldots,g_n\in G\Bigr\rangle.$$ This is just the $\mathbf{w}$-verbal subgroup of $G$, of course. But you can consider the set of all elements of $G$ that "don't matter" relative to $\mathbf{w}$: that is, $$\mathbf{w}^*(G) = \left\langle x\in G\left| \begin{array}{l} \text{for all }g_1,\ldots,g_n\in G,\quad 1\leq i\leq n,\\ \mathbf{w}(g_1,\ldots,g_{i-1},xg_i,g_{i+1},\ldots,g_n) =\mathbf{w}(g_1,\ldots,g_n)\\\ \mathbf{w}(g_1,\ldots,g_{i-1},g_ix,g_{i+1},\ldots,g_n) = \mathbf{w}(g_1,\ldots,g_n).\end{array}\right.\right\rangle.$$ It's easy to check that $\mathbf{w}^*(G)$ is a subgroup of $G$, called the "marginal subgroup of $G$ relative to $\mathbf{w}$." In fact, it is always characteristic.
(In the case of $\mathbf{w}(x,y) = [x,y]$, which yields the commutator subgroup as a verbal subgroup, has marginal subgroup equal to the center, as we saw above.)
Given a variety $\mathfrak{V}$, we can consider the marginal subgroup relative to $\mathfrak{V}$, $\mathfrak{V}^*(G)$, by taking the intersection of all $\mathbf{w}^*(G)$ for $\mathbf{w}\in\mathfrak{V}(\mathbf{F}_{\infty})$.
Marginal subgroups are not quite as nice as verbal subgroup: they are always characteristic, but seldom fully invariant, and they don't have nice functorial properties like the verbal subgroups do. But of course they are closely related to the verbal subgroups.
Other examples of marginal subgroups: if you consider the word $\mathbf{w}_c(x_1,\ldots,x_{c}) = [x_1,x_2,\ldots,x_c]$ (where $[a,b,c]=[[a,b],c]$), then the verbal subgroup $\mathbf{w}_c(G)$ is precisely the $c$th term of the lower central series of $G$; the corresponding marginal subgroup $\mathbf{w}_c^*(G)$ is the $c-1$st center of $G$, $Z_{c-1}(G)$, the $c-1$st term of the upper central series of $G$; and so, a group $G$ is nilpotent of class at most $c$ if and only if $\mathbf{w}_{c+1}(G) = \{1\}$, if and only if $\mathbf{w}_{c+1}^*(G) = G$; that is, if and only if $G_{c+1}=\{1\}$, if and only if $Z_c(G) = G$. The word $\mathbf{w}_{c+1}$ defines to the variety of nilpotent groups of class at most $c$. The fact that the upper central series occurs precisely as the marginal subgroups of the lower central series (which is defined as a series of verbal subgroups) is behind many of the nice "dualities" between the two.
As another example, if you take $\mathbf{w}(x) = x^n$, this word corresponds to the Burnside variety $\mathfrak{B}_n$ of all groups of exponent $n$ (groups in which every element is of exponent $n$; some call them "groups of exponent dividing $n$"), so named due to the Burnside and restricted Burnside problems; the Burnside problem for exponent $n$ is true if and only if $\mathbf{F}_k/\mathfrak{B}_n(\mathbf{F}_k)$ is finite for every $k$, where $\mathbf{F}_k$ is the free group of rank $k$. The marginal subgroup corresponding to $\mathbf{w}$ is the set of all central element of exponent $n$; that is, $\mathbf{w}^*(G) = \{x\in Z(G)\mid x^n = 1\}$.
Some duality
So there is duality between verbal subgroups and the corresponding marginal subgroups, and it is exploited when dealing with things like the Schur multiplier, Baer invariants, and isologism, among other topics. You also see it at play when you look at nilpotent groups, and consider the upper and lower central series of $G$, which play kind of "dual roles" (one going up, the other going down; if $G$ is nilpotent of class exactly $c$, then $Z_{i+1}(G)\subseteq G_{c-i}$, where $G_{c-i}$ is the $(c-i)$th term of the lower central series of $G$, etc.)
For another example of a duality between the verbal and marginal subgroups, consider the following; given a variety $\mathfrak{V}$, the following are equivalent:
- $G\in\mathfrak{V}$.
- $\mathfrak{V}(G) = \{1\}$.
- $\mathfrak{V}^*(G) = G$.
The first two equivalences are just from the definition of $\mathfrak{V}(G)$ as the smallest normal subgroup of $G$ for which $G/\mathfrak{V}(G)\in\mathfrak{V}$. To see the final equivalence, note that if $\mathbf{w}(x_1,\ldots,x_n)$ is an identity of $\mathfrak{V}$ (an identity that every group in $\mathfrak{V}$ must satisfy), then for all $g_1,\ldots,g_n\in\mathfrak{V}^*(G)$ we have $$\mathbf{w}(g_1,\ldots,g_n) = \mathbf{w}(1g_1,\ldots,1g_n) = \mathbf{w}(1,\ldots,1) = 1.$$ Therefore, if $\mathfrak{V}^*(G)=G$, then $\mathfrak{V}(G)=\{1\}$. Converserly, if $\mathfrak{V}(G)=1$, then for all words $\mathbf{w}$ corresponding to $\mathfrak{V}$, all $x_1,\ldots,x_n\in G$, all $i$, $1\leq i\leq n$, and all $g\in G$ we have \begin{align*} 1 &= \mathbf{w}(x_1,\ldots,x_{i-1},x_i,x_{i+1},\ldots,x_n) = \mathbf{w}(x_1,\ldots,x_{i-1},gx_{i},x_{i+1},\ldots,x_n)\\ &= \mathbf{w}(x_1,\ldots,x_{i-1},x_ig,x_{i+1},\ldots,x_n), \end{align*} so $g\in\mathfrak{V}^*(G)$ for all $g\in G$.
References
A great place to learn about varieties of groups is Hanna Neumann's classic book, Varieties of Groups Ergebnisse der Mathematik und ihrer Grenzgebiete Band 37, Springer-Verlag, 1967. It is of course a bit dated (many of the problems posed were solved in the 70s, and there is work of Kleiman in the late 70s and early 80s that essentially shows that things can get pretty nasty when dealing with arbitrary varieties; I'm sad to say that the varietal point of view has all fallen by the way side in today's group theory).
Marginal subgroups were introduced in a very short note by Phillip Hall: Verbal and marginal subgroups, Phillip Hall, J. Reine Angew. Math. 182 (1940), pp. 156-157, MR 2, 125i.
If you want to see verbal and marginal subgroups play a major role and their interplay with homology, there is the tour de force by C.R. Leedham-Green and S. McKay, Baer invariants, isologisms, varietal laws, and homology, Acta Math. 136 (1976) no. 1-2, pp. 99-150, MR 0435250 (55 #8210). This one is very hard going, though, so since you are only beginning, you might want to set it aside for a while before trying to dig in.
From a slightly more general point of view, varieties of groups (and marginal subgroups) are the intersection of Group Theory and General Algebra (also called "Universal Algebra"); the idea of "identities" giving important subcategories of the collection of all your objects is central to General Algebra, and a lot of the universal and functorial properties of verbal subgroups are true in much more general settings.
You can get a pretty good introduction to the viewpoint of varieties of groups (if not to the specific properties they have by virtue of they being varieties of groups) with the oft-recommended (by me, anyway) General Algebra book by George Bergman. The files are in postscript, but you can get them converted to PDF with freely available software/websites like ps2pdf.
Let $H$ be a subgroup of $G$ containing all commutators. To show $H$ is normal it would suffice to take any $h\in H$ and $g\in G$ and show that $g^{-1}hg\in H$. But this element is the product of $h$ with a commutator (namely $h^{-1}g^{-1}hg$) so by assumption it is in $H$.
"Dual" means many things, and "categorically dual" is only one of them. To me the center and the commutator are very different constructions. The commutator is nothing more than the kernel of the abelianization. Abelianization $G \mapsto G/[G, G]$ defines a functor $\text{Grp} \to \text{Ab}$ which is left adjoint to the forgetful functor $\text{Ab} \to \text{Grp}$, realizing $\text{Ab}$ as a reflective subcategory of $\text{Grp}$.
The center, however, is not a functorial construction. Rather it should be thought of as a kind of "higher automorphism group" of $G$. More precisely, every group $G$ defines a category $C_G$ with one object whose morphisms are the elements of $G$ (composing like the elements of $G$ do), and it turns out that $Z(G)$ is precisely the automorphism group of the identity functor $C_G \to C_G$ (that is, it's the group of natural equivalences from this functor to itself).
Edit: in particular the center is not universal for homomorphisms from an abelian group into $G$. More precisely, the forgetful functor $\text{Ab} \to \text{Grp}$ does not have a right adjoint, and this is because it does not preserve coproducts.
Of course a more typical way to think of $Z(G)$ is as the kernel of the map $G \to \text{Aut}(G)$ given by conjugation. Again, this isn't functorial, but it is natural in a certain sense.