Generalized repetitions of letters with limited amount of adjacent letters
The following answer is based upon the Goulden-Jackson Cluster Method which provides a convenient technique to solve problems of this kind. We consider the set of words of length $y\geq 0$ built from an alphabet $$\mathcal{V}=\{A_1,A_2,\ldots A_x\}$$ and the set $B=\{A_1^{z+1},A_2^{z+1},\ldots,A_x^{z+1}\}$ of bad words each of length $z+1$, which are not allowed to be part of the words we are looking for. We derive a generating function $f(t)$ with the coefficient of $t^y$ being the number of wanted words of length $y$.
According to the paper (p.7) the generating function $f(t)$ is \begin{align*} f(t)=\frac{1}{1-xt-\text{weight}(\mathcal{C})}\tag{1} \end{align*} with $x=|\mathcal{V}|$ the size of the alphabet and $\mathcal{C}$ the weight-numerator of bad words with \begin{align*} \text{weight}(\mathcal{C})=\text{weight}(\mathcal{C}[A_1^{z+1}])+\text{weight}(\mathcal{C}[A_2^{z+1}]) +\cdots+\text{weight}(\mathcal{C}[A_x^{z+1}])\tag{2} \end{align*}
We calculate according to the paper for $1\leq j\leq x$: \begin{align*} \text{weight}(\mathcal{C}[A_j^{z+1}])&=-t^{z+1}-(t+t^2+\cdots+t^{z})\text{weight}(\mathcal{C}[A_j^{z+1}])\tag{3}\\ \end{align*} where the long expression at the right-hand side of (3) accounts for overlappings of $A_{j}^{z+1}$ with itself. We obtain from (3): \begin{align*} \text{weight}(\mathcal{C}[A_j^{z+1}])&=-\frac{t^{z+1}}{1+t+\cdots t^{z}}\\ &=-\frac{t^{z+1}(1-t)}{1-t^{z+1}}\qquad\qquad 1\leq j \leq x\\ \end{align*} It follows from (1) - (3): \begin{align*} \color{blue}{f(t)}&=\frac{1}{1-xt-\text{weight}(\mathcal{C})}\\ &=\frac{1}{1-xt+x\frac{t^{z+1}(1-t)}{1-t^{z+1}}}\\ &\,\,\color{blue}{=\frac{1-t^{z+1}}{1-xt-(1-x)t^{z+1}}}\tag{4} \end{align*} and conclude the number of valid words of length $y\geq 0$ is the coefficient of $t^y$ in (4). We use the coefficient of operator $[t^y]$ to denote the coefficient of $y^n$ of a series. Applying standard techniques to extract the coefficient $[t^y]$ from $f(t)$ we obtain \begin{align*} [y^t]f(t)&=\sum_{k=0}^{\left\lfloor\frac{y}{z}\right\rfloor}\binom{y-zk}{k}(1-x)^kx^{y-(z+1)k}\\ &\qquad- \sum_{k=0}^{\left\lfloor\frac{y-z-1}{z}\right\rfloor}\binom{y-z-1-zk}{k}(1-x)^kx^{y-z-1-(z+1)k}\tag{5} \end{align*}
Example: $x=2,y=3,z=2$
We look at OPs example where the alphabet $\mathcal{V}=\{A,B\}$ consists of two characters. Invalid words are those containing substrings which are elements from the set of $\{AAA,BBB\}$ of bad words. According to (4) the generating function is given as
\begin{align*} f(t)&=\left.\frac{1-t^{z+1}}{1-xt-(1-x)t^{z+1}}\right|_{x=2,z=2}\\ &=\frac{1-t^3}{1-2t+t^3}\\ &=1 + 2 t + 4 t^2 + 6t^3 + \color{blue}{10} t^4 + 16 t^5\\ &\qquad + 26 t^6 + 42t^7 + 68 t^8 +\cdots\\ \end{align*}
The last line was calculated with some help of Wolfram Alpha. Note, the coefficient $6$ of $t^{3}$ according to the six valid words stated by OP. The blue marked coefficient of $t^4$ is $10$ and the $2^4-10=6$ invalid words are \begin{align*} &\rm{\color{blue}{aaa}a}&\rm{a\color{blue}{bbb}}\\ &\rm{\color{blue}{aaa}b}&\rm{b\color{blue}{bbb}}\\ &\rm{b\color{blue}{aaa}}&\rm{\color{blue}{bbb}a}\\ \end{align*}
Applying (5) we find \begin{align*} \color{blue}{[t^3]f(t)}&=\sum_{k=0}^1\binom{3-2k}{k}(-1)^k2^{3-3k}-\sum_{k=0}^0\binom{0-2k}{k}(-1)^k2^{0-3k}\\ &=\binom{3}{0}(-1)^02^3+\binom{1}{1}(-1)^12^0-\binom{0}{0}(-1)^02^0\\ &=8-1+1\\ &\,\,\color{blue}{=6}\\ \\ \color{blue}{[t^4]f(t)}&=\sum_{k=0}^2\binom{4-2k}{k}(-1)^k2^{4-3k}-\sum_{k=0}^0\binom{0-2k}{k}(-1)^k2^{1-2k}\\ &=\binom{4}{0}(-1)^02^4+\binom{2}{1}(-1)^12^1+\binom{0}{2}(-1)^22^{-2}-\binom{0}{0}(-1)^02^1\\ &=16-4+0-2\\ &\,\,\color{blue}{=10} \end{align*}
in accordance with the results above.
Partial Answer (assuming $2z \geq y$ and $x \geq 2$)
First note that if $z \geq y$, there is no restriction on which letters can be placed where. If $P$ denotes the number of possible sequences, this is to say $P = x^y$ in this case.
Now suppose $z < y$. We can use the subtraction principle as you alluded to. We need to find the number of sequences that have more than $z$ adjacent letters. Let's call this number $R$. In order to construct $R$, we will need to form a sum, each term being all possible "bad" sequences of a fixed number of repeating letters more than $z$. I find an example illustrative.
Suppose $x =3$, $y= 6$, and $z = 3$. Then any sequence with 4 or more adjacent letters is "bad" and we need to include it in $R$. One such sequence, given this $x,y,z$, is: $$ AAAABB . $$ We note there are a few ways to "place" the block of $A$'s within the length 6 sequence, in this case $3$. In general, if our string of adjacent letters has length $L \leq y$, the number of ways to place the block is $y-L + 1$. Moreover, this block of adjacent letters could be any of $x$ letters, so we multiply the term by $x$. In our example, we could have: $$ CCCCBB. $$ Finally, we want to count the number of ways the slots unoccupied by the block of adjacent letters could be chosen. We want to be careful not to double count. In the example above, notice the last two slots are the unoccupied ones. Since we are summing over length of adjacent letter blocks, $L$, notice if we let slot $5$ be $C$ we would double count when we increased $L$ to $5$ (currrently $L = 4$). So we allow slot $5$ to be any of the $x-1$ remaining letters, in this case $A$ or $B$. On the contrary, slot $6$ can be any of the $x$ letters, $A$,$B$, or $C$ in this case. Let $j$ denote the number of adjacent positions to the block of $L$ letters. We see in any case that $j \in \{0,1,2\}$. For example, $j = 0$ for: $$ AAAAAA, $$ while $j = 2$ for: $$ BAAAAB. $$ A key is that the number of slots which can have $x - 1$ letters as opposed to $x$ letters is dependent on the placement of the $L$ adjacent blocks. To account for this, define the sequence: $$ S_L = \begin{cases} (1, 2, \dots, 2, 1) & y - L -1 \geq 0, \\ (0) & y - L - 1 < 0 \end{cases} $$ where the number of $2$'s is equal to $y - L - 1$, in the top case. We stated earlier the number of ways to place the block of $L$ adjacent letters is $y - L +1$, and observe this is precisely the length of $S_L$ in each case. Moreover, the terms in $S_L$ are precisely the number of slots adjacent to the block of $L$ letters, i.e. slots that can have only $x-1$ letters as options. We can imagine this as placing the block of $L$ letters as far to the left as possible, and consider sliding it over one slot at a time until it reaches the right hand side. In context of our example, we have $L = 4$, so we have $y - L + 1 = 3$ options for where to place a block of $L$ letters ($A$'s in this example): $$ AAAA \: \_\_ \: \_\_ \; , \; \_\_ \: AAAA \: \_\_ \; , \; \_\_ \: \_\_ \:AAAA $$ The sequence that corresponds to this is $S_L = (1,2,1)$, since $y - L - 1 = 1$, so we have one $2$, which corresponds to the two adjacent slots for the middle placement above (and the $1$'s are of course the number of adjacent slots for the first and last placements of the $A$'s).
With this, we can write down the sum (it will actually be a double sum, accounting for the slots adjacent to the $L$ letter block in each placement). The number of invalid sequences of letters for given $x,y,z$ are: $$ R = \sum_{L = z+1}^{y} \sum_{j \in S_L} x(x-1)^{j}x^{y-L-j}. $$ By subtraction principle, we have $P = x^y - R$, which is to say, the number of sequences of length $y$ from an $x$ letter alphabet with no more than $z$ adjacent repeating letters is: $$ P = x^y - \sum_{L = z+1}^{y} \sum_{j \in S_L} x(x-1)^{j}x^{y-L-j}. $$
Note: this formula agrees with your given example of $x = 2, y = 3, z =2$, as this forces $L = 3$, so that $S_L = 0$ and: $$ P = x^y - x(x-1)^0 x^{y-y-0} = x^y - x. $$
If $z = 1$, you can pick any of the $x$ letters to be first in the sequence. At each step, you have to use a different letter than the one you just used, so you have $x-1$ choices, for a total of
$$x(x-1)^{y-1}$$
options.
When $z \geq 2$, the analysis is complicated by the fact that the blocks are not of equal sizes. So let's find the answer in the case where we know exactly how many blocks of each size there are. Suppose there are:
- $B_1$ blocks of length 1
- $B_2$ blocks of length 2
- $B_3$ blocks of length 3...
and so on, up to $B_z$ blocks of size $z$. Since the total length of the string is $y$, the only restriction on these block sizes is $0 \leq B_i$ for each $1 \leq i \leq z$, and
$$B_1 + 2B_2 + ... + zB_z = y.$$
There are
$$\frac{(B_1 + B_2 + ... + B_z)!}{B_1! B_2! ... B_z!}$$
ways that we can order the blocks within the sequence, and adjacent blocks must use different letters. So we can pick the first block to be any of the $x$ letters, and each new block must use a different letter from the preceding block, of which there are $(x-1)$. Since the total number of blocks is $B_1 + ... + B_z$, we thus have, for a specific sequence of blocks, that there are
$$x(x-1)^{B_1 + ... + B_z - 1}$$
ways to pick the letter for each block. So if I know the block sizes $B_1$ through $B_z$, the number of possible sequences with those block sizes is given by
$$\frac{(B_1 + B_2 + ... + B_z)!}{B_1! B_2! ... B_k!} * x(x-1)^{B_1 + ... + B_z - 1},$$
and the answer to OP's question is then the sum
$$\sum_{B_1 + 2B_2 + ... + zB_z = y} \frac{(B_1 + B_2 + ... + B_z)!}{B_1! B_2! ... B_k!} * x(x-1)^{B_1 + ... + B_z - 1}$$
over all tuples of nonnegative integers $(B_1, ..., B_z)$ satisfying the condition $B_1 + 2B_2 + ... + zB_z = y$.