What is known about this cohomology operation?

I'm going to refer to $H$ as the cup-$1$ product.

So when $p = |x|$ is odd and $x$ is a cycle, the boundary formula says $$d(x \cup_1 x) = 2x^2$$ which is not necessarily zero. In this case, we don't get a cohomology operation unless (e.g.) when $2 = 0$ in $R$, and then the cohomology operation is "mostly like $Sq^{p-1}$".

When $p$ is even, then $d(x \cup_1 x) = 0$ and so we do get a cohomology operation (once we verify that it's well-defined, of course). However, in this case the next product (the cup-2 product) has an identity $$ d(x \cup_2 x) = 2 x\cup_1 x. $$ Here are some consequences.

• If 2 is invertible in the ring $R$, then $x \cup_1 x$ is automatically zero in cohomology.

• If $R = \Bbb Z$, then $x \cup_2 x$ is a cochain which reduces, mod 2, to $Sq^{p-2} r(x)$ (the Steenrod square on the mod-2 reduction $r(x)$). However, the integral Bockstein $\beta$ of $Sq^{p-2} r(x)$ is calculated by taking a cocycle representative over $\Bbb Z/2$, lifting it to a cochain representative over $\Bbb Z$, taking the coboundary, and dividing by $2$. This means that $x \cup_1 x$ is a representative for $\beta Sq^{p-2} r(x)$. (Note that if we then mod-2 reduce, we see it become $Sq^1 Sq^{p-2} r(x) = Sq^{p-1} r(x)$ because $p$ is even.)

• If $R = \Bbb Z/4$, I believe that a similar calculation shows that $x \cup_1 x$ is a representative for $\beta' Sq^{p-2} r(x)$ where $r$ is reduction mod $2$ and $\beta'$ is the Bockstein associated to the exact sequence $$0 \to \Bbb Z/4 \to \Bbb Z/8 \to \Bbb Z/2 \to 0.$$ However, in my time available I can only make this work if $x$ lifts to mod-$8$ cohomology, and so there's possibly a correction factor. EDIT: This computation works, but you need to use a cup-3 product to show that there are no extra correction factors.

More information on the cup-$i$ products is available in the McClure-Smith's paper (http://arxiv.org/abs/math/0106024) referenced in Henrik Rüping's answer, and in Mosher and Tangora's book.

Here are some examples. And since this really just adresses the computation part of the question and not what that map is, it is really just a partial answer.

LEt me first describe the sequence operad in words (and with $\mathbb{F}_2$-coefficients to ignore signs - makes everything much simpler). For any surjection $f:\{1,\ldots,n+k\}\rightarrow \{1,\ldots,n\}$ there is associated a natural $\langle f\rangle: C^*(X)^{\otimes n}\rightarrow C^{*-k}(X)$. I will use the abbreviation $f=(f(1),f(2),f(3))$ and further I will leave out the commas and brackets (If $n\ge 10$ things would be too ugly anyway), so $121$ really denotes a surjection $\{1,2,3\}\rightarrow \{1,2\}$.

Further $C^*(X)$ could either denote the singular cochain complex of $X$ or the normalized simplicial cochain complex of a simplicial set. I will work with the latter. We have $\langle 121\rangle:C^*(X)^{\otimes 2}\rightarrow C^{*-1}(X)$ $$\langle 121\rangle (\Psi_1\otimes \Psi_2)(\sigma:\Delta^n\rightarrow X):= \sum_{0\le i_1\le i_2\le n} \pm \Psi_1(\sigma|_{0,\ldots,i_1,i_2,\ldots,n})\cdot \Psi_2(\sigma|_{i_1,\ldots,i_2}).$$

In words, we look at all ways to divide $0,\ldots,n$ into 3 parts, we plug in the first and third into $\Psi_1$ and the second into $\Psi_2$.

Just a reminder that those maps are NOT chain maps. Here are some reminders/conventions: If the map $f$ is not surjective, then $\langle f\rangle $ is zero. The same holds if a symbol occurs twice in a row (like $112$) since then the simplex that we would feed into $\Psi_1$ is automatically degenerate (it contains $\ldots,i_1,i_1,\ldots$) etc.

The boundary of $\langle f\rangle$ viewed as an element in $Hom(C^*(X),C^*(X))$ can be computed the following way. Look at all valid ways to delete a number in $f$ and take the alternating sum. So for example $$d(\langle 12123\rangle) =\pm \langle 2123\rangle \pm \langle 1213\rangle$$; we can only delete the leftmost 1 or the rightmost 2; everything else would result in either a repitition or a missing number.

Similarly there is a composition formula. Let us compute $\langle 121 \rangle\circ(\langle 12\rangle \otimes \langle 1\rangle)$. Note that $\langle 1\rangle$ is just the identity so we really plug in something in the first coordinate. That composition takes three arguments $\Psi_1\otimes \Psi_2\otimes \Psi_3$, the first two get plugged in into $12$ and that result and $\Psi_3$ are then plugged in into 121. So we have to rename the 2 in 121 into three and do something with 12 in the other spots. So first we end up with $\_3\_$ and then we have to shuffle 12 in the empty spots, meaning that the point where we switch from 1 to two could either lie on the left or on the right of 3, so we end up with $$\langle 121 \rangle\circ(\langle 12\rangle \otimes \langle 1\rangle) = \langle 1312\rangle \pm \langle 1232\rangle. $$ Recall that $\langle 12\rangle$ is just the cup product. If we like we could now write each summand again as a different composition : Let us retranslate that computation back to more familiar notation. This equation really means that $$(\Psi_1\Psi_2)\cup_1\Psi_3 = (\Psi_1\cup_1\Psi_3)\Psi_2 \pm \Psi_1(\Psi_2\cup_1\Psi_3).$$ Coincidencially we have now shown that $\langle 121\rangle (\_\otimes \Psi_3)$ is a derivation. (Remarkably $\Psi_3\cup_1\_ $ is not!).

Now if we want we can turn all those maps into a chain operad; let us define the module of $n$-ary operations that shift by $k$ as the free $R$-module generated by all surjections without repetitions $\{1,\ldots,n+k\}\rightarrow \{1,\ldots,n\}$. Then the formula for the boundary turns the $n$-ary operations into a chain complex and the composition formula gives us the composition in that operad.

Another small remark is that $d(\langle 121\rangle)=\langle 12\rangle \pm \langle 21\rangle$, so we see that $\langle 121\rangle$ has the defining property of the $\cup_1$-Product.

So let us now see how this operad can be used in computations. The problem is now that the Cartan relations are quite powerful. For this I need a nice, small space given as the geometric realization of a simplicial set and then we can compute things. Otherwise we could use the observation that $\langle 121\rangle:C^1(X)^{\otimes 2}\rightarrow C^1(X)$ is just the pointwise multplication of two cochains (see Tyler Lawsons comment on my question DGAs with pointwise Multiplication). So to really have a look at an example, let us consider $$ H^1 (\mathbb{R}P^\infty;\mathbb{Z}/4)\rightarrow H^1 (\mathbb{R}P^\infty;\mathbb{Z}/4).$$ Since that space is a model for $B\mathbb{Z}/2$, we can use the bar construction. So that space is really the realization of a simplicial set. $C^1(BG)$ consists of the set theoretic maps from $G\setminus \{e\}$ to $\mathbb{Z}/4$. And a cochain is a cocycle iff it is a group homomorphism. Thus the nontrivial element of $H^1$ is given by the map that sends the nontrivial group element in $\mathbb{Z}/2$ to $2$. This cycle is of the form $2c$ where $c$ is some chain (and not a cycle!).

Thus we get $$\langle 121\rangle (2c\otimes 2c) = 4 \langle 121\rangle (c\otimes c)=0$$ and thus that operation is zero.

So morally the fact that $Sq^0$ is the identity with prime coefficients has something to do with Fermats little theorem $(x^p-x =0 mod p)$ and it "fails" with $\mathbb{Z/4}$-coefficients as seen in this example.

It seems like one really did not need the sequence operad for that example, but If you would like to compute what happens in bigger degrees it should be useful. (Also then you would have to keep track of signs....)