When will $1^2+2^2+3^2+5^2+7^2+11^2+\cdots\cdots + p^2$ (where $p$ is prime) be greater than $10^{10}$?

Since you mentioned TakeWhile and FoldWhile, let me first give you a one-line with NestWhile:

NestWhile[With[{c=First[#]+1,s=Last[#]},{c,s+Prime[c]^2}]&,{1,1},#[[2]]<10^10&]
(* {823, 10025552438} *)

What follows is my original answer that shows binary search which is not the most appropriate thing in this situation but still very fast. Consider the vector {a,b,c} and note that the dot-product of it with itself is a^2+b^2+c^2. This naturally leads to a simple function that calculates your sum of prime squares:

f[n_] := #.# &[Prime[Range[n]]]

A quick check shows that an upper bound is n=1000. Now, implement a quick binary search which will converge fast:

search[_, _, lo_, hi_] := {lo + 1, Prime[lo + 1]} /; hi - lo <= 1;
search[f_, goal_, lo_, hi_] := With[{cent = Round[(hi + lo)/2]},
  If[f[cent] > goal,
   search[f, goal, lo, cent],
   search[f, goal, cent, hi]
   ]
  ]

It starts with a lower and upper bound and always divides this range.

search[f, 10^10, 10, 1000] // AbsoluteTiming
(* {0.00287, {823, 6323}} *)

So the answer is $2^2+3^2\ldots+p_{823}^2$. Note, that I violated your definition since I'm only summing real primes and don't include the 1. The answer is still correct though :) Quick check

f[823]
(* 10025552442 *)

and

f[822]
(* 9985572113 *)

seems correct. You should note that it only took exactly 10 steps. You can show this by introducing a step-variable that is incremented.

Is that it or did I make a silly mistake?

n = 0;
sum = 1; (*for 1^1*)
While[
 sum < 10^10,
 sum += Prime[++n]^2
]

sum          (*10025552443*)
n            (*823*)
Prime[n]     (*6323*) 

you can try this:

For[{s = 1, n = 1}, s < 10^10, n++, s = s + Prime[n]^2]; {Prime[n - 1],s}