Electric Field within the cavity of a conducting sphere?

ANY point charge in this situation DOES contribute to the field inside the cavity. Negative charges on the interior wall don't contribute to the flux through a surface inside the cavity, which is not to be confused.

Here are more precisions : 1) The misconception you seem to have is a classical one : you look at an equation stating that $A = B$ and you tend to interpret it as "B causes A" when in fact it only says "B equals A". In this case, the flux (surface integral of the E-field) is EQUAL to the interior charges on epsilon does NOT means that the E-field is only caused by the interior charges. In fact, ALL charges contribute to the E-field even though its flux is equal to a multiple of the interior charges.

Here's a counter example that makes this point obvious : consider a long charged wire with charge density $A$. To find the E-field, one would use a cylindrical gaussian surface of length $L$ and assume that the total interior charges are $AL$. Then one would assume that the E-field has the same cylindrical symmetry has the wire, which would simplify the surface integal to E multiplied by the lateral surface $2\pi r L$. Now, part of the wire is obviously outside the gaussian surface. What if we remove all the charges there ? The interior charges would remain the same, but the symmetry would be lost, making it impossible to reduce the integral to $E 2\pi r L$. In fact, part of the flux would now be through the ends of the cylindrical surface. That shows the the "E" in the integral is the resulting E caused by ALL charges.

2) in the specific case of a spherical uniform distribution of charges, with or without a cavity, isolating or conducting, the resulting E-field cancels inside the cavity. This can be shown without Gauss law, using superposition. It was done by Newton for the first time (he considered gravity, but the maths is the same : consider any point inside the cavity and diametrically opposed solid angles radiating from that point. These angles enclose a small patch of charges. the size of the patch is proportional to $r^2$ but the E-field that the patch generates at the considered point goes as $\frac{1}{r^2}$. So the E-field produces by the patch doesn't depend on $r$. That means that patches in opposing directions ALWAYS cause cancelling E-field, wherever the point you're considering is located inside the cavity).

It might help this situation make more intuitive sense to point out that the electric fields produced by all the infinitesimally small sections inside a uniformly-charged circle or sphere yield a net field of zero regardless of position.

In other words, the field cancels itself out; when closer to one side of a positively charged circle or sphere, the forces pushing a positive charge away from that side are stronger, but there are more sections of charge pushing the point charge back into that side.

Complex integration can be done to prove this, but it's not really necessary to go through that as long as it makes sense to you conceptually that this rule could be feasible.