Why aren't interactions between molecules of an ideal gas and walls of container negligible?

One important answer is simply that experimentally ignoring the interaction with the walls is clearly a terrible approximation. If that were true any gas would instantly escape from any container we put it in.

More theoretically, an idea gas does not assume there are no interactions between particles, it assumes that the interactions have 0 range (i.e. the particles have to be in contact) We can apply the same idea to the wall of the container, but we get a very different result, because the wall has a finite cross sectional area, rather than being point like, so the particle will always hit the wall if it travels far enough, but it will almost never hit another particle.

When they say "assume interactions are negligible," they really mean "assume the only interaction is when they elastically bounce off each other". What this really means is you are ignoring any attractive or repulsive force between the molecules, but you are allowing them to bounce off each other like billiard balls. You make the same approximation for the boundary of the box: the molecules bounce elastically off the boundary of the box, but they don't otherwise interact with the boundary.

I'll add that you can also derive the ideal gas law assuming the molecules never even bounce off each other, but this is an unnecessary restriction. You can't derive the ideal gas law by assuming molecules don't bounce off the walls of the box, since then you'd never be able to find the pressure of the gas.

For particles to interact meaningfully, they have to be very close.

If the density is low, the probability that 2 particles will reach interaction distance is small, and thus the effect can be ignored.

The probability that a particle will reach interaction distance from a wall is 100%. If it travels far enough it reaches the wall. The wall is contiguous and dense so the particle will interact with it once it reaches it. And the wall is still there no matter how low the density of the gas is.