MySQL: How do I get my "Maximum possible memory usage" down?
You have a server with 256M, but you can't use all of that -- remember there's some OS overhead. Add to that with the fact you're over committing as other folks have mentioned and you'll definitely thrash here. 256M is only enough for a small DB, 20 connections is a lot with what you've got configured.
1) reduce your max connections to 4 (you're using 3 out of 20)
2) optimize your query cache better; 8M is really large, and 64M total is a lot based on your hits/prunes; try a 4/32 combo and see how it goes. Really I think a 2/24 combo would work for you.
3) you have no sorts requiring temp tables, why is that max_heap_table_size verb in there? Comment that out, use the defaults
4) do you actually have 128 tables? Try cutting that table_cache in half to 64 or 48
5) reduce thread_cache_size to 4
6) optimize those tables to reduce fragmenting
Those are some things to start with. It looks like you threw a bunch of numbers in a config without any actual profiling to know what you needed and have created a mess; if all else fails go back to the defaults and get rid of your custom settings and start over using some performance tuning guides you can find on Google. Get the output of SHOW VARIABLES and SHOW STATUS, find any one of a bajillion tuning guides and plug in your actual, real numbers into their equations and that'll tell you the exact-ish numbers you need to put in your config file.
I'm not a MySQL guru and I can't diagnose the problem with this information, but I tried searching for the formula in the source code. Here it is:
server_buffers + total_per_thread_buffers * max_connections
server_buffers = key_buffer_size + innodb_buffer_pool_size + innodb_additional_mem_pool_size + innodb_log_buffer_size + query_cache_size
total_per_thread_buffers = read_buffer_size + read_rnd_buffer_size + sort_buffer_size + thread_stack + max_allowed_packet + join_buffer_size
Now you have to check each of these values and figure out which one is responsible for this huge number. And don't trust this script unreservedly - I tried running it on one of my DB servers and it calculated that the maximum memory is 140% of the total amount of physical memory, but the system has been running for years without any stability issues.