Maximum memory usable by a 32 bit RHEL 6 system
Well, I do not expect a concise answer than the one available from here.
What I understand about 32-bit OS is, the address is expressed in 32 bits, so at most the OS could use 2^32 = 4GB memory space
The most that the process can address is 4GB. You are potentially confusing memory with address space. A process can have more memory than address space. That is perfectly legal and quite common in video processing and other memory intensive applications. A process can be allocated dozens of GB of memory and swap it into and out of the address space at will. Only 2 GB can go into the user address space at a time.
If you have a four-car garage at your house, you can still own fifty cars. You just can't keep them all in your garage. You have to have auxiliary storage somewhere else to store at least 46 of them; which cars you keep in your garage and which ones you keep in the parking lot down the street is up to you.
Does this mean any 32-bit OS, be it Windows or unix, if the machine has RAM + page file on hard disk more than 4GB, for example 8GB RAM and 20GB page file, there will never be "memory used up"?
Absolutely it does not mean that. A single process could use more memory than that! Again the amount of memory a process uses is almost completely unrelated to the amount of virtual address space a process uses. Just like the number of cars you keep in your garage is completely unrelated to the number of cars you own.
Moreover, two processes can share non-private memory pages. If twenty processes all load the same DLL, the processes all share the memory pages for that code. They don't share virtual memory address space, they share memory.
My point, in case it is not clear, is that you should stop thinking of memory and address space as the same thing, because they're not the same thing at all.
if this 32-bit OS machine has 2GB RAM and 2GB page file, increasing the page file size won't help the performance. Is this true?
You have fifty cars and a four-car garage, and a 100 car parking lot down the street. You increase the size of the parking lot to 200 spots. Do any of your cars get faster as a result of you now having 150 extra parking spaces instead of 50 extra parking spaces?
The answer by Ramesh is completely wrong. A process can't have more memory than address space, and simply because it can't address it !! To use more than 4GB the process needs to access it by the address, like using ponters in C/C++. If you supposedly can have, for instance, 10GB of RAM, how can you tell the process to access the data located at 5GB, if your pointers, in 32-bit, can reach at most 4gb? It's impossible. All his explanation is related to the operating system. It can activate certain blocks of 4GB at most, from a pool of, let's say, 64GB. So there are always concurrently 4GB maximum, and any process can access at most 4GB. Then, for a different process, the SO can activate a different block of 4GB from the pool, but the process again will be limited to 4GB. Actually, even though a 32-bit pointer can address up to 4GB, the limit for a 32-bit process is 3GB.