Sharepoint - When should we separate a site collection into a different web application
As far as I know or according to my experience you should be creating Site Collections for departments and let each department have there own content database as a best practice.
Web applications should be created on the basis of mode of authentication.
I would recommend you to go through these topology diagrams provided by Microsoft to understand the concept.
e.g.
While the scope of your question is really too broad and many answers can fit to suit as needed, I would share a starting point specific to determine the need of site collection vs web application.
SharePoint Factors : As thanby added , A web application is generally a high level abstraction in the "arrangement of your portal sites". Generally they are defined to host large number of site collections. Web Application typically runs on their dedicated application pools and they should be really limited to as minimum as you can get. The reason is application pools are memory hungry and you certainly don't want your WFEs ending up consuming large amounts of RAM. So if you have 20 departments , you should not end up having 20 application pools. Thats the general idea.
Site Collections as the name says are designed to host large number of sites. Your sites are designed to host large number of documents and items. Said so , all content ultimately resides in a content database which is the most critical factor that comes into consideration for scaling up portal sites in SharePoint. So the question really is do you need separate content database per site collection or a single content database for all site collection.
Typically, 1 single web application and multiple large site collections mapped to their dedicated content database should suffice. Small set of site collections are mapped to a single content database.
Sizing factors : Quick set of questions to decide whether you need a site collection for a department with a dedicated content database or single content database serving multiple department sites.
- No of users per department
- Estimate the volume documents to be uploaded and download per department
- Check for requirements pertaining to search , excel services , ECM and managed metadata.
- You may decide for an extended web application in case there is a need to connect partners to the same content using different authentication mechanisms.
Further you must read through the capacity planning section here to get some decent insights on this subject:-
SQL IO Optimization and Sizing for SharePoint 2010
In general it is advisable to keep it as simple as possible. You'll only want to split your content across multiple web apps if security is a huge concern or if you have enough serious traffic that you'll want to load-balance. For performance, the best way to load balance it to start by splitting administrative tasks like timer jobs and search indexing onto another web app, and even better is if you can put that on a whole separate server to keep your web front-end as light and responsive as possible.
Each web app takes up dedicated resources, even if there are no users on it. They all have default overhead requirements that cannot be turned on and off as needed. Typically, if you have a single server farm, you'll want to keep all your content together for ease of use and efficiency. If you have two servers, again the best way to work it is to put the overhead jobs on another web app on the other server, and still keep all your content on one app on the web front-end.
Did that help answer your question?
Edit: Further reading is here if you haven't seen that one already.