When do we need an Operating System in Embedded System Design?
My rule of thumb is that you should consider an operating system if the product requires one or more of the following: a TCP/IP stack (or other complex networking stack), a complex GUI (perhaps one with GUI objects such as windows and events), or a file system.
If you've done some bare metal coding then you're probably familiar with the super-loop program architecture. If the product's firmware requirements are simple enough to be implemented with a super-loop that is maintainable (and hopefully somewhat extensible) then you probably don't need an operating system.
As the software requirements increase, the super-loop gets more complex. When the software requirements are so many that the super-loop becomes too complex or cannot fulfill the real-time requirements of the system then it is time to consider another architecture.
A RTOS architecture allows you to divide the software requirements into tasks. If done properly, this simplifies the implementation of each task. And with task prioritization an RTOS can make it easier to fulfill real-time requirements. An RTOS is not a panacea, however. An RTOS increases the overall system complexity and opens you up to new types of bugs (such as deadlocks). As an alternative to the RTOS you might consider and event-based state machine architecture (such as QP).
If your product has networking, a complex GUI, and a file-system then you might be at the point where you should consider full featured operating systems such as VxWorks, Windows, or Linux. Full featured operating systems will include drivers for the low-level details and allow you to focus on your application.
It really depends on your definition of an 'embedded system'. There may be some who would claim that if it isn't bare-metal programming, it's not embedded (which precludes your question), but I would disagree with that - I would argue that any system which is designed to perform only one function, i.e., to only run one specific 'application', could be called an embedded system.
That said, it should be fairly easy to imagine situations that would benefit from the services of a full blown OS. For instance, where I work it is quite common to find people building test equipment on top of an instrumentation design suite that runs on top of windows. These systems are configured to boot into the test station configuration and lock out general usage (to prevent corruption of the station) and are arguably therefore embedded systems.
However, just buying off-the-shelf I/O modules, plugging them into a rack mount PC, and whipping up a configuration in a GUI may fail to qualify as embedded system design to some. For a little less off-the-shelf situation, consider a custom process controller with an FPGA, for which you want to do some fancy data logging. You might embed a soft-core processor system (with an existing BSP) and run a realtime linux in order to run a network stack (for your logging and NTP etc) and do everything else in logic.
My (very vague) rule of thumb is if you need more than one thread of control (say at least one device that involves a protocol or a state machine plus something else to do) then some for of OSish software will make your life easier