Startup.cs vs Program.cs in ASP.NET Core 2

Could we remove this class altogether and keep entire configuration in Program.cs ?

Yes

Documentation explains

Convenience methods

To configure services and the request processing pipeline without using a Startup class, call ConfigureServices and Configure convenience methods on the host builder. Multiple calls to ConfigureServices append to one another. If multiple Configure method calls exist, the last Configure call is used.

It is more about the configuring of the builder than the actual Program.cs. That is just the default template class used to hold main entry to the application.

Reference App startup in ASP.NET Core

Program.cs is where the application starts.

Startup.cs is where lot of the configuration happens.

The idea for this separation is based on SOLID principles' first principle- SRP (Single Responsibility Principle). SOLID principles make software designs more understandable, flexible, and maintainable.

Single Responsibility Principle (SRP) states that a class or a method should only do one thing (or should have only one job). If you look at the Startup.cs, it does exactly this making it easy to read and understand the code.