.NET Core Identity vs IdentityServer4

It depends.

IdentityServer will provide you with OAuth 2.0 and OpenID Connect implementation, and it will handle all details for you (providing you endpoints, token management, scopes, grants and so on). It runs independently so you can use it for multiple clients (SPA, mobile, web apps) and it is nicely isolated from rest of your app. If you wish so, you can use it together with ASP.NET Core Identity.

If you don't use IdentityServer, you will have to write some of these things yourself because ASP.NET Core Identity is a membership system and it does not provide any ready to use endpoints and neither token management or support for different ways how to authorize.

You need to evaluate whether it is better for you to write these things yourself but have a more straightforward setup because you probably don't need everything IdentityServer provides although it might limit you in future.

You can also have a look at OpenIddict that is less complicated than IdentityServer.

You really can't compare the two.

ASP.NET Identity is a database API to manage users, roles, claims, email confirmation tokens etc. Something you can use for implementing signup, login, change password etc.

IdentityServer is an OpenID Connect and OAuth 2.0 implementation. It gives you features like single sign-on and API access control. This is useful if you want to share users between multiple client applications.

You can combine both though - use IdentityServer for the protocol work, and ASP.NET Identity for the user management on your central login page.