How to set badge count in Oreo without showing a notification?

Sorry, but there is no SDK-level support for showing numbers or other badges on launcher icons, other than the Notification scenario that you described.

set the importance of the notification channel to

IMPORTANCE_MIN

like int importance = NotificationManager.IMPORTANCE_MIN; and then create the channel as -

NotificationChannel nChannel = new NotificationChannel
                    (channelId, title, importance);

This will the set the badge count(shown on the long press of the icon) without notifying the user about any notification in the system tray. Though the notification will be in the tray, but will not pop up and quietly reside there.