How to mail-enable an existing Azure AD security group?

There is no way to mail-enable an existing security group in AzureAD. you would need to recreate the group as mail-enabled through the Office365 Portal. Exchange ECP, or PowerShell. The latter can help you transfer settings and members.

If the security group is dynamic, you would lose this capability when converting. You can have a dynamic distro group (but it's not security group), or dynamic security group (that is not a distro group).