Is it possible to make a circular NSButton?

NSButton is a subclass of NSView, so all methods in NSView, such as drawRect(_:), are also available in NSButton.

So create a new Button.swift which you draw your custom layout

import Cocoa

class Button: NSButton {

    override func drawRect(dirtyRect: NSRect) {
        super.drawRect(dirtyRect)

        // Drawing code here.
        let path = NSBezierPath(ovalIn: dirtyRect)
        NSColor.green.setFill()
        path.fill()
    }

}

Great Tutorial Here. It is iOS , but quite similar!

Don't play around with corner radius. A circle doesn't have corners. To make the button appear as a circle, mask the button's layer to a circle.

You are setting the radius based upon the width of the button. By the look of your screenshot, the button is not a square when you start, so rounding the corners cannot create a circle.