Highlighting a NSMenuItem with a custom view?

If you're adding a view to a menu item, that view has to draw the highlight itself. You don't get that for free, I'm afraid. From the Menu Programming Topics:

A menu item with a view does not draw its title, state, font, or other standard drawing attributes, and assigns drawing responsibility entirely to the view.

Here's a rather less long-winded version of the above. It's worked well for me. (backgroundColour is an ivar.)

- (void)drawRect:(NSRect)rect
{
    if ([[self enclosingMenuItem] isHighlighted]) {
        [[NSColor selectedMenuItemColor] set];
    } else if (backgroundColour) {
        [backgroundColour set];
    }
    NSRectFill(rect);
}

Update for 2019:

class CustomMenuItemView: NSView {
    private var effectView: NSVisualEffectView

    override init(frame: NSRect) {
        effectView = NSVisualEffectView()
        effectView.state = .active
        effectView.material = .selection
        effectView.isEmphasized = true
        effectView.blendingMode = .behindWindow

        super.init(frame: frame)
        addSubview(effectView)
        effectView.frame = bounds
    }

    required init?(coder decoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    override func draw(_ dirtyRect: NSRect) {
        effectView.isHidden = !(enclosingMenuItem?.isHighlighted ?? false)
    }
}

Set one of those to your menuItem.view.

(Credit belongs to Sam Soffes who helped me figure this out and sent me almost that code verbatim.)