How do I use a subclass of NSDocumentController in XCode 4?

You can subclass easily in Swift 3:

class AppDelegate: NSObject {
    let docController = DocController()
}

class DocController: NSDocumentController {
}

In your application delegate:

// LukeAppDelegate.h
#import "LukeAppDelegate.h"
#import "VRDocumentController"

- (void)applicationWillFinishLaunching:(NSNotification *)notification {
    VRDocumentController *dc = [[VRDocumentController alloc] init];
}

This will make sure that an instance of VRDocumentController is created and registered as the shared document controller, preventing Cocoa from using the default NSDocumentController.

As to why you haven’t been able to use a custom object in your nib file, make sure that that you select Object (blue cube) instead of Object Controller (blue cube inside a green sphere) when dragging a new object into the nib file.

---

Edit: If you’re targeting an OS X version that supports restoration, -applicationWillFinishLaunching: may be too late to register a custom document controller. If the application delegate is placed inside MainMenu.xib, it should be instantiated by the nib loading process before any documents are restored, hence you can move the NSDocumentController subclass initialisation to the application delegate’s init method:

// LukeAppDelegate.h
#import "LukeAppDelegate.h"
#import "VRDocumentController"

- (id)init {
    self = [super init];
    VRDocumentController *dc = [[VRDocumentController alloc] init];
    return self;
}

The answer marked correct (@Bavarious'answer) fails to work on document-based apps because documents often load before any applicationWillFinishLaunching: is called. After reading the helpful clues in @Graham Perks answer, I tried a few different approaches and this seems to work reliably:

@implementation AppDelegate

- (id)init
{
    self = [super init];
    if (self) {
        MyDocumentController *dc = [[MyDocumentController alloc] init];
        if (dc) {};
    }
    return self;
}

Note Obviously (if not already created) you'll need to create an AppDelegate yourself first and link it in your MainMenu.XIB.

Easier method of getting the Custom Document Controller created in time is to implement the + (void) load method:

This is literally all you need to do it implement a custom document controller.

@implementation AppDocumentController

+ (void) load
{
    [AppDocumentController new];
}

// ... your overriding goes here

@end

This is cool because there's no separation of the codez. Not 100% sure if the controller is created too early, but works well for me.