Custom window style with minimize animation

If you handle the WM_NCCALCSIZE message by returning 0, handle the WM_NCHITTEST message using either your own code (if you want to do manual hit-testing) or also returning 0, and set the WindowStyle to SingleBorder, the window will function like a borderless window but it will have the animations enabled.

If completely necessary, you may also need to handle the WM_GETMINMAXINFO to fix the maximize size - it clips the borders off because the window's style is SingleBorder.

Edited the answer after experimenting a bit.

There are two options: 1. You can change the Style just before minimising and activating the window:

private void Button_OnClick(object sender, RoutedEventArgs e)
{
    //change the WindowStyle to single border just before minimising it
    this.WindowStyle = WindowStyle.SingleBorderWindow;
    this.WindowState = WindowState.Minimized;
}

private void MainWindow_OnActivated(object sender, EventArgs e)
{
    //change the WindowStyle back to None, but only after the Window has been activated
    Dispatcher.BeginInvoke(DispatcherPriority.ApplicationIdle, new Action(() => WindowStyle = WindowStyle.None));
}

This solution has one limitation - it doesn't animate the window if you minimise it from the taskbar.

2. Minimise the Window by sending it WM_SYSCOMMAND message with SC_MINIMIZE parameter and changing the border style by hooking into the message (HwndSource.FromHwnd(m_hWnd).AddHook(WindowProc)).

internal class ApiCodes
{
    public const int SC_RESTORE = 0xF120;
    public const int SC_MINIMIZE = 0xF020;
    public const int WM_SYSCOMMAND = 0x0112;
}

private IntPtr hWnd;

[DllImport("user32.dll")]
public static extern int SendMessage(IntPtr hWnd, int wMsg, IntPtr wParam, IntPtr lParam);


private void Window_Loaded(object sender, RoutedEventArgs e)
{
    hWnd = new WindowInteropHelper(this).Handle;
    HwndSource.FromHwnd(hWnd).AddHook(WindowProc);
}

private void Button_Click(object sender, RoutedEventArgs e)
{
    SendMessage(hWnd, ApiCodes.WM_SYSCOMMAND, new IntPtr(ApiCodes.SC_MINIMIZE), IntPtr.Zero);
}

private IntPtr WindowProc(IntPtr hwnd, int msg, IntPtr wParam, IntPtr lParam, ref bool handled)
{
    if (msg == ApiCodes.WM_SYSCOMMAND)
    {
        if (wParam.ToInt32() == ApiCodes.SC_MINIMIZE)
        {
            WindowStyle = WindowStyle.SingleBorderWindow;
            WindowState = WindowState.Minimized;
            handled = true;
        }
        else if (wParam.ToInt32() == ApiCodes.SC_RESTORE)
        {
            WindowState = WindowState.Normal;
            WindowStyle = WindowStyle.None;
            handled = true;
        }
    }
    return IntPtr.Zero;
}

Neither of the above methods are great, because they are just hacks. The biggest downside is that you can actually see the border reappearing for a moment when you click the button. I'd like to see what others come up with as I don't consider this as a good answer myself.

A newer feature of .NET has solved this problem. Leave your WindowStyle="SingleBorder" or "ThreeDBorder" Leave ResizeMode="CanResize"

Then add this to the xaml inside the

<Window>
  <WindowChrome.WindowChrome>
    <WindowChrome GlassFrameThickness="0" CornerRadius="0" CaptionHeight="0" UseAeroCaptionButtons="False" ResizeBorderThickness="7"/>
  </WindowChrome.WindowChrome>
</Window>

The window will not have any of the default border, but will still allow resizing and will not cover the task bar when maximized. It will also show the minimize animation as before.

EDIT

Unfortunately, when using WindowStyle="None" it still disables the animation and covers the taskbar. So this method does not work if you're trying to make a transparent window.