Why is compilation OK, when I use Invoke method, and not OK when I return Func<int,int> directly?

There are two things you need to know to understand this behaviour.

1. All delegates derive from System.Delegate, but different delegates have different types and therefore cannot be assigned to each other.
2. The C# language provides special handling for assigning a method or lambda to a delegate.

Because different delegates have different types, that means you can't assign a delegate of one type to another.

For example, given:

delegate void test1(int i);
delegate void test2(int i);

Then:

test1 a = Console.WriteLine; // Using special delegate initialisation handling.
test2 b = a;                 // Using normal assignment, therefore does not compile.

The first line above compiles OK because it is using the special handling for assigning a lambda or a method to a delegate.

In fact, this line is effectively rewritten like this by the compiler:

test1 a = new test1(Console.WriteLine);

The second line above does not compile because it is trying to assign an instance of one type to another incompatible type.

As far at the types go, there is no compatible assignment between test1 and test2 because they are different types.

If it helps to think about it, consider this class hierarchy:

class Base
{
}

class Test1 : Base
{
}

class Test2 : Base
{
}

The following code will NOT compile, even though Test1 and Test2 derive from the same base class:

Test1 test1 = new Test1();
Test2 test2 = test1; // Compile error.

This explains why you can't assign one delegate type to another. That's just the normal C# language.

However, the crucial thing is to understand why you're allowed to assign a method or lambda to a compatible delegate. As noted above, this is part of the C# language support for delegates.

So finally to answer your question:

When you use Invoke() you are assigning a METHOD call to the delegate using the special C# language handling for assigning methods or lambdas to a delegate rather than trying to assign an incompatible type - hence it compiles OK.

To be completely clear, the code which compiles in your OP:

public test Success()
{
    Func<int, int> f = x => x;
    return f.Invoke; // <- code successfully compiled 
}

Is actually converted conceptually to something like:

public test Success()
{
    Func<int, int> f = x => x;
    return new test(f.Invoke);
}

Whereas the failing code is attempting to assign between two incompatible types:

public test Fail()
{
    Func<int, int> f = x => x;
    return f; // Attempting to assign one delegate type to another: Fails
}

In the second case, f is of type Func<int, int>, but the method is said to return a test. These are unrelated (delegate) types, that are unconvertible to each other, so a compiler error occurs. You can go to this section of the language spec, and search for "delegate". You will find no mention of conversions between delegates that have the same signatures.

In the first case however, f.Invoke is a method group expression, which doesn't actually have a type. The C# compiler will convert method group expressions to specific delegate types according to the context, through a method group conversion.

(Quoting the 5th bullet here, emphasis mine)

An expression is classified as one of the following:

• ...

• A method group, which is a set of overloaded methods resulting from a member lookup. [...] A method group is permitted in an invocation_expression, a delegate_creation_expression and as the left hand side of an is operator, and can be implicitly converted to a compatible delegate type.

In this case, it is converted to the test delegate type.

In other words, return f doesn't work because f already has a type, but f.Invoke doesn't have a type yet.

Issue out here is Type compatibility:

Following is the definition of Func delegate from MSDN Sources:

public delegate TResult Func<in T, out TResult>(T arg);

If you see there's no direct relation between the Func mentioned above and your defined Delegate:

public delegate int test(int i);

Why 1st snippet compiles:

public test Success()
{
    Func<int, int> f = x => x;
    return f.Invoke; // <- code successfully compiled 
 }

Delegates are compared using signature, which is input parameters and Output result, ultimately a Delegate is a Function pointer and two functions can be compared only via signature. At runtime the method invoked via Func is assigned to the Test delegate, since Signature is same it works seamlessly. It's a function pointer assignment, where Test delegate will now invoke the method pointed by the Func delegate

Why 2nd Snippet fails to compile

Between Func and test delegate, there's no type / assignment compatibility, Func cannot fill in as part of the Type system rules. Even when its result can be assigned and filled in test delegate as done in the first case.