LibraryUnload fails

Todd Gayley once mentioned that LibraryLoad is for pre-loading depend libraries you need. So for instance, if you write a function which needs some numeric-library, you can load this before calling the functions in your library.

You don't need to pre-load the WolframLibrary library you have developed because you implicitly load it by loading the functions in it.

Therefore, I believe the solution to your problem is that you have to call LibraryFunctionLoad on one of the functions you are developing and after that LibraryUnload should work as expected.

Furthermore, you should unload all functions with LibraryFunctionUnload before unloading the library.

(Information worth sharing for Windows users despite the linux tag in original question.)

As this comment mentioned, there is an additional problem on Windows platform, which I will try to describe here, as long as a solution to it.

The LibraryUnload failure problem on Windows

Suppose one has a "standalone" library myLib.dll, which has nothing to do with Mathematica or LibraryLink. Then one has a libLink.dll, which uses myLib and provides functions to Mathematica through LibraryLink.

As libLink depends on myLib, one may try to pre-load myLib before loading libLink, and it works with no problem:

LibraryLoad["myLib.dll"]
myfunc = LibraryFunctionLoad["libLink.dll", ... ]

But then, on Windows, if one tries to unload them with

LibraryUnload["libLink.dll"]
LibraryUnload["myLib.dll"]

The former one will succeed but the latter one will fail with an error message

LibraryFunction::unloadlib: The library myLib.dll cannot be unloaded.

Using tools such as Process Explorer one can see the file's handle has not been released by the kernel.

The solution

The described problem can be frustrating when one is developing myLib. One has to kill the kernel each time before re-compiling myLib. The solution, which was found and kindly shared to me by ilian (who has very generously reviewed my code and looked into the problem.), is

NOT to manually pre-load myLib with LibraryLoad at first place.

Instead, one may modify the OS environment so Windows can find all the depended libraries and manage them automatically:

SetEnvironment["PATH" -> Environment["PATH"] <> ";" <> "path_to_folder_of_myLib.dll"]
myfunc = LibraryFunctionLoad["libLink.dll", ... ]

Unloading will now be as simple as:

LibraryUnload["libLink.dll"]

After that, if there is no other reference to myLib.dll, it will be editable.