Custom Linux GUI: Where to begin?

You want to build your own DE (desktop environment). Common examples are GNOME and KDE, although more of them were popular. It actually used to be very popular to build your own DE, and dedicated scripting tools were often created.

You will usually need to combine multiple tools: a window manager, a toolbar program, a desktop manager (icons), session manager, possibly more. Out of that only a window manager is a required part (it usually doesn't make any sense not to run any WM inside X server), and others are optional. You will usually not write these tools from scratch (it is a lot of work), but use already available components.

Note that both GNOME and KDE actually consists of these elements (well integrated). For example GNOME consists of a window manager (metacity), toolbars (gnome-panel), desktop manager (nautilus) and so on. You can change one element into another if you want: it used to be popular to use sawfish as a window manager instead of metacity, keeping all the other elements intact. Now it is popular to use compiz instead of metacity.

I used to use FVWM, which is a window manager with an additional (quite good) scripting capabilities that could be used to create toolbars, menus and other things. One of nice examples is FVWM-Crystal, which is a complete DE built on FVWM and few other tools, written in a mix of FVWM scripting and Python. You could actually change any kind of behaviour at runtime by just opening FVWM's scripting console and typing commands. FVWM-Crystal can be a good starting point if you want to start from something already usable; bare FVWM is good if you want to build your DE from scratch.

[UPDATE: it seems that gnome-shell is very similar in this regard to FVWM. Most of its behavior is scripted using JavaScript (instead of a custom language in FVWM), and is very easy to change. It is also very actively developed, as it is an important part of GNOME3, with very good support for composition. It's a hacker's heaven ;-)]

[UPDATE2: so it seems I was right on gnome-shell. There's a Cinnamon, which uses the same framework (mutter window manager) as gnome-shell, yet builds a desktop which works in a different way than gnome-shell.]

There was a website that was a good source of ideas to implement: lynucs.org. It is down for two years now, but it got archived.

Warning: lots of these components are not maintained any more, especially since GNOME/KDE started to become actually usable. Development in this area is usually driven more by specific requirements of embedded devices: phones or netbooks.

Now, to answer your actual questions:

• Linux/OS flaver: anything easily customizable: Debian, Arch, Gentoo, LFS...
• Tools/frameworks: it depends on what do you want to achieve. If you want to write a custom toolbar you might just use some scripting languages (like FVWM's), custom tools like adesklets, or write your own in Qt (if most of your other components is in Qt) or GTK (if most of your other components is in GTK).
• Personal UIs: there were lots of them, and sites like lynucs.org was a nice compilation of them (lynucs listed components used in each desktop).
• Complex transparency effects: older tools usually do not handle it natively, or do a simple things like just reusing part of desktop wallpaper as its own background (so-called fake transparency). Most probably you'll need to write your own code, f.e. as a plugin to compiz.

If you fast and cool UI I would suggest to look into Enlightenment. Its older than Gnome and I found it extreme fast and low resource consuming option.

I'd probably start with an Ubuntu install and the OpenBox window manager and build up from there.

OpenBox is very minimal but you can add applications (eg Docky, Awn, etc) to provide desktop interface elements or write your own.