Security BY obscurity is horrible. Is security AND obscurity good?
Don't roll your own crypto!
From a purely cryptographic point of view, any length-preserving bijective function cannot reduce security. In fact, even the identity function, defined as f(x) = x, will not reduce security, assuming the keys used for the standard cipher and your homebrew cipher are mutually independent. The only possible way it could reduce security is if your homebrew function does not use a different, independent key and leaks the key in the ciphertext, for example with fk(x) = x ⊕ k done on each individual block of input x, a classic XOR cipher vulnerable to known-plaintext attacks.
From a practical standpoint, there are gotchas that can matter. I mentioned length-preserving above for good reason. A compression function is still a function, and sometimes compression and encryption can lead to very bad results. This is partially why your second example, with your custom algorithm applied before standard encryption, is indeed worse. It can leak information about the plaintext through length. There can also be bugs in your implementation that result in other security vulnerabilities. From a purely theoretical point of view, they are out-of-scope.
It was pointed out to me in a comment that I may not be sufficiently emphasizing just how bad of an idea this is. While it may be fine in theory, the real world works differently. Actually using your own homebrew crypto is a very bad idea, no matter how you use it. The only time you should ever actually do this is if you are a professional cryptographer. Bernstein can do this. Rivest can do this. Rijmen can do this. You cannot. Don't shoot yourself in the foot and instead use proper algorithms.
There is a risk involved when you apply your custom encryption algorithm first.
This is based on the fact that an encryption like AES does leak information about the length of the plaintext.
Suppose the extremely hypothetical (and unpractical) example where you custom algorithm 'encrypts' a single byte plaintext like 0x40 into 64 zeros and a two byte plaintext like 0x02 0x00 into 512 zeros.
When you encrypt this using AES an attacker will still know the length of the custom encrypted result just by looking at the length of the AES encrypted text.
With this information the attacker can 'decrypt' this into the original plaintext without having the AES encryption key.
In summary: a very bad custom algorithm can indeed harm the security of a following AES encryption.
The first question to ask when assessing any sort of security system is "Who is the attacker, and what can they do?" At the low end, you're against an attacker who can just see the encrypted output of one of your messages. At the high end you're up against an attacker who knows the plaintext and ciphertext pairs of hundreds of other messages, who can coerce your computer into encrypting other messages with the same key, who can monitor the power lines to your building and measure the tiny variations in power draw as the encryption process proceeds, and all kinds of similar attacks. Standard encryption algorithms are vetted to make sure they're resistent even to these powerful adversaries.
If you use your own encryption first, even if the second layer is military grade encryption, you risk decreasing your security compared to just using the standard encryption.
For example, suppose that you are encrypting text. You first use a simple substitution cipher and then use your choice of heavy hitting algorithm. Unfortunately, substituion ciphers are inherently reliant on doing array lookups from private information (i.e. your plaintext and your key). This makes them vulnerable to timing attacks: because of the way CPU caches work if two letters in the plaintext are close together they will likely encrypt faster than two letters that are further appart. That doesn't sound like much information, but if an attacker can watch enough encryptions they can potentially work out what the plaintext is without having to crack AES or whatever you're using on top.
If your custom encryption layer does not interact at all with secret information: i.e. it works with data that has already been securely enough encrypted for transmission and doesn't touch the keys of the standard encryption, then it may be safe. At least, it won't be possible to bypass the standard encryption. There are still risks, though, because every extra piece of software carries additional risks. Perhaps a bug in your custom layer lets a remote adversary run arbitrary commands on your machine; then they could just email themselves the plaintext!
The take home message, then, is that having one lump of excellent security code that is well vetted and bug free like a standard encryption function can still be undermined if you put a buggy home grown program (of any sort, encryption or otherwise) onto the same system.