Noise (capacitance-related?) in serial signal
So, I followed the advice given by the two kind people that commented... Here are the results.
Ali Chen indicated the slow edges can be attributed to the capacitance of the RCA cable; and "Anonymous" recommended attaching directly to the board with a jack with no wires. I followed their advice, stripped the tablet down to expose the PCB, plugged a naked jack and probed it - but the results were unfortunately the same: very slow, clearly-capacitance-driven edges. It wasn't the RCA wires - instead, it appears that whoever designed the tablet didn't care much for the serial signal coming out from the headphone jack (probably used some other way to interface with the board). I did try probing all over the PCB in hopes of finding a cleaner serial signal, but I failed.
Anonymous also recommended lessening the baud rate; unfortunately, there is no documented way to influence the boot process of my tablet so as to configure the baud rate used during u-boot (which is what I was interested in)...
But it is possible to do so AFTER the boot is completed, from within an ADB shell - since I've managed to compile my own kernel and become root.
So I was able to do this...
$ su
# stty -F /dev/ttyHSL0 9600
# while true ; do echo UUUUUUU > /dev/ttyHSL0 ; sleep 0.1 ; done
And indeed, the result is much nicer:
I am pretty sure this signal can be decoded fine, if I use a shifter (it's at 1.8V, so my 3.3V USB-TTL still can't decode it).
So, to conclude: my tablet's "serial port inside the headphone jack" can only really be used AFTER the boot is completed, and the UART slowed down to 9600 baud; which is unfortunate, since the serial output is needed most during the boot process (if something fails, that is) - and during that time, the UART speed is hardcoded in my tablet's boot code at 115200 baud.
P.S. I also tried a suggestion from a friend, to use a 3.3K pull-up towards the 3.3V rail in the serial signal sent out by the headphone jack - to no avail.
UPDATE, 3 days later
I persevered :-)
Following the advice from Chris Stratton - that a good shifter can cope even with this kind of signal - I bought a soldering iron, a BSS138, a breadboard and a bunch of cables. After what is probably the worst soldering job EVER, I managed to solder the pin headers on the BSS138, and then proceeded to attach it to the breadboard, and create this tangled mess:
What I didn't expect, was that after spawning minicom and issuing a "fastboot reboot", to my utter amazement, I saw this:
Unbelievable - after BSS138 "lifts" the signal from 1.8 up to 3.3V, that miserable, capacitance-riddled signal can actually be decoded! I can finally see why my tablet isn't booting.
Hello, little tablet - I OWN you now :-)