Are You Weak at Level 80?
Although you're correct in your statement that you won't get as much of a benefit from gaining the last 30-50 levels as you would early on, when you're leveling "core" skills to your character, there are a few other things to consider.
One is that there is a maximum level for your enemies:
Enemy types also seem to reach a plateau where they stop getting stronger. The strongest bandits (non-boss) are mid-20s. The strongest generic vampire is 54, and guards seem to stop scaling at 50. This implies that the difficulty of many areas will not increase beyond certain levels, except perhaps in frequency of difficult encounters.
This means that past about the mid-50's, you can expect monster difficulty to plateau. There are a few exceptions, such as Magic Anomalies and Legendary Dragons (in Dawnguard), however.
At the same time, you're still going to be gaining health, magicka, stamina, and perk points. If you spend these level upgrades wisely, there's no reason why you can't be gaining some advantage, while most of your enemies are gaining none.
Another factor that occurs to me is that gear is important - depending on at what point you max these trees, you might get a significant boost from alchemy, enchanting, and blacksmithing. If you hadn't invested in crafting skills by the time you were in the 30's or 40's, you've still got significant gains to be gotten from making/improving your own equipment.
I have played Skyrim many times over and all my characters reached at least level 60 (one of them got 75 i think).
I always play on master difficulty. The main way i stayed alive with all characters was enchanting. You choose an attack skill (One Handed, Archery or Two Handed) then enchant four pieces (depending on what skill you use) with X% more damage. With four pieces, you gain a 160% damage increase with those weapons. I also tend to imbue the items with some form of elemental or magic resistance as well to stop getting smashed at higher levels by mages.
If you prefer to play as a mage, just enchant your gear with X% spell cost reduction. One of my characters used Alteration and Destruction. I had four pieces of gear each with 25% reduction costs for both schools of magic. I could repeatedly cast any Alteration (Dragonhide, Paralyze, Mass Paralyze etc.) and any Destruction spell for free. The Master destruction spells can be particularly nasty as they can be held down for a constant shit storm upon the enemy. Combine those with Mass Paralyze and the reckoning is at hand.
Armour can get you the maximum 85% damage reduction Stones, perks and enchantments can get you the maximum 85% magic or elemental resistance
Never underestimate the value and power of your crafting skills. Alchemy can be great as you can get almost all magically effects from potions (Restore Health, Restore Stamina Invisibility, Paralyze etc.). Smithing is excellent as you can make your weapons and armour legendary for HUGE damage and defense increases. Enchanting is (in my opinion) essential. Play around with your own crafting and enchanting recipes and see what you come up with.
Another tip is if you like to play as a Vampire get the Restoration perk "Necromage" it makes all your enchantments stronger. Instead of 160% more damage it goes upwards of 200%.
Dont stress about being a high level, you really can become the most incredible force in all of Skyrim. You just need to know how and its really quite daunting not knowing where to look for info or how to make a supreme being.
There are some very helpful sites around and places like this that can help with your confusion. Always ask questions mate, you can only gain knowledge by asking or searching!
Hope this helps on your quests my friend! Goodluck and just have fun man, dont stress about levels just get out there with some sweet gear and kick some arse!