How to display special German characters "ÄÜÖß" in a map?

try opening the dbase-file with libreoffice (or openoffice) and try different encodings, if utf-8 doesnot work try iso-8859-1 and look if the chars öäüß are displayed correctly

Your shapefile is most probably in UTF-8, but qgis 1.8.0 has a bug in correct encoding of shapefiles. Until this is solved, you can try this workaround in Windows:

Browse to C:\Programs\Quantum GIS Lisboa\bin on Windows XP or C:\Programs (x86)\Quantum GIS Lisboa\bin on Windows 7. Look for qgis.bat and open it with a suitable editor (I have installed notepad++ for such things). After line 1, insert:

SET SHAPE_ENCODING=UTF-8

In newer versions of QGIS you can right click the layer and select properties. In the first tab there is a section about layer information with the options for data source encoding. You can select UTF-8 there. Sorry for the German screen shot but since the question was about German characters...