Do I have to care about bad boxes?

It is worth noting that TeX doesn't make the value judgements here.

The user, or more likely, the class file on behalf of the user, has set constraints on the amount by which boxes may overflow, the amount of stretching allowed on short pages etc. TeX only warns if these user-set constraints are exceeded.

So if you are setting difficult material with lots of big unbreakable chunks, that means that the constraints are probably wrong. (They were designed mostly for copy that is mostly text that can be hyphenated and with enough interword stretch to get tight fitting boxes.)

Rather than TeX trying to meet un-achievable constraints and then complaining about failing to meet them, it is better to modify (relax) the constraints. But before doing that you should make sure that they really are un-achievable and it is not just user error preventing tex from finding a good layout.

Some general notes added as requested:

Settings that only affect warning messages.

There are several setting that only affect the messages that TeX sends out. For example if you know that some boxes will be overfull by 2pt (and you don't mind this) but you want to be warned for a box that is 1cm too full then set

\hfuzz=2pt

this will make no difference to the typesetting, but will suppress warnings for boxes that are only slightly overfull.

\vfuzz

is the same thing for vertical boxes.

Similar are \hbadness and vbadness which are a measure of how bad a box is, typically how much white space has had to be stretched. the exact number is not usually that relevant but 0 is good and 10000 is infinitely bad (TeX's badness calculation arbitrarily forces any very bad boxes to this amount).

The parameter \tolerance is what tells Tex how much stretching should be allowed. TeX tries to line-break a paragraph in such a way as to keep the badness below the specified tolerance.

Good typographic quality would indicate that you shouldn't ignore (or turn off) these warnings, However sometimes it makes sense. The PDF version of the MathML recommendation for example is set automatically by LaTeX from sources primarily designed for the normative HTML version. Because it has many one or two line paragraphs and many indented tables and examples, the right hand margin is very uneven anyway and so allowing the occasional overlarge example to protrude isn't so bad, it is set with

\hfuzz=20pt
\vfuzz=20pt
\hbadness=2000
\vbadness=\maxdimen

Settings that don't affect the typesetting of good boxes

TeX3 introduced a parameter \emergencystretch which allows extra stretch to be more uniformly added to a paragraph if the default stretching does not produce a good break. Setting this to a non zero value can sometimes help.

Settings that do affect the typesetting.

LaTeX has a command \sloppy that sets up looser typesetting

\def\sloppy{%
  \tolerance 9999%
  \emergencystretch 3em%
  \hfuzz .5\p@
  \vfuzz\hfuzz}

The setting of tolerance makes it rather more sloppy than is perhaps desirable, and it is probably worth experimenting with just setting \emergencystretch.

The default fussy settings used by LaTeX are

\def\fussy{%
  \emergencystretch\z@
  \tolerance 200%
  \hfuzz .1\p@
  \vfuzz\hfuzz}

For vertical page breaking, if you are getting underful boxes in the output routine you may prefer \raggedbottom rather than \flushbottom as this adds stretchable space at the end of each page so no page will be underful (but short pages will be silently accepted). It is possible of course to have something between these two extremes, that adds glue that only extends a finite amount rather than fill glue.

The above are the main global settings however every aspect of the document design affects this. In particular, if the space between display elements such as lists and displayed maths, and the space between paragraphs, is stretchy then it it is much easier for TeX fo find a "good" page break, however this means that there is no vertical alignment between lines on different pages. especially for book designs using two page spreads, designers often aim to keep text on a grid where text always aligns on fixed positions. This usually means that you have to make all the vertical spaces much more rigid, which increases the chance of Tex giving warnings about over or under-full vboxes.

Yes, you should definitely take care of them.

1. Overfull \hbox messages tell you that some line sticks out over the right margin;

2. Underfull \hbox messages tell you that some line is poorly typeset (or that you've improperly used \\ to leave a vertical space (for example, typing two \\ in a row);

3. Underfull \vbox messages usually tell you that a page is poorly typeset.

How much badness requires action is difficult to say, but I would never accept a final PDF where there's some Overfull \hbox.