Avoid just nearly filled last lines

Adding a \parfillskip that has a minimum length and infinite stretchability seems like it would do it.

For example, the below sets parfillskip such that the end of the paragraph must have AT LEAST whitespace equal to the minimum of 2 x characters:

\documentclass{article}
\usepackage[paperwidth=3in,paperheight=3in]{geometry}
\begin{document}
\thispagestyle{empty}
%% Measure the width of 'x' in this font
%% and set \parfillskip equal to two times
%% that width.
\setbox0=\hbox{x} \parfillskip=2\wd0 plus1fil 

asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
sasdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdfxx 
\end{document}

Commenting out the \parfillskip=2\wd0 plus1fil line gives:

But it shows this when parfillskip is set:

I'm not sure I'm serious here, but if you are able (and or willing) to retypeset your paragaphs by collecting them first then there is one feature of TeX that would allow you to measure this and act upon it: TeX knows about the length of the last line of a paragraph (or partial paragraph to be precise) if that is followed by a math display. In that case the length of the line above the display is known within the formula as \predisplaysize. Thus you could

• collect the paragraph material in a macro, say
• trial typeset it and check out how close its last line is to the full line
• depending on the result retypeset the paragraph in earnest using a suitable value of \parfillskip, e.g., set it to 0pt or set it to 2em plus 1fil or whatever.

Here is the code to check it out:

\newdimen\mydim
\def\checkit{%
  \mydim\hsize
  \advance\mydim by -\predisplaysize
  \advance\mydim by 2em
  \typeout{There is \the\mydim\space  space available on the last line}%
}


asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdfxx 
$$\checkit$$

The 2em are substracted because the \predisplaysize is the length of the last line + 2em. For your example that gives us:

There is 4.98819pt space available on the last line

There are some special conditions: if the last line is not typeset at its natural width then the size given as \maxdimen, e.g., if \parfillskipwas set to 0pt, and if there was no previous line then it is set to -\maxdimen but both cases could be controlled in your situation.

This can be done in LuaTeX using the pre_linebreak_filter callback: Before the "normal" linebreaking, do a "trial" linebreak and measure the effective parfill. If it is below the current value of \parindent, set the parfill for this paragraph to zero. Additionally, it can sometimes happen that \parfillskip=0pt fails. In this case this will stretch the last line without affecting the remaining paragraph, leading to a potentially very stretched line. If you prefer to just accept a nearly filled line in this case, just delete the part surrounded by

luatexbase.add_to_callback("post_linebreak_filter", function(head)
  ...
end, "drop_short_parfill")

\documentclass{article}
\usepackage[paperwidth=3in,paperheight=3in]{geometry}
\usepackage{luacode}
\begin{luacode*}
  local glue = node.id'glue'
  luatexbase.add_to_callback("pre_linebreak_filter", function(head)
    local copy = node.copy_list(head)
    local parfill = node.tail(copy)
    if not parfill.id == glue or not parfill.subtype == 15 then
      texio.write_nl'Unable to find the parfill. Disabling special handler.'
      node.flush_list(copy)
      return true
    end
    local lines, infos = tex.linebreak(copy)
    local last = node.tail(lines)
    if tex.parindent > node.effective_glue(parfill, last) then
      node.flush_list(lines)
      copy = node.copy_list(head)
      parfill = node.tail(copy)
      node.setglue(parfill)
      lines, infos = tex.linebreak(copy)
      local t = node.tail(lines)
      if t.width == node.rangedimensions(t, t.head) then
        parfill = node.tail(head)
        node.setglue(parfill)
      end
    end
    node.flush_list(lines)
    return true
  end, "drop_short_parfill")
  luatexbase.add_to_callback("post_linebreak_filter", function(head)
    local t = node.tail(head)
    local tt = node.slide(t.head)
    while tt.id ~= glue or tt.subtype ~= 15 do
      tt = tt.prev
    end
    if tex.parindent > node.effective_glue(tt, t) then
      node.setglue(tt)
      local n = node.hpack(t.head, t.width, "exactly")
      head = node.insert_before(head, t, n)
      n.next, t.head = nil, nil
      node.free(t)
    end
    return head
  end, "drop_short_parfill")
\end{luacode*}
\begin{document}
\thispagestyle{empty}
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
sasdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf

asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
sasdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdfx

asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
sasdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdf sadf
asdfjk adsf af dsasdf f dasf fdsa fds afsd fdsaf asdfsdafdsa asdf dsaf asdfxx
\end{document}

In the example, the first paragraph does not need any change. The third automatically gets \parfillskip=0pt and for the second paragraph, only the last line is stretched.

In case you wonder why the result of the "trial" linebreak is always discarded instead of being reused as the actual linebreak, especially if no change is necessary: That would be implemented through linebreak_filter, which is broken in current LuaTeX versions.