Find a free X11 display number

Why not exploit the fact that every X11 server puts a "Lock" file into /tmp?

This is called /tmp/.Xn-lock where "n" is the Display id. (Also note the leading . in the filename).

This is the mechanism that Xserver itself uses to check for duplication, and it appears to be consistent on all *nix platforms I have tried (HP-UX, Linux, ...)

So you could adapt your script thus (forgive me for syntax errors, I'm nore accustomed to C shell than Bourne/Korn shell scripting)

DISPLAY_NUM=0

do

  if ( -e /tmp/.X$DISPLAY_NUM-lock ) then

     let DISPLAY_NUM=$DISPLAY_NUM+1

  else

     Xvfb :$DISPLAY_NUM -screen 0 1280x1024x24 -ac   (or whatever args take your fancy)

  fi

done

Possibly a bit off-topic, but if you use xvfb-run to start your Xserver-needing command, then just running

# xvfb-run -a your command

does the trick.

With full credit to this answer to the related question How high do X11 display numbers go?:

Recent X servers as of version 1.13 (Xvfb, too) support the -displayfd <fd> command line option: It will make the X server choose the display itself and write the display number back to the file descriptor <fd>. It's a bit convoluted, but this would be a safe and race-condition-free way to ask Xvfb to use any free display. A bash example:

exec 6>display.log
Xvfb -displayfd 6
# The display number of the new Xvfb instance has been written to display.log 
# Kill Xvfb
exec 6>&-

There's no point in trying to find a free display number. As you have guessed, between the time you find a free one and the time Xvfb starts, another X server might have taken the port you thought was free. So, better to just try to launch Xvfb, handle the failure if the port is taken, and then retry at the next port until you succeed or run out of ports to try.

#!/bin/bash
DISPLAY_NUM=0
unset TEST_HAS_RUN
until [ $TEST_HAS_RUN ] || (( $DISPLAY_NUM > 10 ))
do
 Xvfb :$DISPLAY_NUM &
 jobs
 sleep 2  # assumption here is that Xvfb will exit quickly if it can't launch
 if jobs | grep Xvfb
 then  
   echo launching test on :$DISPLAY_NUM
   xterm -display :$DISPLAY_NUM
   TEST_HAS_RUN=1
   kill %-
 else   
   let DISPLAY_NUM=$DISPLAY_NUM+1
 fi
done