How to remove temp files without changing the directory's modification timestamp?

You'll have to reset the timestamp on the directory after removing the files. Assuming GNU tools, something like this should work:

mtime=$(stat -c %y dir)            # get the timestamp, store in $mtime
rm dir/somefile dir/someotherfile  # do whatever you need
touch -d "$mtime" dir              # set the timestamp back

That resets the modification (mtime) and access (atime) timestamps on the directory to the original modification timestamp, but also sets the change timestamp (ctime) to the current time. Changing the ctime is unavoidable, but you probably don't care about it or atime.

The touch command provides the -r option which allows you to touch a file with the same timestamps as a reference file. As such, you could do this:

touch -r /dir/somefile /tmp/_thetimestamps  # save the timestamps in /tmp/_thetimestamps
rm /dir/somefile                            # do whatever you need
touch -r /tmp/_thetimestamps /dir           # set the timestamps back

Linux Man:

-r, --reference=FILE
      use this file's times instead of current time

Unix Man:

-r       Use the access and modifications times from the specified file
         instead of the current time of day.