How do I know if a remote machine is using Windows or Linux?

It isn't definitive but nmap will do this with the command nmap -O -v (see docs for more details) If you're running windows or want a gui, look at zenmap

If you're on an IPv4 network, just use ping. If the response has a TTL of 128, the target is probably running Windows. If the TTL is 64, the target is probably running some variant of Unix.

: Presumes ping service enabled on Windows local and remote hosts
:
del _IX.txt, Windows.txt
ping -n 1 [computername|ipaddress] | findstr /i /c:"Reply" > ttl.txt
for /f "tokens=1-9* delims=:=< " %%a in (ttl.txt) do (
    if %%i leq 130 (
       if %%i geq 100 (
          echo Windows & rem or echo %%c >> Windows.txt
       ) else (
          if %%i equ 64 (
             echo *IX & rem or echo %%c >> _IX.txt
          )
       )
    )
)