What is the proper way to calculate distance in km from degrees

Convert your points to UTM first using Rgdal, for example:

dir <- 'C:/your/path'
setwd(dir)#set working directory

pointfile <- 'points.csv'
data <- read.csv(pointfile)#read in data

library(rgdal)

#load in points as lat long
coords <- SpatialPoints(cbind(data$lon, data$lat), proj4string = CRS("+proj=longlat"))

#convert points to WGS84 UTM zone 56S (western AUS)
coord.UTM <- spTransform(coords, CRS("+init=epsg:32756")) 

Or, you can use the package 'raster' to compute the distances in meters directly from the latlong points, though the estimates will vary slightly due to lack of precision in the lat and long coordinates.

library(raster)

dists <- pointDistance(coord.UTM, lonlat = FALSE)

dists_lonlat <- pointDistance(coords, lonlat = TRUE) #specify for lonlat

I'm not familiar with the marmap package and what it returns to the user. But when calculating distances, I think it's safer to have your source data projected to a CRS that is already in meters. So, what I would try is:

1. Get the data using the getNOAA.bathy()
2. Convert the data to raster using the as.raster() from the same package
3. Project it to a CRS suitable for your area
4. Calculate the distances