R - Create multiple linestrings from multiple coordinates

split-apply-combine is the general pattern you're looking for, e.g.

library(sf)
df <-
tibble::tribble(~ID,        ~X,      ~Y, ~prevXval, ~prevYval,
                  1,    -0.121,  65.001,     0.067,    65.117,
                  2,   180.000, -50.039,  -179.879,   -50.156,
                  3,    -0.075,  47.787,     0.038,    47.904,
                  4,   179.754, -73.960,  -179.900,   -73.979)

rows <- split(df, seq(nrow(df)))
lines <- lapply(rows, function(row) {
    lmat <- matrix(unlist(row[2:5]), ncol = 2, byrow = TRUE)
    st_linestring(lmat)
  })
lines <- st_sfc(lines)
lines_sf <- st_sf('ID' = df$ID, 'geometry' = lines)

purrr::map can be used in place of lapply.

Here's my version of your function to create a line string from a single row:

st_segment = function(r){st_linestring(t(matrix(unlist(r), 2, 2)))}

then with my test dataset:

df = data.frame(X=runif(10),Y=runif(10), PX=runif(10), PY=runif(10))

I can create a geometry column by using sapply on that data frame:

df$geom = st_sfc(sapply(1:nrow(df), 
    function(i){st_segment(df[i,])},simplify=FALSE))

giving:

head(df)
           X          Y         PX        PY                           geom
1 0.40675125 0.18953097 0.31176190 0.8423786 LINESTRING (0.4067513 0.189...
2 0.76997740 0.05169539 0.04452896 0.6224400 LINESTRING (0.7699774 0.051...
3 0.01624562 0.55156073 0.27115739 0.2459570 LINESTRING (0.01624562 0.55...
4 0.73451115 0.74298509 0.89374478 0.4820546 LINESTRING (0.7345111 0.742...
5 0.28522902 0.93939136 0.15557674 0.7243534 LINESTRING (0.285229 0.9393...
6 0.79610820 0.59341010 0.36035310 0.9634315 LINESTRING (0.7961082 0.593...

Which I can then turn into a spatial data frame:

> df = st_sf(df)

and if I plot that I see the geometry...