In igraph in R, is it possible to create dotted lines around the vertex objects?

You can define your own shapes: https://igraph.org/r/doc/shapes.html and an example of a point with a dotted border is given at https://r.789695.n4.nabble.com/Drawing-a-dotted-circle-td4655331.html. A full example of creating a new shape given at https://lists.gnu.org/archive/html/igraph-help/2013-03/msg00030.html. Also see more examples at ?add_shape. The example below tweaks the code from lists.gnu.org to combine everything.

Function to create new igraph shape

myimg <- function(coords, v=NULL, params) {
  vertex.color <- params("vertex", "color")
  if (length(vertex.color) != 1 && !is.null(v)) {
    vertex.color <- vertex.color[v]
  }
  vertex.size  <- 1/200 * params("vertex", "size")
  if (length(vertex.size) != 1 && !is.null(v)) {
    vertex.size <- vertex.size[v]
  }
  vertex.frame.color <- params("vertex", "frame.color")
  if (length(vertex.frame.color) != 1 && !is.null(v)) {
    vertex.frame.color <- vertex.frame.color[v]
  }
  vertex.frame.width <- params("vertex", "frame.width")
  if (length(vertex.frame.width) != 1 && !is.null(v)) {
    vertex.frame.width <- vertex.frame.width[v]
  }
  ltype <- params("vertex", "ltype")
  if (length(ltype) != 1 && !is.null(v)) {
    ltype <- ltype[v]
  }   

  mapply(coords[,1], coords[,2], vertex.color, vertex.frame.color,
         vertex.size, vertex.frame.width, ltype, 
         FUN=function(x, y, bg, fg, size, lwd, lty) {
           symbols(x=x, y=y, bg=bg, fg=fg, lwd=lwd, lty=lty,
                   circles=size, add=TRUE, inches=FALSE)
         })
  }

You then make igraph recognise the shape using add_shape. You set default parameter values using the parameters argument.

library(igraph)

g <- make_undirected_graph(edges = c(1, 3, 2, 1, 2, 4, 3, 4, 4, 5), n = 5)

add_shape("myimg",  plot=myimg, 
          parameters = list(
            vertex.frame.color=1, 
            vertex.frame.width=1,
            vertex.ltype=1))

Then plot

plot(g,  vertex.shape="myimg", 
         vertex.frame.color=1:5, 
         vertex.frame.width=5, 
         vertex.ltype=1:5,
         vertex.color=6:10,
         vertex.size=seq(50, 80, length=5))

To get all the borders dotted just use vertex.ltype="dotted" or vertex.ltype=3.

I'm not sure how to do this within igraph, but one option would be to print it using ggraph, a package that bridges igraph with ggplot2. Then we can build up the graph in layers and specify the appearance of each one:

library(ggraph)
ggraph(g) +
  geom_edge_link(color = "gray60") +
  geom_node_circle(aes(r = 0.1), lty = "dashed", fill = "orange") +
  geom_node_text(aes(label = ggraph.orig_index)) +
  coord_equal() +
  theme_void()

I don't use igraph but I found the network package a good replacement of it, and it's plotting function is more intuitive (if you are familiar with the base plot parameters) and easier to customize. But I am 100% sure you could find a similar function in igraph that lets you adjust the plot parameters.

require(network)

# build the network
g = as.network.matrix(x = matrix(c(1, 3, 2, 1, 2, 4, 3, 4, 4, 5), nrow = 5, byrow = T), matrix.type = 'edgelist', directed = F)

# vertex.lty is the parameter you are looking for, and you can pass an array to it.
# lty = 2 for dashed lines and lty = 3 for dotted lines.
plot(g, label = 1:5, label.pos = 5, 
     vertex.cex =6, vertex.col = "#ffdd88",
     vertex.lty = c(1,2,2,3,3), vertex.lwd = c(2,3,4,2,3)) 

(Edited for better formatting.)