Why do engineers use derivatives in discontinuous functions? Is it correct?
In general, computing the extrema of a continuous function and rounding them to integers does not yield the extrema of the restriction of that function to the integers. It is not hard to construct examples.
However, your particular function is convex on the domain $k>0$. In this case the extremum is at one or both of the two integers nearest to the unique extremum of the continuous function.
It would have been nice to explicitly state this fact when determining the minimum by this method, as it is really not obvious, but unfortunately such subtleties are often forgotten (or never known in the first place) in such applied fields. So I commend you for noticing the problem and asking!
The main question here seems to be "why can we differentiate a function only defined on integers?". The proper answer, as divined by the OP, is that we can't--there is no unique way to define such a derivative, because we can interpolate the function in many different ways. However, in the cases that you are seeing, what we are really interested in is not the derivative of the function, per se, but rather the extrema of the function. The derivative is just a tool used to find the extrema.
So what's really going on here is that we start out with a function $f:\mathbb{N}\rightarrow \mathbb{R}$ defined only on positive integers, and we implicitly extend $f$ to another function $\tilde{f}:\mathbb{R}\rightarrow\mathbb{R}$ defined on all real numbers. By "extend" we mean that values of $\tilde{f}$ coincide with those of $f$ on the integers. Now, here's the crux of the matter: If we can show that there is some integer $n$ such that $\tilde{f}(n)\geq \tilde{f}(m)$ for all integers $m$, i.e. $n$ is a maximum of $\tilde{f}$ over the integers, then we know the same is true for $f$, our original function. The advantage of doing this is that now can use calculus and derivatives to analyze $\tilde{f}$. It doesn't matter how we extend $f$ to $\tilde{f}$, because at the end of the day we're are only using $\tilde{f}$ as a tool to find properties of $f$, like maxima.
In many cases, there is a natural way to extend $f$ to $\tilde{f}$. In your case, $f=\text{ExecutionTime}$, and to extend it you just take the formula $\Delta \left(4k + \frac{2n}{k} - 4\right)$ and allow $n$ and $k$ to be real-valued instead of integer-valued. You could have extended it a different way--e.g. $\Delta \left(4k + \frac{2n}{k} - 4\right) + \sin(2\pi k)$ is also a valid extension of $\text{ExecutionTime}(n,k)$, but this is not as convenient. And all we are trying to do is find a convenient way to analyze the original, integer-valued function, so if there's a straightforward way to do it we might as well use it.
As an illustrative example, an interesting (and non-trivial) case of this idea of extending an integer-valued function to a continuous-valued one is the gamma function $\Gamma$, which is a continuous extension of the integer-valued factorial function. $\Gamma$ is not the only way to extend the factorial function, but it is for most purposes (in fact, all purposes that I know of) the most convenient.
You are confusing a mathematical model of the system with the system itself. The map is not the territory.
Obviously in the real system both $n$ and $k$ must be integers. On the other hand, the math formula for the execution time is a perfectly good function for any real (or even complex!) values of $n$ and $k$ except when $k = 0$.
So you can certainly find the minimum value if $k$ according to the math model, even though the answer $k = \sqrt{(n/2)}$ gives a fractional value of $k$ for most integer values of $n$.
If you want to make the math more rigorous, since the function is convex you can then say that
For any integer $k \ge \sqrt{n/2}$, $E(n,k+1) > E(n,k)$, and
For any integer $k \le \sqrt{n/2}$ and $k > 1$, $E(n,k-1) > E(n,k)$.
Therefore, the minimum value of $E$ when $k$ is an integer is one of the (one or two) integers in the range $\sqrt{n/2} - 1 < k < \sqrt{n/2} + 1$.
On the other hand, engineers are interested in getting results, not doing rigorous math, and if you sketch a graph of the general shape of $E(n,k)$ this rigorous mathematical argument is "obvious".