What is a potential?
In the language of vector calculus:
The word potential is generally used to denote a function which, when differentiated in a special way, gives you a vector field. These vector fields that arise from potentials are called conservative. Given a vector field $\vec F$, the following conditions are equivalent:
- $\nabla \times \vec F=0$
- $\vec F= -\nabla \phi$
- $\oint_C \vec F \cdot \text{d}\vec \ell=0$ for any closed loop $C$ (Hence the name "conservative")
The function $\phi$ appearing in $(2)$ is called the potential of $\vec F.$ So any irrotational vector field can be written as the gradient of a potential function.
In electromagnetism specifically, Faraday's law tells us that $\nabla \times \vec E = -\frac{\partial \vec B}{ \partial t}$. For magnetic fields that do not vary with time (electrostatics) we get that $\nabla \times \vec E = 0$ and thus $\vec E = - \nabla V$ where $V$ is the potential of $\vec E$. This is exactly what we call the electric potential or "voltage" if you're a non-physicist. In the electrodynamics case where $\frac{\partial \vec B}{ \partial t} \neq 0$ a notion of electric potential still exists as we can break the electric field up into the sum of an irrotational field and a solenoidal field (this is called the Helmholtz theorem). We can then use Maxwell's equations to get that $\vec E = - \nabla V- \frac{\partial \vec A}{\partial t}$ where $V$ is the same electric potential and $\vec A$ is a vector field that we call the vector potential.
The case of gravity is analogous. If $\vec g$ is an irrotational gravitational field (which is always the case in Newtonian gravity) then $\vec g = -\nabla \phi$ where $\phi$ is the gravitational potential. This is closely related to gravitational potential energy in that a mass $m$ placed in the gravitational field $\vec g$ will have potential energy $U=m \phi$.
Electric potential and electric potential energy are two different concepts but they are closely related to each other. Consider an electric charge $q_1$ at some point $P$ near charge $q_2$ (assume that the charges have opposite signs).
Now, if we release charge $q_1$ at $P$, it begins to move toward charge $q_2$ and thus has kinetic energy. Energy cannot appear by magic (there is no free lunch), so from where does it come? It comes from the electric potential energy $U$ associated with the attractive 'conservative' electric force between the two chages. To account for the potential energy $U$, we define an electric potential $V_2$ that is set up at point $P$ by charge $q_2$.
The electric potential exists regardless of whether $q_1$ is at point $P$. If we choose to place charge $q_1$ there, the potential energy of the two charges is then due to charge $q_1$ and that pre-existing electric potential $V_2$ such that:
$$U=q_1V_2$$
P.S. You can use the same argument if you consider chage $q_2$, in that case the potential energy is the same and is given by:
$$U=q_2V_1$$