ALL "forces" as manifestations of properties of space-time
Quote from "Space-Time Structure" by Erwin Schrodinger:
"At any rate the very foundation of the theory, viz. the basic principle of equivalence of acceleration and any gravitational field, clearly means that there is no room for any kind of "force" to produce acceleration save gravitation, which however is not to be regarded as a force but resides on the geometry of space-time. Thus in fact, though not always in the wording, the mystic concept of force is wholly abandoned."
The situation is a bit difficult if you want to take the strong interaction effects and so on into account, because it's a question what you permit to be "geometrical". The fundamental forces involving charges are decribed by qunatum field theories, gauge theories of the the Yang–Mills type and as far as this whole connection-business goes, it is actually a pretty geometric setting. So from a mathematical side, there are several similarities to general relativity. These theories are not concerned with a curved base manifold and the tangent space in the exact same sense as Riemann-geometry, but there is also a notiond of curvature. I don't know if these will help you but here are two more links regarding a united mathematical perspective.
For starter, lets keep this pre quantum. The obvious thing that comes to mind here is Kaluza Klein theory. However, as you'll read in the very first paragraph, this is a theory in five dimensions. If you don't mind this, then I'd push you further in a stringly direction, although this also differs pretty much from conventional general relativity. It's not unlikely that some people here might write an answer from that perspective.
The incorporation of a force like electromagnetism in a classical four dimensional Pseudo-Riemann geometry type theory like general relativity turns out to be problematic. Without going into computations involving features like spacetime metrics, Christoffel symbols and the possibility to choose a local inertial frame, here is an argument involving the related equivalence principle. I quote the weak, Einsteinian and strong version from wikipedia here and highlight some words:
Weak: "All test particles at the alike spacetime point in a given gravitational field will undergo the same acceleration, independent of their properties, including their rest mass."
Einsteinian: "The outcome of any local non-gravitational experiment in a freely falling laboratory is independent of the velocity of the laboratory and its location in spacetime."
Strong: "The gravitational motion of a small test body depends only on its initial position in spacetime and velocity, and not on its constitution and the outcome of any local experiment (gravitational or not) in a freely falling laboratory is independent of the velocity of the laboratory and its location in spacetime."
Let's consider a situation in general relativity where you have a big electrically negative charged object $X$ and the corresponding spacetime curvature is governed by the Reissner–Nordström metric. You are sitting in your cosy laboratory, falling through space and you have optionally two to four particles $A,B,C$ and $D$ of charge $Q_A=0, Q_B=-1, Q_C=+1$ and $Q_D=-9001$. Lets say at the beginning of your experiments they are never moving relative to you. You and the uncharged particle will fall freely (along a geodesic produces by the Reissner–Nordström metric) towards $X$, while the other particles are never free falling, interact electromagnetically with $X$ and therefore move in different ways, depending on their charge. They also might interact with each other in various ways. There is only one particle mass, but netral, positive and negative electric charges so that different species are affected differently by the surroundings.
Given the fact that the physical observations so far agree with the results of general relativity: If you incorporate elecromagnetism in your four dimensional curved spacetime of your geometrical theory, how would you realize the equivalence principle?
Your « wild supposition » has been pursued by eminent names such as Einstein, Klein, Weyl, Schroedinger, and Mills. It has never been quite made to work, but it does not seem to have any really blatant fault except that it is not quantum. Not yet, anyway. I suppose that Yang--Mills might prove to be the right way to go, and if so, that would be a positive answer to your question, and quantised too. We'll see. It's a millenium problem.