Use a smartphone as a Dial Up Modem?
No. Analog telephone modems (a.k.a. POTS - "Plain Old Telephone Service" modems) worked because when the traditional telephone network (a.k.a the PSTN - "Public Switched Telephone Network") digitized the audio, it did so with 8-bit samples 8,000 times per second, for a total of 8 bits * 8 kHz = 64,000 bps digital audio. It wasn't possible to fully utilize that full 64kbps for data because of problems with the analog <-> digital conversion (ADC/DAC), but if you had a digital modem on a digital phone line (like an ISDN B-channel), you could send at 53-56kbps through a single DAC. But from the analog line, you could only send 33.8kbps through a single ADC.
But the problem with cell phones is that they've never provided a full 64kbps digital audio channel like the landline PSTN did. They use lossy codecs to compress your voice audio down to fit into a 9600 bps channel. So if you want to try to do audio-modulated signaling over that channel, I think you'd be lucky to get 4800 bps (4.8kbps = 0.6 KibiBytes/sec) out of it. And that's assuming you design custom modulation schemes to get the most out of what that audio channel gives you. I have no idea if any of the traditional modulation schemes used by POTS modems would work well over GSM voice channels. For all I know, an off the shelf POTS modem may just completely fall on its face and not even be able to sustain the ages-old 300 bps rate over a GSM voice channel.
Update: I found a reference online that said that doing analog audio modulation over GSM voice channels was typically limited to 2400 bps, so my estimate above was quite generous.
This goes a little outside the scope of your question, but note that technologies existed to make dialup-like connections over digital cellular networks but without the severely rate-limiting analog complication of hooking a POTS modem to the headset jack of the cell phone.
One solution was called "Circuit-Switched Data" or CSD, and usually involved hooking your computer to your cell phone via a serial connection of some type (using traditional RS-232 or RS-422 style serial interfaces or IrDA in the old days, and USB or Bluetooth more recently), and having your cell phone give that serial interface direct access to the 9600 bps data connection to the wireless carrier's equipment. Inside the wireless carrier's network, your 9600 bps data stream would be fed into a digital modem hooked up to a digital phone line, so you could use this technology to make a 9600 bps modem call to any modem on the PSTN.
CSD was followed by High-Speed CSD (HSCSD), which was like CSD except your cell phone would use 4 GSM voice channels simultaneously, so you'd be able to make ~56kbps connections, just like (in the downstream channel at least) the last era of POTS modems (V.90).
Over digital cellular technologies like GPRS and EDGE, it was possible to use that serial connection to the phone to do PPP, but that wouldn't have worked to just any modem hooked up to any phone line; the other end of your PPP connection would need to be a PPP server that receives its PPP connections not from modems, but over IP, tunneled in L2TP. This was common among ISPs that outsourced their dial-up modem POPs to third parties like Level 3 and UUNet, and is still common today among DSL ISPs (that's why so many DSL services require PPPoA or PPPoE).
tl;dr: To answer your specific question, No, it is probably not possible to hook a POTS modem via headset jack to a cell phone and make a successful connection to a modem on the PSTN. If it connects at all, it will probably only use extremely low data rates (even as POTS modems go), and be glitchy and unreliable. However, there are (or have been) other ways to connect a computer to a cell phone to make dial-up modem calls or PPP connections.
Not a direct answer but a curious one.
While it is impossible to use legacy modems with cellular voice channels, it is, in fact, possible to design a specialized modem which will work over it (more or less by plugging into a handset jack, as OP desires). This is a useful feature, because:
- Data coverage may not be available in the remote areas where data communications may still be required (to support ATMs and EFT terminals, to name the most common need).
- Voice channel has highest communication priority and will keep operating even if data services are completely overflown (by a massive crowd or spike in usage).
IEEE article describing a possible approach:
https://ieeexplore.ieee.org/document/4383352
had previously connected a cell phone to analog modem, before data tethering was available.
used a cell phone dock that had connections for an external speaker and external microphone, and then used a heathkit ham radio phone patch to isolate (null) audio in from audio out and provide a balanced telephone connection.
made a virtual phone line with filtered dc to make the modem happy, and connected the phone patch to it.
manually dialed the modem dial up number on the cell phone, had the modem connect with an atdt command that ignored lack of dial tone and dialed one digit.
this worked for many calls to a landline based modem
at the time 9600 baud was the norm, would think you could probably get 28,800 or 33.6k out of the setup
tried it with an ameritech (land line carrier) connected cell phone, the cell phone calls were echoed without the setup and would not do a connection at all with a modem
had another same model cell phone on cellular one (wireless carrier), the normal cell calls were crystal clear, when hooked to the "setup" it provided dial up modem calls from all around my state and nearby states, as well as long distance modem calls from outside the area
also had a normal desk phone so they could do test calls thru the setup, if there was a question about it working.