asyncio.run() cannot be called from a running event loop

The asyncio.run() documentation says:

This function cannot be called when another asyncio event loop is running in the same thread.

The problem in your case is that jupyter (IPython) is already running an event loop (for IPython ≥ 7.0):

You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races.

That's the reason why you don't need to start the event loop yourself in jupyter and you can directly call await main(url) even outside asynchronous functions.

In jupyter

async def main():
    print(1)
    
await main()

In plain Python (≥3.7)

import asyncio

async def main():
    print(1)
    
asyncio.run(main())

In your code that would give:

url = ['url1', 'url2']
result = await main(url)

for text in result:
    pass # text contains your html (text) response

To add to cglacet's answer - if one wants to detect whether a loop is running and adjust automatically (ie run main() on the existing loop, otherwise asyncio.run()), here is one suggestion that I tried (if indeed one wants to do that):

try:
    loop = asyncio.get_running_loop()
except RuntimeError:  # if cleanup: 'RuntimeError: There is no current event loop..'
    loop = None

if loop and loop.is_running():
    print('Async event loop already running')
    tsk = loop.create_task(main())
    # ^-- https://docs.python.org/3/library/asyncio-task.html#task-object
    tsk.add_done_callback(                                          # optional
        lambda t: print(f'Task done: '                              # optional
                        f'{t.result()=} << return val of main()'))  # optional (using py38)
else:
    print('Starting new event loop')
    asyncio.run(main())

Just use this:

https://github.com/erdewit/nest_asyncio

import nest_asyncio
nest_asyncio.apply()