asyncio.gather vs asyncio.wait (vs asyncio.TaskGroup)

by

Although similar in general cases (“run and get results for many tasks”), each function has some specific functionality for other cases (and see also TaskGroup for Python 3.11+ below):

asyncio.gather()

Returns a Future instance, allowing high level grouping of tasks:

import asyncio
from pprint import pprint

import random


async def coro(tag):
    print(">", tag)
    await asyncio.sleep(random.uniform(1, 3))
    print("<", tag)
    return tag


loop = asyncio.get_event_loop()

group1 = asyncio.gather(*[coro("group 1.{}".format(i)) for i in range(1, 6)])
group2 = asyncio.gather(*[coro("group 2.{}".format(i)) for i in range(1, 4)])
group3 = asyncio.gather(*[coro("group 3.{}".format(i)) for i in range(1, 10)])

all_groups = asyncio.gather(group1, group2, group3)

results = loop.run_until_complete(all_groups)

loop.close()

pprint(results)

All tasks in a group can be cancelled by calling group2.cancel() or even all_groups.cancel(). See also .gather(..., return_exceptions=True),

asyncio.wait()

Supports waiting to be stopped after the first task is done, or after a specified timeout, allowing lower level precision of operations:

import asyncio
import random


async def coro(tag):
    print(">", tag)
    await asyncio.sleep(random.uniform(0.5, 5))
    print("<", tag)
    return tag


loop = asyncio.get_event_loop()

tasks = [coro(i) for i in range(1, 11)]

print("Get first result:")
finished, unfinished = loop.run_until_complete(
    asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED))

for task in finished:
    print(task.result())
print("unfinished:", len(unfinished))

print("Get more results in 2 seconds:")
finished2, unfinished2 = loop.run_until_complete(
    asyncio.wait(unfinished, timeout=2))

for task in finished2:
    print(task.result())
print("unfinished2:", len(unfinished2))

print("Get all other results:")
finished3, unfinished3 = loop.run_until_complete(asyncio.wait(unfinished2))

for task in finished3:
    print(task.result())

loop.close()

TaskGroup (Python 3.11+)

Update: Python 3.11 introduces TaskGroups which can “automatically” await more than one task without gather() or await():

# Python 3.11+ ONLY!
async def main():
    async with asyncio.TaskGroup() as tg:
        task1 = tg.create_task(some_coro(...))
        task2 = tg.create_task(another_coro(...))
    print("Both tasks have completed now.")

Related Contents:

  1. Asyncio.gather vs asyncio.wait
  2. Simplest async/await example possible in Python
  3. Python: what are the advantages of async over threads? [closed]
  4. Duplication of code for synchronous and asynchronous implementations
  5. Asynchronous method call in Python?
  6. How to limit concurrency with Python asyncio?
  7. What is the core difference between asyncio and trio?
  8. Combine awaitables like Promise.all
  9. How can I wrap a synchronous function in an async coroutine?
  10. What does asyncio.create_task() do?
  11. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  12. Using asyncio.Queue for producer-consumer flow
  13. how to add a coroutine to a running asyncio loop?
  14. How to use `async for` in Python?
  15. How to combine Celery with asyncio?
  16. What does the “yield from” syntax do in asyncio and how is it different from “await”
  17. How to use asyncio with existing blocking library?
  18. How can I send an HTTP request from my FastAPI app to another site (API)?
  19. How to use await in a python lambda
  20. How can I await inside future-like object’s __await__?
  21. Why can’t I ‘yield from’ inside an async function?
  22. multiprocessing vs multithreading vs asyncio in Python 3
  23. How can I run an external command asynchronously from Python?
  24. socket.shutdown vs socket.close
  25. asyncio.ensure_future vs. BaseEventLoop.create_task vs. simple coroutine?
  26. How to test Python 3.4 asyncio code?
  27. “asyncio.run() cannot be called from a running event loop” when using Jupyter Notebook
  28. asyncio.sleep() vs time.sleep()
  29. Learning asyncio: “coroutine was never awaited” warning error
  30. multiprocessing vs multithreading vs asyncio
  31. When and how to use Tornado? When is it useless?
  32. Test if function or method is normal or asynchronous
  33. Asynchronous exception handling in Python
  34. How to use ‘yield’ inside async function?
  35. Mocking async call in python 3.5
  36. How to combine python asyncio with threads?
  37. When using asyncio, how do you allow all running tasks to finish before shutting down the event loop
  38. python-asyncio TypeError: object dict can’t be used in ‘await’ expression
  39. How to call a async function from a synchronized code Python
  40. How can I call an async function without await?
  41. Asynchronous context manager
  42. Please explain “Task was destroyed but it is pending!” after cancelling tasks
  43. asynchronous programming in python
  44. what’s Python asyncio.Lock() for?
  45. Using queues results in asyncio exception “got Future attached to a different loop”
  46. Asyncio two loops for different I/O tasks?
  47. Python native coroutines and send()
  48. How do I add two integers together with Twisted?
  49. How to run multiple coroutines concurrently using asyncio?
  50. How to save UploadFile in FastAPI

Leave a Comment