ProcessPoolExecutor from concurrent.futures way slower than multiprocessing.Pool

When using map from concurrent.futures, each element from the iterable is submitted separately to the executor, which creates a Future object for each call. It then returns an iterator which yields the results returned by the futures. Future objects are rather heavyweight, they do a lot of work to allow all the features they provide …

Read more

What is the difference between concurrent.futures and asyncio.futures?

The asyncio documentation covers the differences: class asyncio.Future(*, loop=None) This class is almost compatible with concurrent.futures.Future. Differences: result() and exception() do not take a timeout argument and raise an exception when the future isn’t done yet. Callbacks registered with add_done_callback() are always called via the event loop’s call_soon_threadsafe(). This class is not compatible with the …

Read more

Use tqdm with concurrent.futures?

You can wrap tqdm around the executor as the following to track the progress: list(tqdm(executor.map(f, iter), total=len(iter)) Here is your example: import time import concurrent.futures from tqdm import tqdm def f(x): time.sleep(0.001) # to visualize the progress return x**2 def run(f, my_iter): with concurrent.futures.ThreadPoolExecutor() as executor: results = list(tqdm(executor.map(f, my_iter), total=len(my_iter))) return results my_iter = …

Read more

How does ThreadPoolExecutor().map differ from ThreadPoolExecutor().submit?

The problem is that you transform the result of ThreadPoolExecutor.map to a list. If you don’t do this and instead iterate over the resulting generator directly, the results are still yielded in the original order but the loop continues before all results are ready. You can test this with this example: import time import concurrent.futures …

Read more