ProcessPoolExecutor and ThreadPoolExecutor

ProcessPoolExecutor and ThreadPoolExecutor (which is used for multithreading) are both part of Python’s concurrent.futures module. They serve different purposes and are suitable for different types of tasks:

Key Differences

Feature	ProcessPoolExecutor	ThreadPoolExecutor
Concurrency Model	Multiprocessing: uses multiple processes	Multithreading: uses multiple threads
Use Case	CPU-bound tasks	I/O-bound tasks
Parallelism	True parallelism (each process has its own memory)	No true parallelism (due to Python’s GIL)
Overhead	Higher (processes have separate memory space)	Lower (threads share memory)
Global Interpreter Lock (GIL)	Not affected (each process has its own interpreter)	Affected (threads are limited by GIL in Python)
Memory Usage	Higher (each process has its own memory space)	Lower (threads share memory)
Task Communication	More complex (inter-process communication required)	Easier (threads can share memory)
Fault Isolation	Higher (if one process crashes, others are unaffected)	Lower (a crash in one thread can affect others)

---

ThreadPoolExecutor

Handling multiple database queries

Best for I/O-bound tasks: Multithreading is typically used for tasks that involve a lot of waiting (I/O-bound), such as network requests, file I/O, or database operations. Threads share memory space, so switching between them is fast, but they are limited by the Global Interpreter Lock (GIL) in Python.

Example Use Cases:

Making concurrent HTTP requests.

Reading/writing files simultaneously.

Example of concurrent :

import concurrent.futures
import time

def io_bound_task(*args, **kwargs):
    time.sleep(args[0])
    print('sleep time', args[0])
    return args[0], kwargs.get('task')

with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = [executor.submit(io_bound_task, 3, task = 'one' ), executor.submit(io_bound_task, 6, task ='two')]

    for future in concurrent.futures.as_completed(futures):

        print(future.result())    

        a,b = future.result()      
        print(b)        

Example of multi processing

if __name__ == "__main__":
    with concurrent.futures.ProcessPoolExecutor() as executor:

        futures = [executor.submit(io_bound_task, 50, task = 'three' ), executor.submit(io_bound_task, 60, task ='four')]

        for future in concurrent.futures.as_completed(futures):

            print(future.result())    

            a,b = future.result()      
            print(b)