There are many ways to run functions in parallel, but there might be many caveats to deal with. We can overcome all those caveats by using Ray is, a simple, universal API for building distributed applications.
What Ray can do for you:
- Providing simple primitives for building and running distributed applications.
- Enabling end-users to parallelize single machine code with little to zero code changes.
- Including a large ecosystem of applications, libraries, and tools on top of the core Ray to enable complex applications.
How to install Ray:
sudo pip3 install Ray
A basic Ray example:
| #!/usr/bin/env python3 | |
| import ray | |
| @ray.remote | |
| def func1(): | |
| from datetime import datetime | |
| import time | |
| now = datetime.now() | |
| current_time = now.strftime(“%H:%M:%S”) | |
| print(“func 1 Start Time =”, current_time) | |
| time.sleep(2) | |
| now = datetime.now() | |
| current_time = now.strftime(“%H:%M:%S”) | |
| print(“func 1 End Time =”, current_time) | |
| @ray.remote | |
| def func2(): | |
| from datetime import datetime | |
| import time | |
| now = datetime.now() | |
| current_time = now.strftime(“%H:%M:%S”) | |
| print(“func 2 Start Time =”, current_time) | |
| time.sleep(2) | |
| now = datetime.now() | |
| current_time = now.strftime(“%H:%M:%S”) | |
| print(“func 2 End Time =”, current_time) | |
| if __name__ == ‘__main__’: | |
| ray.init() | |
| ray.get([func1.remote(), func2.remote()]) |
- Line 2 imports ray into our project
@ray.remote decoratoris used to define which functions will run in parallel- Line 31
ray.init()initializes Ray - Line 32
ray.get([func1.remote(),func2.remote()])the list of functions to run in parallel
Ray provides a very nice output of the executing functions
./test1.py
(func2 pid=1596) func 2 Start Time = 11:51:48
(func1 pid=1591) func 1 Start Time = 11:51:48
(func2 pid=1596) func 2 End Time = 11:51:50
(func1 pid=1591) func 1 End Time = 11:51:50
This is a very basic example of Ray that executes functions in parallel, those functions do not return anything, let’s see a more advanced example of parallel execution with functions that accept parameters and return results
| #!/usr/bin/env python3 | |
| import ray | |
| @ray.remote | |
| def func1(x): | |
| return x+1 | |
| @ray.remote | |
| def func2(y): | |
| return y+1 | |
| if __name__ == ‘__main__’: | |
| ray.init() | |
| ret1, ret2 = ray.get([func1.remote(x=2), func2.remote(y=2)]) | |
| print(ret1) | |
| print(ret2) |
- Line 14 —u sing
ray.getwe execute the functions in parallel with parameters, and they return their results in variablesret1andret2.
ray.get function accepts functions as parameters in a list form, also returns results as a list, this means that we can do the following which can be very useful when the number of functions to be executed is varied each time.
| #!/usr/bin/env python3 | |
| import ray | |
| @ray.remote | |
| def func1(x): | |
| return x+1 | |
| @ray.remote | |
| def func2(y): | |
| return y+1 | |
| if __name__ == ‘__main__’: | |
| ray.init() | |
| funcs = [] | |
| funcs.append(func1.remote(x=2)) | |
| funcs.append(func2.remote(y=2)) | |
| results = ray.get(funcs) | |
| print(results) |