Parsl
The Parallel Scripting Library enables you to divide computing tasks among multiple techniques, leveraging a similar syntax to Python’s existing syntax. Pool
objects. This feature enables the integration of diverse computational tasks into complex workflows, allowing for flexible execution scenarios, including parallel, sequential, and pipeline processing.
Parsl enables the execution of native Python functions alongside external utilities via shell instructions. The @operate decorator that initiates your work’s execution is the distinguishing feature of your Python code. The job-submission system offers granular control over issue execution on target nodes, including the allocation of cores per user, memory resources per user, CPU affinity settings, timeout polling frequencies, and more.
Parsl boasts an impressive feature set, including prebuilt templates that enable efficient workload distribution to numerous high-performance computing resources. While traditional high-performance computing architectures like AWS or HPC clusters are often discussed, access to advanced supercomputing resources such as Blue Waters, ASPIRE 1, Frontera, etc., also warrants consideration. Parsl was co-developed in collaboration with numerous institutions that designed and built similar hardware.