June 2025 OLCF User Conference Call: Dask on HPC Systems

The OLCF hosts monthly User Conference Calls. These calls are your opportunity to speak with center personnel to get the latest updates, express any concerns you may have, etc. No registration is required for this event. An email with an invite to the event will be sent to OLCF users approximately one week beforehand.

Monthly Topic:  Dask on HPC Systems

Speaker: Mark Coletti (ORNL)

Abstract:

Scientific workflows automate research tasks that include data pre-processing, model training, running simulations, analyzing output, and producing visualizations. The use of scientific workflow tools is especially important in high-performance computing (HPC) environments to reduce errors, automate complex tasks, and otherwise increase productivity. While numerous workflow management systems exist, they often impose rigid, top-down architectures that limit flexibility and integration with evolving research code.

In this talk, I present a bottom-up approach to scientific workflow design using Python and Dask, a flexible parallel computing library. By constructing task graphs directly from domain-specific code, researchers can retain full control over data movement, computation, and parallelism. I will demonstrate how Dask’s dynamic execution model enables the creation of scalable, custom workflows tailored to HPC systems and discuss real-world case studies from my own research. This session is intended for scientific practitioners and developers seeking expressive and adaptable strategies for managing large-scale computational pipelines.

Looking for a previous User Call recording? Check out our Training Archive and Vimeo Channel.