Unified Communication Framework

_{Unified Communication Framework (UCF) – Collaboration between industry,laboratories, and academia to create production grade communication frameworks and open standards for data-centric and high-performance applications. In this talk we will present recent advances in development UCF projects including Open UCX, Apache Spark UCX as well incubation projects in the area of SmartNIC programming, benchmarking, and other areas of accelerated compute.}

_{Gilad Shainer serves as senior vice-president of marketing for Mellanox networking at NVIDIA, focusing on high- performance computing, artificial intelligence and the InfiniBand technology. Mr. Shainer joined Mellanox in 2001 as a design engineer and later served in senior marketing management roles since 2005. Mr. Shainer serves as the chairman of the HPC-AI Advisory Council organization, the president of UCF and CCIX consortiums, a member of IBTA and a contributor to the PCISIG PCI-X and PCIe specifications. Mr. Shainer holds multiple patents in the field of high-speed networking. He is a recipient of 2015 R&D100 award for his contribution to the CORE-Direct In-Network Computing technology and the 2019 R&D100 award for his contribution to the Unified Communication X (UCX) technology. Gilad Shainer holds a MSc degree and a BSc degree in Electrical Engineering from the Technion Institute of Technology in Israel.}

_{Pavel Shamis is a Principal Research Engineer at Arm. His work is focused on co-design software, and hardware building blocks for high-performance interconnect technologies, development of communication middleware, and novel programming models. Prior to joining ARM, he spent five years at Oak Ridge National Laboratory (ORNL) as a research scientist at Computer Science and Math Division (CSMD). In this role, Pavel was responsible for research and development multiple projects in high-performance communication domains including Collective Communication Offload (CORE-Direct & Cheetah), OpenSHMEM, and OpenUCX. Before joining ORNL, Pavel spent ten years at Mellanox Technologies, where he led Mellanox HPC team and was one of the key drivers in the enablement Mellanox HPC software stack, including OFA software stack, OpenMPI, MVAPICH, OpenSHMEM, and other. Pavel is a board member of UCF consortium and co-maintainer of Open UCX. He holds multiple patents in the area of in-network accelerator. Pavel is a recipient of 2015 R&D100 award for his contribution to the development CORE-Direct in-network computing technology and the 2019 R&D100 award for the development of Open Unified Communication X (Open UCX) software framework for HPC, data analytics, and AI.}

_{Seamless support of GPU buffers communication introduces yet another dimension for designing optimal data transfer protocols. In this talk we will have open discussion about outstanding issues and challenges, including out-of-box performance, PCI topology support, rendezvous protocols, memory type cache and memory hooks, and more.}

_{Yossi Itigin is a UCX team lead at NVIDIA, focuses on high-performance communication middleware, and a maintainer of OpenUCX project. Prior to joining NVIDIA, Mr. Itigin spent nine years at Mellanox Technologies in different technical roles, all related to developing and optimizing RDMA software.}

_{An update on UCX usage within the MPICH library. Topics covered include point-to-point communication, RMA, multi-threading, and more.}

_{Ken is a Principal Software Development Specialist at Argonne National Laboratory.}

_{The capability of using GPU memory in RDMA operations (a.k.a GPU Direct RDMA) is important for scale-out configuration of GPU computation. In this talk, a dma-buf based approach is presented to enable such capability in the upstream Linux kernel and user space rdma-core libraries.}

_{Jianxin Xiong works for Intel. His experience covers various layers of interconnection software stack, such as RDMA drivers in Linux kernel, RDMA device virtualization, Open Fabric Interface, DAPL, Tag Matching Interface, and MPI. His current focus is GPU/accelerator scale-out with RDMA devices.}

_{Utilizing accelerators in Apache Spark presents opportunities for significant speedup of ETL, ML and DL applications. In this talk i’ll describe acceleration of Spark data shuffle across network and GPU accelerators using UCX library. Will show our existing solutions on speeding up host-to-host data transfer, using one-sided communications, device-to-device acceleration for Rapids Spark using tag API, and our roadmap to unify these two solutions.}

_{Peter Rudenko is a software engineer in High Performance Computing team, focusing on accelerating data intensive applications, developing UCX communication library and various big data solutions.}

_{Abstract: Efficient communication is paramount to data science, at the same time being very challenging, given it’s dominated by high-level languages such as Python. We will show how UCX-Py leverages UCX performance greatly for distributed frameworks such as Dask and GPU-accelerated ecosystem RAPIDS. The performance achieved is seen in the COVID-19 research on Summit, the world’s largest supercomputer at ORNL.}

_{Benjamin Zaitlen is a system software manager for NVIDIA-RAPIDS, focusing on connecting the PyData Ecosystem to accelerated computing and networking}

_{Peter Entschev is a system software engineer at NVIDIA, working on distributed computing and communications on the RAPIDS team.}

_{Matthew graduated from East Tennessee State University (ETSU) with a masters in applied computer science. Matthew’s past work has included working on programming models and high performance network libraries. Matthew’s current work includes monitoring systems at large scale and allowing applications to make decisions at run time.}

_{Steve is the Chief Architect for Next Generation Platforms at Los Alamos National Laboratory}

_{In order to achieve out-of-box performance, UCX must select the optimal protocol for the given scenario, considering factors such as buffer length, memory locality, and data layout. In this talk we will present and discuss the next version of protocol selection mechanism, and make sure it will be able to handle that task.}

_{Yossi Itigin is a UCX team lead at NVIDIA, focuses on high-performance communication middleware, and a maintainer of OpenUCX project. Prior to joining NVIDIA, Mr. Itigin spent nine years at Mellanox Technologies in different technical roles, all related to developing and optimizing RDMA software.}

_{Active messages is a common messaging interface for various PGAS (Partitioned Global Address Space) APIs/libraries. In this talk will give an overview of the existing UCP active messaging API, including recently introduced rendezvous protocol capabilities.}

_{Mikhail Brinskii is a UCX developer with main focus on networking and HPC solutions. Before joining Nvidia, Mikhail had been working at Intel developing highly optimized Intel MPI Library.}

_{Huawei is unique vendor in the HPC landscape, producing all cluster components: Compute (Kunpeng 920 CPU), Storage (OceanStor solutions), Interconnect (RoCE NICs and Switches) and accelerators. This talk will present some of our recent work on UCX – and how it fits in the bigger picture of our HPC solutions.}

_{Alex is CS Ph.D. candidate in the area of MPI collective operations, software architect and developer in the areas surrounding MPI performance and low-latency communication. Today, team leader of the cluster computing team in Tel-Aviv Research center, Huawei.}

_{In this talk, we will give the community an update on what has been done over the last year with OpenSNAPI. We will also discuss two potential use case for OpenSNAPI for both SmartNICs as well as Computational Storage devices}

_{Steve is the Chief Architect for Next Generation Platforms at Los Alamos National Laboratory}

_{BlazingSQL is an open-source SQL engine built on GPUs as part of the RAPIDS ecosystem. We have recently added the capacity to leverage UCX as a communication protocol in our engine. We will discuss how we integrated UCX into BlazingSQL. First, we re-implemented our communication library to support multiple networking protocols. Second, we alternated between the NB and NBR APIs to find the most appropriate fit for our code-base. We’ll discuss our implementation, current results when compared to our non-UCX communication protocols, and our future plans for UCX.}

_{Rodrigo one of the cofounders of BlazingSQL, an open-source GPU accelerated SQL engine contained within a Python package. Rodrigo started his career as a consultant at Deloitte building analytics systems for the financial services industry before working on startups initially delivering analytics projects in Peru to multiple government entities and Latin American multi-nationals. Rodrigo started working on BlazingSQL with his brother, and they were inspired by work that came out of a consulting project for the Peruvian government. That work now makes its home within the RAPIDS GPU data science ecosystem to enable end-to-end workloads entirely on GPUs.}

_{Felipe is a maker. From aquaponics, beer and cheese-making to home automation. He is obsessed with creating. Before being CTO of BlazingSQL he and his brother had a consulting company based out of Peru where they originally built BlazingSQL as a tool to help them with their own consulting work. Before this he was the CTO of kWhOURs which provided a SaaS solution for energy auditing. Through BlazingSQL he has become a high performance junkie that spends nights dreaming about how hybrid processing systems are going to change the world.}

_{The Charm++ parallel programming system supports a variety of networking hardware through its machine layers, with UCX as a recent addition. In this talk, we will discuss the issues and results seen with the UCX machine layer over the last one year. Additionally, we will also talk about the upcoming feature of support for direct GPU-GPU communication with UCX in Charm++.}

_{Nitin Bhat is a software engineer at Charmworks and has been a core developer of Charm++ since 2015. Prior to Charmworks, he graduated with an MS in Computer Science from University of Illinois Urbana-Champaign, where he was working as a Research Assistant at the Parallel Programming Laboratory.}

_{Jaemin Choi is a PhD candidate and research assistant at the University of Illinois Urbana-Champaign. His research revolves around support for GPUs in the Charm++ parallel programming system, including asynchronous progress of GPU tasks and optimizations for inter-GPU communication.}

_{This talk will present various improvements done to improve performance of ROCm GPUs in UCX. I’ll also talk about future plans and what’s required to achieve them.}

_{Sourav Chakraborty works for the Radeon Technologies Group in AMD. His PhD research at The Ohio State University includes MPI semantics and implementation, fast interconnects, and scalable and reliable distributed systems.}

_{Score-P is a data acquisition tool which allows collecting profiling and trace data of MPI applications, part of the VI-HPS project. This talk will present (a) a brief introduction to some of the main VI-HPS (Score-P, Cube and Vampir), (b) UCX counters trace data acquisition using Score-P with the Huawei UCX statistics plugin, and (c) Visualization over Vampir.}

_{An Electrical and SW engineer with background in Statistical Signal Processing, Wireless Communications and Data Science. Past experience: Developing ARM based low-power IoT devices of different types: Cellular Handsets / Digital TV Receivers / Automotive (V2X) and GPON FTTH switches.}

_{This talk describes the building-blocks available from Arm for modern SmartNICs/DPUs along with relevant standards enabling operating systems to boot on Arm-based SmartNICs without modification. Support for interfaces like CXL in Arm IP that targets future SmartNIC architectures is also discussed.}

_{Kshitij the Technical Assistant to the GM of Arm’s Infrastructure business unit. He is a systems architect and focuses on enabling Arm customers targeted different market segments using Arm IP. Kshitij holds a PhD in computer architecture from University of Utah.}

_{UCC is a community-driven effort to develop collective API and library implementation for applications in various domains, including High-Performance Computing, Artificial Intelligence, Data Center, and I/O. Over the last few months, the UCC WG group has met weekly to develop the UCC specification. In this talk, I will highlight some of the design principles of the UCC v1.0 specification. I will also share the status of UCC implementation and upcoming plans of the working group. Further, we will share results from the experimental implementation – XCCL, which has helped make an informed decision regarding UCC interfaces and semantics.}

_{Manjunath Gorentla Venkata is an HPC software architect at NVIDIA. His focus is on programming models and network libraries for HPC systems. Previously, he was a research scientist and the Languages Team lead at Oak Ridge National Laboratory. He’s served on open standards committees for parallel programming models, including OpenSHMEM and MPI for many years, and he is the author of more than 50 research papers in this area. Manju earned Ph.D. and M.S. degrees in computer science from the University of New Mexico.}

_{Though UCX stands for unified communication, the focus has been on P2P. This talk presents the first part of our work on broadcast communication – inter-process communication within the same host. This new UCT transport significantly accelerates MPI’s Barrier, Bcast, Reduce, Allreduce, Scatter and Gather.}

_{Alex is CS Ph.D. candidate in the area of MPI collective operations, software architect and developer in the areas surrounding MPI performance and low-latency communication. Today, team leader of the cluster computing team in Tel-Aviv Research center, Huawei.}

_{Though UCX stands for unified communication, the focus has been on P2P. This talk presents the first part of our work on broadcast communication – inter-process communication within the same host. This new UCT transport significantly accelerates MPI’s Barrier, Bcast, Reduce, Allreduce, Scatter and Gather.}

_{Computer Engineering Bs.C., previously a Firmware Engineer for NICs at Mellanox in the areas of Quality of Service and Virtualization, now member of the cluster computing team in Tel-Aviv Research center, Huawei.}

_{A lot has happened since UCG’s 1st pull-request, back in 2018: the API has matured, optimizations across UCX have been implemented (some accepted upstream) by team members in China and in Israel, and this talk will present the journey so far – and the road ahead.}

_{Alex is CS Ph.D. candidate in the area of MPI collective operations, software architect and developer in the areas surrounding MPI performance and low-latency communication. Today, team leader of the cluster computing team in Tel-Aviv Research center, Huawei.}

_{Review interesting changes in the RDMA world over the last year, with a focus on elements interesting to the UXC community}

_{Jason is the maintainer for RDMA in the Linux kernel and userspace rdma-core.}

_{Recommender models comprise over 50% of the AI training workload in Facebook’s data centers. Facebook recently open-sourced a deep learning recommendation model (DLRM) that can be used as a tool for co-designing datacenter hardware and software for large-scale recommender model training. To these ends, we have implemented a new UCC process group for distributed PyTorch and have exposed various all-to-all and allreduce collectives through the UCC teams abstraction. We present an assessment of the suitability of the current UCC API for recommender workloads by running DLRM over the UCC process group on the Selene A100 SuperPOD on up to 256 GPUs and ConnectX-6 HCAs. We identify streams, collective topologies, and offload-devices as key abstractions that should be considered for inclusion to UCC.}

_{Josh Ladd is a senior director of networking software at Mellanox/Nvidia where he leads the HPC middleware research and development team. Prior to joining Mellanox/Nvidia, Josh was a computer scientist at ORNL, where his research centered on the design and implemented of collective communication frameworks for extreme-scale supercomputers. Josh earned his PhD in mathematics from Colorado State University.}

_{Application workflows use files to communicate between stages of data processing and analysis kernel executions. To address speed and granularity issues with this method, we employ persistent memory (PMEM) devices that provide DRAM-like speeds and byte granular access combined with persistent storage capabilities. We deploy an Arm-based Mellanox Bluefield SmartNIC with attached NVDIMM-N modules. Both SmartNIC and PMEM introduce API design and system software integration challenges. We address this with the design and implementation for an innovative client-server software architecture with a client API extension to the OpenSHMEM library.}

_{Megan Grodowitz has been a Staff Research Engineer at Arm, Ltd since joining Arm in 2017. Before Arm, she worked as computer science research staff at DoE and DoD labs since receiving her PhD from University of Notre Dame in 2011.}