Halo exchange of device buffers for the OpenMP target backend, following
cuda/sendrecv.f90.
With more than one rank the device addresses go straight to MPI, which
therefore has to be GPU-aware, as it has to be for the CUDA backend.
UCX tells device from host buffers with a memory type cache fed by
intercepting CUDA allocations, and it can miss those the OpenMP runtime
makes: it then reads the buffers as host memory, failing in
process_vm_readv with "Bad address". Run with UCX_MEMTYPE_CACHE=n so UCX
queries the driver for every buffer instead.
The send buffers are filled by a kernel that the runtime may still have
queued (see device_sync), so the device is synchronised before MPI
reads them, as the CUDA backend synchronises its stream.
Nodes of different colours represent the following:
Solid arrows point from a submodule to the (sub)module which it is
descended from. Dashed arrows point from a module or program unit to
modules which it uses.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a submodule to the (sub)module which it is
descended from. Dashed arrows point from a module or program unit to
modules which it uses.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Subroutines
public subroutine sendrecv_fields(f_recv_s, f_recv_e, f_send_s, f_send_e, n_data, nproc, prev, next)