m_omptgt_sendrecv Module

Halo exchange of device buffers for the OpenMP target backend, following cuda/sendrecv.f90.

With more than one rank the device addresses go straight to MPI, which therefore has to be GPU-aware, as it has to be for the CUDA backend. UCX tells device from host buffers with a memory type cache fed by intercepting CUDA allocations, and it can miss those the OpenMP runtime makes: it then reads the buffers as host memory, failing in process_vm_readv with "Bad address". Run with UCX_MEMTYPE_CACHE=n so UCX queries the driver for every buffer instead. The send buffers are filled by a kernel that the runtime may still have queued (see device_sync), so the device is synchronised before MPI reads them, as the CUDA backend synchronises its stream.


Uses

  • module~~m_omptgt_sendrecv~~UsesGraph module~m_omptgt_sendrecv m_omptgt_sendrecv iso_c_binding iso_c_binding module~m_omptgt_sendrecv->iso_c_binding m_mpi m_mpi module~m_omptgt_sendrecv->m_mpi module~m_common m_common module~m_omptgt_sendrecv->module~m_common module~m_omptgt_common m_omptgt_common module~m_omptgt_sendrecv->module~m_omptgt_common module~m_common->m_mpi module~m_logger m_logger module~m_common->module~m_logger module~m_omptgt_common->iso_c_binding module~m_omptgt_common->module~m_common module~m_omptgt_common->module~m_logger omp_lib omp_lib module~m_omptgt_common->omp_lib module~m_logger->m_mpi iso_fortran_env iso_fortran_env module~m_logger->iso_fortran_env

Used by

  • module~~m_omptgt_sendrecv~~UsedByGraph module~m_omptgt_sendrecv m_omptgt_sendrecv module~m_omptgt_backend m_omptgt_backend module~m_omptgt_backend->module~m_omptgt_sendrecv module~m_omptgt_exec_dist m_omptgt_exec_dist module~m_omptgt_backend->module~m_omptgt_exec_dist module~m_omptgt_exec_dist->module~m_omptgt_sendrecv module~m_backend_setup m_backend_setup module~m_backend_setup->module~m_omptgt_backend program~x3d2_poisson_solver x3d2_poisson_solver program~x3d2_poisson_solver->module~m_backend_setup program~xcompact xcompact program~xcompact->module~m_backend_setup

Subroutines

public subroutine sendrecv_fields(f_recv_s, f_recv_e, f_send_s, f_send_e, n_data, nproc, prev, next)

Arguments

Type IntentOptional Attributes Name
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f_recv_s
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f_recv_e
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f_send_s
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f_send_e
integer, intent(in) :: n_data
integer, intent(in) :: nproc
integer, intent(in) :: prev
integer, intent(in) :: next

public subroutine sendrecv_3fields(f1_recv_s, f1_recv_e, f2_recv_s, f2_recv_e, f3_recv_s, f3_recv_e, f1_send_s, f1_send_e, f2_send_s, f2_send_e, f3_send_s, f3_send_e, n_data, nproc, prev, next)

Arguments

Type IntentOptional Attributes Name
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f1_recv_s
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f1_recv_e
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f2_recv_s
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f2_recv_e
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f3_recv_s
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f3_recv_e
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f1_send_s
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f1_send_e
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f2_send_s
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f2_send_e
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f3_send_s
real(kind=dp), intent(in), pointer, contiguous, dimension(:, :, :) :: f3_send_e
integer, intent(in) :: n_data
integer, intent(in) :: nproc
integer, intent(in) :: prev
integer, intent(in) :: next