Halo exchange of device buffers for the OpenMP target backend, following cuda/sendrecv.f90.
With more than one rank the device addresses go straight to MPI, which therefore has to be GPU-aware, as it has to be for the CUDA backend. UCX tells device from host buffers with a memory type cache fed by intercepting CUDA allocations, and it can miss those the OpenMP runtime makes: it then reads the buffers as host memory, failing in process_vm_readv with "Bad address". Run with UCX_MEMTYPE_CACHE=n so UCX queries the driver for every buffer instead. The send buffers are filled by a kernel that the runtime may still have queued (see device_sync), so the device is synchronised before MPI reads them, as the CUDA backend synchronises its stream.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f_recv_s | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f_recv_e | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f_send_s | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f_send_e | ||
| integer, | intent(in) | :: | n_data | |||
| integer, | intent(in) | :: | nproc | |||
| integer, | intent(in) | :: | prev | |||
| integer, | intent(in) | :: | next |
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f1_recv_s | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f1_recv_e | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f2_recv_s | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f2_recv_e | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f3_recv_s | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f3_recv_e | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f1_send_s | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f1_send_e | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f2_send_s | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f2_send_e | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f3_send_s | ||
| real(kind=dp), | intent(in), | pointer, contiguous, dimension(:, :, :) | :: | f3_send_e | ||
| integer, | intent(in) | :: | n_data | |||
| integer, | intent(in) | :: | nproc | |||
| integer, | intent(in) | :: | prev | |||
| integer, | intent(in) | :: | next |