m_cuda_spectral Module

Forward paired split of one dim2 mode against its mirror. The rule is symmetric, so the partner entry is pair_fw(partner, own, i2_rev).

Arguments

Return Value complex(kind=dp)

Backward counterpart of pair_fw, symmetric in the same way.

Arguments

Return Value complex(kind=dp)

Source index of entry i of a non-periodic line of n points gathered into a periodic one: the odd entries ascending, then the even entries descending, so that a plain FFT can stand in for the cosine transform. For an odd n the last entry lands at the centre, n/2 + 1. The single definition of the fold for the periodicity kernels: the enforce kernels gather through it and the undo kernels scatter through it, which keeps their reads contiguous. The OpenMP backend has the same function.

Arguments

Return Value real

Forward postprocess of one spectral entry in dim3 then dim1.

Arguments

Backward counterpart of postprocess_dim13_fw.

Arguments

Divide one entry by its spectral equivalence constant, with the eps_wave guard that keeps roundoff noise out of the pressure field.

Arguments

Copy data between x3d2 padded arrays and cuFFTMp descriptors

Arguments

Copy with transpose: src(nx, ny, nz) -> dst(ny, nx, nz) Used for 100 case forward FFT

Arguments

Copy with transpose back: src(ny, nx, nz) -> dst(nx, ny, nz) Used for 100 case backward FFT

Arguments

Used before R2C FFT for 110 case to put Z (periodic) in fast dim.

Arguments

Transpose back: src(nz, nx, ny) -> dst(nx, ny, nz) Used after C2R FFT for 110 case.

Arguments

Post-processes the divergence of velocity in spectral space, including scaling w.r.t. grid size.

Arguments

Post-processes the divergence of velocity in spectral space, including scaling w.r.t. grid size.

Arguments

Post-processes the divergence of velocity in spectral space, including scaling w.r.t. grid size.

Arguments

Solve the Poisson equation at cell centres with non-periodic BC along y

Arguments

Post-processes the divergence of velocity in spectral space, including scaling w.r.t. grid size.

Arguments

Stage 1 (forward): normalisation followed by the dim3 and dim1 postprocess. Every mode is handled on its own here.

Arguments

Stage 2 (forward): the dim2 paired split, then the Poisson solve.

Arguments

Stage 3 (backward): the dim2 paired recombination, then the dim3 and dim1 postprocess.

Arguments

Copies the first dim2 plane of the local spectral slab into a contiguous buffer so it can be handed to MPI.

Arguments

Step 1 (forward): normalise + Z periodic post-process Z is dim1 (serial j loop), periodic R2C — no sign flip needed since j only goes to nz/2+1. nx, ny are the local loop bounds (equal to nx_glob, ny_glob on a single rank); the normalisation divisor always needs the globals.

Arguments

Step 2 (forward): X paired even/odd split X is dim2 (thread i). Only i in [2, nx/2+1] executes. Writes to i and nx-i+2. Race-free (pair doesn't enter).

Arguments

Step 3 (forward): Y paired even/odd split Y is dim3 (blockIdx%y = k). Only k in [2, ny/2+1] executes. Writes to k and ny-k+2. Race-free (pair block doesn't enter).

Arguments

Step 4: Poisson solve — divide by waves x and y are cosine transformed, so neither has a Nyquist mode and the only singular entry is the mean, which the eps_wave test handles. The x Nyquist zeroing of the 010 case must not be copied here: in this layout x index nx/2 + 1 is the regular cosine mode N/2 (#465).

Arguments

Step 5 (backward): Y paired even/odd recombine

Arguments

Step 6 (backward): X paired even/odd recombine

Arguments

Multi-rank counterpart of process_spectral_110_x_pair_fw: the X paired split, but the partner mode lives on another rank, reached through the dim2 mirror exchange (see exchange_dim2_mirror). Unlike the single-rank kernel this only updates this rank's own modes; the partner rank updates its own copy the same way. At the global Nyquist mode (even grid size) the mirror partner is this rank's own pre-update value, so the formula below self-pairs it exactly as the single-rank kernel does.

Arguments

Multi-rank counterpart of process_spectral_110_x_pair_bw, see process_spectral_110_x_pair_fw_mirror.

Arguments

Step 7 (backward): Z periodic undo

Arguments

Packs the physical Z-slab field block (nx, ny, nz_l) into the MPI_Alltoall send buffer, selecting for each destination rank s the Y range [sny_l+1, (s+1)ny_l] it owns in the Y-slab layout.

Arguments

Unpacks the MPI_Alltoall receive buffer into the padded real view (2(nz/2+1), nx, ny_l) of the cuFFTMp Y-slab descriptor: data received from source rank s reconstructs the global Z range [snz_l+1, (s+1)*nz_l].

Arguments

Inverse of unpack_110_yslab: packs the Y-slab real view into the MPI_Alltoall send buffer, selecting for each destination rank s the Z range it originally owned in the physical Z-slab layout.

Arguments

Inverse of pack_110_xyz_to_yslab: unpacks into the physical Z-slab field block (nx, ny, nz_l). dst must be zeroed by the caller first, as fft_backward_110_cuda already does for the single rank path.

Arguments

Arguments

Arguments

Arguments

Arguments

Combined X and Y periodicity enforcement (interleave shuffle).

Arguments

Combined X and Y periodicity undo (reverse interleave shuffle).

Arguments