MFC
Exascale flow solver
Loading...
Searching...
No Matches
m_mpi_common Module Reference

MPI communication layer: domain decomposition, halo exchange, reductions, and parallel I/O setup. More...

Functions/Subroutines

impure subroutine s_initialize_mpi_common_module (exchange_all_chemistry_temperatures_in, use_rdma_transport_in, num_beta_vars_in)
 Initialize the module.
impure subroutine s_mpi_initialize
 Initialize the MPI execution environment and query the number of processors and local rank.
impure subroutine s_initialize_mpi_data (q_cons_vf, ib_markers, ib_mpi_data, beta, qbmm_pb, qbmm_mv)
 Set up MPI I/O data views and variable pointers for parallel file output.
subroutine s_initialize_mpi_data_ds (m_ds, n_ds, p_ds, q_cons_vf)
 Set up MPI I/O data views for downsampled (coarsened) parallel file output.
impure subroutine s_mpi_gather_data (my_vector, counts, gathered_vector, root)
 Gather variable-length real vectors from all MPI ranks onto the root process.
impure subroutine mpi_bcast_time_step_values (proc_time, time_avg)
 Gather per-rank time step wall-clock times onto rank 0 for performance reporting.
impure subroutine s_prohibit_abort (condition, message)
 Print a case file error with the prohibited condition and message, then abort execution.
impure subroutine s_mpi_reduce_stability_criteria_extrema (max_loc, min_loc, bubs_loc, max_glb, min_glb, bubs_glb)
 The goal of this subroutine is to determine the global extrema of the stability criteria in the computational domain. This is performed by sifting through the local extrema of each stability criterion. Note that each of the local extrema is from a single process, within its assigned section of the computational domain. Finally, note that the global extrema values are only bookkept on the rank 0 processor. max_loc/max_glb hold the max-reduced criteria (ICFL, VCFL, CCFL, TCFL) and min_loc/min_glb hold the min-reduced criteria (Rc).
subroutine s_mpi_reduce_int_sum (var_loc, sum)
 Reduce a local integer value to its global sum across all MPI ranks.
impure subroutine s_mpi_allreduce_sum (var_loc, var_glb)
 Reduce a local real value to its global sum across all MPI ranks.
impure subroutine s_mpi_allreduce_vectors_sum (var_loc, var_glb, num_vectors, vector_length)
 Reduce an array of vectors to their global sums across all MPI ranks.
impure subroutine s_mpi_allreduce_integer_sum (var_loc, var_glb)
 Reduce a local integer value to its global sum across all MPI ranks.
impure subroutine s_mpi_allreduce_integer_min (var_loc, var_glb)
 Reduce a local integer value to its global minimum across all MPI ranks.
impure subroutine s_mpi_allreduce_integer_max (var_loc, var_glb)
 Reduce a local integer value to its global maximum across all MPI ranks.
impure subroutine s_mpi_allreduce_integer_sum_vec (var_loc, var_glb)
 Reduce an integer(8) array to its element-wise global sum across all MPI ranks.
impure subroutine s_mpi_allreduce_min (var_loc, var_glb)
 Reduce a local real value to its global minimum across all MPI ranks.
impure subroutine s_mpi_allreduce_min_vec (var_loc, var_glb)
 Reduce a local real vector to its elementwise global minimum across all MPI ranks.
impure subroutine s_mpi_allreduce_max (var_loc, var_glb)
 Reduce a local real value to its global maximum across all MPI ranks.
impure subroutine s_mpi_allreduce_array_max (arr, arr_len)
 In-place elementwise max-reduction of a real array across all ranks: each element is written (identically) by its owner(s) and sentinel-low elsewhere, so MAX recovers the global array.
impure subroutine s_mpi_reduce_min (var_loc)
 Reduce a local real value to its global minimum across all ranks.
impure subroutine s_mpi_reduce_maxloc (var_loc)
 Reduce a 2-element variable to its global maximum value with the owning processor rank (MPI_MAXLOC). Reduce a local value to its global maximum with location (rank) across all ranks.
impure subroutine s_mpi_abort (prnt, code)
 The subroutine terminates the MPI execution environment.
impure subroutine s_check_mpi_file_open (ierr, file_loc)
 Abort naming the file if MPI_FILE_OPEN failed. File errors default to MPI_ERRORS_RETURN, so an unchecked failure carries on with an invalid handle and crashes somewhere unrelated.
impure subroutine s_mpi_barrier
 Halts all processes until all have reached barrier.
impure subroutine s_mpi_finalize
 The subroutine finalizes the MPI execution environment.
subroutine s_mpi_sendrecv_variables_buffers (q_comm, mpi_dir, pbc_loc, nvar, pb_in, mv_in, q_t_sf)
 The goal of this procedure is to populate the buffers of the cell-average conservative variables by communicating with the neighboring processors.
subroutine s_mpi_reduce_beta_variables_buffers (q_comm, kahan_comp, mpi_dir, pbc_loc, nvar)
 The goal of this procedure is to populate the buffers of the cell-average conservative variables by communicating with the neighboring processors.
subroutine s_mpi_decompose_computational_domain (write_silo_ghost_offsets, adjust_local_domains, output_offsets, local_domains)
 The purpose of this procedure is to optimally decompose the computational domain among the available processors. This is performed by attempting to award each processor, in each of the coordinate directions, approximately the same number of cells, and then recomputing the affected global parameters.
subroutine, private s_apply_decomposition_policies (proc_counts, remainders, local_cells, global_cells, write_silo_ghost_offsets, adjust_local_domains, output_offsets, local_domains)
 Apply executable-configured output and local-domain policies after the shared Cartesian decomposition.
subroutine s_mpi_sendrecv_grid_variable_buffer (cell_boundaries, cell_centers, cell_widths, num_cells, bc_bounds, pbc_loc, offset)
 The goal of this procedure is to populate the buffers of the grid variables by communicating with the neighboring processors. Note that only the buffers of the cell-width distributions are handled in such a way. This is because the buffers of cell-boundary locations may be calculated directly from those of the cell-width distributions.
subroutine s_apply_grid_from_global_dim (x_cb_glb, m_dim_glb, m_dim, sidx, bc_beg, bc_end, cb_lo, cb_hi, cw_lo, cw_hi, x_cb_loc, x_cc_loc, dx_loc)
 Populate the local cell-boundary, cell-center, and cell-width arrays in one direction directly from the global cell-boundary array. This guarantees that every rank sees bitwise-identical values at any shared physical cell or boundary.
impure subroutine s_finalize_mpi_common_module
 Module deallocation and/or disassociation procedures.

Variables

integer, private v_size
real(wp), dimension(:), allocatable, private buff_send
 Primitive variable send buffer for halo exchange.
real(wp), dimension(:), allocatable, private buff_recv
 Primitive variable receive buffer for halo exchange Variables for EL bubbles communication.
type(int_bounds_info), dimension(3) comm_coords
integer, dimension(3) comm_size
integer, parameter num_beta_vars_max = 11
 q_beta indices to communicate, set by s_populate_beta_buffers
integer, dimension(1:num_beta_vars_max) beta_vars = 0
integer num_beta_vars = 0
 Most beta fields exchanged at once; sizes the beta halo.
integer(kind=8) halo_size
logical, private exchange_all_chemistry_temperatures = .false.
logical, private use_rdma_transport = .false.

Detailed Description

MPI communication layer: domain decomposition, halo exchange, reductions, and parallel I/O setup.

Function/Subroutine Documentation

◆ mpi_bcast_time_step_values()

impure subroutine m_mpi_common::mpi_bcast_time_step_values ( real(wp), dimension(0:num_procs - 1), intent(inout) proc_time,
real(wp), intent(inout) time_avg )

Gather per-rank time step wall-clock times onto rank 0 for performance reporting.

Definition at line 805 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_apply_decomposition_policies()

subroutine, private m_mpi_common::s_apply_decomposition_policies ( integer, dimension(3), intent(in) proc_counts,
integer, dimension(3), intent(in) remainders,
integer, dimension(3), intent(in) local_cells,
integer, dimension(3), intent(in) global_cells,
logical, intent(in) write_silo_ghost_offsets,
logical, intent(in) adjust_local_domains,
type(int_bounds_info), dimension(3), intent(inout), optional output_offsets,
type(bounds_info), dimension(3), intent(inout), optional local_domains )
private

Apply executable-configured output and local-domain policies after the shared Cartesian decomposition.

Definition at line 3524 of file m_mpi_common.fpp.f90.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ s_apply_grid_from_global_dim()

subroutine m_mpi_common::s_apply_grid_from_global_dim ( real(wp), dimension(-1:m_dim_glb), intent(in) x_cb_glb,
integer, intent(in) m_dim_glb,
integer, intent(in) m_dim,
integer, intent(in) sidx,
integer, intent(in) bc_beg,
integer, intent(in) bc_end,
integer, intent(in) cb_lo,
integer, intent(in) cb_hi,
integer, intent(in) cw_lo,
integer, intent(in) cw_hi,
real(wp), dimension(-1 - cb_lo:m_dim + cb_hi), intent(inout) x_cb_loc,
real(wp), dimension(-cw_lo:m_dim + cw_hi), intent(inout) x_cc_loc,
real(wp), dimension(-cw_lo:m_dim + cw_hi), intent(inout) dx_loc )

Populate the local cell-boundary, cell-center, and cell-width arrays in one direction directly from the global cell-boundary array. This guarantees that every rank sees bitwise-identical values at any shared physical cell or boundary.

Definition at line 3643 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_check_mpi_file_open()

impure subroutine m_mpi_common::s_check_mpi_file_open ( integer, intent(in) ierr,
character(len=*), intent(in) file_loc )

Abort naming the file if MPI_FILE_OPEN failed. File errors default to MPI_ERRORS_RETURN, so an unchecked failure carries on with an invalid handle and crashes somewhere unrelated.

Definition at line 1129 of file m_mpi_common.fpp.f90.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ s_finalize_mpi_common_module()

impure subroutine m_mpi_common::s_finalize_mpi_common_module

Module deallocation and/or disassociation procedures.

Definition at line 3721 of file m_mpi_common.fpp.f90.

◆ s_initialize_mpi_common_module()

impure subroutine m_mpi_common::s_initialize_mpi_common_module ( logical, intent(in) exchange_all_chemistry_temperatures_in,
logical, intent(in) use_rdma_transport_in,
integer, intent(in) num_beta_vars_in )

Initialize the module.

Definition at line 470 of file m_mpi_common.fpp.f90.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ s_initialize_mpi_data()

impure subroutine m_mpi_common::s_initialize_mpi_data ( type(scalar_field), dimension(sys_size), intent(in) q_cons_vf,
type(integer_field), intent(in), optional ib_markers,
type(mpi_io_ib_var), intent(inout), optional ib_mpi_data,
type(scalar_field), intent(in), optional beta,
type(pres_field), intent(in), optional qbmm_pb,
type(pres_field), intent(in), optional qbmm_mv )

Set up MPI I/O data views and variable pointers for parallel file output.

Definition at line 652 of file m_mpi_common.fpp.f90.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ s_initialize_mpi_data_ds()

subroutine m_mpi_common::s_initialize_mpi_data_ds ( integer, intent(in) m_ds,
integer, intent(in) n_ds,
integer, intent(in) p_ds,
type(scalar_field), dimension(sys_size), intent(in), optional q_cons_vf )

Set up MPI I/O data views for downsampled (coarsened) parallel file output.

Definition at line 735 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_abort()

impure subroutine m_mpi_common::s_mpi_abort ( character(len=*), intent(in), optional prnt,
integer, intent(in), optional code )

The subroutine terminates the MPI execution environment.

Definition at line 1099 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_allreduce_array_max()

impure subroutine m_mpi_common::s_mpi_allreduce_array_max ( real(wp), dimension(arr_len), intent(inout) arr,
integer, intent(in) arr_len )

In-place elementwise max-reduction of a real array across all ranks: each element is written (identically) by its owner(s) and sentinel-low elsewhere, so MAX recovers the global array.

Definition at line 1049 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_allreduce_integer_max()

impure subroutine m_mpi_common::s_mpi_allreduce_integer_max ( integer, intent(in) var_loc,
integer, intent(out) var_glb )

Reduce a local integer value to its global maximum across all MPI ranks.

Definition at line 968 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_allreduce_integer_min()

impure subroutine m_mpi_common::s_mpi_allreduce_integer_min ( integer, intent(in) var_loc,
integer, intent(out) var_glb )

Reduce a local integer value to its global minimum across all MPI ranks.

Definition at line 952 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_allreduce_integer_sum()

impure subroutine m_mpi_common::s_mpi_allreduce_integer_sum ( integer(kind=8), intent(in) var_loc,
integer(kind=8), intent(out) var_glb )

Reduce a local integer value to its global sum across all MPI ranks.

Definition at line 936 of file m_mpi_common.fpp.f90.

◆ s_mpi_allreduce_integer_sum_vec()

impure subroutine m_mpi_common::s_mpi_allreduce_integer_sum_vec ( integer(kind=8), dimension(:), intent(in) var_loc,
integer(kind=8), dimension(:), intent(out) var_glb )

Reduce an integer(8) array to its element-wise global sum across all MPI ranks.

Definition at line 984 of file m_mpi_common.fpp.f90.

◆ s_mpi_allreduce_max()

impure subroutine m_mpi_common::s_mpi_allreduce_max ( real(wp), intent(in) var_loc,
real(wp), intent(out) var_glb )

Reduce a local real value to its global maximum across all MPI ranks.

Definition at line 1032 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_allreduce_min()

impure subroutine m_mpi_common::s_mpi_allreduce_min ( real(wp), intent(in) var_loc,
real(wp), intent(out) var_glb )

Reduce a local real value to its global minimum across all MPI ranks.

Definition at line 1000 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_allreduce_min_vec()

impure subroutine m_mpi_common::s_mpi_allreduce_min_vec ( real(wp), dimension(:), intent(in) var_loc,
real(wp), dimension(:), intent(out) var_glb )

Reduce a local real vector to its elementwise global minimum across all MPI ranks.

Definition at line 1016 of file m_mpi_common.fpp.f90.

◆ s_mpi_allreduce_sum()

impure subroutine m_mpi_common::s_mpi_allreduce_sum ( real(wp), intent(in) var_loc,
real(wp), intent(out) var_glb )

Reduce a local real value to its global sum across all MPI ranks.

Definition at line 899 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_allreduce_vectors_sum()

impure subroutine m_mpi_common::s_mpi_allreduce_vectors_sum ( real(wp), dimension(:,:), intent(in) var_loc,
real(wp), dimension(:,:), intent(inout) var_glb,
integer, intent(in) num_vectors,
integer, intent(in) vector_length )

Reduce an array of vectors to their global sums across all MPI ranks.

Definition at line 915 of file m_mpi_common.fpp.f90.

◆ s_mpi_barrier()

impure subroutine m_mpi_common::s_mpi_barrier

Halts all processes until all have reached barrier.

Definition at line 1146 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_decompose_computational_domain()

subroutine m_mpi_common::s_mpi_decompose_computational_domain ( logical, intent(in) write_silo_ghost_offsets,
logical, intent(in) adjust_local_domains,
type(int_bounds_info), dimension(3), intent(inout), optional output_offsets,
type(bounds_info), dimension(3), intent(inout), optional local_domains )

The purpose of this procedure is to optimally decompose the computational domain among the available processors. This is performed by attempting to award each processor, in each of the coordinate directions, approximately the same number of cells, and then recomputing the affected global parameters.

Non-optimal number of processors in the x-, y- and z-directions

Definition at line 3142 of file m_mpi_common.fpp.f90.

Here is the call graph for this function:

◆ s_mpi_finalize()

impure subroutine m_mpi_common::s_mpi_finalize

The subroutine finalizes the MPI execution environment.

Definition at line 1157 of file m_mpi_common.fpp.f90.

◆ s_mpi_gather_data()

impure subroutine m_mpi_common::s_mpi_gather_data ( real(wp), dimension(counts), intent(in) my_vector,
integer, intent(in) counts,
real(wp), dimension(:), intent(out), allocatable gathered_vector,
integer, intent(in) root )

Gather variable-length real vectors from all MPI ranks onto the root process.

Parameters
[in]countsArray of vector lengths for each process
[in]my_vectorInput vector on each process
[in]rootRank of the root process
[out]gathered_vectorGathered vector on the root process

Definition at line 772 of file m_mpi_common.fpp.f90.

◆ s_mpi_initialize()

impure subroutine m_mpi_common::s_mpi_initialize

Initialize the MPI execution environment and query the number of processors and local rank.

Definition at line 616 of file m_mpi_common.fpp.f90.

◆ s_mpi_reduce_beta_variables_buffers()

subroutine m_mpi_common::s_mpi_reduce_beta_variables_buffers ( type(scalar_field), dimension(1:), intent(inout) q_comm,
type(scalar_field), dimension(1:), intent(inout) kahan_comp,
integer, intent(in) mpi_dir,
integer, intent(in) pbc_loc,
integer, intent(in) nvar )

The goal of this procedure is to populate the buffers of the cell-average conservative variables by communicating with the neighboring processors.

Parameters
q_cons_vfCell-average conservative variables
mpi_dirMPI communication coordinate direction
pbc_locProcessor boundary condition (PBC) location

Definition at line 2578 of file m_mpi_common.fpp.f90.

◆ s_mpi_reduce_int_sum()

subroutine m_mpi_common::s_mpi_reduce_int_sum ( integer, intent(in) var_loc,
integer, intent(out) sum )

Reduce a local integer value to its global sum across all MPI ranks.

Definition at line 883 of file m_mpi_common.fpp.f90.

Here is the caller graph for this function:

◆ s_mpi_reduce_maxloc()

impure subroutine m_mpi_common::s_mpi_reduce_maxloc ( real(wp), dimension(2), intent(inout) var_loc)

Reduce a 2-element variable to its global maximum value with the owning processor rank (MPI_MAXLOC). Reduce a local value to its global maximum with location (rank) across all ranks.

Definition at line 1082 of file m_mpi_common.fpp.f90.

◆ s_mpi_reduce_min()

impure subroutine m_mpi_common::s_mpi_reduce_min ( real(wp), intent(inout) var_loc)

Reduce a local real value to its global minimum across all ranks.

Definition at line 1063 of file m_mpi_common.fpp.f90.

◆ s_mpi_reduce_stability_criteria_extrema()

impure subroutine m_mpi_common::s_mpi_reduce_stability_criteria_extrema ( real(wp), dimension(:), intent(in) max_loc,
real(wp), dimension(:), intent(in) min_loc,
integer, intent(in) bubs_loc,
real(wp), dimension(:), intent(out) max_glb,
real(wp), dimension(:), intent(out) min_glb,
integer, intent(out) bubs_glb )

The goal of this subroutine is to determine the global extrema of the stability criteria in the computational domain. This is performed by sifting through the local extrema of each stability criterion. Note that each of the local extrema is from a single process, within its assigned section of the computational domain. Finally, note that the global extrema values are only bookkept on the rank 0 processor. max_loc/max_glb hold the max-reduced criteria (ICFL, VCFL, CCFL, TCFL) and min_loc/min_glb hold the min-reduced criteria (Rc).

Definition at line 839 of file m_mpi_common.fpp.f90.

◆ s_mpi_sendrecv_grid_variable_buffer()

subroutine m_mpi_common::s_mpi_sendrecv_grid_variable_buffer ( real(wp), dimension(-1 - offset%beg:), intent(inout), contiguous cell_boundaries,
real(wp), dimension(-buff_size:), intent(inout), contiguous cell_centers,
real(wp), dimension(-buff_size:), intent(inout), contiguous cell_widths,
integer, intent(in) num_cells,
type(int_bounds_info), intent(in) bc_bounds,
integer, intent(in) pbc_loc,
type(int_bounds_info), intent(in) offset )

The goal of this procedure is to populate the buffers of the grid variables by communicating with the neighboring processors. Note that only the buffers of the cell-width distributions are handled in such a way. This is because the buffers of cell-boundary locations may be calculated directly from those of the cell-width distributions.

Definition at line 3588 of file m_mpi_common.fpp.f90.

◆ s_mpi_sendrecv_variables_buffers()

subroutine m_mpi_common::s_mpi_sendrecv_variables_buffers ( type(scalar_field), dimension(1:), intent(inout) q_comm,
integer, intent(in) mpi_dir,
integer, intent(in) pbc_loc,
integer, intent(in) nvar,
real(stp), dimension(idwbuff(1)%beg:,idwbuff(2)%beg:,idwbuff(3)%beg:,1:,1:), intent(inout), optional pb_in,
real(stp), dimension(idwbuff(1)%beg:,idwbuff(2)%beg:,idwbuff(3)%beg:,1:,1:), intent(inout), optional mv_in,
type(scalar_field), intent(inout), optional q_t_sf )

The goal of this procedure is to populate the buffers of the cell-average conservative variables by communicating with the neighboring processors.

Definition at line 1169 of file m_mpi_common.fpp.f90.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ s_prohibit_abort()

impure subroutine m_mpi_common::s_prohibit_abort ( character(len=*), intent(in) condition,
character(len=*), intent(in) message )

Print a case file error with the prohibited condition and message, then abort execution.

Definition at line 819 of file m_mpi_common.fpp.f90.

Here is the call graph for this function:

Variable Documentation

◆ beta_vars

integer, dimension(1:num_beta_vars_max) m_mpi_common::beta_vars = 0

Definition at line 420 of file m_mpi_common.fpp.f90.

◆ buff_recv

real(wp), dimension(:), allocatable, private m_mpi_common::buff_recv
private

Primitive variable receive buffer for halo exchange Variables for EL bubbles communication.

Definition at line 415 of file m_mpi_common.fpp.f90.

◆ buff_send

real(wp), dimension(:), allocatable, private m_mpi_common::buff_send
private

Primitive variable send buffer for halo exchange.

Definition at line 413 of file m_mpi_common.fpp.f90.

◆ comm_coords

type(int_bounds_info), dimension(3) m_mpi_common::comm_coords

Definition at line 416 of file m_mpi_common.fpp.f90.

◆ comm_size

integer, dimension(3) m_mpi_common::comm_size

Definition at line 417 of file m_mpi_common.fpp.f90.

◆ exchange_all_chemistry_temperatures

logical, private m_mpi_common::exchange_all_chemistry_temperatures = .false.
private

Definition at line 464 of file m_mpi_common.fpp.f90.

◆ halo_size

integer(kind=8) m_mpi_common::halo_size

Definition at line 450 of file m_mpi_common.fpp.f90.

◆ num_beta_vars

integer m_mpi_common::num_beta_vars = 0

Most beta fields exchanged at once; sizes the beta halo.

Definition at line 421 of file m_mpi_common.fpp.f90.

◆ num_beta_vars_max

integer, parameter m_mpi_common::num_beta_vars_max = 11

q_beta indices to communicate, set by s_populate_beta_buffers

Definition at line 419 of file m_mpi_common.fpp.f90.

◆ use_rdma_transport

logical, private m_mpi_common::use_rdma_transport = .false.
private

Definition at line 465 of file m_mpi_common.fpp.f90.

◆ v_size

integer, private m_mpi_common::v_size
private

Definition at line 399 of file m_mpi_common.fpp.f90.