|
MFC
Exascale flow solver
|
|
MFC
Exascale flow solver
|
MPI communication layer: domain decomposition, halo exchange, reductions, and parallel I/O setup. More...
Functions/Subroutines | |
| impure subroutine | s_initialize_mpi_common_module (exchange_all_chemistry_temperatures_in, use_rdma_transport_in) |
| Initialize the module. | |
| impure subroutine | s_mpi_initialize |
| Initialize the MPI execution environment and query the number of processors and local rank. | |
| impure subroutine | s_initialize_mpi_data (q_cons_vf, ib_markers, ib_mpi_data, beta, qbmm_pb, qbmm_mv) |
| Set up MPI I/O data views and variable pointers for parallel file output. | |
| subroutine | s_initialize_mpi_data_ds (m_ds, n_ds, p_ds, q_cons_vf) |
| Set up MPI I/O data views for downsampled (coarsened) parallel file output. | |
| impure subroutine | s_mpi_gather_data (my_vector, counts, gathered_vector, root) |
| Gather variable-length real vectors from all MPI ranks onto the root process. | |
| impure subroutine | mpi_bcast_time_step_values (proc_time, time_avg) |
| Gather per-rank time step wall-clock times onto rank 0 for performance reporting. | |
| impure subroutine | s_prohibit_abort (condition, message) |
| Print a case file error with the prohibited condition and message, then abort execution. | |
| impure subroutine | s_mpi_reduce_stability_criteria_extrema (max_loc, min_loc, bubs_loc, max_glb, min_glb, bubs_glb) |
| The goal of this subroutine is to determine the global extrema of the stability criteria in the computational domain. This is performed by sifting through the local extrema of each stability criterion. Note that each of the local extrema is from a single process, within its assigned section of the computational domain. Finally, note that the global extrema values are only bookkept on the rank 0 processor. max_loc/max_glb hold the max-reduced criteria (ICFL, VCFL, CCFL, TCFL) and min_loc/min_glb hold the min-reduced criteria (Rc). | |
| subroutine | s_mpi_reduce_int_sum (var_loc, sum) |
| Reduce a local integer value to its global sum across all MPI ranks. | |
| impure subroutine | s_mpi_allreduce_sum (var_loc, var_glb) |
| Reduce a local real value to its global sum across all MPI ranks. | |
| impure subroutine | s_mpi_allreduce_vectors_sum (var_loc, var_glb, num_vectors, vector_length) |
| Reduce an array of vectors to their global sums across all MPI ranks. | |
| impure subroutine | s_mpi_allreduce_integer_sum (var_loc, var_glb) |
| Reduce a local integer value to its global sum across all MPI ranks. | |
| impure subroutine | s_mpi_allreduce_integer_sum_vec (var_loc, var_glb) |
| Reduce an integer(8) array to its element-wise global sum across all MPI ranks. | |
| impure subroutine | s_mpi_allreduce_min (var_loc, var_glb) |
| Reduce a local real value to its global minimum across all MPI ranks. | |
| impure subroutine | s_mpi_allreduce_min_vec (var_loc, var_glb) |
| Reduce a local real vector to its elementwise global minimum across all MPI ranks. | |
| impure subroutine | s_mpi_allreduce_max (var_loc, var_glb) |
| Reduce a local real value to its global maximum across all MPI ranks. | |
| impure subroutine | s_mpi_reduce_min (var_loc) |
| Reduce a local real value to its global minimum across all ranks. | |
| impure subroutine | s_mpi_reduce_maxloc (var_loc) |
| Reduce a 2-element variable to its global maximum value with the owning processor rank (MPI_MAXLOC). Reduce a local value to its global maximum with location (rank) across all ranks. | |
| impure subroutine | s_mpi_abort (prnt, code) |
| The subroutine terminates the MPI execution environment. | |
| impure subroutine | s_check_mpi_file_open (ierr, file_loc) |
| Abort naming the file if MPI_FILE_OPEN failed. File errors default to MPI_ERRORS_RETURN, so an unchecked failure carries on with an invalid handle and crashes somewhere unrelated. | |
| impure subroutine | s_mpi_barrier |
| Halts all processes until all have reached barrier. | |
| impure subroutine | s_mpi_finalize |
| The subroutine finalizes the MPI execution environment. | |
| subroutine | s_mpi_sendrecv_variables_buffers (q_comm, mpi_dir, pbc_loc, nvar, pb_in, mv_in, q_t_sf) |
| The goal of this procedure is to populate the buffers of the cell-average conservative variables by communicating with the neighboring processors. | |
| subroutine | s_mpi_reduce_beta_variables_buffers (q_comm, kahan_comp, mpi_dir, pbc_loc, nvar) |
| The goal of this procedure is to populate the buffers of the cell-average conservative variables by communicating with the neighboring processors. | |
| subroutine | s_mpi_decompose_computational_domain (write_silo_ghost_offsets, adjust_local_domains, output_offsets, local_domains) |
| The purpose of this procedure is to optimally decompose the computational domain among the available processors. This is performed by attempting to award each processor, in each of the coordinate directions, approximately the same number of cells, and then recomputing the affected global parameters. | |
| subroutine, private | s_apply_decomposition_policies (proc_counts, remainders, local_cells, global_cells, write_silo_ghost_offsets, adjust_local_domains, output_offsets, local_domains) |
| Apply executable-configured output and local-domain policies after the shared Cartesian decomposition. | |
| subroutine | s_mpi_sendrecv_grid_variable_buffer (cell_boundaries, cell_centers, cell_widths, num_cells, bc_bounds, pbc_loc, offset) |
| The goal of this procedure is to populate the buffers of the grid variables by communicating with the neighboring processors. Note that only the buffers of the cell-width distributions are handled in such a way. This is because the buffers of cell-boundary locations may be calculated directly from those of the cell-width distributions. | |
| subroutine | s_apply_grid_from_global_dim (x_cb_glb, m_dim_glb, m_dim, sidx, bc_beg, bc_end, cb_lo, cb_hi, cw_lo, cw_hi, x_cb_loc, x_cc_loc, dx_loc) |
| Populate the local cell-boundary, cell-center, and cell-width arrays in one direction directly from the global cell-boundary array. This guarantees that every rank sees bitwise-identical values at any shared physical cell or boundary. | |
| impure subroutine | s_finalize_mpi_common_module |
| Module deallocation and/or disassociation procedures. | |
Variables | |
| integer, private | v_size |
| real(wp), dimension(:), allocatable, private | buff_send |
| Primitive variable send buffer for halo exchange. | |
| real(wp), dimension(:), allocatable, private | buff_recv |
| Primitive variable receive buffer for halo exchange Variables for EL bubbles communication. | |
| type(int_bounds_info), dimension(3) | comm_coords |
| integer, dimension(3) | comm_size |
| integer, dimension(1:3) | beta_vars = [1, 2, 5] |
| q_beta indices to communicate: 1=void fraction, 2=d(beta)/dt, 5=energy source | |
| integer(kind=8) | halo_size |
| logical, private | exchange_all_chemistry_temperatures = .false. |
| logical, private | use_rdma_transport = .false. |
MPI communication layer: domain decomposition, halo exchange, reductions, and parallel I/O setup.
| impure subroutine m_mpi_common::mpi_bcast_time_step_values | ( | real(wp), dimension(0:num_procs - 1), intent(inout) | proc_time, |
| real(wp), intent(inout) | time_avg ) |
Gather per-rank time step wall-clock times onto rank 0 for performance reporting.
Definition at line 765 of file m_mpi_common.fpp.f90.
|
private |
Apply executable-configured output and local-domain policies after the shared Cartesian decomposition.
Definition at line 3446 of file m_mpi_common.fpp.f90.
| subroutine m_mpi_common::s_apply_grid_from_global_dim | ( | real(wp), dimension(-1:m_dim_glb), intent(in) | x_cb_glb, |
| integer, intent(in) | m_dim_glb, | ||
| integer, intent(in) | m_dim, | ||
| integer, intent(in) | sidx, | ||
| integer, intent(in) | bc_beg, | ||
| integer, intent(in) | bc_end, | ||
| integer, intent(in) | cb_lo, | ||
| integer, intent(in) | cb_hi, | ||
| integer, intent(in) | cw_lo, | ||
| integer, intent(in) | cw_hi, | ||
| real(wp), dimension(-1 - cb_lo:m_dim + cb_hi), intent(inout) | x_cb_loc, | ||
| real(wp), dimension(-cw_lo:m_dim + cw_hi), intent(inout) | x_cc_loc, | ||
| real(wp), dimension(-cw_lo:m_dim + cw_hi), intent(inout) | dx_loc ) |
Populate the local cell-boundary, cell-center, and cell-width arrays in one direction directly from the global cell-boundary array. This guarantees that every rank sees bitwise-identical values at any shared physical cell or boundary.
Definition at line 3565 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_check_mpi_file_open | ( | integer, intent(in) | ierr, |
| character(len=*), intent(in) | file_loc ) |
Abort naming the file if MPI_FILE_OPEN failed. File errors default to MPI_ERRORS_RETURN, so an unchecked failure carries on with an invalid handle and crashes somewhere unrelated.
Definition at line 1038 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_finalize_mpi_common_module |
Module deallocation and/or disassociation procedures.
Definition at line 3643 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_initialize_mpi_common_module | ( | logical, intent(in) | exchange_all_chemistry_temperatures_in, |
| logical, intent(in) | use_rdma_transport_in ) |
Initialize the module.
Definition at line 453 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_initialize_mpi_data | ( | type(scalar_field), dimension(sys_size), intent(in) | q_cons_vf, |
| type(integer_field), intent(in), optional | ib_markers, | ||
| type(mpi_io_ib_var), intent(inout), optional | ib_mpi_data, | ||
| type(scalar_field), intent(in), optional | beta, | ||
| type(pres_field), intent(in), optional | qbmm_pb, | ||
| type(pres_field), intent(in), optional | qbmm_mv ) |
Set up MPI I/O data views and variable pointers for parallel file output.
Definition at line 615 of file m_mpi_common.fpp.f90.
| subroutine m_mpi_common::s_initialize_mpi_data_ds | ( | integer, intent(in) | m_ds, |
| integer, intent(in) | n_ds, | ||
| integer, intent(in) | p_ds, | ||
| type(scalar_field), dimension(sys_size), intent(in), optional | q_cons_vf ) |
Set up MPI I/O data views for downsampled (coarsened) parallel file output.
Definition at line 698 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_abort | ( | character(len=*), intent(in), optional | prnt, |
| integer, intent(in), optional | code ) |
The subroutine terminates the MPI execution environment.
Definition at line 1006 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_allreduce_integer_sum | ( | integer(kind=8), intent(in) | var_loc, |
| integer(kind=8), intent(out) | var_glb ) |
Reduce a local integer value to its global sum across all MPI ranks.
Definition at line 894 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_allreduce_integer_sum_vec | ( | integer(kind=8), dimension(:), intent(in) | var_loc, |
| integer(kind=8), dimension(:), intent(out) | var_glb ) |
Reduce an integer(8) array to its element-wise global sum across all MPI ranks.
Definition at line 910 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_allreduce_max | ( | real(wp), intent(in) | var_loc, |
| real(wp), intent(out) | var_glb ) |
Reduce a local real value to its global maximum across all MPI ranks.
Definition at line 956 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_allreduce_min | ( | real(wp), intent(in) | var_loc, |
| real(wp), intent(out) | var_glb ) |
Reduce a local real value to its global minimum across all MPI ranks.
Definition at line 926 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_allreduce_min_vec | ( | real(wp), dimension(:), intent(in) | var_loc, |
| real(wp), dimension(:), intent(out) | var_glb ) |
Reduce a local real vector to its elementwise global minimum across all MPI ranks.
Definition at line 940 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_allreduce_sum | ( | real(wp), intent(in) | var_loc, |
| real(wp), intent(out) | var_glb ) |
Reduce a local real value to its global sum across all MPI ranks.
Definition at line 859 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_allreduce_vectors_sum | ( | real(wp), dimension(:,:), intent(in) | var_loc, |
| real(wp), dimension(:,:), intent(inout) | var_glb, | ||
| integer, intent(in) | num_vectors, | ||
| integer, intent(in) | vector_length ) |
Reduce an array of vectors to their global sums across all MPI ranks.
Definition at line 873 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_barrier |
Halts all processes until all have reached barrier.
Definition at line 1055 of file m_mpi_common.fpp.f90.
| subroutine m_mpi_common::s_mpi_decompose_computational_domain | ( | logical, intent(in) | write_silo_ghost_offsets, |
| logical, intent(in) | adjust_local_domains, | ||
| type(int_bounds_info), dimension(3), intent(inout), optional | output_offsets, | ||
| type(bounds_info), dimension(3), intent(inout), optional | local_domains ) |
The purpose of this procedure is to optimally decompose the computational domain among the available processors. This is performed by attempting to award each processor, in each of the coordinate directions, approximately the same number of cells, and then recomputing the affected global parameters.
Non-optimal number of processors in the x-, y- and z-directions
Definition at line 3051 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_finalize |
The subroutine finalizes the MPI execution environment.
Definition at line 1066 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_gather_data | ( | real(wp), dimension(counts), intent(in) | my_vector, |
| integer, intent(in) | counts, | ||
| real(wp), dimension(:), intent(out), allocatable | gathered_vector, | ||
| integer, intent(in) | root ) |
Gather variable-length real vectors from all MPI ranks onto the root process.
| [in] | counts | Array of vector lengths for each process |
| [in] | my_vector | Input vector on each process |
| [in] | root | Rank of the root process |
| [out] | gathered_vector | Gathered vector on the root process |
Definition at line 735 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_initialize |
Initialize the MPI execution environment and query the number of processors and local rank.
Definition at line 579 of file m_mpi_common.fpp.f90.
| subroutine m_mpi_common::s_mpi_reduce_beta_variables_buffers | ( | type(scalar_field), dimension(1:), intent(inout) | q_comm, |
| type(scalar_field), dimension(1:), intent(inout) | kahan_comp, | ||
| integer, intent(in) | mpi_dir, | ||
| integer, intent(in) | pbc_loc, | ||
| integer, intent(in) | nvar ) |
The goal of this procedure is to populate the buffers of the cell-average conservative variables by communicating with the neighboring processors.
| q_cons_vf | Cell-average conservative variables |
| mpi_dir | MPI communication coordinate direction |
| pbc_loc | Processor boundary condition (PBC) location |
Definition at line 2487 of file m_mpi_common.fpp.f90.
| subroutine m_mpi_common::s_mpi_reduce_int_sum | ( | integer, intent(in) | var_loc, |
| integer, intent(out) | sum ) |
Reduce a local integer value to its global sum across all MPI ranks.
Definition at line 843 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_reduce_maxloc | ( | real(wp), dimension(2), intent(inout) | var_loc | ) |
Reduce a 2-element variable to its global maximum value with the owning processor rank (MPI_MAXLOC). Reduce a local value to its global maximum with location (rank) across all ranks.
Definition at line 989 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_reduce_min | ( | real(wp), intent(inout) | var_loc | ) |
Reduce a local real value to its global minimum across all ranks.
Definition at line 970 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_mpi_reduce_stability_criteria_extrema | ( | real(wp), dimension(:), intent(in) | max_loc, |
| real(wp), dimension(:), intent(in) | min_loc, | ||
| integer, intent(in) | bubs_loc, | ||
| real(wp), dimension(:), intent(out) | max_glb, | ||
| real(wp), dimension(:), intent(out) | min_glb, | ||
| integer, intent(out) | bubs_glb ) |
The goal of this subroutine is to determine the global extrema of the stability criteria in the computational domain. This is performed by sifting through the local extrema of each stability criterion. Note that each of the local extrema is from a single process, within its assigned section of the computational domain. Finally, note that the global extrema values are only bookkept on the rank 0 processor. max_loc/max_glb hold the max-reduced criteria (ICFL, VCFL, CCFL, TCFL) and min_loc/min_glb hold the min-reduced criteria (Rc).
Definition at line 799 of file m_mpi_common.fpp.f90.
| subroutine m_mpi_common::s_mpi_sendrecv_grid_variable_buffer | ( | real(wp), dimension(-1 - offset%beg:), intent(inout), contiguous | cell_boundaries, |
| real(wp), dimension(-buff_size:), intent(inout), contiguous | cell_centers, | ||
| real(wp), dimension(-buff_size:), intent(inout), contiguous | cell_widths, | ||
| integer, intent(in) | num_cells, | ||
| type(int_bounds_info), intent(in) | bc_bounds, | ||
| integer, intent(in) | pbc_loc, | ||
| type(int_bounds_info), intent(in) | offset ) |
The goal of this procedure is to populate the buffers of the grid variables by communicating with the neighboring processors. Note that only the buffers of the cell-width distributions are handled in such a way. This is because the buffers of cell-boundary locations may be calculated directly from those of the cell-width distributions.
Definition at line 3510 of file m_mpi_common.fpp.f90.
| subroutine m_mpi_common::s_mpi_sendrecv_variables_buffers | ( | type(scalar_field), dimension(1:), intent(inout) | q_comm, |
| integer, intent(in) | mpi_dir, | ||
| integer, intent(in) | pbc_loc, | ||
| integer, intent(in) | nvar, | ||
| real(stp), dimension(idwbuff(1)%beg:,idwbuff(2)%beg:,idwbuff(3)%beg:,1:,1:), intent(inout), optional | pb_in, | ||
| real(stp), dimension(idwbuff(1)%beg:,idwbuff(2)%beg:,idwbuff(3)%beg:,1:,1:), intent(inout), optional | mv_in, | ||
| type(scalar_field), intent(inout), optional | q_t_sf ) |
The goal of this procedure is to populate the buffers of the cell-average conservative variables by communicating with the neighboring processors.
Definition at line 1078 of file m_mpi_common.fpp.f90.
| impure subroutine m_mpi_common::s_prohibit_abort | ( | character(len=*), intent(in) | condition, |
| character(len=*), intent(in) | message ) |
Print a case file error with the prohibited condition and message, then abort execution.
Definition at line 779 of file m_mpi_common.fpp.f90.
| integer, dimension(1:3) m_mpi_common::beta_vars = [1, 2, 5] |
q_beta indices to communicate: 1=void fraction, 2=d(beta)/dt, 5=energy source
Definition at line 404 of file m_mpi_common.fpp.f90.
|
private |
Primitive variable receive buffer for halo exchange Variables for EL bubbles communication.
Definition at line 400 of file m_mpi_common.fpp.f90.
|
private |
Primitive variable send buffer for halo exchange.
Definition at line 398 of file m_mpi_common.fpp.f90.
| type(int_bounds_info), dimension(3) m_mpi_common::comm_coords |
Definition at line 401 of file m_mpi_common.fpp.f90.
| integer, dimension(3) m_mpi_common::comm_size |
Definition at line 402 of file m_mpi_common.fpp.f90.
|
private |
Definition at line 447 of file m_mpi_common.fpp.f90.
| integer(kind=8) m_mpi_common::halo_size |
Definition at line 433 of file m_mpi_common.fpp.f90.
|
private |
Definition at line 448 of file m_mpi_common.fpp.f90.
|
private |
Definition at line 384 of file m_mpi_common.fpp.f90.