|
MFC
Exascale flow solver
|
Rank decomposition, box arithmetic and block ownership (SFC cut, owner lookups, owned-block lists). More...
Functions/Subroutines | |
| pure subroutine, public | s_amr_rank_decomp (r, sidx, ext) |
| Rank r's coarse-grid decomposition (start_idx + local extent m/n/p), computed O(1) from its cartesian coords. The domain cart is MPI_CART_CREATE(reorder=.false.), so ranks keep MPI_COMM_WORLD order and coords are row-major in r. The per-dim split is the integer split-with-remainder of s_mpi_decompose_computational_domain. | |
| impure subroutine, public | s_amr_validate_decomp () |
| Guard: the computed decomposition must reproduce this rank's actual one (each rank checks itself, so all r are covered). | |
| pure integer function | f_amr_cell_coord (g, base, rem) |
| Closed-form inverse of the per-dim split in s_amr_rank_decomp: the cart coord owning global coarse cell g, with base=(gd+1)/pd, rem=mod(gd+1,pd). Exact for g in [0,gd]; callers clamp for out-of-range g (ghost reach). | |
| pure subroutine | s_amr_coord_range (blo, bhi, clo, chi) |
| Per-dim contiguous rank-coord range [clo:chi] whose owned coarse slab meets box [blo:bhi], clamped to [0,pd-1]. The boundary coords' coarse_range is extended by buff_size at the domain edge (s_amr_rank_coarse_range), and clamping reproduces both the interior (scatter) and coarse_range (gather) sets. Collapsed dims contribute coord 0. | |
| pure subroutine, public | s_amr_ranks_overlapping (blo, bhi, ranks, nr) |
| Ascending rank list overlapping coarse box [blo:bhi]: the coord brick is enumerated cx->cy->cz so r = cx*(Py*Pz)+cy*Pz+cz ascends, reproducing the 0..num_procs-1 scan order. Owner not excluded. Caller sizes ranks(:) >= f_amr_overlap_count. | |
| pure logical function, public | f_amr_rank_overlaps (blo, bhi, r) |
| Does rank r's subdomain overlap coarse box [blo:bhi]? Answers membership without enumerating the overlap set, which is O(num_procs) for a box spanning the machine. | |
| pure integer function, public | f_amr_overlap_count (blo, bhi) |
| Overlap count only (allocation sizing), = product of the per-dim coord-range widths. | |
| pure subroutine, public | s_amr_rank_coarse_range (r, crlo, crhi) |
| Rank r's contiguous owned coarse-cell range per dim: the interior plus buff_size ghosts where the subdomain touches the domain edge. One span, so box intersections identify contributors without a per-cell scan. | |
| pure subroutine, public | s_amr_box_isect (alo, ahi, blo, bhi, olo, ohi) |
| Per-dim intersection of two global boxes [alo:ahi] and [blo:bhi] -> [olo:ohi] (empty when olo > ohi in some dim). | |
| impure subroutine | s_amr_block_cost (cost) |
| Per-block cost over each block's level-0 footprint (1 per cell + K_ib per IB-marked cell), summed over each rank's owned coarse cells and replicated by one MPI_ALLREDUCE(SUM). | |
| subroutine, public | s_amr_sfc_cut (keys, wt, n, cut, owner) |
| Cost-weighted SFC partition of n items into num_procs contiguous Morton-key ranges: walk the key-sorted items (stable merge sort) and advance the owner rank when the running weight crosses the next even share. Writes owner(1:n) and cut(0:num_procs-1) (running Morton upper bound per rank; ranks with no item inherit the predecessor's bound, so cut is non-decreasing). | |
| pure subroutine, public | s_amr_sort_by_key (keys, n, ord) |
| ord(1:n) = item indices in ascending key order. Stable iterative bottom-up merge sort, a pure function of the input so every rank derives identical orders from replicated keys (differing assignments would deadlock or abort). | |
| impure subroutine, public | s_amr_refresh_my_blocks () |
| Refresh the owned-block list if any owner write has happened since the last refresh. Lazy, since s_amr_assign_block_owners is not the only writer of amr_block_owner. | |
| impure subroutine, public | s_amr_refresh_lists () |
| Rebuild the epoch-keyed block lists when the mesh epoch has moved (one O(global blocks) walk per regrid, not per RK stage). Also caches the parent index (f_amr_parent_block is an O(global blocks) scan) and the children adjacency used by s_amr_sibling_face_weights. | |
| impure subroutine | s_amr_place_overlap (k, g, vol, rk, nr) |
| Coarse-side overlap per rank of block k's region grown by g coarse cells: level 1 against the level-0 rank bricks, level >= 2 against the owners of the level-(l-1) blocks. Adds into vol(0:) and lists touched ranks in rk(1:nr); the caller zeroes vol(rk(1:nr)) after use. | |
| impure subroutine | s_amr_place_owner (key, wt, aidx, na, own) |
| Owner-aware placement (amr_place_owner) of one level's na blocks (slots aidx, weights wt, Morton keys key): each block goes to the rank owning most of the coarse cells under it. A rank above (1 + place_tol) x the mean sheds its last blocks in Morton order, first to the least-loaded rank next to the block, then to the least-loaded rank anywhere, never pushing the receiver above the cap. Replicated inputs in a fixed order, so every rank computes the same owners. | |
| impure subroutine, public | s_amr_assign_block_owners () |
| impure subroutine, public | s_amr_adopt_block_owners (own) |
| Install a recorded owner table instead of recomputing it (restart at the writer's rank count): the cost weights read live state (body markers) that differs between the regrid before the write and the restart. Each level's cut is the running max Morton key of each rank's blocks; s_amr_validate_owner rejects a table not contiguous in that order (unless amr_place_owner). | |
| impure subroutine | s_amr_report_balance (wt, maxlev) |
| Per-level and total load-balance report (rank 0, rank_time_wrt): max/mean assigned block weight over ranks (1 = perfect). no_blocks_ranks is the granularity floor of a level with fewer boxes than ranks, not idleness. | |
| pure integer function | f_amr_owner (k) |
| Owner rank of block k by binary search of its Morton key in its level's cut amr_fine_cut(:, level); reproduces s_amr_assign_block_owners' assignment exactly. | |
| impure subroutine, public | s_amr_validate_owner () |
| Guard: the SFC cut-point accessor must reproduce the stored owner table exactly. | |
| impure subroutine, public | s_amr_compute_isect (lo, hi) |
| Set this rank's intersection of box lo:hi with its subdomain (amr_isect_lo/hi) and whether it holds fine cells (amr_rank_owns_block: nonempty in all active dims). Call with the coarse grid state in m/n/p, never inside the fine advance. | |
| pure subroutine, public | s_amr_rank_interior (r, ilo, ihi) |
| Rank r's coarse interior box (global, no ghosts) from s_amr_rank_decomp; restriction targets are identified by interior overlap alone. | |
| impure subroutine, public | s_amr_body_bbox (i, mrg, blo, bhi) |
| Margin-padded global coarse-index bounding box of immersed body i (supported analytic geometries only; aborts on others). Reads the body's live centroid and orientation, so a moving body's box tracks it. | |
| impure subroutine, public | s_amr_expand_box_over_bodies (lo, hi) |
| Expand a candidate regrid box (global indices) to fully contain every immersed body it overlaps, with a margin; aborts if a body exceeds the per-rank block cap. | |
| pure subroutine, public | s_amr_tile_box (lo, hi, out, nt, cap, capped, tsz) |
| Split box [lo:hi] into an even grid of adjacent sub-boxes each <= amr_maxc_fit per dim (the max a rank can whole-own), appended to out(nt+1:); sets capped=1 and stops if the cap is hit. Collapsed dims stay [0:0]. | |
Rank decomposition, box arithmetic and block ownership (SFC cut, owner lookups, owned-block lists).
|
private |
Closed-form inverse of the per-dim split in s_amr_rank_decomp: the cart coord owning global coarse cell g, with base=(gd+1)/pd, rem=mod(gd+1,pd). Exact for g in [0,gd]; callers clamp for out-of-range g (ghost reach).
Definition at line 436 of file m_amr_distribution.fpp.f90.
| pure integer function, public m_amr_distribution::f_amr_overlap_count | ( | integer, dimension(3), intent(in) | blo, |
| integer, dimension(3), intent(in) | bhi ) |
Overlap count only (allocation sizing), = product of the per-dim coord-range widths.
Definition at line 508 of file m_amr_distribution.fpp.f90.
|
private |
Owner rank of block k by binary search of its Morton key in its level's cut amr_fine_cut(:, level); reproduces s_amr_assign_block_owners' assignment exactly.
Definition at line 1032 of file m_amr_distribution.fpp.f90.
| pure logical function, public m_amr_distribution::f_amr_rank_overlaps | ( | integer, dimension(3), intent(in) | blo, |
| integer, dimension(3), intent(in) | bhi, | ||
| integer, intent(in) | r ) |
Does rank r's subdomain overlap coarse box [blo:bhi]? Answers membership without enumerating the overlap set, which is O(num_procs) for a box spanning the machine.
Definition at line 494 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_adopt_block_owners | ( | integer, dimension(:), intent(in) | own | ) |
Install a recorded owner table instead of recomputing it (restart at the writer's rank count): the cost weights read live state (body markers) that differs between the regrid before the write and the restart. Each level's cut is the running max Morton key of each rank's blocks; s_amr_validate_owner rejects a table not contiguous in that order (unless amr_place_owner).
Definition at line 961 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_assign_block_owners |
Definition at line 909 of file m_amr_distribution.fpp.f90.
|
private |
Per-block cost over each block's level-0 footprint (1 per cell + K_ib per IB-marked cell), summed over each rank's owned coarse cells and replicated by one MPI_ALLREDUCE(SUM).
Definition at line 544 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_body_bbox | ( | integer, intent(in) | i, |
| integer, intent(in) | mrg, | ||
| integer, dimension(3), intent(out) | blo, | ||
| integer, dimension(3), intent(out) | bhi ) |
Margin-padded global coarse-index bounding box of immersed body i (supported analytic geometries only; aborts on others). Reads the body's live centroid and orientation, so a moving body's box tracks it.
Definition at line 1092 of file m_amr_distribution.fpp.f90.
| pure subroutine, public m_amr_distribution::s_amr_box_isect | ( | integer, dimension(3), intent(in) | alo, |
| integer, dimension(3), intent(in) | ahi, | ||
| integer, dimension(3), intent(in) | blo, | ||
| integer, dimension(3), intent(in) | bhi, | ||
| integer, dimension(3), intent(out) | olo, | ||
| integer, dimension(3), intent(out) | ohi ) |
Per-dim intersection of two global boxes [alo:ahi] and [blo:bhi] -> [olo:ohi] (empty when olo > ohi in some dim).
Definition at line 533 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_compute_isect | ( | integer, dimension(3), intent(in) | lo, |
| integer, dimension(3), intent(in) | hi ) |
Set this rank's intersection of box lo:hi with its subdomain (amr_isect_lo/hi) and whether it holds fine cells (amr_rank_owns_block: nonempty in all active dims). Call with the coarse grid state in m/n/p, never inside the fine advance.
Definition at line 1068 of file m_amr_distribution.fpp.f90.
|
private |
Per-dim contiguous rank-coord range [clo:chi] whose owned coarse slab meets box [blo:bhi], clamped to [0,pd-1]. The boundary coords' coarse_range is extended by buff_size at the domain edge (s_amr_rank_coarse_range), and clamping reproduces both the interior (scatter) and coarse_range (gather) sets. Collapsed dims contribute coord 0.
Definition at line 451 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_expand_box_over_bodies | ( | integer, dimension(3), intent(inout) | lo, |
| integer, dimension(3), intent(inout) | hi ) |
Expand a candidate regrid box (global indices) to fully contain every immersed body it overlaps, with a margin; aborts if a body exceeds the per-rank block cap.
Definition at line 1119 of file m_amr_distribution.fpp.f90.
|
private |
Coarse-side overlap per rank of block k's region grown by g coarse cells: level 1 against the level-0 rank bricks, level >= 2 against the owners of the level-(l-1) blocks. Adds into vol(0:) and lists touched ranks in rk(1:nr); the caller zeroes vol(rk(1:nr)) after use.
Definition at line 817 of file m_amr_distribution.fpp.f90.
|
private |
Owner-aware placement (amr_place_owner) of one level's na blocks (slots aidx, weights wt, Morton keys key): each block goes to the rank owning most of the coarse cells under it. A rank above (1 + place_tol) x the mean sheds its last blocks in Morton order, first to the least-loaded rank next to the block, then to the least-loaded rank anywhere, never pushing the receiver above the cap. Replicated inputs in a fixed order, so every rank computes the same owners.
Definition at line 852 of file m_amr_distribution.fpp.f90.
| pure subroutine, public m_amr_distribution::s_amr_rank_coarse_range | ( | integer, intent(in) | r, |
| integer, dimension(3), intent(out) | crlo, | ||
| integer, dimension(3), intent(out) | crhi ) |
Rank r's contiguous owned coarse-cell range per dim: the interior plus buff_size ghosts where the subdomain touches the domain edge. One span, so box intersections identify contributors without a per-cell scan.
Definition at line 520 of file m_amr_distribution.fpp.f90.
| pure subroutine, public m_amr_distribution::s_amr_rank_decomp | ( | integer, intent(in) | r, |
| integer, dimension(3), intent(out) | sidx, | ||
| integer, dimension(3), intent(out) | ext ) |
Rank r's coarse-grid decomposition (start_idx + local extent m/n/p), computed O(1) from its cartesian coords. The domain cart is MPI_CART_CREATE(reorder=.false.), so ranks keep MPI_COMM_WORLD order and coords are row-major in r. The per-dim split is the integer split-with-remainder of s_mpi_decompose_computational_domain.
Definition at line 399 of file m_amr_distribution.fpp.f90.
| pure subroutine, public m_amr_distribution::s_amr_rank_interior | ( | integer, intent(in) | r, |
| integer, dimension(3), intent(out) | ilo, | ||
| integer, dimension(3), intent(out) | ihi ) |
Rank r's coarse interior box (global, no ghosts) from s_amr_rank_decomp; restriction targets are identified by interior overlap alone.
Definition at line 1079 of file m_amr_distribution.fpp.f90.
| pure subroutine, public m_amr_distribution::s_amr_ranks_overlapping | ( | integer, dimension(3), intent(in) | blo, |
| integer, dimension(3), intent(in) | bhi, | ||
| integer, dimension(:), intent(out) | ranks, | ||
| integer, intent(out) | nr ) |
Ascending rank list overlapping coarse box [blo:bhi]: the coord brick is enumerated cx->cy->cz so r = cx*(Py*Pz)+cy*Pz+cz ascends, reproducing the 0..num_procs-1 scan order. Owner not excluded. Caller sizes ranks(:) >= f_amr_overlap_count.
Definition at line 471 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_refresh_lists |
Rebuild the epoch-keyed block lists when the mesh epoch has moved (one O(global blocks) walk per regrid, not per RK stage). Also caches the parent index (f_amr_parent_block is an O(global blocks) scan) and the children adjacency used by s_amr_sibling_face_weights.
Definition at line 696 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_refresh_my_blocks |
Refresh the owned-block list if any owner write has happened since the last refresh. Lazy, since s_amr_assign_block_owners is not the only writer of amr_block_owner.
Definition at line 674 of file m_amr_distribution.fpp.f90.
|
private |
Per-level and total load-balance report (rank 0, rank_time_wrt): max/mean assigned block weight over ranks (1 = perfect). no_blocks_ranks is the granularity floor of a level with fewer boxes than ranks, not idleness.
Definition at line 986 of file m_amr_distribution.fpp.f90.
| subroutine, public m_amr_distribution::s_amr_sfc_cut | ( | integer(kind=8), dimension(n), intent(in) | keys, |
| real(wp), dimension(n), intent(in) | wt, | ||
| integer, intent(in) | n, | ||
| integer(kind=8), dimension(0:num_procs - 1), intent(out) | cut, | ||
| integer, dimension(n), intent(out) | owner ) |
Cost-weighted SFC partition of n items into num_procs contiguous Morton-key ranges: walk the key-sorted items (stable merge sort) and advance the owner rank when the running weight crosses the next even share. Writes owner(1:n) and cut(0:num_procs-1) (running Morton upper bound per rank; ranks with no item inherit the predecessor's bound, so cut is non-decreasing).
Definition at line 595 of file m_amr_distribution.fpp.f90.
| pure subroutine, public m_amr_distribution::s_amr_sort_by_key | ( | integer(kind=8), dimension(n), intent(in) | keys, |
| integer, intent(in) | n, | ||
| integer, dimension(n), intent(out) | ord ) |
ord(1:n) = item indices in ascending key order. Stable iterative bottom-up merge sort, a pure function of the input so every rank derives identical orders from replicated keys (differing assignments would deadlock or abort).
Definition at line 636 of file m_amr_distribution.fpp.f90.
| pure subroutine, public m_amr_distribution::s_amr_tile_box | ( | integer, dimension(3), intent(in) | lo, |
| integer, dimension(3), intent(in) | hi, | ||
| type(t_box), dimension(:), intent(inout) | out, | ||
| integer, intent(inout) | nt, | ||
| integer, intent(in) | cap, | ||
| integer, intent(inout) | capped, | ||
| integer, dimension(3), intent(in), optional | tsz ) |
Split box [lo:hi] into an even grid of adjacent sub-boxes each <= amr_maxc_fit per dim (the max a rank can whole-own), appended to out(nt+1:); sets capped=1 and stops if the cap is hit. Collapsed dims stay [0:0].
| [in] | tsz | per-dim tile size (default amr_maxc_fit) |
Definition at line 1148 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_validate_decomp |
Guard: the computed decomposition must reproduce this rank's actual one (each rank checks itself, so all r are covered).
Definition at line 422 of file m_amr_distribution.fpp.f90.
| impure subroutine, public m_amr_distribution::s_amr_validate_owner |
Guard: the SFC cut-point accessor must reproduce the stored owner table exactly.
Definition at line 1054 of file m_amr_distribution.fpp.f90.