|
MFC
Exascale flow solver
|
Adaptive Mesh Refinement says what AMR does; this page says how it is built: modules, index spaces, data layout, the step, regrid and communication, and the invariants that are easy to break. Routines are cited by name (greppable in src/simulation/m_amr*.fpp, m_global_parameters.fpp, m_time_steppers.fpp).
Each module may use only those above it; m_amr re-exports the chain, so callers write use m_amr, only: ....
| Module | Responsibility |
|---|---|
| m_box | Box type, per-rank partition arithmetic and the Morton key (common) |
| m_amr_restart_io | Fine-level restart file layout shared by the writer, the reader and post_process (common) |
| m_global_parameters | Block metadata pools, the working-slot mirrors, s_amr_select_slot |
| m_amr_state | Per-block module state: flat store, wave buffers, swap mirrors (sw_*) |
| m_amr_distribution | Block cost, SFC cut and owner placement, owned-block lists, box arithmetic |
| m_amr_store | Flat store: local index, reserve/compact, slot alloc/free, solver bridge load/store |
| m_amr_wave | Nonblocking exchange engine: pooled sides, post/send/wait |
| m_amr_ghost | Coarse-to-fine ghost fill: slab table, linear or quartic fill, quartic guard, volume-fraction closure |
| m_amr_seam | Fine-fine seams between same-level blocks, coarse conservative halo |
| m_amr_exchange | Coarse-patch gathers and the fill waves |
| m_amr_frame | Grid-state swap to a block or batch and back; 6-equation fine relaxation |
| m_amr_registers | Flux registers: capture during the RHS and reflux apply |
| m_amr_transfer | Prolongation, restriction, reflux and restriction waves |
| m_amr_advance | Batched fine RHS + RK advance, IB and Lagrangian fine services |
| m_amr_cluster | Berger-Rigoutsos clustering |
| m_amr_regrid | Tagging, box shaping, migration and slot rebuild |
| m_amr_nest | Child nesting for levels above 1, and the shared tag sensor |
| m_amr_restart | Hierarchy checkpoint (serial and MPI-IO) |
| m_amr_stage | Hooks called by s_tvd_rk: stage begin, fine stage, subcycled fine step, end-of-step fold |
| m_phase_timing | rank_time_wrt phase brackets, AMR memory table and report |
| m_stage_hooks | the post-RK hook lists shared by the coarse stage and the fine blocks |
Four integer index spaces are in play and the compiler cannot tell them apart; mixing two gives a silent wrong answer.
"Coarse" in the coupling routines means the block's parent level, which is level 0 only for level-1 blocks.
amr_cons_st(x, y, z, var, loc) holds the conserved state of every owned block (amr_stor_st is the regrid stash); all slots share the buffered extents mbuf_lo:mbuf_hi, sized for the largest block. The shared solver (s_compute_rhs and the physics hooks) takes scalar_field arrays, so a batch is copied into the bridge amr_cons_br (s_amr_br_load_batch), advanced, and copied back. A batch stacks up to amr_bat_lim blocks of one level along the last dimension (offsets amr_bat_off); a smaller block joins a larger leader when padding wastes at most amr_bat_pad of its cells. Fine blocks share one pooled q_prim/rhs scratch.
The store grows by s_amr_st_reserve (never shrinks), on the device up to amr_grow_dev_bytes and through the host above it.
The solver reads module-level grid state (m, n, p, idwint, idwbuff, x_cc, dx, ...). s_amr_swap_to_fine installs a block's (or batch's) grid, recomputes the spacing-dependent coefficients and pushes them to the device; s_amr_restore_coarse puts the base grid back. Any module variable derived from the grid that a kernel reads during the fine advance must be swapped there (see the swap contract at the sw_* declarations in m_amr_state) or refreshed at its use site, including its device copy; CCE OpenACC reads a stale device copy.
AMR fails closed on physics switches: with amr = T the case validator rejects any logical switch set true that is not in AMR_VETTED_FLAGS (toolchain/mfc/case_validator.py). New physics joins that list only after it meets the swap contract above and has an AMR test.
s_tvd_rk calls, per RK stage:
and after the stage loop:
s_amr_regrid, every amr_regrid_int steps: halo, tag (s_amr_regrid_tag_cells), cluster (s_amr_regrid_cluster_tags), shape (pad, clamp, cap, keep clear of bodies, sources and bubble clouds), nest, check caps and disjointness, and return early if the box set is unchanged (amr_snap widens this). Otherwise s_amr_regrid_stash_migrate stashes each old block at its old local index, assigns owners and ships stashes to new owners, and s_amr_regrid_rebuild_slots builds the new blocks level by level: overlap with an old block is copied from the stash, the rest is prolonged. Old slots are freed only after their last reader (last_use), then s_amr_reconcile_slots compacts the store.
Every exchange is a wave on m_amr_wave: all receives posted, sends packed and posted, one wait. Both sides derive the same transfer list from replicated metadata, so no handshake is needed, and messages carry keyed tags, so matching does not depend on posting order. Waves: coarse halo, level-1 and parent fill waves, seam halo, reflux faces, restriction and level>=2 register waves, regrid migration. With amr_device_pack the fill waves pack and unpack in one fused kernel per family. Regrid adds a few collectives (tag reductions, the box list, owners).
A block cell that the block's edges cannot reach during a run advances exactly as the same cell of a uniform fine grid, so the two runs must agree there to roundoff. ./mfc.sh amr_check case.py tests this: it runs the case with one static block (the case's amr_block_beg/amr_block_end, else the central half of the domain) and on a uniform grid at the block's resolution, averages the fine run onto the block's cells and prints the largest relative difference by distance from the block edge. The core is the cells farther from every edge than one reconstruction stencil per Runge-Kutta stage and step; the check passes when the core differs by less than --tol (default 1e-8).
The case needs a fixed dt, t_step_start = 0, and an initial condition that survives interpolation from the coarse grid: piecewise-constant patches whose faces lie on coarse cell faces, with part of the flow inside the core. The command reports when either the initial data differs or the core never changes. The two runs are made in amr_check/amr and amr_check/fine under the case directory; parameters naming a file there are passed as absolute paths, but input files the solver opens by a fixed relative name (e.g. input/lag_bubbles.dat) are not carried over. Options after -- go to ./mfc.sh run.
A switch joins AMR_VETTED_FLAGS once a case that uses it passes this check and an AMR test covers it.
rank_time_wrt = T brackets the step phases (PH_* in m_phase_timing) with device-synchronizing timers and prints mean, max and imbalance per phase, plus the block-balance report at each regrid. With it off the brackets return immediately.