|
MFC
Exascale flow solver
|
Experimental. AMR is off by default (amr = F). The case validator rejects unsupported combinations by name.
AMR adds refined rectangular blocks on top of the base grid (level 0) where the flow needs resolution: shocks, interfaces, wakes, bodies. The base grid is advanced by the ordinary solver; refined blocks run the same solver on their own grids, nested up to amr_max_level levels at amr_ref_ratio:1 per level.
With amr_regrid_int > 0, the blocks are rebuilt around the cells where the density gradient exceeds amr_tag_eps: once per level before the first step of a fresh run, so they start on the initial features, and then every amr_regrid_int steps. With amr_regrid_int = 0 a single block stays at amr_block_beg..amr_block_end.
On a fresh run the blocks start from patch_icpp evaluated at their own cell centers, as on a uniform grid at that resolution. If the patches alone do not reproduce the level-0 data (hard-coded or STL patches, a perturbed or smoothed initial condition, old data), the simulation prints a note and the blocks start from data interpolated from level 0. The step-0 files are pre_process's level-0 data: under a block the run starts from the average of the fine data instead, which differs from them where a patch varies within a coarse cell. For a smooth analytic profile sampled at cell centers that difference is second order in the cell width; an expression that is the exact cell average (it may use dx) agrees on every level.
AMR lives in simulation (src/simulation/m_amr*.fpp); post_process can overlay the blocks. Internals: AMR Implementation Reference.
Conservation across coarse/fine faces holds to round-off for single- and multi-fluid, viscous and chemistry cases, and a uniform state stays uniform with regrid on. Interiors keep the WENO order; the coarse/fine ghost fill is second order (linear) or fifth order (quartic). For a smooth wave crossing a static block, the measured global L2 rate is 2.5 with the linear fill (its second-order ghost values leave a second-order error in the few cells at each interface) and 5 with the quartic fill, as for WENO5 on a uniform grid; the Convergence -> 1D -> AMR tests track it. Results do not depend on message timing, and agree across rank counts to round-off when amr_max_grid_size is set (with amr_max_grid_size = 0 the block size cap, and therefore the box set, depends on the decomposition).
Each block is owned whole by one rank. Owners come from a cost-weighted space-filling-curve cut over the blocks (cost: one per coarse cell plus K_ib per immersed-boundary cell), recomputed at every regrid. amr_place_owner = T instead places each block on the rank that owns most of the coarse cells under it, capped at 5% above the mean load, which cuts communication. A block may straddle rank boundaries; its coarse-side coupling moves point-to-point between the owner and the ranks it overlaps. All exchanges are posted in a fixed order, so runs are deterministic.
On GPUs every block array is device-resident. A rank advances its blocks of equal level in batches stacked along the last dimension, one solver call per batch; this is why AMR needs a uniform Cartesian grid. The toolchain enables amr_device_pack (fused exchange kernels) when amr_max_grid_size <= 64 and sets amr_snap = min(2, amr_buf - 2) (regrid hysteresis) under dynamic regrid, unless they are set explicitly.
rank_time_wrt = T prints a per-phase time budget of the AMR step, the block-balance report, and at the end of the run the memory each group of AMR arrays holds per rank ([amr-mem]: block store, advance scratch, coarse patches, flux registers, the fine IB ghost-point park, and the host-side per-slot IB marker copies).
Required: model_eqns = 2 or 3, time_stepper = 1, 2 or 3, WENO or MUSCL (recon_type = 1 or 2) or IGR, a uniform Cartesian grid, and mpp_lim = T when num_fluids > 1.
| Feature | Status |
|---|---|
| Single- and multi-fluid Euler, 6-equation model | Supported |
| Viscous, hypoelastic (with continuum damage), body forces | Supported |
| Chemistry (reactions, diffusion) | Lock-step only; no reaction substepping |
| Immersed boundaries (fixed, prescribed or force-driven motion; not STL, thermal_bc, surface_reaction, ib_surface_wrt, ib_heat_wrt, ib_mass_relax or ib_uncover_fill) | Supported; blocks always contain each body; moving bodies are lock-step only; multi-level with IB is single-rank; forces and torques are summed on the finest level covering each face (the coarse grid only with amr_subcycle); the default ib_mass_relax = -1 resolves to off, so a moving body differs from a uniform-grid run, where it is on |
| Lagrangian bubbles | Lock-step; the cloud stays on the base grid (blocks are kept clear of it) |
| Acoustic sources | On the base grid; blocks are kept clear of the source support |
| Body forces (bf_x/y/z, bf_spatial_support) | Supported; the source is evaluated on each block's own grid |
| MHD / RMHD | 1D only |
| IGR | Single-rank, lock-step; coarse/fine conservation is truncation-order |
| Synthetic turbulence forcing | Supported; evaluated on each block's own grid |
| Phase change (relax) | Supported; each block is relaxed at the end of the step, then restricted |
| Euler-Euler bubbles, QBMM, surface tension | Not supported |
| Riemann-extrapolation boundaries (bc = -4) | Supported, except with null_weights |
| Cylindrical coordinates, stretched grids | Not supported |
Full descriptions: Case Files.
| Parameter | Default | Description |
|---|---|---|
| amr | F | Enable AMR |
| amr_block_beg(i), amr_block_end(i) | 0 | Initial block, level-0 cell indices (inclusive) |
| amr_regrid_int | 0 | Steps between regrids; 0 keeps the initial block |
| amr_tag_eps | 0.1 | Tagging threshold on the normalized density jump (> 0 with regrid) |
| amr_tag_abs | F | Tag on the absolute jump; amr_tag_eps then has the units of the tagged variable |
| amr_tag_var | 1 | Tagged variable: 1 total density, 2 volume fraction of fluid 1, 3 velocity magnitude (needs amr_tag_abs), 4 pressure (five-equation model) |
| amr_buf | 3 | Padding around tagged cells, coarse cells (>= 1 with regrid) |
| amr_cluster_eff | 0.7 | Minimum tag efficiency of a clustered box |
| amr_blocking_factor | 4 | Minimum box width, coarse cells |
| amr_max_grid_size | 0 | Maximum box width, coarse cells; 0 derives it from the decomposition |
| amr_max_blocks | 1024 | Maximum number of blocks |
| amr_max_level | 1 | Number of refined levels |
| amr_ref_ratio | 2 | Refinement ratio, 2 or 4 (4: one level only) |
| amr_subcycle | F | Subcycle refined levels in time (ratio 2, up to two levels) |
| amr_prolong | 1 | 1: limited linear, 2: quartic (ratio 2) |
| amr_place_owner | F | Owner-aware block placement |
| amr_snap | 0 | Regrid hysteresis, coarse cells (<= amr_buf - 2) |
| amr_device_pack | F | Fused device pack/unpack of the coarse-patch exchange |
Complete cases: examples/2D_amr_droplet (a tracked interface), examples/2D_amr_blast_multilevel (two refined levels), examples/2D_amr_ibm_cylinder (an immersed body) and examples/3D_amr_blast (3D, subcycled).
AMR costs more per cell than a uniform grid: extra kernels for ghost fills, flux registers and restriction, plus the regrids. On a GPU it pays only when the grid it replaces is large enough that the cell count, and not that fixed cost, sets the time per step.
Measured on one MI250X GCD (amdflang, OpenMP offload): examples/2D_amr_droplet with one refined level, against the same case on a uniform grid at the fine resolution. Time is the solver's time to the same physical time; below 1 the AMR run is faster.
| Base grid | Uniform fine grid | Fine cells held by AMR | Lock-step / uniform fine | Subcycled / uniform fine |
|---|---|---|---|---|
| 128^2 | 256^2 (0.07 M cells) | 11% | 2.98 | 2.19 |
| 512^2 | 1024^2 (1.0 M cells) | 7% | 2.05 | 1.54 |
| 2048^2 | 4096^2 (16.8 M cells) | 6% | 1.05 | 0.70 |
The last row is from the time per step over 400 steps. A 3D spherical blast (examples/3D_amr_blast, 64^3 base against a uniform 128^3, the blocks holding 54% of the fine cells) took 1.86 (lock-step) and 1.40 (subcycled) times the uniform run.