MFC
Exascale flow solver
Loading...
Searching...
No Matches
m_amr.fpp.f90
Go to the documentation of this file.
1# 1 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
2!>
3!!@file
4!!@brief Contains module m_amr
5
6# 7 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
7# 8 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
8# 1 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp" 1
9# 1 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 1
10# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
11# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
12# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
13# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
14# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
15# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
16
17# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
18# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
19# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
20
21# 12 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
22# 1 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp" 1
23! Default for Fypp runs outside the toolchain (e.g. documentation). Searched last, so the
24! build/include/<target>/thermochem.fpp the toolchain writes from m_thermochem takes precedence.
25# 4 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
26# 5 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
27# 13 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 2
28
29# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
30# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
31
32# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
33
34# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
35
36# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
37
38# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
39
40# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
41
42# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
43
44# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
45
46# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
47! New line at end of file is required for FYPP
48# 2 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
49# 1 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp" 1
50# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
51# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
52# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
53# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
54# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
55# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
56
57# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
58# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
59# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
60
61# 12 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
62# 1 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp" 1
63! Default for Fypp runs outside the toolchain (e.g. documentation). Searched last, so the
64! build/include/<target>/thermochem.fpp the toolchain writes from m_thermochem takes precedence.
65# 4 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
66# 5 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
67# 13 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 2
68
69# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
70# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
71
72# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
73
74# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
75
76# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
77
78# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
79
80# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
81
82# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
83
84# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
85
86# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
87! New line at end of file is required for FYPP
88# 2 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp" 2
89
90# 4 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
91# 5 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
92# 6 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
93# 7 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
94# 8 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
95
96# 20 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
97
98# 49 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
99
100# 54 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
101
102# 59 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
103
104# 64 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
105
106# 69 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
107
108# 74 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
109
110# 87 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
111
112# 101 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
113
114# 106 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
115
116# 111 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
117
118# 116 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
119
120# 121 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
121
122# 126 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
123
124# 131 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
125
126# 136 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
127
128# 141 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
129
130# 171 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
131
132# 231 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
133
134# 245 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
135
136# 270 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
137
138# 281 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
139
140# 283 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
141# 294 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
142
143# 343 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
144
145# 353 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
146
147# 363 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
148
149# 372 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
150
151# 389 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
152
153# 399 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
154
155# 406 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
156
157# 412 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
158
159# 418 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
160
161# 424 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
162
163# 430 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
164
165# 436 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
166! New line at end of file is required for FYPP
167# 3 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
168# 1 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp" 1
169# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
170# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
171# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
172# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
173# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
174# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
175
176# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
177# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
178# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
179
180# 12 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
181# 1 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp" 1
182! Default for Fypp runs outside the toolchain (e.g. documentation). Searched last, so the
183! build/include/<target>/thermochem.fpp the toolchain writes from m_thermochem takes precedence.
184# 4 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
185# 5 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
186# 13 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 2
187
188# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
189# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
190
191# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
192
193# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
194
195# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
196
197# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
198
199# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
200
201# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
202
203# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
204
205# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
206! New line at end of file is required for FYPP
207# 2 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp" 2
208
209# 7 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
210
211# 17 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
212
213# 22 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
214
215# 27 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
216
217# 32 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
218
219# 37 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
220
221# 42 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
222
223# 47 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
224
225# 52 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
226
227# 57 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
228
229# 62 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
230
231# 73 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
232
233# 78 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
234
235# 83 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
236
237# 88 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
238
239# 103 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
240
241# 131 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
242
243# 161 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
244
245# 176 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
246
247# 194 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
248
249# 216 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
250
251# 245 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
252
253# 260 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
254
255# 270 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
256
257# 279 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
258
259# 295 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
260
261# 305 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
262
263# 312 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
264! New line at end of file is required for FYPP
265# 4 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
266
267! GPU parallel region (scalar reductions, maxval/minval)
268# 23 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
269
270! GPU parallel loop over threads (most common GPU macro)
271# 44 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
272
273! Required closing for GPU_PARALLEL_LOOP
274# 56 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
275
276! Mark routine for device compilation
277# 113 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
278
279! Declare device-resident data
280# 131 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
281
282! Inner loop within a GPU parallel region
283# 146 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
284
285! Scoped GPU data region
286# 165 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
287
288! Host code with device pointers (for MPI with GPU buffers)
289# 191 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
290
291! Allocate device memory (unscoped)
292# 205 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
293
294! Free device memory
295# 217 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
296
297! Atomic operation on device
298# 229 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
299
300! End atomic capture block
301# 240 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
302
303! Copy data between host and device
304# 252 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
305
306! Synchronization barrier
307# 264 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
308
309! Import GPU library module (openacc or omp_lib)
310# 273 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
311
312! Emit code only for AMD compiler
313# 280 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
314
315! Emit code for non-Cray compilers
316# 287 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
317
318! Emit code only for Cray compiler
319# 294 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
320
321! Cray compiler directive, emitted from Fypp so the formatter cannot turn it into a comment
322# 301 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
323
324! Emit code for non-NVIDIA compilers
325# 308 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
326
327# 310 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
328# 311 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
329! New line at end of file is required for FYPP
330# 2 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp" 2
331
332# 14 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
333
334! Caution: This macro requires the use of a binding script to set CUDA_VISIBLE_DEVICES, such that we have one GPU device per MPI
335! rank. That's because for both cudaMemAdvise (preferred location) and cudaMemPrefetchAsync we use location = device_id = 0. For an
336! example see misc/nvidia_uvm/bind.sh.
337# 52 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
338
339! Allocate and create GPU device memory
340# 72 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
341
342! Free GPU device memory and deallocate
343# 80 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
344
345! Cray-specific GPU pointer setup for vector fields
346# 104 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
347
348! Cray-specific GPU pointer setup for scalar fields
349# 120 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
350
351! Cray-specific GPU pointer teardown for scalar fields - exact inverse of ACC_SETUP_SFs. Removes the
352! present-table entries ACC_SETUP_SFs added (the scalar_field descriptor and its %sf copyin). REQUIRED when
353! an ACC_SETUP'd field is freed MID-RUN (e.g. AMR slot free on regrid/restart): Cray 'exit data delete'
354! decrements the reference counter, so the lone @:DEALLOCATE(arg%sf) only undoes the @:ALLOCATE create ref
355! and leaves the descriptor + the ACC_SETUP %sf copyin dangling. Call BEFORE @:DEALLOCATE(arg%sf).
356# 140 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
357
358! Cray-specific GPU pointer setup for acoustic source spatials
359# 165 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
360
361! Whether stress index idx is a shear component. Spelled out because NVHPC 26.9 at -O2 vectorizes any(idx == shear_indices)
362! inside a loop to always true, which doubled the normal-stress elastic energy.
363# 171 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
364
365# 177 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
366
367# 184 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
368! New line at end of file is required for FYPP
369# 9 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp" 2
370
371!> @brief Block-structured AMR: up to amr_max_blocks refined blocks (2:1 or 4:1 per amr_ref_ratio), optionally nested to
372!! amr_max_level, advanced with the shared solver via grid-state swap and conservatively coupled to each block's parent level (ghost
373!! prolongation, Berger-Colella flux reflux, restriction) and dynamic regrid.
374module m_amr
375
376#ifdef MFC_MPI
377 use mpi !< mpi-io for the parallel_io amr restart file
378#endif
379
380 use m_derived_types ! scalar_field, t_box, int_bounds_info
382 use m_mpi_proxy, only: s_mpi_abort
384 use m_ibm, only: s_ibm_alloc_fine
386 use m_amr_state
388 use m_amr_wave
389 use m_amr_store
391 use m_amr_seam
392 use m_amr_frame
394 use m_amr_advance
395
396 implicit none
397
398 !> Re-exports every AMR service module.
399
400contains
401
402 !> Build the static refined level-1 block (no-op unless amr); runs after the level-0 grid and time-steppers are set up.
403 impure subroutine s_initialize_amr_module()
404
405 amr_dim = [.true., n_glb > 0, p_glb > 0]
406 amr_sidx = 0; amr_sidx(1:num_dims) = start_idx
407 amr_ext = merge([m, n, p], 0, amr_dim)
408
409 if (.not. amr) return
410
411#ifdef MFC_GPU
412 amr_fw_dev = rdma_mpi
413#endif
415
416 call s_amr_alloc_pool()
417 amr_block_level = 1 ! init default (level-1); regrid re-tags each block's level for nesting
420 amr_cur = 1
421 amr_seam_pairs_dirty = .true.; amr_seam_pairs_nblk = -1 ! force a seam-list build on the first fine-fine halo
423
424 call s_amr_init_extents()
425 call s_amr_init_report()
426 call s_amr_scr_init()
428 call s_amr_build_global_cb(0) ! the fine-distribution owner rebuilds whole-block fine coordinates from these
430 ! the coarse decomposition is a structured split computed O(1) per rank by s_amr_rank_decomp; validate it against this rank
432 ! per-slot fine-grid IB marker fields, sized like q_cons
433 if (ib) call s_ibm_alloc_fine(amr_max_blocks, mbuf_lo(1), mbuf_hi(1), mbuf_lo(2), mbuf_hi(2), mbuf_lo(3), mbuf_hi(3))
435 call s_amr_init_tags()
436
437 end subroutine s_initialize_amr_module
438
439 !> Block caps and preallocation extents. buff_size is unknown at checker time, so the geometric aborts live here.
440 impure subroutine s_amr_init_extents()
441
442 integer :: d, bad_loc, bad_glb, fit_d
443
444 call s_amr_compute_isect(amr_block_beg, amr_block_end)
445
446 ! the fine ghost shell and reflux outside cells must stay inside the global domain
447 if (amr_block_beg(1) < buff_size .or. amr_block_end(1) > m_glb - buff_size .or. (n_glb > 0 .and. (amr_block_beg(2) &
448 & < buff_size .or. amr_block_end(2) > n_glb - buff_size)) .or. (p_glb > 0 .and. (amr_block_beg(3) < buff_size &
449 & .or. amr_block_end(3) > p_glb - buff_size))) then
450 call s_mpi_abort('amr block must lie at least buff_size cells inside the domain boundaries')
451 end if
452
453 ! The fine advance reuses solver scratch sized to the local grid, so an IB block (never tiled) must fit a rank's local
454 ! half-extent; non-IB blocks are tiled to <= amr_maxc_fit.
455 bad_loc = 0
456 if (ib) then
457 if (amr_ref_ratio*(amr_block_end(1) - amr_block_beg(1) + 1) - 1 > m) bad_loc = 1
458 if (n_glb > 0 .and. amr_ref_ratio*(amr_block_end(2) - amr_block_beg(2) + 1) - 1 > n) bad_loc = 1
459 if (p_glb > 0 .and. amr_ref_ratio*(amr_block_end(3) - amr_block_beg(3) + 1) - 1 > p) bad_loc = 1
460 end if
461 call s_mpi_allreduce_integer_max(bad_loc, bad_glb)
462 if (bad_glb == 1) then
463 call s_mpi_abort('amr fine extent exceeds a rank local grid (solver scratch is local-sized): an immersed-body block ' &
464 & // 'is owned whole and un-tiled, so it may cover at most about half of any rank subdomain per ' &
465 & // 'dimension; shrink the body region or use fewer ranks')
466 end if
467
468 ! max coarse block cells per dim; 1 for collapsed dims
469 amr_maxc = merge(([m_glb, n_glb, p_glb] + 1)/amr_ref_ratio, 1, amr_dim)
470
471 ! Regrid box cap: min over ranks of the local half-extent, or the absolute amr_max_grid_size (rank-count-invariant box set,
472 ! scratch sized to the cap). Every box is clamped to amr_maxc_fit, which sizes the fine/coord arrays.
474 do d = 1, num_dims
475 call s_mpi_allreduce_integer_min((amr_ext(d) + 1)/amr_ref_ratio, fit_d)
476 if (amr_max_grid_size > 0) then
477 amr_maxc_fit(d) = min(amr_maxc(d), amr_max_grid_size)
478 else
479 amr_maxc_fit(d) = min(amr_maxc(d), fit_d)
480 end if
481 end do
482
483 ! max fine extents and buffered bounds
484 max_f = merge(amr_ref_ratio*amr_maxc_fit - 1, 0, amr_dim)
485 call s_amr_set_mbuf()
486
487 end subroutine s_amr_init_extents
488
489 !> Rank-0 advisories and the memory demand. Collective (reduces over ranks).
490 impure subroutine s_amr_init_report()
491
492 integer :: nonuni, nonuni_glb
493 real(wp) :: slot_gib, cells, nfam
494
495 ! stacked members reuse the batch leader's coordinates, exact only on bitwise-uniform spacing; warn if it differs at
496 ! roundoff
497
498 nonuni = 0
499 if (any(dx(0:m) /= dx(0))) nonuni = 1
500 if (n_glb > 0) then; if (any(dy(0:n) /= dy(0))) nonuni = 1; end if
501 if (p_glb > 0) then; if (any(dz(0:p) /= dz(0))) nonuni = 1; end if
502 call s_mpi_allreduce_integer_max(nonuni, nonuni_glb)
503 if (proc_rank /= 0) return
504
505 if (nonuni_glb == 1) print '(A)', &
506 & ' [amr] NOTE: the grid''s cell spacing is not bitwise uniform: stacked blocks reuse the batch leader''s ' &
507 & // 'coordinate arrays, so members after the first see the leader''s spacing'
508 ! the SFC map spreads whole blocks, so fewer blocks than ranks leaves some ranks without fine work
509 if (num_procs > amr_max_blocks) print '(A,I0,A,I0,A)', ' [amr] WARNING: amr_max_blocks (', amr_max_blocks, &
510 & ') < num_procs (', num_procs, &
511 & '): the fine level can occupy at most amr_max_blocks ranks - raise amr_max_blocks for better fine-level balance'
512 ! amr_regrid_int = 0 is static AMR and never adapts
513 if (amr_regrid_int == 0) print '(A)', &
514 & ' [amr] NOTE: amr_regrid_int = 0 - the block set is STATIC and never adapts. ' &
515 & // 'Set amr_regrid_int > 0 (4-8 is a reasonable start) for adaptive refinement.'
516 if (amr_max_grid_size == 0 .and. num_procs > 1) print '(A)', &
517 & ' [amr] NOTE: amr_max_grid_size = 0 derives the block cap from the ' &
518 & // 'decomposition, so it SHRINKS as ranks are added and the box set depends on rank ' &
519 & // 'count. Pin it to make the box set rank-invariant.'
520 if (amr_regrid_int > 0 .and. amr_regrid_int < 4) print '(A,I0,A)', ' [amr] NOTE: amr_regrid_int = ', amr_regrid_int, &
521 & ' regrids often; the tag sweep is per-CELL and flat in box count. ' &
522 & // 'Raise it unless the refined feature moves quickly.'
523
524 ! Memory demand: a block costs 2 field families (q_cons, q_cons_stor; q_prim/rhs are pooled) x sys_size arrays on the mbuf
525 ! extents. Exceeding device memory presents as a hang (one rank dies in __tgt_target_data_begin_mapper, the rest block in
526 ! MPI).
527 cells = real(product(mbuf_hi - mbuf_lo + 1), wp)
528 nfam = 2._wp
529 slot_gib = cells*real(sys_size, wp)*nfam*real(storage_size(1._wp)/8, wp)/1024._wp**3
530 print '(A,I0,A,I0,A,ES10.3,A,F8.3,A)', ' [amr] per-block slot: ', nint(cells), ' cells x sys_size x ', nint(nfam), &
531 & ' fields = ', cells*real(sys_size, wp)*nfam, ' words (', slot_gib, ' GiB per owned block)'
532 print '(A,F9.2,A,I0,A)', ' [amr] worst case if one rank owned every block: ', slot_gib*real(amr_max_blocks, wp), &
533 & ' GiB (amr_max_blocks = ', amr_max_blocks, '). Typical is amr_max_blocks/num_procs blocks per rank.'
534
535 end subroutine s_amr_init_report
536
537 !> Coarse-patch gather buffer (see amr_cg): the largest block's coarse footprint + 2*amr_cpat_mar halo, device-mapped.
538 impure subroutine s_amr_init_coarse_patch()
539
540 type(scalar_field), allocatable :: tmp_cg(:)
541 integer :: i
542
544 amr_cpat_hi = merge(amr_maxc_fit - 1 + 2*amr_cpat_mar, 0, amr_dim)
545 ! CCE OpenMP-offload: allocating a bare module-scope scalar_field array aborts (lib-4425); allocate a local, then
546 ! move_alloc.
547 allocate (tmp_cg(1:sys_size))
548#ifdef MFC_DEBUG
549# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
550 block
551# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
552 use iso_fortran_env, only: output_unit
553# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
554
555# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
556 print *, 'm_amr.fpp:187: ', '@:ALLOCATE(amr_slab_tab(1:15, 1:64))'
557# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
558
559# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
560 call flush (output_unit)
561# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
562 end block
563# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
564#endif
565# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
566 allocate (amr_slab_tab(1:15, 1:64))
567# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
568
569# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
570
571# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
572#if defined(MFC_OpenACC)
573# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
574!$acc enter data create(amr_slab_tab)
575# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
576#elif defined(MFC_OpenMP)
577# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
578!$omp target enter data map(always,alloc:amr_slab_tab)
579# 187 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
580#endif
581# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
582#ifdef MFC_DEBUG
583# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
584 block
585# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
586 use iso_fortran_env, only: output_unit
587# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
588
589# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
590 print *, 'm_amr.fpp:188: ', '@:ALLOCATE(amr_cfl_rf(0:m, 0:n, 0:p))'
591# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
592
593# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
594 call flush (output_unit)
595# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
596 end block
597# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
598#endif
599# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
600 allocate (amr_cfl_rf(0:m, 0:n, 0:p))
601# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
602
603# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
604
605# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
606#if defined(MFC_OpenACC)
607# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
608!$acc enter data create(amr_cfl_rf)
609# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
610#elif defined(MFC_OpenMP)
611# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
612!$omp target enter data map(always,alloc:amr_cfl_rf)
613# 188 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
614#endif
615# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
616#ifdef MFC_DEBUG
617# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
618 block
619# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
620 use iso_fortran_env, only: output_unit
621# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
622
623# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
624 print *, 'm_amr.fpp:189: ', '@:ALLOCATE(amr_cov_box(1:6, 1:amr_max_blocks))'
625# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
626
627# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
628 call flush (output_unit)
629# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
630 end block
631# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
632#endif
633# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
634 allocate (amr_cov_box(1:6, 1:amr_max_blocks))
635# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
636
637# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
638
639# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
640#if defined(MFC_OpenACC)
641# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
642!$acc enter data create(amr_cov_box)
643# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
644#elif defined(MFC_OpenMP)
645# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
646!$omp target enter data map(always,alloc:amr_cov_box)
647# 189 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
648#endif
649# 190 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
650 call move_alloc(tmp_cg, amr_cg)
651
652# 191 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
653#if defined(MFC_OpenACC)
654# 191 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
655!$acc enter data create(amr_cg)
656# 191 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
657#elif defined(MFC_OpenMP)
658# 191 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
659!$omp target enter data map(always,alloc:amr_cg)
660# 191 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
661#endif
662# 192 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
663 do i = 1, sys_size
664#ifdef MFC_DEBUG
665# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
666 block
667# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
668 use iso_fortran_env, only: output_unit
669# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
670
671# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
672 print *, 'm_amr.fpp:193: ', '@:ALLOCATE(amr_cg(i)%sf(0:amr_cpat_hi(1), 0:amr_cpat_hi(2), 0:amr_cpat_hi(3)))'
673# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
674
675# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
676 call flush (output_unit)
677# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
678 end block
679# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
680#endif
681# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
682 allocate (amr_cg(i)%sf(0:amr_cpat_hi(1), 0:amr_cpat_hi(2), 0:amr_cpat_hi(3)))
683# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
684
685# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
686
687# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
688#if defined(MFC_OpenACC)
689# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
690!$acc enter data create(amr_cg(i)%sf)
691# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
692#elif defined(MFC_OpenMP)
693# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
694!$omp target enter data map(always,alloc:amr_cg(i)%sf)
695# 193 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
696#endif
697# 194 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
698 amr_cg(i)%sf = 0._stp ! padding beyond a block's patch is never read; keep it finite for the device copy
699#ifdef _CRAYFTN
700# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
701 block
702# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
703#ifdef MFC_DEBUG
704# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
705 block
706# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
707 use iso_fortran_env, only: output_unit
708# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
709
710# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
711 print *, 'm_amr.fpp:195: ', '@:ACC_SETUP_SFs(amr_cg(i))'
712# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
713
714# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
715 call flush (output_unit)
716# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
717 end block
718# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
719#endif
720# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
721
722# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
723
724# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
725#if defined(MFC_OpenACC)
726# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
727!$acc enter data copyin(amr_cg(i))
728# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
729#elif defined(MFC_OpenMP)
730# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
731!$omp target enter data map(to:amr_cg(i))
732# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
733#endif
734# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
735 if (associated(amr_cg(i)%sf)) then
736# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
737
738# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
739#if defined(MFC_OpenACC)
740# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
741!$acc enter data copyin(amr_cg(i)%sf)
742# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
743#elif defined(MFC_OpenMP)
744# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
745!$omp target enter data map(to:amr_cg(i)%sf)
746# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
747#endif
748# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
749 end if
750# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
751 end block
752# 195 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
753#endif
754# 196 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
755 end do
756
757 end subroutine s_amr_init_coarse_patch
758
759 !> Place the initial block(s) and set their geometry. With dynamic regrid and bodies the block gets the same body-containment
760 !! expansion regrid boxes get; non-IB blocks are tiled to <= amr_maxc_fit, IB keeps one block. Slot arrays are allocated lazily
761 !! by s_amr_reconcile_slots.
762 impure subroutine s_amr_init_first_blocks()
763
764 type(t_box), allocatable :: tiled(:)
765 integer :: blk_lo(3), blk_hi(3), nt, capt, kk
766
767 blk_lo = amr_block_beg; blk_hi = amr_block_end
768 if (ib .and. amr_regrid_int > 0) call s_amr_expand_box_over_bodies(blk_lo, blk_hi)
769 allocate (tiled(amr_max_blocks)); nt = 0; capt = 0
770 if (ib) then
771 nt = 1; tiled(1)%lo = blk_lo; tiled(1)%hi = blk_hi
772 else
773 call s_amr_tile_box(blk_lo, blk_hi, tiled, nt, amr_max_blocks, capt)
774 end if
775 amr_num_blocks = nt
776 ! set regions first: the owner assignment reads amr_region_*_all and s_set_amr_fine_geometry needs the owners
777 do kk = 1, nt
778 amr_region_lo_all(:,kk) = tiled(kk)%lo; amr_region_hi_all(:,kk) = tiled(kk)%hi
779 end do
780 call s_amr_assign_block_owners()
781 call s_amr_reconcile_slots() ! allocate this rank's owned blocks
782 do kk = 1, nt
783 amr_cur = kk
784 call s_set_amr_fine_geometry(tiled(kk)%lo, tiled(kk)%hi)
785 end do
786 call s_amr_reduce_xchg_flag()
787 call s_amr_select_slot(1) ! refresh the per-block mirrors
788 deallocate (tiled)
789
790 end subroutine s_amr_init_first_blocks
791
792 !> Tag bases: narrow-reduction tags sit above the per-box tags; keyed bands start at the next 65536 boundary above them.
793 impure subroutine s_amr_init_tags()
794
795#ifdef MFC_MPI
796 integer(kind=MPI_ADDRESS_KIND) :: tag_ub
797 logical :: tag_ub_set
798 integer :: ierr
799#endif
800
801 amr_tag_narrow = amr_max_blocks + 400
802 amr_tag_base = ((amr_max_blocks + 800)/65536 + 1)*65536
803#ifdef MFC_MPI
804 call mpi_comm_get_attr(mpi_comm_world, mpi_tag_ub, tag_ub, tag_ub_set, ierr)
805 if (.not. (tag_ub_set)) then
806# 246 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
807 call s_mpi_abort("m_amr.fpp:246: " // "Assertion failed: tag_ub_set. " &
808# 246 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
809 & // "MPI_TAG_UB attribute unavailable")
810# 246 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
811 end if
812# 247 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
813 if (.not. (amr_max_blocks + 800 <= tag_ub)) then
814# 247 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
815 call s_mpi_abort("m_amr.fpp:247: " // "Assertion failed: amr_max_blocks + 800 <= tag_ub. " &
816# 247 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
817 & // "AMR tag space exceeds MPI_TAG_UB: amr_max_blocks is too large for this MPI's tag range")
818# 247 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
819 end if
820# 249 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
821 if (.not. (amr_tag_base + amr_n_bands*65536 <= tag_ub)) then
822# 249 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
823 call s_mpi_abort("m_amr.fpp:249: " // "Assertion failed: amr_tag_base + amr_n_bands*65536 <= tag_ub. " &
824# 249 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
825 & // "AMR keyed-tag band space exceeds MPI_TAG_UB")
826# 249 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
827 end if
828# 250 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
829#endif
830
831 end subroutine s_amr_init_tags
832
833 impure subroutine s_finalize_amr_module()
834
835 integer :: i, islot
836
837 if (.not. amr) return
838 if (allocated(amr_cons_st)) call s_mem_note(mem_store, storage_size(amr_cons_st, kind=8)/8_8*(size(amr_cons_st, &
839 & kind=8) + size(amr_stor_st, kind=8)))
840# 262 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
841 if (allocated(amr_scr_prim)) call s_mem_note_fields(mem_scratch, amr_scr_prim)
842# 263 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
843# 262 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
844 if (allocated(amr_scr_rhs)) call s_mem_note_fields(mem_scratch, amr_scr_rhs)
845# 263 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
846# 262 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
847 if (allocated(amr_scr_prim_blk)) call s_mem_note_fields(mem_scratch, amr_scr_prim_blk)
848# 263 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
849# 262 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
850 if (allocated(amr_cons_br)) call s_mem_note_fields(mem_scratch, amr_cons_br)
851# 263 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
852# 264 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
853 if (allocated(amr_cg)) call s_mem_note_fields(mem_patch, amr_cg)
854 if (allocated(amr_pc)) call s_mem_note(mem_patch, storage_size(amr_pc, kind=8)/8_8*size(amr_pc, kind=8))
855 do islot = 1, amr_max_blocks
856 call s_amr_free_slot(islot)
857 end do
858 call s_amr_free_pool()
859 ! gfortran/ifx abort on deallocating an unallocated array: guard each
860 if (allocated(amr_fw_rblk)) deallocate (amr_fw_rblk)
861 if (allocated(amr_sw_sq)) deallocate (amr_sw_sq, amr_sw_rq)
862# 274 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
863 if (allocated(amr_fw_sq)) then
864# 275 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
865 if (amr_fw_dev) then
866
867# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
868#if defined(MFC_OpenACC)
869# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
870!$acc exit data delete(amr_fw_sq)
871# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
872#elif defined(MFC_OpenMP)
873# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
874!$omp target exit data map(release:amr_fw_sq)
875# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
876#endif
877# 277 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
878 end if
879 deallocate (amr_fw_sq)
880# 279 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
881 end if
882# 274 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
883 if (allocated(amr_fw_rq)) then
884# 275 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
885 if (amr_fw_dev) then
886
887# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
888#if defined(MFC_OpenACC)
889# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
890!$acc exit data delete(amr_fw_rq)
891# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
892#elif defined(MFC_OpenMP)
893# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
894!$omp target exit data map(release:amr_fw_rq)
895# 276 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
896#endif
897# 277 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
898 end if
899 deallocate (amr_fw_rq)
900# 279 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
901 end if
902# 281 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
903# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
904 if (allocated(amr_my_blk)) deallocate (amr_my_blk)
905# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
906# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
907 if (allocated(amr_l1r_blk)) deallocate (amr_l1r_blk)
908# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
909# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
910 if (allocated(amr_l1p_blk)) deallocate (amr_l1p_blk)
911# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
912# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
913 if (allocated(amr_fch_blk)) deallocate (amr_fch_blk)
914# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
915# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
916 if (allocated(amr_own_blk)) deallocate (amr_own_blk)
917# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
918# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
919 if (allocated(amr_parent_blk)) deallocate (amr_parent_blk)
920# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
921# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
922 if (allocated(amr_child_ptr)) deallocate (amr_child_ptr)
923# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
924# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
925 if (allocated(amr_child_idx)) deallocate (amr_child_idx)
926# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
927# 283 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
928 if (allocated(amr_cg_pz)) deallocate (amr_cg_pz)
929# 284 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
930# 285 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
931 do i = 1, sys_size
932#ifdef MFC_DEBUG
933# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
934 block
935# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
936 use iso_fortran_env, only: output_unit
937# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
938
939# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
940 print *, 'm_amr.fpp:286: ', '@:DEALLOCATE(amr_cg(i)%sf)'
941# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
942
943# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
944 call flush (output_unit)
945# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
946 end block
947# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
948#endif
949# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
950
951# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
952#if defined(MFC_OpenACC)
953# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
954!$acc exit data delete(amr_cg(i)%sf)
955# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
956#elif defined(MFC_OpenMP)
957# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
958!$omp target exit data map(release:amr_cg(i)%sf)
959# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
960#endif
961# 286 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
962 deallocate (amr_cg(i)%sf)
963# 287 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
964 end do
965#ifdef MFC_DEBUG
966# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
967 block
968# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
969 use iso_fortran_env, only: output_unit
970# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
971
972# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
973 print *, 'm_amr.fpp:288: ', '@:DEALLOCATE(amr_cg)'
974# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
975
976# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
977 call flush (output_unit)
978# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
979 end block
980# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
981#endif
982# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
983
984# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
985#if defined(MFC_OpenACC)
986# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
987!$acc exit data delete(amr_cg)
988# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
989#elif defined(MFC_OpenMP)
990# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
991!$omp target exit data map(release:amr_cg)
992# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
993#endif
994# 288 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
995 deallocate (amr_cg)
996# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
997#ifdef MFC_DEBUG
998# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
999 block
1000# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1001 use iso_fortran_env, only: output_unit
1002# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1003
1004# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1005 print *, 'm_amr.fpp:289: ', '@:DEALLOCATE(amr_slab_tab)'
1006# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1007
1008# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1009 call flush (output_unit)
1010# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1011 end block
1012# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1013#endif
1014# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1015
1016# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1017#if defined(MFC_OpenACC)
1018# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1019!$acc exit data delete(amr_slab_tab)
1020# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1021#elif defined(MFC_OpenMP)
1022# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1023!$omp target exit data map(release:amr_slab_tab)
1024# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1025#endif
1026# 289 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1027 deallocate (amr_slab_tab)
1028# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1029#ifdef MFC_DEBUG
1030# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1031 block
1032# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1033 use iso_fortran_env, only: output_unit
1034# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1035
1036# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1037 print *, 'm_amr.fpp:290: ', '@:DEALLOCATE(amr_cfl_rf)'
1038# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1039
1040# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1041 call flush (output_unit)
1042# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1043 end block
1044# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1045#endif
1046# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1047
1048# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1049#if defined(MFC_OpenACC)
1050# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1051!$acc exit data delete(amr_cfl_rf)
1052# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1053#elif defined(MFC_OpenMP)
1054# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1055!$omp target exit data map(release:amr_cfl_rf)
1056# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1057#endif
1058# 290 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1059 deallocate (amr_cfl_rf)
1060# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1061#ifdef MFC_DEBUG
1062# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1063 block
1064# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1065 use iso_fortran_env, only: output_unit
1066# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1067
1068# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1069 print *, 'm_amr.fpp:291: ', '@:DEALLOCATE(amr_cov_box)'
1070# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1071
1072# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1073 call flush (output_unit)
1074# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1075 end block
1076# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1077#endif
1078# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1079
1080# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1081#if defined(MFC_OpenACC)
1082# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1083!$acc exit data delete(amr_cov_box)
1084# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1085#elif defined(MFC_OpenMP)
1086# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1087!$omp target exit data map(release:amr_cov_box)
1088# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1089#endif
1090# 291 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1091 deallocate (amr_cov_box)
1092# 292 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1093 if (allocated(amr_pc)) then
1094#ifdef MFC_DEBUG
1095# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1096 block
1097# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1098 use iso_fortran_env, only: output_unit
1099# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1100
1101# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1102 print *, 'm_amr.fpp:293: ', '@:DEALLOCATE(amr_pc)'
1103# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1104
1105# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1106 call flush (output_unit)
1107# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1108 end block
1109# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1110#endif
1111# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1112
1113# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1114#if defined(MFC_OpenACC)
1115# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1116!$acc exit data delete(amr_pc)
1117# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1118#elif defined(MFC_OpenMP)
1119# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1120!$omp target exit data map(release:amr_pc)
1121# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1122#endif
1123# 293 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1124 deallocate (amr_pc)
1125# 294 "/home/runner/work/MFC/MFC/src/simulation/m_amr.fpp"
1126 end if
1127 call s_amr_free_swap_buffers()
1128
1129 end subroutine s_finalize_amr_module
1130
1131end module m_amr
Fine-block stage advance and the immersed-boundary/Lagrange fine services.
Rank decomposition, box arithmetic and block ownership (SFC cut, owner lookups, owned-block lists).
impure subroutine, public s_amr_compute_isect(lo, hi)
Set this rank's intersection of box lo:hi with its subdomain (amr_isect_lo/hi) and whether it holds f...
impure subroutine, public s_amr_validate_decomp()
Guard: the computed decomposition must reproduce this rank's actual one (each rank checks itself,...
Parent/child data exchange: gather plans, pack/unpack and the fill waves.
Block-frame swap (fine grid state in/out of the shared solver).
Same-level fine-fine seams and the coarse conservative halo.
Module-scope AMR state (block pool, flat field store, wave buffers) and the leaf helpers every AMR mo...
integer amr_cpat_mar
coarse-cell stencil reach (f_amr_cpat_mar)
impure subroutine s_amr_build_global_cb(pad)
Assemble the global coarse cell-boundary arrays over (-1-pad : G+pad): each rank writes its own cells...
integer, dimension(3) amr_maxc
max coarse block cells per dim: (m_glb+1)/2 etc.; 1 for collapsed dims
pure integer function f_amr_cpat_mar()
Coarse-patch margin amr_cpat_mar: the coarse cells a buff_size fine ghost layer spans plus the prolon...
integer amr_seam_pairs_nblk
integer amr_br_batch
Blocks per batched s_compute_rhs call (amr_bat_max once initialized); bounded to avoid exhausting dev...
integer, dimension(3) max_f
Per-slot sizing: max fine cells per dim and the buffered array bounds. amr_slot_live(k): slot k's fie...
integer, dimension(3) amr_cpat_hi
amr_cg upper local bounds per dim (0 in collapsed dims)
logical amr_fw_dev
Device-resident wire pools: with rdma_mpi MPI sends/receives them by device address,...
impure subroutine s_amr_scr_init()
Allocate the pooled q_prim/rhs advance scratch (idempotent). The lockstep driver associates it for ev...
logical amr_seam_pairs_dirty
integer, dimension(3) mbuf_lo
integer, dimension(:,:), allocatable amr_slab_tab
Per-block gathered coarse patch: the owner may not hold the coarse cells its block refines,...
integer, dimension(3) amr_maxc_fit
Regrid box size cap per dim (fixed for the run, identical on all ranks; 1 in collapsed dims): a box o...
integer, dimension(3) mbuf_hi
Flat per-block field store: dense slot indexing, reservation, alloc/free and the prim/bridge loaders.
impure subroutine, public s_amr_alloc_pool()
Block-metadata pool for amr_max_blocks slots (geometry, ownership, level, SFC cuts,...
impure subroutine, public s_amr_set_mbuf()
The buffered per-slot array bounds from the max fine extents (collapsed dims 0:0).
impure subroutine, public s_amr_init_swap_buffers()
Bounce buffers for the copy-based coordinate swap; shaped like the base-level *_alloc arrays,...
Prolongation, restriction and Berger-Colella reflux, including the restrict/reflux/freg waves.
MPI point-to-point waves for AMR exchange: post all receives, pack, post all sends,...
Block-structured AMR: up to amr_max_blocks refined blocks (2:1 or 4:1 per amr_ref_ratio),...
impure subroutine s_amr_init_report()
Rank-0 advisories and the memory demand. Collective (reduces over ranks).
impure subroutine s_finalize_amr_module()
impure subroutine s_amr_init_extents()
Block caps and preallocation extents. buff_size is unknown at checker time, so the geometric aborts l...
impure subroutine s_amr_init_coarse_patch()
Coarse-patch gather buffer (see amr_cg): the largest block's coarse footprint + 2*amr_cpat_mar halo,...
impure subroutine s_amr_init_tags()
Tag bases: narrow-reduction tags sit above the per-box tags; keyed bands start at the next 65536 boun...
impure subroutine s_amr_init_first_blocks()
Place the initial block(s) and set their geometry. With dynamic regrid and bodies the block gets the ...
impure subroutine s_initialize_amr_module()
Re-exports every AMR service module.
Shared derived types for field data, patch geometry, bubble dynamics, and MPI I/O structures.
Global parameters for the computational domain, fluid properties, and simulation algorithm configurat...
integer, dimension(3) amr_sidx
integer proc_rank
Rank of the local processor.
integer(8) amr_mesh_epoch
Monotone mesh epoch, incremented wherever amr_seam_pairs_dirty is set and at the end of each slot rec...
integer, dimension(:), allocatable amr_block_level
Refinement level of each active block (1..amr_max_level) and the deepest level populated....
integer buff_size
Number of ghost cells for boundary condition storage.
logical, dimension(3) amr_dim
The active dimensions, and this rank's coarse subdomain origin and extent padded to 3-vectors (0 in c...
integer num_procs
Number of processors.
integer amr_num_blocks
Number of active AMR fine-block slots (>= 1) and the working slot index selecting which one the per-b...
integer, dimension(3) amr_ext
real(wp), dimension(:), allocatable, target dy
real(wp), dimension(:), allocatable, target dz
real(wp), dimension(:), allocatable, target dx
Ghost-node immersed boundary method: locates ghost/image points, computes interpolation coefficients,...
impure subroutine, public s_ibm_alloc_fine(nslots, f1_lo, f1_hi, f2_lo, f2_hi, f3_lo, f3_hi)
Allocate the per-slot fine-IB marker fields, sized to the max buffered fine extents; ghost-point list...
MPI communication layer: domain decomposition, halo exchange, reductions, and parallel I/O setup.
impure subroutine s_mpi_abort(prnt, code)
The subroutine terminates the MPI execution environment.
impure subroutine s_mpi_allreduce_integer_min(var_loc, var_glb)
Reduce a local integer value to its global minimum across all MPI ranks.
impure subroutine s_mpi_allreduce_integer_max(var_loc, var_glb)
Reduce a local integer value to its global maximum across all MPI ranks.
MPI halo exchange, domain decomposition, and buffer packing/unpacking for the simulation solver.
Per-phase wall-time budget of the AMR step, reported when rank_time_wrt is on. Brackets are disjoint ...
impure subroutine, public s_mem_note(id, bytes)
Add bytes to group id of the end-of-run AMR memory table.
impure subroutine, public s_mem_note_fields(id, f)
s_mem_note for the fields of f that are allocated.
integer, parameter, public mem_scratch
batched-advance scratch (prim, rhs) and the block bridge
integer, parameter, public mem_patch
coarse patch pool and its subcycling time cache
integer, parameter, public mem_store
AMR allocations by group, noted by each owner before it frees them (s_mem_note) and printed with the ...
Derived type annexing a scalar field (SF).