MFC
Exascale flow solver
Loading...
Searching...
No Matches
m_start_up.fpp.f90
Go to the documentation of this file.
1# 1 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2!>
3!! @file
4!! @brief Contains module m_start_up
5
6# 1 "/home/runner/work/MFC/MFC/src/common/include/case.fpp" 1
7! This file exists so that Fypp can be run without generating case.fpp files for
8! each target. This is useful when generating documentation, for example. This
9! should also let MFC be built with CMake directly, without invoking mfc.sh.
10
11! For pre-process.
12# 8 "/home/runner/work/MFC/MFC/src/common/include/case.fpp"
13
14! For moving immersed boundaries in simulation
15# 12 "/home/runner/work/MFC/MFC/src/common/include/case.fpp"
16# 6 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp" 2
17# 1 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp" 1
18# 1 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 1
19# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
20# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
21# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
22# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
23# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
24# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
25
26# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
27# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
28# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
29
30# 12 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
31# 1 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp" 1
32! Default for Fypp runs outside the toolchain (e.g. documentation). Searched last, so the
33! build/include/<target>/thermochem.fpp the toolchain writes from m_thermochem takes precedence.
34# 4 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
35# 5 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
36# 13 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 2
37
38# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
39# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
40
41# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
42
43# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
44
45# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
46
47# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
48
49# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
50
51# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
52
53# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
54
55# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
56! New line at end of file is required for FYPP
57# 2 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
58# 1 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp" 1
59# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
60# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
61# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
62# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
63# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
64# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
65
66# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
67# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
68# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
69
70# 12 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
71# 1 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp" 1
72! Default for Fypp runs outside the toolchain (e.g. documentation). Searched last, so the
73! build/include/<target>/thermochem.fpp the toolchain writes from m_thermochem takes precedence.
74# 4 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
75# 5 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
76# 13 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 2
77
78# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
79# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
80
81# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
82
83# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
84
85# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
86
87# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
88
89# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
90
91# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
92
93# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
94
95# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
96! New line at end of file is required for FYPP
97# 2 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp" 2
98
99# 4 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
100# 5 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
101# 6 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
102# 7 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
103# 8 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
104
105# 20 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
106
107# 43 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
108
109# 48 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
110
111# 53 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
112
113# 58 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
114
115# 63 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
116
117# 68 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
118
119# 81 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
120
121# 95 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
122
123# 100 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
124
125# 105 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
126
127# 110 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
128
129# 115 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
130
131# 120 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
132
133# 125 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
134
135# 130 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
136
137# 135 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
138
139# 165 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
140
141# 219 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
142
143# 233 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
144
145# 258 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
146
147# 269 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
148
149# 271 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
150# 282 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
151
152# 332 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
153
154# 342 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
155
156# 352 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
157
158# 361 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
159
160# 378 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
161
162# 388 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
163
164# 395 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
165
166# 401 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
167
168# 407 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
169
170# 413 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
171
172# 419 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
173
174# 425 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
175! New line at end of file is required for FYPP
176# 3 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
177# 1 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp" 1
178# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
179# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
180# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
181# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
182# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
183# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
184
185# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
186# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
187# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
188
189# 12 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
190# 1 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp" 1
191! Default for Fypp runs outside the toolchain (e.g. documentation). Searched last, so the
192! build/include/<target>/thermochem.fpp the toolchain writes from m_thermochem takes precedence.
193# 4 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
194# 5 "/home/runner/work/MFC/MFC/src/common/include/defaults/thermochem.fpp"
195# 13 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 2
196
197# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
198# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
199
200# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
201
202# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
203
204# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
205
206# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
207
208# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
209
210# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
211
212# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
213
214# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
215! New line at end of file is required for FYPP
216# 2 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp" 2
217
218# 7 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
219
220# 17 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
221
222# 22 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
223
224# 27 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
225
226# 32 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
227
228# 37 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
229
230# 42 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
231
232# 47 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
233
234# 52 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
235
236# 57 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
237
238# 62 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
239
240# 73 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
241
242# 78 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
243
244# 83 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
245
246# 88 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
247
248# 103 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
249
250# 131 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
251
252# 160 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
253
254# 175 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
255
256# 193 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
257
258# 215 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
259
260# 244 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
261
262# 259 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
263
264# 269 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
265
266# 278 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
267
268# 294 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
269
270# 304 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
271
272# 311 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
273! New line at end of file is required for FYPP
274# 4 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
275
276! GPU parallel region (scalar reductions, maxval/minval)
277# 23 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
278
279! GPU parallel loop over threads (most common GPU macro)
280# 43 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
281
282! Required closing for GPU_PARALLEL_LOOP
283# 55 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
284
285! Mark routine for device compilation
286# 112 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
287
288! Declare device-resident data
289# 130 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
290
291! Inner loop within a GPU parallel region
292# 145 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
293
294! Scoped GPU data region
295# 164 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
296
297! Host code with device pointers (for MPI with GPU buffers)
298# 193 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
299
300! Allocate device memory (unscoped)
301# 207 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
302
303! Free device memory
304# 219 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
305
306! Atomic operation on device
307# 231 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
308
309! End atomic capture block
310# 242 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
311
312! Copy data between host and device
313# 254 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
314
315! Synchronization barrier
316# 266 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
317
318! Import GPU library module (openacc or omp_lib)
319# 275 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
320
321! Emit code only for AMD compiler
322# 282 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
323
324! Emit code for non-Cray compilers
325# 289 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
326
327! Emit code only for Cray compiler
328# 296 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
329
330! Emit code for non-NVIDIA compilers
331# 303 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
332
333# 305 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
334# 306 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
335! New line at end of file is required for FYPP
336# 2 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp" 2
337
338# 14 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
339
340! Caution: This macro requires the use of a binding script to set CUDA_VISIBLE_DEVICES, such that we have one GPU device per MPI
341! rank. That's because for both cudaMemAdvise (preferred location) and cudaMemPrefetchAsync we use location = device_id = 0. For an
342! example see misc/nvidia_uvm/bind.sh.
343# 52 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
344
345! Allocate and create GPU device memory
346# 72 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
347
348! Free GPU device memory and deallocate
349# 80 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
350
351! Cray-specific GPU pointer setup for vector fields
352# 104 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
353
354! Cray-specific GPU pointer setup for scalar fields
355# 120 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
356
357! Cray-specific GPU pointer setup for acoustic source spatials
358# 145 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
359
360# 151 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
361
362# 158 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
363! New line at end of file is required for FYPP
364# 7 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp" 2
365
366!> @brief Reads input files, loads initial conditions and grid data, and orchestrates solver initialization and finalization
367module m_start_up
368
369 use m_derived_types
370 use m_global_parameters
371 use m_mpi_proxy
372 use m_mpi_common
373 use m_variables_conversion
374 use m_eos
375 use m_weno
376 use m_muscl
377 use m_thinc
378 use m_riemann_solvers
379 use m_cbc
380 use m_boundary_common
381 use m_boundary_io
382 use m_acoustic_src
383 use m_rhs
384 use m_pressure_relaxation, only: s_report_pressure_relaxation
385 use m_chemistry
386 use m_data_output
387 use m_time_steppers
388 use m_qbmm
389 use m_derived_variables
390 use m_hypoelastic
391 use m_phase_change
392 use m_viscous
393 use m_bubbles_ee
394 use m_bubbles_el
395 use ieee_arithmetic
396 use m_helper_basic
397 use m_helper
398
399#if defined(MFC_OpenACC)
400# 41 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
401 use openacc
402# 41 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
403#elif defined(MFC_OpenMP)
404# 41 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
405 use omp_lib
406# 41 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
407#endif
408# 42 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
409
410 use m_nvtx
411 use m_ibm
412 use m_ib_patches
413 use m_model
414 use m_collisions
415 use m_compile_specific
416 use m_checker_common
417 use m_checker
418 use m_surface_tension
419 use m_body_forces
420 use m_sim_helpers
421 use m_igr
422 use m_constants, only: model_eqns_6eq, time_stepper_rk1, time_stepper_rk2, time_stepper_rk3, recon_type_weno, recon_type_muscl
423
424 implicit none
425
426 private; public :: s_read_input_file, s_check_input_file, s_read_data_files, s_read_serial_data_files, &
427 & s_read_parallel_data_files, s_initialize_internal_energy_equations, s_initialize_modules, s_initialize_gpu_vars, &
428 & s_initialize_mpi_domain, s_finalize_modules, s_perform_time_step, s_save_data, s_save_performance_metrics
429
430 type(scalar_field), allocatable, dimension(:) :: q_cons_temp
431 real(wp) :: dt_init
432
433contains
434
435 !> Read data files. Dispatch subroutine that replaces procedure pointer.
436 impure subroutine s_read_data_files(q_cons_vf)
437
438 type(scalar_field), dimension(sys_size), intent(inout) :: q_cons_vf
439
440 if (.not. parallel_io) then
441 call s_read_serial_data_files(q_cons_vf)
442 else
443 call s_read_parallel_data_files(q_cons_vf)
444 end if
445
446 end subroutine s_read_data_files
447
448 !> Verify the input file exists and read it
449 impure subroutine s_read_input_file
450
451 character(LEN=name_len), parameter :: file_path = './simulation.inp'
452 logical :: file_exist !< Logical used to check the existence of the input file
453 integer :: iostatus
454 ! Integer to check iostat of file read
455
456 character(len=1000) :: line
457
458# 1 "/home/runner/work/MFC/MFC/build/include/simulation/generated_namelist.fpp" 1
459! AUTO-GENERATED - do not edit directly. Regenerate: cmake reconfigure
460!
461# 22 "/home/runner/work/MFC/MFC/build/include/simulation/generated_namelist.fpp"
462namelist /user_inputs/ adc_kappa, bx0, ca, r0ref, re_inv, web, acoustic, acoustic_source, adap_dt, adap_dt_max_iters, &
463 & adap_dt_tol, adv_n, alf_factor, alpha_bar, alt_soundspeed, avg_state, bc_x, bc_y, bc_z, bf_spatial_support, bf_x, bf_y, &
464 & bf_z, bub_pp, bubble_model, bubbles_euler, bubbles_lagrange, case_dir, cfl_adap_dt, cfl_const_dt, cfl_target, chem_params, &
465 & coefficient_of_restitution, collision_model, collision_temporal_resolution, collision_time, cont_damage, cont_damage_s, &
466 & cyl_coord, down_sample, dt, fd_order, fft_wrt, file_per_process, fluid_pp, g_x, g_y, g_z, hll_u_interface, hyper_cleaning, &
467 & hyper_cleaning_speed, hyper_cleaning_tau, hypo_hll_interface_rhs, hypoelasticity, ib, ib_airfoil, &
468 & ib_coefficient_of_friction, ib_force_stride, ib_force_wrt, ib_ip_min_dist, ib_neighborhood_radius, ib_second_order_vel, &
469 & ib_state_wrt, ic_beta, ic_eps, int_comp, k_x, k_y, k_z, lag_params, low_mach, m, many_ib_patch_parallelism, mixture_err, &
470 & model_eqns, mp_weno, mpp_lim, muscl_eps, n, n_start, null_weights, num_bc_patches, num_ibs, num_igr_iters, &
471 & num_igr_warm_start_iters, num_particle_clouds, num_probes, num_source, num_stl_models, num_turbulent_sources, &
472 & nv_uvm_igr_temps_on_gpu, nv_uvm_out_of_core, nv_uvm_pref_gpu, p, p_x, p_y, p_z, palpha_eps, parallel_io, particle_cloud, &
473 & patch_ib, pi_fac, poly_sigma, polydisperse, polytropic, precision, prim_vars_wrt, probe, probe_wrt, ptgalpha_eps, qbmm, &
474 & ramp_ratio, rburn, rdma_mpi, reactive_burn, relax, relax_model, riemann_hypo_adc, riemann_solver, run_time_info, sigma, &
475 & spatial_bf, stl_models, surface_tension, synth_l, synth_u_inf, synth_amp_shell, synth_k_shell, synth_n_shells, &
476 & synth_n_waves_per_shell, synth_seed, synthetic_turbulence, t_save, t_step_old, t_step_print, t_step_save, t_step_start, &
477 & t_step_stop, t_stop, tau_star, teno_ct, thermal, time_stepper, turb_pos, w_x, w_y, w_z, wave_speeds, weno_re_flux, &
478 & weno_avg, weno_eps, &
479 & igr, igr_iter_solver, igr_order, igr_pres_lim, mapped_weno, mhd, muscl_lim, muscl_order, nb, num_fluids, recon_type, &
480 & relativity, teno, viscous, weno_order, wenoz, wenoz_q
481# 42 "/home/runner/work/MFC/MFC/build/include/simulation/generated_namelist.fpp"
482# 92 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp" 2
483
484 inquire (file=trim(file_path), exist=file_exist)
485
486 if (file_exist) then
487 open (1, file=trim(file_path), form='formatted', action='read', status='old')
488 read (1, nml=user_inputs, iostat=iostatus)
489
490 if (iostatus /= 0) then
491 backspace(1)
492 read (1, fmt='(A)') line
493 print *, 'Invalid line in namelist: ' // trim(line)
494 call s_mpi_abort('Invalid line in simulation.inp. It is ' // 'likely due to a datatype mismatch. Exiting.')
495 end if
496
497 close (1)
498
499 if ((bf_x) .or. (bf_y) .or. (bf_z) .or. (bf_spatial_support)) then
500 bodyforces = .true.
501 end if
502
503 m_glb = m
504 n_glb = n
505 p_glb = p
506
507 call s_update_cell_bounds(cells_bounds, m, n, p)
508
509 if (cfl_adap_dt .or. cfl_const_dt) cfl_dt = .true.
510
511 if (any((/bc_x%beg, bc_x%end, bc_y%beg, bc_y%end, bc_z%beg, bc_z%end/) == -17) .or. num_bc_patches > 0) then
512 bc_io = .true.
513 end if
514
515 if (bc_x%beg == bc_periodic .and. bc_x%end == bc_periodic) periodic_bc(1) = .true.
516 if (bc_y%beg == bc_periodic .and. bc_y%end == bc_periodic) periodic_bc(2) = .true.
517 if (bc_z%beg == bc_periodic .and. bc_z%end == bc_periodic) periodic_bc(3) = .true.
518 else
519 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
520 end if
521
522 end subroutine s_read_input_file
523
524 !> Validate that all user-provided inputs form a consistent simulation configuration
525 impure subroutine s_check_input_file
526
527 character(LEN=path_len) :: file_path
528 logical :: file_exist
529
530 file_path = trim(case_dir) // '/.'
531
532 call my_inquire(file_path, file_exist)
533
534 if (file_exist .neqv. .true.) then
535 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
536 end if
537
538 call s_check_inputs_common(check_total_cells=.false., n_global=0_8)
539 call s_check_inputs()
540
541 end subroutine s_check_input_file
542
543 !> Read serial initial condition and grid data files and compute cell-width distributions
544 impure subroutine s_read_serial_data_files(q_cons_vf)
545
546 type(scalar_field), dimension(sys_size), intent(inout) :: q_cons_vf
547 character(LEN=path_len + 2*name_len) :: t_step_dir !< Relative path to the starting time-step directory
548 character(LEN=path_len + 3*name_len) :: file_path !< Relative path to the grid and conservative variables data files
549 logical :: file_exist !< Logical used to check the existence of the input file
550 integer :: i, r
551
552 if (cfl_dt) then
553 write (t_step_dir, '(A,I0,A,I0)') trim(case_dir) // '/p_all/p', proc_rank, '/', n_start
554 else
555 write (t_step_dir, '(A,I0,A,I0)') trim(case_dir) // '/p_all/p', proc_rank, '/', t_step_start
556 end if
557
558 file_path = trim(t_step_dir) // '/.'
559 call my_inquire(file_path, file_exist)
560
561 if (file_exist .neqv. .true.) then
562 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
563 end if
564
565 if (bc_io) then
566 call s_read_serial_boundary_condition_files(t_step_dir, bc_type)
567 else
568 call s_assign_default_bc_type(bc_type)
569 end if
570
571 file_path = trim(t_step_dir) // '/x_cb.dat'
572
573 inquire (file=trim(file_path), exist=file_exist)
574
575 if (file_exist) then
576 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
577 read (2) x_cb(-1:m); close (2)
578 else
579 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
580 end if
581
582 dx(0:m) = x_cb(0:m) - x_cb(-1:m - 1)
583 x_cc(0:m) = x_cb(-1:m - 1) + dx(0:m)/2._wp
584
585 if (n > 0) then
586 file_path = trim(t_step_dir) // '/y_cb.dat'
587
588 inquire (file=trim(file_path), exist=file_exist)
589
590 if (file_exist) then
591 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
592 read (2) y_cb(-1:n); close (2)
593 else
594 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
595 end if
596
597 dy(0:n) = y_cb(0:n) - y_cb(-1:n - 1)
598 y_cc(0:n) = y_cb(-1:n - 1) + dy(0:n)/2._wp
599 end if
600
601 if (p > 0) then
602 file_path = trim(t_step_dir) // '/z_cb.dat'
603
604 inquire (file=trim(file_path), exist=file_exist)
605
606 if (file_exist) then
607 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
608 read (2) z_cb(-1:p); close (2)
609 else
610 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
611 end if
612
613 dz(0:p) = z_cb(0:p) - z_cb(-1:p - 1)
614 z_cc(0:p) = z_cb(-1:p - 1) + dz(0:p)/2._wp
615 end if
616
617 do i = 1, sys_size
618 write (file_path, '(A,I0,A)') trim(t_step_dir) // '/q_cons_vf', i, '.dat'
619 inquire (file=trim(file_path), exist=file_exist)
620 if (file_exist) then
621 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
622 read (2) q_cons_vf(i)%sf(0:m,0:n,0:p); close (2)
623 else
624 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
625 end if
626 end do
627
628 if (bubbles_euler .or. hypoelasticity) then
629 ! Read pb and mv for non-polytropic qbmm
630 if (qbmm .and. .not. polytropic) then
631 do i = 1, nb
632 do r = 1, nnode
633 write (file_path, '(A,I0,A)') trim(t_step_dir) // '/pb', sys_size + (i - 1)*nnode + r, '.dat'
634 inquire (file=trim(file_path), exist=file_exist)
635 if (file_exist) then
636 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
637 read (2) pb_ts(1)%sf(0:m,0:n,0:p,r, i); close (2)
638 else
639 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
640 end if
641 end do
642 end do
643 do i = 1, nb
644 do r = 1, nnode
645 write (file_path, '(A,I0,A)') trim(t_step_dir) // '/mv', sys_size + (i - 1)*nnode + r, '.dat'
646 inquire (file=trim(file_path), exist=file_exist)
647 if (file_exist) then
648 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
649 read (2) mv_ts(1)%sf(0:m,0:n,0:p,r, i); close (2)
650 else
651 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
652 end if
653 end do
654 end do
655 end if
656 end if
657
658 end subroutine s_read_serial_data_files
659
660 !> Read parallel initial condition and grid data files via MPI I/O
661 impure subroutine s_read_parallel_data_files(q_cons_vf)
662
663 type(scalar_field), dimension(sys_size), intent(inout) :: q_cons_vf
664
665#ifdef MFC_MPI
666 real(wp), allocatable, dimension(:) :: x_cb_glb, y_cb_glb, z_cb_glb
667 integer :: ifile, ierr, data_size
668 integer, dimension(MPI_STATUS_SIZE) :: status
669 integer(KIND=MPI_OFFSET_KIND) :: disp
670 integer(KIND=MPI_OFFSET_KIND) :: m_mok, n_mok, p_mok
671 integer(KIND=MPI_OFFSET_KIND) :: wp_mok, var_mok
672 integer(KIND=MPI_OFFSET_KIND) :: mok
673 character(LEN=path_len + 2*name_len) :: file_loc
674 logical :: file_exist
675 character(len=10) :: t_step_start_string
676 integer :: i, j
677 integer(KIND=MPI_OFFSET_KIND) :: nvars_mok !< variables read per cell
678
679 ! Downsampled data variables
680 integer :: m_ds, n_ds, p_ds
681 integer :: m_glb_ds, n_glb_ds, p_glb_ds
682 integer :: m_glb_read, n_glb_read, p_glb_read !< data size of read
683
684 allocate (x_cb_glb(-1:m_glb))
685 allocate (y_cb_glb(-1:n_glb))
686 allocate (z_cb_glb(-1:p_glb))
687
688 file_loc = trim(case_dir) // '/restart_data' // trim(mpiiofs) // 'x_cb.dat'
689 inquire (file=trim(file_loc), exist=file_exist)
690
691 if (down_sample) then
692 m_ds = int((m + 1)/3) - 1
693 n_ds = int((n + 1)/3) - 1
694 p_ds = int((p + 1)/3) - 1
695
696 m_glb_ds = int((m_glb + 1)/3) - 1
697 n_glb_ds = int((n_glb + 1)/3) - 1
698 p_glb_ds = int((p_glb + 1)/3) - 1
699 end if
700
701 if (file_exist) then
702 data_size = m_glb + 2
703 call mpi_file_open(mpi_comm_world, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
704 call s_check_mpi_file_open(ierr, file_loc)
705 call mpi_file_read(ifile, x_cb_glb, data_size, mpi_p, status, ierr)
706 call mpi_file_close(ifile, ierr)
707 else
708 call s_mpi_abort('File ' // trim(file_loc) // ' is missing. Exiting.')
709 end if
710
711 call s_apply_grid_from_global_dim(x_cb_glb, m_glb, m, start_idx(1), bc_x%beg, bc_x%end, buff_size, buff_size, buff_size, &
712 & buff_size, x_cb, x_cc, dx)
713
714 if (n > 0) then
715 file_loc = trim(case_dir) // '/restart_data' // trim(mpiiofs) // 'y_cb.dat'
716 inquire (file=trim(file_loc), exist=file_exist)
717
718 if (file_exist) then
719 data_size = n_glb + 2
720 call mpi_file_open(mpi_comm_world, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
721 call s_check_mpi_file_open(ierr, file_loc)
722 call mpi_file_read(ifile, y_cb_glb, data_size, mpi_p, status, ierr)
723 call mpi_file_close(ifile, ierr)
724 else
725 call s_mpi_abort('File ' // trim(file_loc) // ' is missing. Exiting.')
726 end if
727
728 call s_apply_grid_from_global_dim(y_cb_glb, n_glb, n, start_idx(2), bc_y%beg, bc_y%end, buff_size, buff_size, &
729 & buff_size, buff_size, y_cb, y_cc, dy)
730
731 if (p > 0) then
732 file_loc = trim(case_dir) // '/restart_data' // trim(mpiiofs) // 'z_cb.dat'
733 inquire (file=trim(file_loc), exist=file_exist)
734
735 if (file_exist) then
736 data_size = p_glb + 2
737 call mpi_file_open(mpi_comm_world, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
738 call s_check_mpi_file_open(ierr, file_loc)
739 call mpi_file_read(ifile, z_cb_glb, data_size, mpi_p, status, ierr)
740 call mpi_file_close(ifile, ierr)
741 else
742 call s_mpi_abort('File ' // trim(file_loc) // 'is missing. Exiting.')
743 end if
744
745 call s_apply_grid_from_global_dim(z_cb_glb, p_glb, p, start_idx(3), bc_z%beg, bc_z%end, buff_size, buff_size, &
746 & buff_size, buff_size, z_cb, z_cc, dz)
747 end if
748 end if
749
750 nvars_mok = int(sys_size, mpi_offset_kind)
751 if ((bubbles_euler .or. hypoelasticity) .and. qbmm .and. .not. polytropic) nvars_mok = nvars_mok + 2*nb*nnode
752
753 if (file_per_process) then
754 if (cfl_dt) then
755 call s_int_to_str(n_start, t_step_start_string)
756 write (file_loc, '(I0,A1,I7.7,A)') n_start, '_', proc_rank, '.dat'
757 else
758 call s_int_to_str(t_step_start, t_step_start_string)
759 write (file_loc, '(I0,A1,I7.7,A)') t_step_start, '_', proc_rank, '.dat'
760 end if
761 file_loc = trim(case_dir) // '/restart_data/lustre_' // trim(t_step_start_string) // trim(mpiiofs) // trim(file_loc)
762 inquire (file=trim(file_loc), exist=file_exist)
763
764 if (file_exist) then
765 call mpi_file_open(mpi_comm_self, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
766 call s_check_mpi_file_open(ierr, file_loc)
767
768 if (down_sample) then
769 call s_initialize_mpi_data_ds(m_ds, n_ds, p_ds)
770 else
771 if (ib) then
772 call s_initialize_mpi_data(q_cons_vf, ib_markers=ib_markers, ib_mpi_data=mpi_io_ib_data, &
773 & qbmm_pb=pb_ts(1), qbmm_mv=mv_ts(1))
774 else
775 call s_initialize_mpi_data(q_cons_vf, qbmm_pb=pb_ts(1), qbmm_mv=mv_ts(1))
776 end if
777 end if
778
779 if (down_sample) then
780 data_size = (m_ds + 3)*(n_ds + 3)*(p_ds + 3)
781 m_glb_read = m_glb_ds + 1
782 n_glb_read = n_glb_ds + 1
783 p_glb_read = p_glb_ds + 1
784 else
785 data_size = (m + 1)*(n + 1)*(p + 1)
786 m_glb_read = m_glb + 1
787 n_glb_read = n_glb + 1
788 p_glb_read = p_glb + 1
789 end if
790 call s_check_restart_file_size(ifile, file_loc, int(data_size, mpi_offset_kind)*int(storage_size(0._stp)/8, &
791 & mpi_offset_kind)*nvars_mok)
792
793 m_mok = int(m_glb_read + 1, mpi_offset_kind)
794 n_mok = int(m_glb_read + 1, mpi_offset_kind)
795 p_mok = int(m_glb_read + 1, mpi_offset_kind)
796 wp_mok = int(storage_size(0._stp)/8, mpi_offset_kind)
797 mok = int(1._wp, mpi_offset_kind)
798
799 if (bubbles_euler .or. hypoelasticity) then
800 do i = 1, sys_size
801 var_mok = int(i, mpi_offset_kind)
802
803 call mpi_file_read(ifile, mpi_io_data%var(i)%sf(:,:,:), data_size*mpi_io_type, mpi_io_p, status, ierr)
804 end do
805 ! Read pb and mv for non-polytropic qbmm
806 if (qbmm .and. .not. polytropic) then
807 do i = sys_size + 1, sys_size + 2*nb*nnode
808 var_mok = int(i, mpi_offset_kind)
809
810 call mpi_file_read(ifile, mpi_io_data%var(i)%sf(:,:,:), data_size*mpi_io_type, mpi_io_p, status, ierr)
811 end do
812 end if
813 else
814 if (down_sample) then
815 do i = 1, sys_size
816 var_mok = int(i, mpi_offset_kind)
817
818 call mpi_file_read(ifile, q_cons_temp(i)%sf, data_size*mpi_io_type, mpi_io_p, status, ierr)
819 end do
820 else
821 do i = 1, sys_size
822 var_mok = int(i, mpi_offset_kind)
823
824 call mpi_file_read(ifile, mpi_io_data%var(i)%sf(:,:,:), data_size*mpi_io_type, mpi_io_p, status, ierr)
825 end do
826 end if
827 end if
828
829 call s_mpi_barrier()
830
831 call mpi_file_close(ifile, ierr)
832 else
833 call s_mpi_abort('File ' // trim(file_loc) // ' is missing. Exiting.')
834 end if
835 else
836 if (cfl_dt) then
837 write (file_loc, '(I0,A)') n_start, '.dat'
838 else
839 write (file_loc, '(I0,A)') t_step_start, '.dat'
840 end if
841 file_loc = trim(case_dir) // '/restart_data' // trim(mpiiofs) // trim(file_loc)
842 inquire (file=trim(file_loc), exist=file_exist)
843
844 if (file_exist) then
845 call mpi_file_open(mpi_comm_world, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
846 call s_check_mpi_file_open(ierr, file_loc)
847
848 if (ib) then
849 call s_initialize_mpi_data(q_cons_vf, ib_markers=ib_markers, ib_mpi_data=mpi_io_ib_data, qbmm_pb=pb_ts(1), &
850 & qbmm_mv=mv_ts(1))
851 else
852 call s_initialize_mpi_data(q_cons_vf, qbmm_pb=pb_ts(1), qbmm_mv=mv_ts(1))
853 end if
854
855 data_size = (m + 1)*(n + 1)*(p + 1)
856
857 m_mok = int(m_glb + 1, mpi_offset_kind)
858 n_mok = int(n_glb + 1, mpi_offset_kind)
859 p_mok = int(p_glb + 1, mpi_offset_kind)
860 wp_mok = int(storage_size(0._stp)/8, mpi_offset_kind)
861 mok = int(1._wp, mpi_offset_kind)
862 call s_check_restart_file_size(ifile, file_loc, m_mok*max(mok, n_mok)*max(mok, p_mok)*wp_mok*nvars_mok)
863
864 if (bubbles_euler .or. hypoelasticity) then
865 do i = 1, sys_size
866 var_mok = int(i, mpi_offset_kind)
867 disp = m_mok*max(mok, n_mok)*max(mok, p_mok)*wp_mok*(var_mok - 1)
868
869 call mpi_file_set_view(ifile, disp, mpi_io_p, mpi_io_data%view(i), 'native', mpi_info_int, ierr)
870 call mpi_file_read(ifile, mpi_io_data%var(i)%sf(:,:,:), data_size*mpi_io_type, mpi_io_p, status, ierr)
871 end do
872 ! Read pb and mv for non-polytropic qbmm
873 if (qbmm .and. .not. polytropic) then
874 do i = sys_size + 1, sys_size + 2*nb*nnode
875 var_mok = int(i, mpi_offset_kind)
876 disp = m_mok*max(mok, n_mok)*max(mok, p_mok)*wp_mok*(var_mok - 1)
877
878 call mpi_file_set_view(ifile, disp, mpi_io_p, mpi_io_data%view(i), 'native', mpi_info_int, ierr)
879 call mpi_file_read(ifile, mpi_io_data%var(i)%sf(:,:,:), data_size*mpi_io_type, mpi_io_p, status, ierr)
880 end do
881 end if
882 else
883 do i = 1, sys_size
884 var_mok = int(i, mpi_offset_kind)
885
886 disp = m_mok*max(mok, n_mok)*max(mok, p_mok)*wp_mok*(var_mok - 1)
887
888 call mpi_file_set_view(ifile, disp, mpi_io_p, mpi_io_data%view(i), 'native', mpi_info_int, ierr)
889 call mpi_file_read_all(ifile, mpi_io_data%var(i)%sf(:,:,:), data_size*mpi_io_type, mpi_io_p, status, ierr)
890 end do
891 end if
892
893 call s_mpi_barrier()
894
895 call mpi_file_close(ifile, ierr)
896 else
897 call s_mpi_abort('File ' // trim(file_loc) // ' is missing. Exiting.')
898 end if
899 end if
900
901 deallocate (x_cb_glb, y_cb_glb, z_cb_glb)
902
903 if (bc_io) then
904 call s_read_parallel_boundary_condition_files(bc_type)
905 else
906 call s_assign_default_bc_type(bc_type)
907 end if
908#endif
909
910 end subroutine s_read_parallel_data_files
911
912#ifdef MFC_MPI
913 !> Abort if an open restart file holds fewer bytes than are about to be read from it, as a job killed while writing it leaves.
914 !! MPI reads past the end return short without an error, so the missing tail would come back as garbage.
915 impure subroutine s_check_restart_file_size(ifile, file_loc, expected_bytes)
916
917 integer, intent(in) :: ifile
918 character(len=*), intent(in) :: file_loc
919 integer(KIND=MPI_OFFSET_KIND), intent(in) :: expected_bytes
920 integer(KIND=MPI_OFFSET_KIND) :: file_bytes
921 integer :: ierr
922 character(len=64) :: sizes
923
924 call mpi_file_get_size(ifile, file_bytes, ierr)
925 if (ierr /= mpi_success) call s_mpi_abort('MPI_FILE_GET_SIZE failed on ' // trim(file_loc) // '. Exiting.')
926 if (file_bytes < expected_bytes) then
927 write (sizes, '(I0," bytes but ",I0)') file_bytes, expected_bytes
928 call s_mpi_abort('Restart file ' // trim(file_loc) // ' holds ' // trim(sizes) // ' are expected. It is ' &
929 & // 'truncated, e.g. by a job killed while writing it; restart from an earlier step.')
930 end if
931
932 end subroutine s_check_restart_file_size
933#endif
934
935 !> Initialize internal-energy equations from phase mass, mixture momentum, and total energy
936 subroutine s_initialize_internal_energy_equations(v_vf)
937
938 type(scalar_field), dimension(sys_size), intent(inout) :: v_vf
939 real(wp) :: rho
940 real(wp) :: dyn_pres
941 real(wp) :: gamma
942 real(wp) :: pi_inf
943 real(wp) :: qv
944 real(wp), dimension(2) :: re
945 real(wp) :: pres, t
946 real(wp) :: alpha_i, alpha_rho_i, e_i
947 integer :: i, j, k, l, c
948 real(wp), dimension(num_species) :: rhoyks
949 real(wp) :: pres_mag
950
951 pres_mag = 0._wp
952
953 t = dflt_t_guess
954
955 do j = 0, m
956 do k = 0, n
957 do l = 0, p
958 call s_convert_to_mixture_variables(v_vf, j, k, l, rho, gamma, pi_inf, qv, re)
959
960 dyn_pres = 0._wp
961 do i = eqn_idx%mom%beg, eqn_idx%mom%end
962 dyn_pres = dyn_pres + 5.e-1_wp*v_vf(i)%sf(j, k, l)*v_vf(i)%sf(j, k, l)/max(rho, sgm_eps)
963 end do
964
965 if (chemistry) then
966 do c = 1, num_species
967 rhoyks(c) = v_vf(eqn_idx%species%beg + c - 1)%sf(j, k, l)
968 end do
969 end if
970
971 if (mhd) then
972 if (n == 0) then
973 pres_mag = 0.5_wp*(bx0**2 + v_vf(eqn_idx%B%beg)%sf(j, k, l)**2 + v_vf(eqn_idx%B%beg + 1)%sf(j, k, l)**2)
974 else
975 pres_mag = 0.5_wp*(v_vf(eqn_idx%B%beg)%sf(j, k, l)**2 + v_vf(eqn_idx%B%beg + 1)%sf(j, k, &
976 & l)**2 + v_vf(eqn_idx%B%beg + 2)%sf(j, k, l)**2)
977 end if
978 end if
979
980 call s_compute_pressure(v_vf(eqn_idx%E)%sf(j, k, l), 0._stp, dyn_pres, pi_inf, gamma, rho, qv, rhoyks, pres, &
981 & t, pres_mag=pres_mag)
982
983 do i = 1, num_fluids
984 alpha_i = v_vf(i + eqn_idx%adv%beg - 1)%sf(j, k, l)
985 alpha_rho_i = v_vf(i + eqn_idx%cont%beg - 1)%sf(j, k, l)
986 call s_phase_internal_energy(pres, alpha_i, alpha_rho_i, i, e_i)
987 v_vf(i + eqn_idx%int_en%beg - 1)%sf(j, k, l) = e_i
988 end do
989 end do
990 end do
991 end do
992
993 end subroutine s_initialize_internal_energy_equations
994
995 !> Advance the simulation by one time step, handling CFL-based dt and time-stepper dispatch
996 impure subroutine s_perform_time_step(t_step, time_avg)
997
998 integer, intent(inout) :: t_step
999 real(wp), intent(inout) :: time_avg
1000 integer :: i, eta_hh, eta_mm, eta_ss
1001 real(wp) :: eta_sec
1002 real(wp) :: dt_floor
1003 character(len=8) :: lim_str !< Time-step limiter tag, e.g. ' (ICFL)'
1004
1005 if (cfl_dt) then
1006 if (cfl_const_dt .and. t_step == 0) call s_compute_dt()
1007
1008 if (cfl_adap_dt) call s_compute_dt()
1009
1010 if (t_step == 0) dt_init = dt
1011
1012 ! the collision restriction deliberately drops dt to collision_time/collision_temporal_resolution, so lower the
1013 ! runaway-dt abort threshold below that cap when it is enabled
1014 dt_floor = 1.e-3_wp*dt_init
1015 if (collision_model > 0 .and. collision_temporal_resolution > 0) then
1016 dt_floor = min(dt_floor, 1.e-3_wp*collision_time/real(collision_temporal_resolution, wp))
1017 end if
1018
1019 if (dt < dt_floor .and. cfl_adap_dt .and. proc_rank == 0) then
1020 print *, "Delta t = ", dt, " limited by ", dt_limiter
1021 call s_mpi_abort("Delta t has become too small")
1022 end if
1023 end if
1024
1025 ! Land exactly on t_stop. A fixed dt already lands on t_step_stop by step count; trimming it to t_step_stop*dt - mytime
1026 ! would only change the run's last dt by round-off, which a run continuing past that step does not see
1027 if (cfl_dt) then
1028 if ((mytime + dt) >= t_stop) then
1029 dt = t_stop - mytime
1030
1031# 639 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1032#if defined(MFC_OpenACC)
1033# 639 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1034!$acc update device(dt)
1035# 639 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1036#elif defined(MFC_OpenMP)
1037# 639 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1038!$omp target update to(dt)
1039# 639 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1040#endif
1041# 640 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1042 end if
1043 end if
1044
1045 if (cfl_dt) then
1046 if (proc_rank == 0 .and. mod(t_step - t_step_start, t_step_print) == 0) then
1047 eta_sec = wall_time_avg*(t_stop - mytime)/max(dt, tiny(dt))
1048 eta_hh = int(eta_sec)/3600
1049 eta_mm = mod(int(eta_sec), 3600)/60
1050 eta_ss = mod(int(eta_sec), 60)
1051 lim_str = ''
1052 if (cfl_adap_dt) lim_str = ' (' // dt_limiter // ')'
1053 print '(" [", I3, "%] t = ", ES11.4, " dt = ", ES11.4, A, " @ step ", I0, " t/step ", ES9.2, "s (avg ", ES9.2, "s) ETA ", I0, ":", I2.2, ":", I2.2)', &
1054 & int(ceiling(100._wp*(mytime/t_stop))), mytime, dt, trim(lim_str), t_step, wall_time, wall_time_avg, eta_hh, &
1055 & eta_mm, eta_ss
1056 end if
1057 else
1058 if (proc_rank == 0 .and. mod(t_step - t_step_start, t_step_print) == 0) then
1059 eta_sec = wall_time_avg*real(t_step_stop - t_step, wp)
1060 eta_hh = int(eta_sec)/3600
1061 eta_mm = mod(int(eta_sec), 3600)/60
1062 eta_ss = mod(int(eta_sec), 60)
1063 print '(" [", I3, "%] step ", I0, " of ", I0, " (t_step ", I0, ") t/step ", ES9.2, "s (avg ", ES9.2, "s) ETA ", I0, ":", I2.2, ":", I2.2)', &
1064 & int(ceiling(100._wp*(real(t_step - t_step_start)/(t_step_stop - t_step_start + 1)))), &
1065 & t_step - t_step_start + 1, t_step_stop - t_step_start + 1, t_step, wall_time, wall_time_avg, eta_hh, &
1066 & eta_mm, eta_ss
1067 end if
1068 end if
1069
1070 if (probe_wrt) then
1071 do i = 1, sys_size
1072
1073# 670 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1074#if defined(MFC_OpenACC)
1075# 670 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1076!$acc update host(q_cons_ts(1)%vf(i)%sf)
1077# 670 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1078#elif defined(MFC_OpenMP)
1079# 670 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1080!$omp target update from(q_cons_ts(1)%vf(i)%sf)
1081# 670 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1082#endif
1083# 671 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1084 end do
1085 if (bubbles_euler) then
1086
1087# 673 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1088#if defined(MFC_OpenACC)
1089# 673 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1090!$acc update host(ptil)
1091# 673 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1092#elif defined(MFC_OpenMP)
1093# 673 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1094!$omp target update from(ptil)
1095# 673 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1096#endif
1097# 674 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1098 end if
1099 end if
1100
1101 ! Total-variation-diminishing (TVD) Runge-Kutta (RK) time-steppers
1102 if (any(time_stepper == (/time_stepper_rk1, time_stepper_rk2, time_stepper_rk3/))) then
1103 call s_tvd_rk(t_step, time_avg, time_stepper)
1104 end if
1105
1106 ! Advance time after RK so source terms see current-step time. With a fixed dt, use the same t_step*dt a restart
1107 ! starts from (p_main): a running sum drifts from it (1290 ulps by step 27000), so a restarted run would see the
1108 ! prescribed IB kinematics, inflow ramps and forcing at slightly different times than the run it continues
1109 if (cfl_dt) then
1110 mytime = mytime + dt
1111 else
1112 mytime = (t_step + 1)*dt
1113 end if
1114
1115 if (relax) call s_infinite_relaxation_k(q_cons_ts(1)%vf)
1116
1117 ! Time-stepping loop controls
1118 t_step = t_step + 1
1119
1120 end subroutine s_perform_time_step
1121
1122 !> Collect per-process wall-clock times and write aggregate performance metrics to file
1123 impure subroutine s_save_performance_metrics(time_avg, time_final, io_time_avg, io_time_final, proc_time, io_proc_time, &
1124 & file_exists)
1125
1126 real(wp), intent(inout) :: time_avg, time_final
1127 real(wp), intent(inout) :: io_time_avg, io_time_final
1128 real(wp), dimension(:), intent(inout) :: proc_time
1129 real(wp), dimension(:), intent(inout) :: io_proc_time
1130 logical, intent(inout) :: file_exists
1131 real(wp) :: grind_time
1132
1133 call s_mpi_barrier()
1134
1135 if (num_procs > 1) then
1136 call mpi_bcast_time_step_values(proc_time, time_avg)
1137
1138 call mpi_bcast_time_step_values(io_proc_time, io_time_avg)
1139 end if
1140
1141 if (proc_rank == 0) then
1142 time_final = 0._wp
1143 io_time_final = 0._wp
1144 if (num_procs == 1) then
1145 time_final = time_avg
1146 io_time_final = io_time_avg
1147 else
1148 time_final = maxval(proc_time)
1149 io_time_final = maxval(io_proc_time)
1150 end if
1151
1152 grind_time = time_final*1.0e9_wp/(real(sys_size, wp)*real(maxval((/1, m_glb/)), wp)*real(maxval((/1, n_glb/)), &
1153 & wp)*real(maxval((/1, p_glb/)), wp))
1154
1155 print *, "Performance:", grind_time, "ns/gp/eq/rhs"
1156 inquire (file='time_data.dat', exist=file_exists)
1157 if (file_exists) then
1158 open (1, file='time_data.dat', position='append', status='old')
1159 else
1160 open (1, file='time_data.dat', status='new')
1161 ! time_final is the fastest single RK stage (one RHS evaluation), not a whole step; see s_tvd_rk
1162 write (1, '(A10, A15, A15)') "Ranks", "s/rhs", "ns/gp/eq/rhs"
1163 end if
1164
1165 write (1, '(I10, 2(F15.8))') num_procs, time_final, grind_time
1166
1167 close (1)
1168
1169 inquire (file='io_time_data.dat', exist=file_exists)
1170 if (file_exists) then
1171 open (1, file='io_time_data.dat', position='append', status='old')
1172 else
1173 open (1, file='io_time_data.dat', status='new')
1174 ! io_time_final is the mean time of one s_save_data call
1175 write (1, '(A10, A15)') "Ranks", "s/save"
1176 end if
1177
1178 write (1, '(I10, F15.8)') num_procs, io_time_final
1179 close (1)
1180 end if
1181
1182 end subroutine s_save_performance_metrics
1183
1184 !> Save conservative variable data to disk at the current time step
1185 impure subroutine s_save_data(t_step, start, finish, io_time_avg, nt)
1186
1187 integer, intent(inout) :: t_step
1188 real(wp), intent(inout) :: start, finish, io_time_avg
1189 integer, intent(inout) :: nt
1190 integer(kind=8) :: i, j, k, l
1191 integer :: stor
1192 integer :: save_count
1193
1194 if (down_sample) then
1195 call s_populate_variables_buffers(bc_type, q_cons_ts(1)%vf, pb_ts(1)%sf, mv_ts(1)%sf, q_t_sf)
1196 end if
1197
1198 stor = 1
1199
1200 if (time_stepper /= time_stepper_rk1) then
1201
1202# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1203
1204# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1205#if defined(MFC_OpenACC)
1206# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1207!$acc parallel loop collapse(4) gang vector default(present) copyin(idwbuff)
1208# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1209#elif defined(MFC_OpenMP)
1210# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1211
1212# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1213
1214# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1215
1216# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1217!$omp target teams loop defaultmap(firstprivate:scalar) bind(teams,parallel) collapse(4) defaultmap(tofrom:aggregate) defaultmap(tofrom:allocatable) defaultmap(tofrom:pointer) map(to:idwbuff)
1218# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1219#endif
1220# 778 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1221 do i = 1, sys_size
1222 do l = idwbuff(3)%beg, idwbuff(3)%end
1223 do k = idwbuff(2)%beg, idwbuff(2)%end
1224 do j = idwbuff(1)%beg, idwbuff(1)%end
1225 q_cons_ts(2)%vf(i)%sf(j, k, l) = q_cons_ts(1)%vf(i)%sf(j, k, l)
1226 end do
1227 end do
1228 end do
1229 end do
1230
1231# 787 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1232#if defined(MFC_OpenACC)
1233# 787 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1234!$acc end parallel loop
1235# 787 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1236#elif defined(MFC_OpenMP)
1237# 787 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1238
1239# 787 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1240!$omp end target teams loop
1241# 787 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1242#endif
1243# 788 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1244 stor = 2
1245 end if
1246
1247 call cpu_time(start)
1248 call nvtxstartrange("SAVE-DATA")
1249 do i = 1, sys_size
1250#ifndef FRONTIER_UNIFIED
1251
1252# 795 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1253#if defined(MFC_OpenACC)
1254# 795 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1255!$acc update host(q_cons_ts(stor)%vf(i)%sf)
1256# 795 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1257#elif defined(MFC_OpenMP)
1258# 795 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1259!$omp target update from(q_cons_ts(stor)%vf(i)%sf)
1260# 795 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1261#endif
1262# 796 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1263#endif
1264 do l = 0, p
1265 do k = 0, n
1266 do j = 0, m
1267 if (ieee_is_nan(real(q_cons_ts(stor)%vf(i)%sf(j, k, l), kind=wp))) then
1268 print *, "NaN(s) in timestep output.", j, k, l, i, proc_rank, t_step, m, n, p
1269 call s_mpi_abort("NaN(s) in timestep output.")
1270 end if
1271 end do
1272 end do
1273 end do
1274 end do
1275
1276 if (qbmm .and. .not. polytropic) then
1277
1278# 810 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1279#if defined(MFC_OpenACC)
1280# 810 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1281!$acc update host(pb_ts(1)%sf)
1282# 810 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1283#elif defined(MFC_OpenMP)
1284# 810 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1285!$omp target update from(pb_ts(1)%sf)
1286# 810 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1287#endif
1288# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1289
1290# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1291#if defined(MFC_OpenACC)
1292# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1293!$acc update host(mv_ts(1)%sf)
1294# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1295#elif defined(MFC_OpenMP)
1296# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1297!$omp target update from(mv_ts(1)%sf)
1298# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1299#endif
1300# 812 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1301 end if
1302
1303 if (cfl_dt) then
1304 save_count = int(mytime/t_save)
1305 else
1306 save_count = t_step
1307 end if
1308
1309 if (bubbles_lagrange) then
1310
1311# 821 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1312#if defined(MFC_OpenACC)
1313# 821 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1314!$acc update host(lag_id, mtn_pos, mtn_posPrev, mtn_vel, intfc_rad, intfc_vel, bub_R0, Rmax_stats, Rmin_stats, bub_dphidt, gas_p, gas_mv, gas_mg, gas_betaT, gas_betaC)
1315# 821 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1316#elif defined(MFC_OpenMP)
1317# 821 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1318!$omp target update from(lag_id, mtn_pos, mtn_posPrev, mtn_vel, intfc_rad, intfc_vel, bub_R0, Rmax_stats, Rmin_stats, bub_dphidt, gas_p, gas_mv, gas_mg, gas_betaT, gas_betaC)
1319# 821 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1320#endif
1321# 823 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1322 do i = 1, n_el_bubs_loc
1323 if (ieee_is_nan(intfc_rad(i, 1)) .or. intfc_rad(i, 1) <= 0._wp) then
1324 call s_mpi_abort("Bubble radius is negative or NaN, please reduce dt.")
1325 end if
1326 end do
1327
1328
1329# 829 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1330#if defined(MFC_OpenACC)
1331# 829 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1332!$acc update host(q_beta(1)%sf)
1333# 829 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1334#elif defined(MFC_OpenMP)
1335# 829 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1336!$omp target update from(q_beta(1)%sf)
1337# 829 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1338#endif
1339# 830 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1340 call s_write_data_files(q_cons_ts(stor)%vf, q_t_sf, q_prim_vf, save_count, bc_type, q_beta(1))
1341
1342# 831 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1343#if defined(MFC_OpenACC)
1344# 831 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1345!$acc update host(Rmax_stats, Rmin_stats, gas_p, gas_mv, intfc_vel)
1346# 831 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1347#elif defined(MFC_OpenMP)
1348# 831 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1349!$omp target update from(Rmax_stats, Rmin_stats, gas_p, gas_mv, intfc_vel)
1350# 831 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1351#endif
1352# 832 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1353 call s_write_restart_lag_bubbles(save_count) ! parallel
1354 if (lag_params%write_bubbles_stats) call s_write_lag_bubble_stats()
1355 else
1356 call s_write_data_files(q_cons_ts(stor)%vf, q_t_sf, q_prim_vf, save_count, bc_type)
1357 end if
1358
1359 ! Write IB kinematic state for restart
1360 if (ib) call s_write_ib_state_file(save_count)
1361
1362 call nvtxendrange
1363 call cpu_time(finish)
1364 if (cfl_dt) then
1365 nt = mytime/t_save
1366 else
1367 nt = int((t_step - t_step_start)/(t_step_save))
1368 end if
1369
1370 if (nt == 1) then
1371 io_time_avg = abs(finish - start)
1372 else
1373 io_time_avg = (abs(finish - start) + io_time_avg*(nt - 1))/nt
1374 end if
1375
1376 end subroutine s_save_data
1377
1378 !> Initialize all simulation sub-modules in the required dependency order
1379 impure subroutine s_initialize_modules
1380
1381 integer :: m_ds, n_ds, p_ds
1382 integer :: i
1383
1384 call s_initialize_global_parameters_module()
1385# 874 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1386 if (bubbles_euler .or. bubbles_lagrange) then
1387 call s_initialize_bubbles_model()
1388 end if
1389 call s_initialize_mpi_common_module(exchange_all_chemistry_temperatures_in=.false., use_rdma_transport_in=rdma_mpi)
1390 call s_initialize_mpi_proxy_module()
1391 call s_initialize_eos_module()
1392 call s_initialize_variables_conversion_module(enforce_density_floor=.true., preserve_qbmm_number=.true.)
1393 if (grid_geometry == 3) call s_initialize_fftw_module()
1394
1395 if (bubbles_euler) call s_initialize_bubbles_ee_module()
1396 if (ib) then
1397 call s_initialize_ibm_module()
1398 end if
1399 if (qbmm) call s_initialize_qbmm_module()
1400
1401 if (acoustic_source) then
1402 call s_initialize_acoustic_src()
1403 end if
1404
1405 if (viscous .and. (.not. igr)) then
1406 call s_initialize_viscous_module()
1407 end if
1408
1409 call s_initialize_rhs_module()
1410
1411 if (surface_tension) call s_initialize_surface_tension_module()
1412
1413 if (relax) call s_initialize_phasechange_module()
1414
1415 call s_initialize_data_output_module()
1416 call s_initialize_derived_variables_module()
1417 call s_initialize_time_steppers_module()
1418
1419 call s_initialize_boundary_common_module(use_dirichlet_buffers=.true.)
1420
1421 if (down_sample) then
1422 m_ds = int((m + 1)/3) - 1
1423 n_ds = int((n + 1)/3) - 1
1424 p_ds = int((p + 1)/3) - 1
1425
1426 allocate (q_cons_temp(1:sys_size))
1427 do i = 1, sys_size
1428 allocate (q_cons_temp(i)%sf(-1:m_ds + 1,-1:n_ds + 1,-1:p_ds + 1))
1429 end do
1430 end if
1431
1432 if (down_sample) then
1433 call s_read_data_files(q_cons_temp)
1434 call s_upsample_data(q_cons_ts(1)%vf, q_cons_temp)
1435 do i = 1, sys_size
1436
1437# 924 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1438#if defined(MFC_OpenACC)
1439# 924 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1440!$acc update device(q_cons_ts(1)%vf(i)%sf)
1441# 924 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1442#elif defined(MFC_OpenMP)
1443# 924 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1444!$omp target update to(q_cons_ts(1)%vf(i)%sf)
1445# 924 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1446#endif
1447# 925 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1448 end do
1449 do i = 1, sys_size
1450 deallocate (q_cons_temp(i)%sf)
1451 end do
1452 deallocate (q_cons_temp)
1453 else
1454 call s_read_data_files(q_cons_ts(1)%vf)
1455 end if
1456
1457 block
1458 type(int_bounds_info), dimension(3) :: grid_offsets
1459
1460 grid_offsets(:)%beg = buff_size
1461 grid_offsets(:)%end = buff_size
1462 if (n == 0) then
1463 call s_populate_grid_variables_buffers(x_cb, x_cc, dx, grid_offsets(1), grid_offsets(2), grid_offsets(3), &
1464 & global_bounds=glb_bounds)
1465 else if (p == 0) then
1466 call s_populate_grid_variables_buffers(x_cb, x_cc, dx, grid_offsets(1), grid_offsets(2), grid_offsets(3), y_cb, &
1467 & y_cc, dy, global_bounds=glb_bounds)
1468 else
1469 call s_populate_grid_variables_buffers(x_cb, x_cc, dx, grid_offsets(1), grid_offsets(2), grid_offsets(3), y_cb, &
1470 & y_cc, dy, z_cb, z_cc, dz, glb_bounds)
1471 end if
1472 end block
1473
1474# 950 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1475#if defined(MFC_OpenACC)
1476# 950 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1477!$acc update device(glb_bounds)
1478# 950 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1479#elif defined(MFC_OpenMP)
1480# 950 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1481!$omp target update to(glb_bounds)
1482# 950 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1483#endif
1484# 951 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1485 dx_min = minval(dx)
1486 if (n > 0) dy_min = minval(dy)
1487 if (p > 0) dz_min = minval(dz)
1488
1489 if (model_eqns == model_eqns_6eq) call s_initialize_internal_energy_equations(q_cons_ts(1)%vf)
1490 if (ib) then
1491 call s_instantiate_stl_models()
1492 call s_initialize_ib_airfoils()
1493 call s_get_neighbor_bounds()
1494 if (cfl_dt .and. n_start > 0) then
1495 call s_read_ib_restart_data(n_start)
1496 else if (t_step_start > 0) then
1497 call s_read_ib_restart_data(t_step_start)
1498 else
1499 call s_read_ib_restart_data(0)
1500 end if
1501 call s_build_ib_neighborhood()
1502 call s_ibm_setup()
1503 if (t_step_start == 0 .or. (cfl_dt .and. n_start == 0)) call s_write_ib_data_file(0)
1504 end if
1505 if (bodyforces .or. synthetic_turbulence) call s_initialize_body_forces_module()
1506 if (acoustic_source) call s_precalculate_acoustic_spatial_sources()
1507
1508 ! Initialize the Temperature cache.
1509 if (chemistry) call s_compute_q_t_sf(q_t_sf, q_cons_ts(1)%vf, idwint)
1510
1511 ! Computation of parameters, allocation of memory, association of pointers, and/or execution of any other tasks that are
1512 ! needed to properly configure the modules. The preparations below DO DEPEND on the grid being complete.
1513 if (igr) then
1514 call s_initialize_igr_module()
1515 end if
1516 if (.not. igr) then
1517 if (recon_type == recon_type_weno) then
1518 call s_initialize_weno_module()
1519 else if (recon_type == recon_type_muscl) then
1520 call s_initialize_muscl_module()
1521 end if
1522 call s_initialize_cbc_module()
1523 call s_initialize_riemann_solvers_module()
1524 end if
1525 if (int_comp > 0) call s_initialize_thinc_module()
1526 call s_initialize_derived_variables()
1527 if (bubbles_lagrange) call s_initialize_bubbles_el_module(q_cons_ts(1)%vf, bc_type)
1528
1529 if (hypoelasticity) call s_initialize_hypoelastic_module()
1530
1531 end subroutine s_initialize_modules
1532
1533 !> Set up the MPI execution environment, bind GPUs, and decompose the computational domain
1534 impure subroutine s_initialize_mpi_domain
1535
1536 integer :: ierr
1537
1538#ifdef MFC_GPU
1539 real(wp) :: starttime, endtime
1540 integer :: num_devices, local_size, num_nodes, ppn, my_device_num
1541 integer :: dev, devnum, local_rank
1542#ifdef MFC_MPI
1543 integer :: local_comm
1544#endif
1545#if defined(MFC_OpenACC)
1546 integer(acc_device_kind) :: devtype
1547#endif
1548#endif
1549
1550 call s_mpi_initialize()
1551
1552#ifdef MFC_GPU
1553#ifndef MFC_MPI
1554 local_size = 1
1555 local_rank = 0
1556#else
1557 call mpi_comm_split_type(mpi_comm_world, mpi_comm_type_shared, 0, mpi_info_null, local_comm, ierr)
1558 call mpi_comm_size(local_comm, local_size, ierr)
1559 call mpi_comm_rank(local_comm, local_rank, ierr)
1560#endif
1561#if defined(MFC_OpenACC)
1562 devtype = acc_get_device_type()
1563 devnum = acc_get_num_devices(devtype)
1564 dev = mod(local_rank, devnum)
1565
1566 call acc_set_device_num(dev, devtype)
1567#elif defined(MFC_OpenMP)
1568 devnum = omp_get_num_devices()
1569 dev = mod(local_rank, devnum)
1570 call omp_set_default_device(dev)
1571#endif
1572#endif
1573
1574 if (proc_rank == 0) then
1575 call s_assign_default_values_to_user_inputs()
1576 call s_read_input_file()
1577 call s_check_input_file()
1578
1579 print '(" Simulating a ", A, " ", I0, "x", I0, "x", I0, " case on ", I0, " rank(s) ", A, ".")', &
1580# 1047 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1581 "regular", &
1582# 1051 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1583 m, n, p, num_procs, &
1584#if defined(MFC_OpenACC)
1585 "with OpenACC offloading"
1586#elif defined(MFC_OpenMP)
1587 "with OpenMP offloading"
1588#else
1589 "on CPUs"
1590#endif
1591 end if
1592
1593 call s_mpi_bcast_user_inputs()
1594
1595 ! Save original BCs before decomposition overwrites them with MPI neighbor ranks
1596 ib_bc_x = bc_x
1597 ib_bc_y = bc_y
1598 ib_bc_z = bc_z
1599
1600 call s_initialize_parallel_io()
1601
1602 call s_mpi_decompose_computational_domain(write_silo_ghost_offsets=.false., adjust_local_domains=.false.)
1603
1604 bc = bc_xyz_info(bc_x, bc_y, bc_z)
1605
1606 end subroutine s_initialize_mpi_domain
1607
1608 !> Transfer initial conservative variable and model parameter data to the GPU device
1609 subroutine s_initialize_gpu_vars
1610
1611 integer :: i
1612
1613 if (.not. down_sample) then
1614 do i = 1, sys_size
1615
1616# 1083 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1617#if defined(MFC_OpenACC)
1618# 1083 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1619!$acc update device(q_cons_ts(1)%vf(i)%sf)
1620# 1083 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1621#elif defined(MFC_OpenMP)
1622# 1083 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1623!$omp target update to(q_cons_ts(1)%vf(i)%sf)
1624# 1083 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1625#endif
1626# 1084 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1627 end do
1628 end if
1629
1630 if (qbmm .and. .not. polytropic) then
1631
1632# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1633#if defined(MFC_OpenACC)
1634# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1635!$acc update device(pb_ts(1)%sf, mv_ts(1)%sf)
1636# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1637#elif defined(MFC_OpenMP)
1638# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1639!$omp target update to(pb_ts(1)%sf, mv_ts(1)%sf)
1640# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1641#endif
1642# 1089 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1643 end if
1644 if (chemistry) then
1645
1646# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1647#if defined(MFC_OpenACC)
1648# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1649!$acc update device(q_T_sf%sf)
1650# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1651#elif defined(MFC_OpenMP)
1652# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1653!$omp target update to(q_T_sf%sf)
1654# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1655#endif
1656# 1092 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1657 end if
1658
1659
1660# 1094 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1661#if defined(MFC_OpenACC)
1662# 1094 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1663!$acc update device(chem_params)
1664# 1094 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1665#elif defined(MFC_OpenMP)
1666# 1094 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1667!$omp target update to(chem_params)
1668# 1094 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1669#endif
1670# 1095 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1671
1672
1673# 1096 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1674#if defined(MFC_OpenACC)
1675# 1096 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1676!$acc update device(rburn)
1677# 1096 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1678#elif defined(MFC_OpenMP)
1679# 1096 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1680!$omp target update to(rburn)
1681# 1096 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1682#endif
1683# 1097 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1684
1685
1686# 1098 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1687#if defined(MFC_OpenACC)
1688# 1098 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1689!$acc update device(R0ref, p0ref, rho0ref, ss, pv, vd, mu_l, mu_v, mu_g, gam_v, gam_g, M_v, M_g, R_v, R_g, Tw, cp_v, cp_g, k_vl, k_gl, gam, gam_m, Eu, Ca, Web, Re_inv, Pe_c, phi_vg, phi_gv, omegaN, bubbles_euler, polytropic, polydisperse, qbmm, ptil, bubble_model, thermal, poly_sigma, adv_n, adap_dt, adap_dt_tol, adap_dt_max_iters, eqn_idx%n, pi_fac, low_Mach)
1690# 1098 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1691#elif defined(MFC_OpenMP)
1692# 1098 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1693!$omp target update to(R0ref, p0ref, rho0ref, ss, pv, vd, mu_l, mu_v, mu_g, gam_v, gam_g, M_v, M_g, R_v, R_g, Tw, cp_v, cp_g, k_vl, k_gl, gam, gam_m, Eu, Ca, Web, Re_inv, Pe_c, phi_vg, phi_gv, omegaN, bubbles_euler, polytropic, polydisperse, qbmm, ptil, bubble_model, thermal, poly_sigma, adv_n, adap_dt, adap_dt_tol, adap_dt_max_iters, eqn_idx%n, pi_fac, low_Mach)
1694# 1098 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1695#endif
1696# 1102 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1697
1698 if (bubbles_euler) then
1699
1700# 1104 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1701#if defined(MFC_OpenACC)
1702# 1104 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1703!$acc update device(weight, R0)
1704# 1104 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1705#elif defined(MFC_OpenMP)
1706# 1104 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1707!$omp target update to(weight, R0)
1708# 1104 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1709#endif
1710# 1105 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1711 if (.not. polytropic) then
1712
1713# 1106 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1714#if defined(MFC_OpenACC)
1715# 1106 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1716!$acc update device(pb0, Pe_T, k_g, k_v, mass_g0, mass_v0, Re_trans_T, Re_trans_c, Im_trans_T, Im_trans_c)
1717# 1106 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1718#elif defined(MFC_OpenMP)
1719# 1106 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1720!$omp target update to(pb0, Pe_T, k_g, k_v, mass_g0, mass_v0, Re_trans_T, Re_trans_c, Im_trans_T, Im_trans_c)
1721# 1106 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1722#endif
1723# 1107 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1724 else if (qbmm) then
1725
1726# 1108 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1727#if defined(MFC_OpenACC)
1728# 1108 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1729!$acc update device(pb0)
1730# 1108 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1731#elif defined(MFC_OpenMP)
1732# 1108 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1733!$omp target update to(pb0)
1734# 1108 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1735#endif
1736# 1109 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1737 end if
1738 end if
1739
1740
1741# 1112 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1742#if defined(MFC_OpenACC)
1743# 1112 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1744!$acc update device(adv_n, adap_dt, adap_dt_tol, adap_dt_max_iters, pi_fac, low_Mach)
1745# 1112 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1746#elif defined(MFC_OpenMP)
1747# 1112 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1748!$omp target update to(adv_n, adap_dt, adap_dt_tol, adap_dt_max_iters, pi_fac, low_Mach)
1749# 1112 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1750#endif
1751# 1113 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1752
1753
1754# 1114 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1755#if defined(MFC_OpenACC)
1756# 1114 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1757!$acc update device(acoustic_source, num_source)
1758# 1114 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1759#elif defined(MFC_OpenMP)
1760# 1114 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1761!$omp target update to(acoustic_source, num_source)
1762# 1114 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1763#endif
1764# 1115 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1765
1766# 1115 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1767#if defined(MFC_OpenACC)
1768# 1115 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1769!$acc update device(sigma, surface_tension)
1770# 1115 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1771#elif defined(MFC_OpenMP)
1772# 1115 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1773!$omp target update to(sigma, surface_tension)
1774# 1115 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1775#endif
1776# 1116 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1777
1778
1779# 1117 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1780#if defined(MFC_OpenACC)
1781# 1117 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1782!$acc update device(dx, dy, dz, x_cb, x_cc, y_cb, y_cc, z_cb, z_cc)
1783# 1117 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1784#elif defined(MFC_OpenMP)
1785# 1117 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1786!$omp target update to(dx, dy, dz, x_cb, x_cc, y_cb, y_cc, z_cb, z_cc)
1787# 1117 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1788#endif
1789# 1118 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1790
1791# 1118 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1792#if defined(MFC_OpenACC)
1793# 1118 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1794!$acc update device(bc_x%beg, bc_x%end, bc_y%beg, bc_y%end, bc_z%beg, bc_z%end)
1795# 1118 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1796#elif defined(MFC_OpenMP)
1797# 1118 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1798!$omp target update to(bc_x%beg, bc_x%end, bc_y%beg, bc_y%end, bc_z%beg, bc_z%end)
1799# 1118 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1800#endif
1801# 1119 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1802
1803# 1119 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1804#if defined(MFC_OpenACC)
1805# 1119 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1806!$acc update device(bc_x%vb1, bc_x%vb2, bc_x%vb3, bc_x%ve1, bc_x%ve2, bc_x%ve3)
1807# 1119 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1808#elif defined(MFC_OpenMP)
1809# 1119 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1810!$omp target update to(bc_x%vb1, bc_x%vb2, bc_x%vb3, bc_x%ve1, bc_x%ve2, bc_x%ve3)
1811# 1119 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1812#endif
1813# 1120 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1814
1815# 1120 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1816#if defined(MFC_OpenACC)
1817# 1120 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1818!$acc update device(bc_y%vb1, bc_y%vb2, bc_y%vb3, bc_y%ve1, bc_y%ve2, bc_y%ve3)
1819# 1120 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1820#elif defined(MFC_OpenMP)
1821# 1120 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1822!$omp target update to(bc_y%vb1, bc_y%vb2, bc_y%vb3, bc_y%ve1, bc_y%ve2, bc_y%ve3)
1823# 1120 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1824#endif
1825# 1121 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1826
1827# 1121 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1828#if defined(MFC_OpenACC)
1829# 1121 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1830!$acc update device(bc_z%vb1, bc_z%vb2, bc_z%vb3, bc_z%ve1, bc_z%ve2, bc_z%ve3)
1831# 1121 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1832#elif defined(MFC_OpenMP)
1833# 1121 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1834!$omp target update to(bc_z%vb1, bc_z%vb2, bc_z%vb3, bc_z%ve1, bc_z%ve2, bc_z%ve3)
1835# 1121 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1836#endif
1837# 1122 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1838
1839
1840# 1123 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1841#if defined(MFC_OpenACC)
1842# 1123 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1843!$acc update device(bc_x%grcbc_in, bc_x%grcbc_out, bc_x%grcbc_vel_out)
1844# 1123 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1845#elif defined(MFC_OpenMP)
1846# 1123 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1847!$omp target update to(bc_x%grcbc_in, bc_x%grcbc_out, bc_x%grcbc_vel_out)
1848# 1123 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1849#endif
1850# 1124 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1851
1852# 1124 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1853#if defined(MFC_OpenACC)
1854# 1124 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1855!$acc update device(bc_y%grcbc_in, bc_y%grcbc_out, bc_y%grcbc_vel_out)
1856# 1124 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1857#elif defined(MFC_OpenMP)
1858# 1124 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1859!$omp target update to(bc_y%grcbc_in, bc_y%grcbc_out, bc_y%grcbc_vel_out)
1860# 1124 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1861#endif
1862# 1125 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1863
1864# 1125 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1865#if defined(MFC_OpenACC)
1866# 1125 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1867!$acc update device(bc_z%grcbc_in, bc_z%grcbc_out, bc_z%grcbc_vel_out)
1868# 1125 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1869#elif defined(MFC_OpenMP)
1870# 1125 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1871!$omp target update to(bc_z%grcbc_in, bc_z%grcbc_out, bc_z%grcbc_vel_out)
1872# 1125 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1873#endif
1874# 1126 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1875
1876
1877# 1127 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1878#if defined(MFC_OpenACC)
1879# 1127 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1880!$acc update device(bc_x%vel_in_ramp, bc_x%vel_in_t0, bc_x%vel_in_frac0)
1881# 1127 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1882#elif defined(MFC_OpenMP)
1883# 1127 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1884!$omp target update to(bc_x%vel_in_ramp, bc_x%vel_in_t0, bc_x%vel_in_frac0)
1885# 1127 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1886#endif
1887# 1128 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1888
1889# 1128 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1890#if defined(MFC_OpenACC)
1891# 1128 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1892!$acc update device(bc_y%vel_in_ramp, bc_y%vel_in_t0, bc_y%vel_in_frac0)
1893# 1128 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1894#elif defined(MFC_OpenMP)
1895# 1128 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1896!$omp target update to(bc_y%vel_in_ramp, bc_y%vel_in_t0, bc_y%vel_in_frac0)
1897# 1128 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1898#endif
1899# 1129 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1900
1901# 1129 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1902#if defined(MFC_OpenACC)
1903# 1129 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1904!$acc update device(bc_z%vel_in_ramp, bc_z%vel_in_t0, bc_z%vel_in_frac0)
1905# 1129 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1906#elif defined(MFC_OpenMP)
1907# 1129 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1908!$omp target update to(bc_z%vel_in_ramp, bc_z%vel_in_t0, bc_z%vel_in_frac0)
1909# 1129 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1910#endif
1911# 1130 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1912
1913
1914# 1131 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1915#if defined(MFC_OpenACC)
1916# 1131 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1917!$acc update device(bc_x%isothermal_in, bc_x%isothermal_out)
1918# 1131 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1919#elif defined(MFC_OpenMP)
1920# 1131 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1921!$omp target update to(bc_x%isothermal_in, bc_x%isothermal_out)
1922# 1131 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1923#endif
1924# 1132 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1925
1926# 1132 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1927#if defined(MFC_OpenACC)
1928# 1132 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1929!$acc update device(bc_y%isothermal_in, bc_y%isothermal_out)
1930# 1132 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1931#elif defined(MFC_OpenMP)
1932# 1132 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1933!$omp target update to(bc_y%isothermal_in, bc_y%isothermal_out)
1934# 1132 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1935#endif
1936# 1133 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1937
1938# 1133 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1939#if defined(MFC_OpenACC)
1940# 1133 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1941!$acc update device(bc_z%isothermal_in, bc_z%isothermal_out)
1942# 1133 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1943#elif defined(MFC_OpenMP)
1944# 1133 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1945!$omp target update to(bc_z%isothermal_in, bc_z%isothermal_out)
1946# 1133 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1947#endif
1948# 1134 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1949
1950# 1134 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1951#if defined(MFC_OpenACC)
1952# 1134 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1953!$acc update device(bc_x%Twall_in, bc_x%Twall_out, bc_y%Twall_in, bc_y%Twall_out, bc_z%Twall_in, bc_z%Twall_out)
1954# 1134 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1955#elif defined(MFC_OpenMP)
1956# 1134 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1957!$omp target update to(bc_x%Twall_in, bc_x%Twall_out, bc_y%Twall_in, bc_y%Twall_out, bc_z%Twall_in, bc_z%Twall_out)
1958# 1134 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1959#endif
1960# 1135 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1961
1962
1963# 1136 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1964#if defined(MFC_OpenACC)
1965# 1136 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1966!$acc update device(bc)
1967# 1136 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1968#elif defined(MFC_OpenMP)
1969# 1136 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1970!$omp target update to(bc)
1971# 1136 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1972#endif
1973# 1137 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1974
1975
1976# 1138 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1977#if defined(MFC_OpenACC)
1978# 1138 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1979!$acc update device(relax, relax_model)
1980# 1138 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1981#elif defined(MFC_OpenMP)
1982# 1138 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1983!$omp target update to(relax, relax_model)
1984# 1138 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1985#endif
1986# 1139 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1987 if (relax) then
1988
1989# 1140 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1990#if defined(MFC_OpenACC)
1991# 1140 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1992!$acc update device(palpha_eps, ptgalpha_eps)
1993# 1140 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1994#elif defined(MFC_OpenMP)
1995# 1140 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1996!$omp target update to(palpha_eps, ptgalpha_eps)
1997# 1140 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1998#endif
1999# 1141 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2000 end if
2001
2002 if (ib) then
2003
2004# 1144 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2005#if defined(MFC_OpenACC)
2006# 1144 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2007!$acc update device(ib_markers%sf)
2008# 1144 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2009#elif defined(MFC_OpenMP)
2010# 1144 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2011!$omp target update to(ib_markers%sf)
2012# 1144 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2013#endif
2014# 1145 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2015 end if
2016# 1147 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2017
2018# 1147 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2019#if defined(MFC_OpenACC)
2020# 1147 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2021!$acc update device(igr, nb, igr_order)
2022# 1147 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2023#elif defined(MFC_OpenMP)
2024# 1147 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2025!$omp target update to(igr, nb, igr_order)
2026# 1147 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2027#endif
2028# 1148 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2029# 1149 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2030
2031 end subroutine s_initialize_gpu_vars
2032
2033 !> Finalize and deallocate all simulation sub-modules in reverse initialization order
2034 impure subroutine s_finalize_modules
2035
2036 if (ib .and. ib_force_wrt) call s_close_ib_force_history()
2037
2038 if (model_eqns == model_eqns_6eq) call s_report_pressure_relaxation()
2039 if (ib .and. chemistry) call s_report_ibm_surface()
2040
2041 call s_finalize_time_steppers_module()
2042 if (hypoelasticity) call s_finalize_hypoelastic_module()
2043 call s_finalize_derived_variables_module()
2044 call s_finalize_data_output_module()
2045 call s_finalize_rhs_module()
2046 if (igr) then
2047 call s_finalize_igr_module()
2048 else
2049 call s_finalize_cbc_module()
2050 call s_finalize_riemann_solvers_module()
2051 if (recon_type == recon_type_weno) then
2052 call s_finalize_weno_module()
2053 else if (recon_type == recon_type_muscl) then
2054 call s_finalize_muscl_module()
2055 end if
2056 end if
2057 if (int_comp > 0) call s_finalize_thinc_module()
2058 call s_finalize_variables_conversion_module()
2059 call s_finalize_eos_module()
2060 if (grid_geometry == 3) call s_finalize_fftw_module
2061 call s_finalize_mpi_common_module()
2062 call s_finalize_global_parameters_module()
2063 call s_finalize_boundary_common_module()
2064 if (relax) call s_finalize_relaxation_solver_module()
2065 if (bubbles_lagrange) call s_finalize_lagrangian_solver()
2066 if (viscous .and. (.not. igr)) then
2067 call s_finalize_viscous_module()
2068 end if
2069 call s_finalize_mpi_proxy_module()
2070
2071 if (surface_tension) call s_finalize_surface_tension_module()
2072 if (bodyforces .or. synthetic_turbulence) call s_finalize_body_forces_module()
2073 if (ib) call s_finalize_ibm_module()
2074
2075 call s_mpi_finalize()
2076
2077 end subroutine s_finalize_modules
2078
2079 !> @brief Fills the properties of a generated particle-cloud IB that the IB state file does not carry (geometry, mass,
2080 !! moving_ibm, inert surface, identity rotation matrix, zeroed step state). This is the only place they are set - pre_process
2081 !! (s_add_cloud_particle) generates only position, kinematics and radius. cloud_ib_idx is the global patch id minus the number
2082 !! of namelist patches; pre_process numbers particles cloud by cloud.
2083 subroutine s_assign_particle_cloud_ib_defaults(cloud_ib_idx, ib_patch)
2084
2085 integer, intent(in) :: cloud_ib_idx
2086 type(ib_patch_parameters), intent(inout) :: ib_patch
2087 integer :: cloud_idx, idx_in_cloud
2088
2089 idx_in_cloud = cloud_ib_idx
2090 do cloud_idx = 1, num_particle_clouds - 1
2091 if (idx_in_cloud <= particle_cloud(cloud_idx)%num_particles) exit
2092 idx_in_cloud = idx_in_cloud - particle_cloud(cloud_idx)%num_particles
2093 end do
2094
2095 ib_patch%geometry = merge(2, 8, num_dims < 3)
2096 ib_patch%step_x_centroid = 0._wp
2097 ib_patch%step_y_centroid = 0._wp
2098 ib_patch%step_z_centroid = 0._wp
2099 ib_patch%step_angles(:) = 0._wp
2100 ib_patch%step_vel(:) = 0._wp
2101 ib_patch%step_angular_vel(:) = 0._wp
2102 ib_patch%force(:) = 0._wp
2103 ib_patch%torque(:) = 0._wp
2104 ib_patch%centroid_offset(:) = 0._wp
2105 ib_patch%rotation_matrix = 0._wp
2106 ib_patch%rotation_matrix(1, 1) = 1._wp
2107 ib_patch%rotation_matrix(2, 2) = 1._wp
2108 ib_patch%rotation_matrix(3, 3) = 1._wp
2109 ib_patch%rotation_matrix_inverse = ib_patch%rotation_matrix
2110 ib_patch%mass = particle_cloud(cloud_idx)%mass
2111 ib_patch%moment = dflt_real
2112 ib_patch%moving_ibm = particle_cloud(cloud_idx)%moving_ibm
2113 ib_patch%slip = .false.
2114 ! Particles are inert surfaces: a cloud IB carries no case-file surface condition, so the thermal,
2115 ! reaction and blowing fields must be set here rather than left as whatever patch_ib held.
2116 ib_patch%thermal_bc = 0
2117 ib_patch%Twall = 0._wp
2118 ib_patch%surface_reaction = 0
2119 ib_patch%v_blow = 0._wp
2120 ib_patch%inj_species = 0
2121 ib_patch%burn_rate_exp = 0._wp
2122 ib_patch%burn_rate_pref = 0._wp
2123 ! Selector for the prescribed-kinematics block; zero leaves the eleven kin_* reals unread.
2124 ib_patch%kin_model = 0
2125
2126 end subroutine s_assign_particle_cloud_ib_defaults
2127
2128 !> @brief Loads the IBs this rank owns from the IB state file for t_step into patch_ib(1:num_ibs), all of which are local. Under
2129 !! file_per_process the rank reads only its own restart_data/lustre_<t_step>/ib_state_<t_step>_<rank>.dat, which holds exactly
2130 !! its IBs; otherwise every rank reads every record of restart_data/ib_state_<t_step>.dat and keeps the ones
2131 !! f_local_rank_owns_location assigns it. Records carry kinematics, position and radius; every other property comes from the
2132 !! namelist patch (global id <= num_ibs) or the particle cloud the IB was generated from. Written by pre_process at t_step = 0
2133 !! (src/pre_process/m_data_output.fpp:s_write_ib_state_0_file) and by s_write_ib_state_file on later steps.
2134 impure subroutine s_read_ib_restart_data(t_step)
2135
2136 integer, intent(in) :: t_step
2137 type(ib_patch_parameters), allocatable, dimension(:) :: namelist_ibs
2138 character(len=path_len + 2*name_len) :: file_loc
2139 integer :: i, ios, file_unit, gbl_id, n_records
2140 integer, parameter :: nfields_per_ib = 20
2141 real(wp) :: ib_buf(nfields_per_ib)
2142 character(len=10) :: t_step_string
2143
2144 moving_immersed_boundary_flag = any(patch_ib(1:num_ibs)%moving_ibm /= 0) &
2145 & .or. any(particle_cloud(1:num_particle_clouds)%moving_ibm /= 0)
2146
2147 allocate (namelist_ibs(num_ibs))
2148 namelist_ibs(:) = patch_ib(1:num_ibs)
2149 num_gbl_ibs = num_ibs + sum(particle_cloud(1:num_particle_clouds)%num_particles)
2150
2151 if (file_per_process) then
2152 call s_int_to_str(t_step, t_step_string)
2153 write (file_loc, '(A,I0,A,i7.7,A)') 'ib_state_', t_step, '_', proc_rank, '.dat'
2154 file_loc = trim(case_dir) // '/restart_data/lustre_' // trim(t_step_string) // '/' // trim(file_loc)
2155 else
2156 write (file_loc, '(A,I0,A)') '/restart_data/ib_state_', t_step, '.dat'
2157 file_loc = trim(case_dir) // trim(file_loc)
2158 end if
2159
2160 open (newunit=file_unit, file=trim(file_loc), form='unformatted', access='stream', status='old', action='read', iostat=ios)
2161 if (ios /= 0) call s_mpi_abort('Error opening IB state file: ' // trim(file_loc))
2162
2163 n_records = num_gbl_ibs
2164 if (file_per_process) then
2165 read (file_unit, iostat=ios) n_records
2166 if (ios /= 0) call s_mpi_abort('Error reading IB state file header: ' // trim(file_loc))
2167 end if
2168
2169 num_ibs = 0
2170 do i = 1, n_records
2171 gbl_id = i
2172 if (file_per_process) read (file_unit, iostat=ios) gbl_id
2173 if (ios == 0) read (file_unit, iostat=ios) ib_buf
2174 if (ios /= 0) call s_mpi_abort('Error reading IB state file: ' // trim(file_loc))
2175
2176 if (.not. file_per_process) then
2177 if (.not. f_local_rank_owns_location(ib_buf(17:19), glb_bounds)) cycle
2178 end if
2179
2180 num_ibs = num_ibs + 1
2181 if (num_ibs > num_local_ibs_max) then
2182# 1300 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2183 call s_prohibit_abort("num_ibs > num_local_ibs_max", "Too many IBs on a single processor rank. Modify case file or increase limit of num_local_ibs_max to resolve.")
2184# 1300 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2185 end if
2186# 1302 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2187 if (gbl_id <= size(namelist_ibs)) then
2188 patch_ib(num_ibs) = namelist_ibs(gbl_id)
2189 else
2190 call s_assign_particle_cloud_ib_defaults(gbl_id - size(namelist_ibs), patch_ib(num_ibs))
2191 end if
2192 patch_ib(num_ibs)%gbl_patch_id = gbl_id
2193 patch_ib(num_ibs)%vel = ib_buf(8:10)
2194 patch_ib(num_ibs)%angular_vel = ib_buf(11:13)
2195 patch_ib(num_ibs)%angles = ib_buf(14:16)
2196 patch_ib(num_ibs)%x_centroid = ib_buf(17)
2197 patch_ib(num_ibs)%y_centroid = ib_buf(18)
2198 patch_ib(num_ibs)%z_centroid = ib_buf(19)
2199 patch_ib(num_ibs)%radius = ib_buf(20)
2200 local_ib_patch_ids(num_ibs) = num_ibs
2201 end do
2202
2203 close (file_unit)
2204 deallocate (namelist_ibs)
2205
2206 num_local_ibs = num_ibs
2207
2208 end subroutine s_read_ib_restart_data
2209
2210 !> @brief Completes this rank's IB neighborhood once s_read_ib_restart_data has loaded only the IBs it owns: every rank sends
2211 !! its own IBs to, and receives the owned IBs of, each distinct rank in ib_neighbor_ranks, appending them to patch_ib after its
2212 !! own.
2213 subroutine s_build_ib_neighborhood()
2214
2215#ifdef MFC_MPI
2216 integer, allocatable, dimension(:) :: nbr_ranks, recv_counts, requests
2217 type(ib_patch_parameters), allocatable, dimension(:,:) :: recv_ibs
2218 integer :: i, n_nbrs, nreqs, patch_bytes, ierr
2219#endif
2220
2221 call s_compute_ib_neighbor_ranks()
2222
2223#ifdef MFC_MPI
2224 if (num_procs > 1) then
2225 ! A rank can fill several table slots (periodicity, few ranks) or be its own neighbor; exchange once per distinct rank
2226 allocate (nbr_ranks(size(ib_neighbor_ranks)))
2227 nbr_ranks = reshape(ib_neighbor_ranks, [size(ib_neighbor_ranks)])
2228 n_nbrs = 0
2229 do i = 1, size(nbr_ranks)
2230 if (nbr_ranks(i) < 0 .or. nbr_ranks(i) == proc_rank) cycle
2231 if (any(nbr_ranks(1:n_nbrs) == nbr_ranks(i))) cycle
2232 n_nbrs = n_nbrs + 1
2233 nbr_ranks(n_nbrs) = nbr_ranks(i)
2234 end do
2235
2236 ! the IBs loaded from restart are this rank's own; neighbors learn their owner from the sender below
2237 patch_ib(1:num_local_ibs)%owner_rank = proc_rank
2238
2239 allocate (recv_counts(n_nbrs), requests(2*n_nbrs))
2240 do i = 1, n_nbrs
2241 call mpi_irecv(recv_counts(i), 1, mpi_integer, nbr_ranks(i), 500, mpi_comm_world, requests(2*i - 1), ierr)
2242 call mpi_isend(num_local_ibs, 1, mpi_integer, nbr_ranks(i), 500, mpi_comm_world, requests(2*i), ierr)
2243 end do
2244 call mpi_waitall(2*n_nbrs, requests, mpi_statuses_ignore, ierr)
2245
2246 patch_bytes = storage_size(patch_ib(1))/8
2247 allocate (recv_ibs(max(1, maxval(recv_counts)), n_nbrs))
2248 nreqs = 0
2249 do i = 1, n_nbrs
2250 if (recv_counts(i) > 0) then
2251 nreqs = nreqs + 1
2252 call mpi_irecv(recv_ibs(:,i), recv_counts(i)*patch_bytes, mpi_byte, nbr_ranks(i), 501, mpi_comm_world, &
2253 & requests(nreqs), ierr)
2254 end if
2255 if (num_local_ibs > 0) then
2256 nreqs = nreqs + 1
2257 call mpi_isend(patch_ib, num_local_ibs*patch_bytes, mpi_byte, nbr_ranks(i), 501, mpi_comm_world, &
2258 & requests(nreqs), ierr)
2259 end if
2260 end do
2261 call mpi_waitall(nreqs, requests, mpi_statuses_ignore, ierr)
2262
2263 do i = 1, n_nbrs
2264 if (num_ibs + recv_counts(i) > num_ib_patches_max_namelist) then
2265# 1379 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2266 call s_prohibit_abort("num_ibs + recv_counts(i) > num_ib_patches_max_namelist", "IB neighborhood exceeds patch_ib capacity. Increase num_ib_patches_max_namelist.")
2267# 1379 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2268 end if
2269# 1381 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2270 patch_ib(num_ibs + 1:num_ibs + recv_counts(i)) = recv_ibs(1:recv_counts(i),i)
2271 patch_ib(num_ibs + 1:num_ibs + recv_counts(i))%owner_rank = nbr_ranks(i)
2272 num_ibs = num_ibs + recv_counts(i)
2273 end do
2274
2275 deallocate (nbr_ranks, recv_counts, requests, recv_ibs)
2276 end if
2277#endif
2278
2279 end subroutine s_build_ib_neighborhood
2280
2281 !> Build ib_neighbor_ranks(-1:1,-1:1,-1:1): MPI ranks of all neighbor domains. Uses two rounds of MPI_SENDRECV cascades - face
2282 !! neighbors are known from bc_*, edge neighbors are obtained in round 1, and (3D) corner neighbors in round 2.
2283 subroutine s_compute_ib_neighbor_ranks()
2284
2285 integer :: ax, k, nbr_idx, nreqs, sx, sy, sz, dx, dy, dz
2286 integer, allocatable :: send_table(:,:,:), recv_tables(:,:,:,:)
2287 integer, dimension(52) :: requests
2288
2289#ifdef MFC_MPI
2290 integer :: ierr
2291 integer, dimension(4) :: buf4, rbuf4
2292 integer, dimension(2) :: buf2, rbuf2
2293
2294 ax = ib_neighborhood_radius
2295
2296 if (allocated(ib_neighbor_ranks)) deallocate (ib_neighbor_ranks)
2297 allocate (ib_neighbor_ranks(-ax:ax,-ax:ax,-ax:ax))
2298 ib_neighbor_ranks = mpi_proc_null
2299 ib_neighbor_ranks(0, 0, 0) = proc_rank
2300
2301 ! Fill radius-1 entries: face neighbors are known from domain decomposition
2302 ib_neighbor_ranks(-1, 0, 0) = bc_x%beg
2303 ib_neighbor_ranks(+1, 0, 0) = bc_x%end
2304 if (num_dims >= 2) then
2305 ib_neighbor_ranks(0, -1, 0) = bc_y%beg
2306 ib_neighbor_ranks(0, +1, 0) = bc_y%end
2307 end if
2308 if (num_dims == 3) then
2309 ib_neighbor_ranks(0, 0, -1) = bc_z%beg
2310 ib_neighbor_ranks(0, 0, +1) = bc_z%end
2311 end if
2312
2313 if (num_dims >= 2) then
2314 ! Round 1a: exchange y/z face ranks with +/-x face neighbors -> xy and xz edge ranks
2315 buf4 = [bc_y%beg, bc_y%end, bc_z%beg, bc_z%end]
2316
2317 ! Send to -x, receive from +x -> edges (+1,+/-1,0) and (+1,0,+/-1)
2318 call mpi_sendrecv(buf4, 4, mpi_integer, merge(bc_x%beg, mpi_proc_null, bc_x%beg >= 0), 310, rbuf4, 4, mpi_integer, &
2319 & merge(bc_x%end, mpi_proc_null, bc_x%end >= 0), 310, mpi_comm_world, mpi_status_ignore, ierr)
2320 if (bc_x%end >= 0) then
2321 ib_neighbor_ranks(+1, -1, 0) = rbuf4(1)
2322 ib_neighbor_ranks(+1, +1, 0) = rbuf4(2)
2323 ib_neighbor_ranks(+1, 0, -1) = rbuf4(3)
2324 ib_neighbor_ranks(+1, 0, +1) = rbuf4(4)
2325 end if
2326
2327 call mpi_sendrecv(buf4, 4, mpi_integer, merge(bc_x%end, mpi_proc_null, bc_x%end >= 0), 311, rbuf4, 4, mpi_integer, &
2328 & merge(bc_x%beg, mpi_proc_null, bc_x%beg >= 0), 311, mpi_comm_world, mpi_status_ignore, ierr)
2329 if (bc_x%beg >= 0) then
2330 ib_neighbor_ranks(-1, -1, 0) = rbuf4(1)
2331 ib_neighbor_ranks(-1, +1, 0) = rbuf4(2)
2332 ib_neighbor_ranks(-1, 0, -1) = rbuf4(3)
2333 ib_neighbor_ranks(-1, 0, +1) = rbuf4(4)
2334 end if
2335 end if
2336
2337 if (num_dims == 3) then
2338 ! Round 1b: exchange z face ranks with +/-y face neighbors -> yz edge ranks
2339 buf2 = [bc_z%beg, bc_z%end]
2340
2341 call mpi_sendrecv(buf2, 2, mpi_integer, merge(bc_y%beg, mpi_proc_null, bc_y%beg >= 0), 312, rbuf2, 2, mpi_integer, &
2342 & merge(bc_y%end, mpi_proc_null, bc_y%end >= 0), 312, mpi_comm_world, mpi_status_ignore, ierr)
2343 if (bc_y%end >= 0) then
2344 ib_neighbor_ranks(0, +1, -1) = rbuf2(1)
2345 ib_neighbor_ranks(0, +1, +1) = rbuf2(2)
2346 end if
2347
2348 call mpi_sendrecv(buf2, 2, mpi_integer, merge(bc_y%end, mpi_proc_null, bc_y%end >= 0), 313, rbuf2, 2, mpi_integer, &
2349 & merge(bc_y%beg, mpi_proc_null, bc_y%beg >= 0), 313, mpi_comm_world, mpi_status_ignore, ierr)
2350 if (bc_y%beg >= 0) then
2351 ib_neighbor_ranks(0, -1, -1) = rbuf2(1)
2352 ib_neighbor_ranks(0, -1, +1) = rbuf2(2)
2353 end if
2354
2355 ! Round 2: exchange z face ranks with xy-diagonal edge neighbors -> corner ranks. Each of the 4 xy diagonals gives 2
2356 ! corners (the +/-z variants). Pattern: send buf2 to mirror diagonal, receive from this diagonal -> that edge's z face
2357 ! ranks.
2358# 1470 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2359 call mpi_sendrecv(buf2, 2, mpi_integer, merge(ib_neighbor_ranks(-1, -1, 0), mpi_proc_null, &
2360# 1471 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2361 & ib_neighbor_ranks(-1, -1, 0) >= 0), 320, rbuf2, 2, mpi_integer, &
2362# 1472 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2363 & merge(ib_neighbor_ranks(1, 1, 0), mpi_proc_null, ib_neighbor_ranks(1, 1, &
2364# 1473 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2365 & 0) >= 0), 320, mpi_comm_world, mpi_status_ignore, ierr)
2366# 1474 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2367 if (ib_neighbor_ranks(1, 1, 0) >= 0) then
2368# 1475 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2369 ib_neighbor_ranks(1, 1, -1) = rbuf2(1)
2370# 1476 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2371 ib_neighbor_ranks(1, 1, +1) = rbuf2(2)
2372# 1477 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2373 end if
2374# 1470 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2375 call mpi_sendrecv(buf2, 2, mpi_integer, merge(ib_neighbor_ranks(-1, 1, 0), mpi_proc_null, &
2376# 1471 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2377 & ib_neighbor_ranks(-1, 1, 0) >= 0), 321, rbuf2, 2, mpi_integer, &
2378# 1472 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2379 & merge(ib_neighbor_ranks(1, -1, 0), mpi_proc_null, ib_neighbor_ranks(1, -1, &
2380# 1473 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2381 & 0) >= 0), 321, mpi_comm_world, mpi_status_ignore, ierr)
2382# 1474 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2383 if (ib_neighbor_ranks(1, -1, 0) >= 0) then
2384# 1475 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2385 ib_neighbor_ranks(1, -1, -1) = rbuf2(1)
2386# 1476 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2387 ib_neighbor_ranks(1, -1, +1) = rbuf2(2)
2388# 1477 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2389 end if
2390# 1470 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2391 call mpi_sendrecv(buf2, 2, mpi_integer, merge(ib_neighbor_ranks(1, -1, 0), mpi_proc_null, &
2392# 1471 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2393 & ib_neighbor_ranks(1, -1, 0) >= 0), 322, rbuf2, 2, mpi_integer, &
2394# 1472 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2395 & merge(ib_neighbor_ranks(-1, 1, 0), mpi_proc_null, ib_neighbor_ranks(-1, 1, &
2396# 1473 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2397 & 0) >= 0), 322, mpi_comm_world, mpi_status_ignore, ierr)
2398# 1474 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2399 if (ib_neighbor_ranks(-1, 1, 0) >= 0) then
2400# 1475 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2401 ib_neighbor_ranks(-1, 1, -1) = rbuf2(1)
2402# 1476 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2403 ib_neighbor_ranks(-1, 1, +1) = rbuf2(2)
2404# 1477 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2405 end if
2406# 1470 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2407 call mpi_sendrecv(buf2, 2, mpi_integer, merge(ib_neighbor_ranks(1, 1, 0), mpi_proc_null, &
2408# 1471 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2409 & ib_neighbor_ranks(1, 1, 0) >= 0), 323, rbuf2, 2, mpi_integer, &
2410# 1472 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2411 & merge(ib_neighbor_ranks(-1, -1, 0), mpi_proc_null, ib_neighbor_ranks(-1, -1, &
2412# 1473 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2413 & 0) >= 0), 323, mpi_comm_world, mpi_status_ignore, ierr)
2414# 1474 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2415 if (ib_neighbor_ranks(-1, -1, 0) >= 0) then
2416# 1475 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2417 ib_neighbor_ranks(-1, -1, -1) = rbuf2(1)
2418# 1476 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2419 ib_neighbor_ranks(-1, -1, +1) = rbuf2(2)
2420# 1477 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2421 end if
2422# 1479 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2423 end if
2424
2425 ! For radius > 1: extend the table by iterative 26-neighbor full-table exchanges. In each round, every rank broadcasts its
2426 ! current table to all 26 immediate neighbors. Their entry at offset (dx,dy,dz) from them = our entry at
2427 ! (dx+sx,dy+sy,dz+sz). One extension round fills the entire next shell, so ax-1 rounds suffice.
2428 if (ax > 1) then
2429 allocate (send_table(-ax:ax,-ax:ax,-ax:ax))
2430 allocate (recv_tables(-ax:ax,-ax:ax,-ax:ax,1:26))
2431
2432 do k = 2, ax
2433 send_table = ib_neighbor_ranks
2434
2435 nreqs = 0
2436 nbr_idx = 0
2437 do sz = -1, 1
2438 do sy = -1, 1
2439 do sx = -1, 1
2440 if (sx == 0 .and. sy == 0 .and. sz == 0) cycle
2441 nbr_idx = nbr_idx + 1
2442 if (ib_neighbor_ranks(sx, sy, sz) < 0) cycle
2443 nreqs = nreqs + 1
2444 call mpi_irecv(recv_tables(:,:,:,nbr_idx), (2*ax + 1)**3, mpi_integer, ib_neighbor_ranks(sx, sy, sz), &
2445 & 400, mpi_comm_world, requests(nreqs), ierr)
2446 end do
2447 end do
2448 end do
2449
2450 do sz = -1, 1
2451 do sy = -1, 1
2452 do sx = -1, 1
2453 if (sx == 0 .and. sy == 0 .and. sz == 0) cycle
2454 if (ib_neighbor_ranks(sx, sy, sz) < 0) cycle
2455 nreqs = nreqs + 1
2456 call mpi_isend(send_table, (2*ax + 1)**3, mpi_integer, ib_neighbor_ranks(sx, sy, sz), 400, &
2457 & mpi_comm_world, requests(nreqs), ierr)
2458 end do
2459 end do
2460 end do
2461
2462 call mpi_waitall(nreqs, requests, mpi_statuses_ignore, ierr)
2463
2464 nbr_idx = 0
2465 do sz = -1, 1
2466 do sy = -1, 1
2467 do sx = -1, 1
2468 if (sx == 0 .and. sy == 0 .and. sz == 0) cycle
2469 nbr_idx = nbr_idx + 1
2470 if (ib_neighbor_ranks(sx, sy, sz) < 0) cycle
2471 do dz = -ax, ax
2472 do dy = -ax, ax
2473 do dx = -ax, ax
2474 if (recv_tables(dx, dy, dz, nbr_idx) == mpi_proc_null) cycle
2475 if (dx + sx < -ax .or. dx + sx > ax) cycle
2476 if (dy + sy < -ax .or. dy + sy > ax) cycle
2477 if (dz + sz < -ax .or. dz + sz > ax) cycle
2478 if (ib_neighbor_ranks(dx + sx, dy + sy, dz + sz) /= mpi_proc_null) cycle
2479 ib_neighbor_ranks(dx + sx, dy + sy, dz + sz) = recv_tables(dx, dy, dz, nbr_idx)
2480 end do
2481 end do
2482 end do
2483 end do
2484 end do
2485 end do
2486 end do
2487
2488 deallocate (send_table, recv_tables)
2489 end if
2490#endif
2491
2492 end subroutine s_compute_ib_neighbor_ranks
2493
2494 subroutine s_get_neighbor_bounds()
2495
2496 real(wp) :: beg_val, end_val, recv_val, bound, max_ib_bound, local_rank_width, min_rank_width
2497 integer :: k, send_neighbor, recv_neighbor, ierr, temporary_radius
2498
2499 ! Default: unbounded in all directions (covers single-rank and no-MPI cases)
2500
2501 neighbor_domain_x%beg = -huge(0._wp)
2502 neighbor_domain_x%end = huge(0._wp)
2503 neighbor_domain_y%beg = -huge(0._wp)
2504 neighbor_domain_y%end = huge(0._wp)
2505 neighbor_domain_z%beg = -huge(0._wp)
2506 neighbor_domain_z%end = huge(0._wp)
2507
2508#ifdef MFC_MPI
2509 ! perform setup if we are doing automatic radius checking
2510 if (ib_neighborhood_radius < 1) then
2511 ib_neighborhood_radius = 0 ! ensure we are starting with 0 neighborhood radius
2512
2513 ! determine the maximum length of space that needs to be contained by the neighborhood
2514 max_ib_bound = -1._wp
2515 do k = 1, num_ibs
2516 call s_get_ib_bound(patch_ib(k), bound)
2517 max_ib_bound = max(max_ib_bound, bound)
2518 end do
2519 do k = 1, num_particle_clouds
2520 max_ib_bound = max(max_ib_bound, particle_cloud(k)%radius)
2521 end do
2522
2523 ! Narrowest rank extent, over every direction as well as every rank. The radius is a count of rank
2524 ! hops, so the distance one hop covers is the extent of the rank it steps over, and the direction
2525 ! needing the most hops to span the body is the one whose ranks are thinnest. Reducing over each
2526 ! rank's widest extent first reports the wrong number whenever ranks are anisotropic, which is the
2527 ! norm on a stretched grid or an elongated domain.
2528 local_rank_width = huge(0._wp)
2529# 1586 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2530 if (num_dims >= 1) local_rank_width = min(local_rank_width, abs(x_cb(m) - x_cb(-1)))
2531# 1587 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2532# 1586 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2533 if (num_dims >= 2) local_rank_width = min(local_rank_width, abs(y_cb(n) - y_cb(-1)))
2534# 1587 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2535# 1586 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2536 if (num_dims >= 3) local_rank_width = min(local_rank_width, abs(z_cb(p) - z_cb(-1)))
2537# 1587 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2538# 1588 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2539 call s_mpi_allreduce_min(local_rank_width, min_rank_width)
2540
2541 ! approximate the size of the neighborhood with a local 1.1x fudge factor for safety, lower bound of 1
2542 ib_neighborhood_radius = max(1, ceiling(1.1_wp*max_ib_bound/(min_rank_width)))
2543 if (proc_rank == 0) print *, "Automatic choice of ib_neighborhood_radius selected: ", ib_neighborhood_radius
2544 end if
2545
2546 ! For each direction, propagate the left/right boundary edges outward ib_neighborhood_radius hops. After k rounds: beg_val =
2547 ! left edge of the rank k hops to the left; end_val = right edge of the rank k hops to the right.
2548# 1598 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2549 if (num_dims >= 1) then
2550# 1599 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2551 beg_val = x_cb(-1)
2552# 1600 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2553 end_val = x_cb(m)
2554# 1601 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2555 do k = 1, ib_neighborhood_radius
2556 send_neighbor = merge(bc_x%end, mpi_proc_null, bc_x%end >= 0)
2557# 1603 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2558 recv_neighbor = merge(bc_x%beg, mpi_proc_null, bc_x%beg >= 0)
2559# 1604 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2560 recv_val = -huge(0._wp)
2561 call mpi_sendrecv(beg_val, 1, mpi_p, send_neighbor, 100, recv_val, 1, mpi_p, recv_neighbor, 100, &
2562# 1606 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2563 & mpi_comm_world, mpi_status_ignore, ierr)
2564 beg_val = recv_val
2565
2566 send_neighbor = merge(bc_x%beg, mpi_proc_null, bc_x%beg >= 0)
2567# 1610 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2568 recv_neighbor = merge(bc_x%end, mpi_proc_null, bc_x%end >= 0)
2569# 1611 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2570 recv_val = huge(0._wp)
2571 call mpi_sendrecv(end_val, 1, mpi_p, send_neighbor, 101, recv_val, 1, mpi_p, recv_neighbor, &
2572# 1613 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2573 & 101, mpi_comm_world, mpi_status_ignore, ierr)
2574# 1614 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2575 end_val = recv_val
2576
2577 ! protect from looping back around on yourself multiple times
2578 if (f_approx_equal(beg_val, x_cb(m)) .or. f_approx_equal(end_val, x_cb(-1))) then
2579# 1618 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2580 beg_val = -huge(0._wp)
2581 end_val = huge(0._wp)
2582 exit
2583 end if
2584 end do
2585 neighbor_domain_x%beg = beg_val
2586# 1624 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2587 neighbor_domain_x%end = end_val
2588# 1625 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2589 end if
2590# 1598 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2591 if (num_dims >= 2) then
2592# 1599 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2593 beg_val = y_cb(-1)
2594# 1600 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2595 end_val = y_cb(n)
2596# 1601 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2597 do k = 1, ib_neighborhood_radius
2598 send_neighbor = merge(bc_y%end, mpi_proc_null, bc_y%end >= 0)
2599# 1603 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2600 recv_neighbor = merge(bc_y%beg, mpi_proc_null, bc_y%beg >= 0)
2601# 1604 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2602 recv_val = -huge(0._wp)
2603 call mpi_sendrecv(beg_val, 1, mpi_p, send_neighbor, 102, recv_val, 1, mpi_p, recv_neighbor, 102, &
2604# 1606 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2605 & mpi_comm_world, mpi_status_ignore, ierr)
2606 beg_val = recv_val
2607
2608 send_neighbor = merge(bc_y%beg, mpi_proc_null, bc_y%beg >= 0)
2609# 1610 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2610 recv_neighbor = merge(bc_y%end, mpi_proc_null, bc_y%end >= 0)
2611# 1611 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2612 recv_val = huge(0._wp)
2613 call mpi_sendrecv(end_val, 1, mpi_p, send_neighbor, 103, recv_val, 1, mpi_p, recv_neighbor, &
2614# 1613 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2615 & 103, mpi_comm_world, mpi_status_ignore, ierr)
2616# 1614 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2617 end_val = recv_val
2618
2619 ! protect from looping back around on yourself multiple times
2620 if (f_approx_equal(beg_val, y_cb(n)) .or. f_approx_equal(end_val, y_cb(-1))) then
2621# 1618 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2622 beg_val = -huge(0._wp)
2623 end_val = huge(0._wp)
2624 exit
2625 end if
2626 end do
2627 neighbor_domain_y%beg = beg_val
2628# 1624 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2629 neighbor_domain_y%end = end_val
2630# 1625 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2631 end if
2632# 1598 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2633 if (num_dims >= 3) then
2634# 1599 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2635 beg_val = z_cb(-1)
2636# 1600 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2637 end_val = z_cb(p)
2638# 1601 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2639 do k = 1, ib_neighborhood_radius
2640 send_neighbor = merge(bc_z%end, mpi_proc_null, bc_z%end >= 0)
2641# 1603 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2642 recv_neighbor = merge(bc_z%beg, mpi_proc_null, bc_z%beg >= 0)
2643# 1604 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2644 recv_val = -huge(0._wp)
2645 call mpi_sendrecv(beg_val, 1, mpi_p, send_neighbor, 104, recv_val, 1, mpi_p, recv_neighbor, 104, &
2646# 1606 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2647 & mpi_comm_world, mpi_status_ignore, ierr)
2648 beg_val = recv_val
2649
2650 send_neighbor = merge(bc_z%beg, mpi_proc_null, bc_z%beg >= 0)
2651# 1610 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2652 recv_neighbor = merge(bc_z%end, mpi_proc_null, bc_z%end >= 0)
2653# 1611 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2654 recv_val = huge(0._wp)
2655 call mpi_sendrecv(end_val, 1, mpi_p, send_neighbor, 105, recv_val, 1, mpi_p, recv_neighbor, &
2656# 1613 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2657 & 105, mpi_comm_world, mpi_status_ignore, ierr)
2658# 1614 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2659 end_val = recv_val
2660
2661 ! protect from looping back around on yourself multiple times
2662 if (f_approx_equal(beg_val, z_cb(p)) .or. f_approx_equal(end_val, z_cb(-1))) then
2663# 1618 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2664 beg_val = -huge(0._wp)
2665 end_val = huge(0._wp)
2666 exit
2667 end if
2668 end do
2669 neighbor_domain_z%beg = beg_val
2670# 1624 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2671 neighbor_domain_z%end = end_val
2672# 1625 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2673 end if
2674# 1627 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2675#endif
2676
2677 end subroutine s_get_neighbor_bounds
2678
2679end module m_start_up
q_cons_vf
type(scalar_field), dimension(sys_size), intent(inout) q_cons_vf
Definition m_phase_change.fpp.f90:1047
m_acoustic_src
One-way acoustic source injection, Maeda and Colonius JCP (2017).
Definition m_acoustic_src.fpp.f90:356
m_acoustic_src::s_precalculate_acoustic_spatial_sources
impure subroutine, public s_precalculate_acoustic_spatial_sources
Pre-compute non-zero spatial source weights before time-stepping.
Definition m_acoustic_src.fpp.f90:1207
m_acoustic_src::s_initialize_acoustic_src
impure subroutine, public s_initialize_acoustic_src
Initialize the acoustic source module.
Definition m_acoustic_src.fpp.f90:545
m_body_forces
Computes gravitational and body force source terms for the momentum equations.
Definition m_body_forces.fpp.f90:356
m_body_forces::s_initialize_body_forces_module
impure subroutine, public s_initialize_body_forces_module
Initialize the body forces module. When synthetic_turbulence is enabled, generates random wave vector...
Definition m_body_forces.fpp.f90:444
m_boundary_common
Noncharacteristic and processor boundary condition application for ghost cells and buffer regions.
Definition m_boundary_common.fpp.f90:367
m_boundary_common::s_populate_grid_variables_buffers
subroutine, public s_populate_grid_variables_buffers(x_cb_in, x_cc_in, dx_in, x_offset, y_offset, z_offset, y_cb_in, y_cc_in, dy_in, z_cb_in, z_cc_in, dz_in, global_bounds)
Populate the buffers of the grid variables, which are constituted of the cell-boundary locations and ...
Definition m_boundary_common.fpp.f90:1125
m_boundary_common::s_populate_variables_buffers
impure subroutine, public s_populate_variables_buffers(bc_type, q_prim_vf, pb_in, mv_in, q_t_sf)
Populate the buffers of the primitive variables based on the selected boundary conditions.
Definition m_boundary_common.fpp.f90:770
m_boundary_common::s_initialize_boundary_common_module
impure subroutine, public s_initialize_boundary_common_module(use_dirichlet_buffers)
Allocate and set up boundary condition buffer arrays for all coordinate directions.
Definition m_boundary_common.fpp.f90:407
m_boundary_io
Boundary condition restart I/O, capillary/IGR buffer population, and grid-variable buffers.
Definition m_boundary_io.fpp.f90:367
m_boundary_io::s_assign_default_bc_type
subroutine s_assign_default_bc_type(bc_type)
Initialize the per-cell boundary condition type arrays with the global default BC values.
Definition m_boundary_io.fpp.f90:794
m_boundary_io::s_read_serial_boundary_condition_files
subroutine s_read_serial_boundary_condition_files(step_dirpath, bc_type)
Read boundary condition type and buffer data from serial (unformatted) restart files.
Definition m_boundary_io.fpp.f90:531
m_boundary_io::s_read_parallel_boundary_condition_files
subroutine s_read_parallel_boundary_condition_files(bc_type, strict)
Read boundary condition type and buffer data from per-rank parallel files using MPI I/O.
Definition m_boundary_io.fpp.f90:597
m_bubbles_ee
Computes ensemble-averaged (Euler–Euler) bubble source terms for radius, velocity,...
Definition m_bubbles_EE.fpp.f90:356
m_bubbles_ee::s_initialize_bubbles_ee_module
impure subroutine s_initialize_bubbles_ee_module
Initialize the Euler-Euler bubble module.
Definition m_bubbles_EE.fpp.f90:414
m_bubbles_el
Tracks Lagrangian bubbles and couples their dynamics to the Eulerian flow via volume averaging.
Definition m_bubbles_EL.fpp.f90:356
m_bubbles_el::s_write_lag_bubble_stats
impure subroutine s_write_lag_bubble_stats()
Write the maximum and minimum radius of each bubble.
Definition m_bubbles_EL.fpp.f90:4832
m_bubbles_el::s_write_restart_lag_bubbles
impure subroutine s_write_restart_lag_bubbles(t_step)
Write restart files for the Lagrangian bubble solver.
Definition m_bubbles_EL.fpp.f90:4616
m_bubbles_el::q_beta
type(scalar_field), dimension(:), allocatable q_beta
Projection of the lagrangian particles in the Eulerian framework.
Definition m_bubbles_EL.fpp.f90:481
m_bubbles_el::s_initialize_bubbles_el_module
impure subroutine s_initialize_bubbles_el_module(q_cons_vf, bc_type)
Initializes the lagrangian subgrid bubble solver.
Definition m_bubbles_EL.fpp.f90:532
m_bubbles_el::intfc_rad
real(wp), dimension(:,:), allocatable intfc_rad
Bubble radius.
Definition m_bubbles_EL.fpp.f90:414
m_cbc
Characteristic boundary conditions (CBCs) for slip walls, non-reflecting subsonic inflow/outflow,...
Definition m_cbc.fpp.f90:368
m_cbc::s_initialize_cbc_module
impure subroutine, public s_initialize_cbc_module
Initialize the CBC module.
Definition m_cbc.fpp.f90:558
m_checker_common
Shared input validation checks for grid dimensions and AMD GPU compiler limits.
Definition m_checker_common.fpp.f90:367
m_checker_common::s_check_inputs_common
impure subroutine, public s_check_inputs_common(check_total_cells, n_global)
Checks compatibility of parameters in the input file. Used by all three stages.
Definition m_checker_common.fpp.f90:382
m_checker
Validates simulation input parameters for consistency and supported configurations.
Definition m_checker.fpp.f90:367
m_checker::s_check_inputs
impure subroutine, public s_check_inputs
Checks compatibility of parameters in the input file. Used by the simulation stage.
Definition m_checker.fpp.f90:382
m_chemistry
Multi-species chemistry interface for thermodynamic properties, reaction rates, and transport coeffic...
Definition m_chemistry.fpp.f90:368
m_chemistry::s_compute_q_t_sf
subroutine s_compute_q_t_sf(q_t_sf, q_cons_vf, bounds)
Initialize the temperature field from conservative variables by inverting the energy equation.
Definition m_chemistry.fpp.f90:462
m_collisions
Ghost-node immersed boundary method: locates ghost/image points, computes interpolation coefficients,...
Definition m_collisions.fpp.f90:357
m_compile_specific
Platform-specific file and directory operations: create, delete, inquire, getcwd, and basename.
Definition m_compile_specific.f90:6
m_compile_specific::my_inquire
impure subroutine my_inquire(fileloc, dircheck)
Inquire on the existence of a directory or file.
Definition m_compile_specific.f90:56
m_constants
Compile-time constant parameters: default values, tolerances, and physical constants.
Definition m_constants.fpp.f90:7
m_constants::dflt_t_guess
real(wp), parameter dflt_t_guess
Default guess for temperature (when a previous value is not available).
Definition m_constants.fpp.f90:66
m_constants::time_stepper_rk2
integer, parameter time_stepper_rk2
Definition m_constants.fpp.f90:174
m_constants::sgm_eps
real(wp), parameter sgm_eps
Segmentation tolerance.
Definition m_constants.fpp.f90:13
m_constants::recon_type_muscl
integer, parameter recon_type_muscl
Definition m_constants.fpp.f90:168
m_constants::time_stepper_rk1
integer, parameter time_stepper_rk1
Definition m_constants.fpp.f90:173
m_constants::time_stepper_rk3
integer, parameter time_stepper_rk3
Definition m_constants.fpp.f90:175
m_constants::nnode
integer, parameter nnode
Number of QBMM nodes.
Definition m_constants.fpp.f90:46
m_constants::recon_type_weno
integer, parameter recon_type_weno
Definition m_constants.fpp.f90:167
m_constants::model_eqns_6eq
integer, parameter model_eqns_6eq
Definition m_constants.fpp.f90:156
m_constants::bc_periodic
integer, parameter bc_periodic
Definition m_constants.fpp.f90:106
m_data_output
Writes solution data, run-time stability diagnostics (ICFL, VCFL, CCFL, Rc), and probe/center-of-mass...
Definition m_data_output.fpp.f90:367
m_data_output::s_initialize_data_output_module
impure subroutine, public s_initialize_data_output_module
Initialize the data output module.
Definition m_data_output.fpp.f90:2569
m_data_output::s_write_data_files
impure subroutine, public s_write_data_files(q_cons_vf, q_t_sf, q_prim_vf, t_step, bc_type, beta)
Write data files. Dispatch subroutine that replaces procedure pointer.
Definition m_data_output.fpp.f90:417
m_data_output::s_write_ib_state_file
impure subroutine, public s_write_ib_state_file(time_step)
Writes IB state records to restart_data/ib_state.dat. Called on every rank.
Definition m_data_output.fpp.f90:1905
m_data_output::s_write_ib_data_file
subroutine, public s_write_ib_data_file(time_step)
Dispatch immersed boundary data output to the serial or parallel writer.
Definition m_data_output.fpp.f90:1579
m_derived_types
Shared derived types for field data, patch geometry, bubble dynamics, and MPI I/O structures.
Definition m_derived_types.fpp.f90:356
m_derived_variables
Derives diagnostic flow quantities (vorticity, speed of sound, numerical Schlieren,...
Definition m_derived_variables.fpp.f90:357
m_derived_variables::s_initialize_derived_variables
impure subroutine, public s_initialize_derived_variables
Allocate and open derived variables. Computing FD coefficients.
Definition m_derived_variables.fpp.f90:657
m_derived_variables::s_initialize_derived_variables_module
impure subroutine, public s_initialize_derived_variables_module
Computation of parameters, allocation procedures, and/or any other tasks needed to properly setup the...
Definition m_derived_variables.fpp.f90:393
m_eos
Equations of state in Gamma/Pi form, rho e = Gamma(rho) p + Pi(rho).
Definition m_eos.fpp.f90:402
m_eos::s_initialize_eos_module
impure subroutine, public s_initialize_eos_module()
Resolve every fluid's EOS coefficients once, before any conversion runs.
Definition m_eos.fpp.f90:424
m_eos::s_phase_internal_energy
subroutine, public s_phase_internal_energy(pres, alpha, alpha_rho, i, e_phase)
Internal energy per unit volume of phase i at pressure pres: alpha (Gamma p + Pi) + alpha_rho qv,...
Definition m_eos.fpp.f90:1781
m_global_parameters
Global parameters for the computational domain, fluid properties, and simulation algorithm configurat...
Definition m_global_parameters.fpp.f90:367
m_global_parameters::mytime
real(wp) mytime
Current simulation time.
Definition m_global_parameters.fpp.f90:1107
m_global_parameters::synth_k_shell
real(wp), dimension(num_synth_shells_max) synth_k_shell
Definition m_global_parameters.fpp.f90:522
m_global_parameters::grid_geometry
integer grid_geometry
Definition m_global_parameters.fpp.f90:410
m_global_parameters::p_glb
integer p_glb
Definition m_global_parameters.fpp.f90:405
m_global_parameters::synth_amp_shell
real(wp), dimension(num_synth_shells_max) synth_amp_shell
Definition m_global_parameters.fpp.f90:522
m_global_parameters::idwint
type(int_bounds_info), dimension(1:3) idwint
Definition m_global_parameters.fpp.f90:758
m_global_parameters::z_cb
real(wp), dimension(:), allocatable, target z_cb
Definition m_global_parameters.fpp.f90:427
m_global_parameters::bc_z
type(int_bounds_info) bc_z
Definition m_global_parameters.fpp.f90:580
m_global_parameters::periodic_bc
logical, dimension(3) periodic_bc
Definition m_global_parameters.fpp.f90:577
m_global_parameters::proc_rank
integer proc_rank
Rank of the local processor.
Definition m_global_parameters.fpp.f90:386
m_global_parameters::mpi_io_ib_data
type(mpi_io_ib_var), public mpi_io_ib_data
Definition m_global_parameters.fpp.f90:748
m_global_parameters::bc_io
logical bc_io
Definition m_global_parameters.fpp.f90:576
m_global_parameters::bodyforces
logical bodyforces
Definition m_global_parameters.fpp.f90:487
m_global_parameters::glb_bounds
type(bounds_info), dimension(3) glb_bounds
Definition m_global_parameters.fpp.f90:428
m_global_parameters::n_el_bubs_loc
integer n_el_bubs_loc
Definition m_global_parameters.fpp.f90:1129
m_global_parameters::idwbuff
type(int_bounds_info), dimension(1:3) idwbuff
Definition m_global_parameters.fpp.f90:774
m_global_parameters::buff_size
integer buff_size
Number of ghost cells for boundary condition storage.
Definition m_global_parameters.fpp.f90:853
m_global_parameters::bc_y
type(int_bounds_info) bc_y
Definition m_global_parameters.fpp.f90:580
m_global_parameters::s_initialize_global_parameters_module
impure subroutine s_initialize_global_parameters_module
Initialize the global parameters module.
Definition m_global_parameters.fpp.f90:1749
m_global_parameters::y_cc
real(wp), dimension(:), allocatable, target y_cc
Definition m_global_parameters.fpp.f90:433
m_global_parameters::pb_ts
type(pres_field), dimension(:), allocatable pb_ts
Definition m_global_parameters.fpp.f90:1108
m_global_parameters::mv_ts
type(pres_field), dimension(:), allocatable mv_ts
Definition m_global_parameters.fpp.f90:1109
m_global_parameters::bc_x
type(int_bounds_info) bc_x
Definition m_global_parameters.fpp.f90:580
m_global_parameters::z_cc
real(wp), dimension(:), allocatable, target z_cc
Definition m_global_parameters.fpp.f90:433
m_global_parameters::wall_time
real(wp) wall_time
Definition m_global_parameters.fpp.f90:380
m_global_parameters::num_procs
integer num_procs
Number of processors.
Definition m_global_parameters.fpp.f90:384
m_global_parameters::x_cc
real(wp), dimension(:), allocatable, target x_cc
Definition m_global_parameters.fpp.f90:433
m_global_parameters::synth_n_waves_per_shell
integer, dimension(num_synth_shells_max) synth_n_waves_per_shell
Definition m_global_parameters.fpp.f90:521
m_global_parameters::turb_pos
real(wp), dimension(num_turb_sources_max, 3) turb_pos
Definition m_global_parameters.fpp.f90:523
m_global_parameters::y_cb
real(wp), dimension(:), allocatable, target y_cb
Definition m_global_parameters.fpp.f90:427
m_global_parameters::cells_bounds
type(cell_num_bounds) cells_bounds
Definition m_global_parameters.fpp.f90:401
m_global_parameters::wall_time_avg
real(wp) wall_time_avg
Definition m_global_parameters.fpp.f90:381
m_global_parameters::m_glb
integer m_glb
Definition m_global_parameters.fpp.f90:405
m_global_parameters::nb
integer nb
Definition m_global_parameters.fpp.f90:934
m_global_parameters::mpi_io_data
type(mpi_io_var), public mpi_io_data
Definition m_global_parameters.fpp.f90:747
m_global_parameters::n_glb
integer n_glb
Definition m_global_parameters.fpp.f90:405
m_global_parameters::dy
real(wp), dimension(:), allocatable, target dy
Definition m_global_parameters.fpp.f90:437
m_global_parameters::synth_l
real(wp), dimension(num_turb_sources_max, 3) synth_l
Definition m_global_parameters.fpp.f90:523
m_global_parameters::dz
real(wp), dimension(:), allocatable, target dz
Definition m_global_parameters.fpp.f90:437
m_global_parameters::dx
real(wp), dimension(:), allocatable, target dx
Definition m_global_parameters.fpp.f90:437
m_global_parameters::x_cb
real(wp), dimension(:), allocatable, target x_cb
Definition m_global_parameters.fpp.f90:427
m_global_parameters::cfl_dt
logical cfl_dt
Definition m_global_parameters.fpp.f90:467
m_helper_basic
Basic floating-point utilities: approximate equality, default detection, and coordinate bounds.
Definition m_helper_basic.fpp.f90:356
m_helper_basic::s_update_cell_bounds
elemental subroutine, public s_update_cell_bounds(bounds, m, n, p)
Update the min and max number of cells in each set of axes.
Definition m_helper_basic.fpp.f90:572
m_helper
Utility routines for bubble model setup, coordinate transforms, array sampling, and special functions...
Definition m_helper.fpp.f90:367
m_helper::s_upsample_data
subroutine, public s_upsample_data(q_cons_vf, q_cons_temp)
Upsample conservative variable fields from a coarsened grid back to the original resolution using int...
Definition m_helper.fpp.f90:1331
m_helper::s_initialize_bubbles_model
impure subroutine, public s_initialize_bubbles_model()
Initialize bubble model arrays for Euler or Lagrangian bubbles with polytropic or non-polytropic gas.
Definition m_helper.fpp.f90:477
m_helper::s_int_to_str
elemental subroutine, public s_int_to_str(i, res)
Convert an integer to its trimmed string representation.
Definition m_helper.fpp.f90:777
m_hypoelastic
Computes hypoelastic stress-rate source terms and damage-state evolution.
Definition m_hypoelastic.fpp.f90:356
m_hypoelastic::s_initialize_hypoelastic_module
impure subroutine, public s_initialize_hypoelastic_module
Initialize the hypoelastic module.
Definition m_hypoelastic.fpp.f90:435
m_ib_patches
Allocate memory and read initial condition data for IC extrusion.
Definition m_ib_patches.fpp.f90:426
m_ib_patches::s_initialize_ib_airfoils
subroutine, public s_initialize_ib_airfoils()
Initialize the NACA surface grids for all airfoil IB patches. Must be called after the grid is establ...
Definition m_ib_patches.fpp.f90:1058
m_ibm
Ghost-node immersed boundary method: locates ghost/image points, computes interpolation coefficients,...
Definition m_ibm.fpp.f90:357
m_ibm::s_ibm_setup
impure subroutine, public s_ibm_setup()
Initializes the values of various IBM variables, such as ghost points and image points.
Definition m_ibm.fpp.f90:763
m_ibm::ib_markers
type(integer_field), public ib_markers
Definition m_ibm.fpp.f90:392
m_ibm::s_initialize_ibm_module
impure subroutine, public s_initialize_ibm_module()
Allocates memory for the variables in the IBM module.
Definition m_ibm.fpp.f90:478
m_igr
Iterative ghost rasterization (IGR) for sharp immersed boundary treatment.
Definition m_igr.fpp.f90:367
m_igr::s_initialize_igr_module
subroutine, public s_initialize_igr_module()
Initialize the IGR module.
Definition m_igr.fpp.f90:502
m_igr::j
integer(kind=8) j
Definition m_igr.fpp.f90:480
m_igr::i
integer(kind=8) i
Definition m_igr.fpp.f90:480
m_igr::l
integer(kind=8) l
Definition m_igr.fpp.f90:480
m_igr::r
integer(kind=8) r
Definition m_igr.fpp.f90:480
m_igr::k
integer(kind=8) k
Definition m_igr.fpp.f90:480
m_model
Binary STL file reader and processor for immersed boundary geometry.
Definition m_model.fpp.f90:357
m_model::s_instantiate_stl_models
subroutine, public s_instantiate_stl_models()
Load, transform, and register STL/OBJ immersed-boundary models onto the simulation grid.
Definition m_model.fpp.f90:1303
m_mpi_common
MPI communication layer: domain decomposition, halo exchange, reductions, and parallel I/O setup.
Definition m_mpi_common.fpp.f90:367
m_mpi_common::s_mpi_abort
impure subroutine s_mpi_abort(prnt, code)
The subroutine terminates the MPI execution environment.
Definition m_mpi_common.fpp.f90:1007
m_mpi_common::s_mpi_barrier
impure subroutine s_mpi_barrier
Halts all processes until all have reached barrier.
Definition m_mpi_common.fpp.f90:1056
m_mpi_common::s_check_mpi_file_open
impure subroutine s_check_mpi_file_open(ierr, file_loc)
Abort naming the file if MPI_FILE_OPEN failed. File errors default to MPI_ERRORS_RETURN,...
Definition m_mpi_common.fpp.f90:1039
m_mpi_common::s_apply_grid_from_global_dim
subroutine s_apply_grid_from_global_dim(x_cb_glb, m_dim_glb, m_dim, sidx, bc_beg, bc_end, cb_lo, cb_hi, cw_lo, cw_hi, x_cb_loc, x_cc_loc, dx_loc)
Populate the local cell-boundary, cell-center, and cell-width arrays in one direction directly from t...
Definition m_mpi_common.fpp.f90:3567
m_mpi_common::s_initialize_mpi_data
impure subroutine s_initialize_mpi_data(q_cons_vf, ib_markers, ib_mpi_data, beta, qbmm_pb, qbmm_mv)
Set up MPI I/O data views and variable pointers for parallel file output.
Definition m_mpi_common.fpp.f90:616
m_mpi_common::mpi_bcast_time_step_values
impure subroutine mpi_bcast_time_step_values(proc_time, time_avg)
Gather per-rank time step wall-clock times onto rank 0 for performance reporting.
Definition m_mpi_common.fpp.f90:766
m_mpi_common::s_initialize_mpi_data_ds
subroutine s_initialize_mpi_data_ds(m_ds, n_ds, p_ds, q_cons_vf)
Set up MPI I/O data views for downsampled (coarsened) parallel file output.
Definition m_mpi_common.fpp.f90:699
m_mpi_common::s_initialize_mpi_common_module
impure subroutine s_initialize_mpi_common_module(exchange_all_chemistry_temperatures_in, use_rdma_transport_in)
Initialize the module.
Definition m_mpi_common.fpp.f90:454
m_mpi_proxy
MPI halo exchange, domain decomposition, and buffer packing/unpacking for the simulation solver.
Definition m_mpi_proxy.fpp.f90:367
m_mpi_proxy::s_initialize_mpi_proxy_module
subroutine s_initialize_mpi_proxy_module()
Initialize the MPI proxy module.
Definition m_mpi_proxy.fpp.f90:426
m_muscl
MUSCL reconstruction with interface sharpening for contact-preserving advection.
Definition m_muscl.fpp.f90:356
m_muscl::s_initialize_muscl_module
subroutine, public s_initialize_muscl_module()
Allocate and initialize MUSCL reconstruction working arrays.
Definition m_muscl.fpp.f90:423
m_nvtx
NVIDIA NVTX profiling API bindings for GPU performance instrumentation.
Definition m_nvtx.f90:6
m_nvtx::nvtxstartrange
subroutine nvtxstartrange(name, id)
Push a named NVTX range for GPU profiling, optionally with a color based on the given identifier.
Definition m_nvtx.f90:62
m_nvtx::nvtxendrange
subroutine nvtxendrange
Pop the current NVTX range to end the GPU profiling region.
Definition m_nvtx.f90:83
m_phase_change
Phase transition relaxation solvers for liquid-vapor flows with cavitation and boiling.
Definition m_phase_change.fpp.f90:367
m_phase_change::s_infinite_relaxation_k
subroutine, public s_infinite_relaxation_k(q_cons_vf)
Apply pT- or pTg-equilibrium relaxation with mass depletion based on the incoming state conditions.
Definition m_phase_change.fpp.f90:420
m_phase_change::s_initialize_phasechange_module
impure subroutine, public s_initialize_phasechange_module
Initialize the phase change module (no module-level state to set up; the pT/pTg relaxation solvers ar...
Definition m_phase_change.fpp.f90:415
m_pressure_relaxation
Pressure relaxation for the six-equation multi-component model via Newton–Raphson equilibration and v...
Definition m_pressure_relaxation.fpp.f90:368
m_pressure_relaxation::s_report_pressure_relaxation
impure subroutine, public s_report_pressure_relaxation
One line at the end of a run if the equilibration ever stopped on its iteration cap....
Definition m_pressure_relaxation.fpp.f90:495
m_qbmm
Quadrature-based moment methods (QBMM) for polydisperse bubble moment inversion and transport.
Definition m_qbmm.fpp.f90:367
m_qbmm::s_initialize_qbmm_module
impure subroutine, public s_initialize_qbmm_module
Initialize the QBMM module.
Definition m_qbmm.fpp.f90:444
m_rhs
Assemble the right-hand side of the governing equations using finite-volume flux differencing,...
Definition m_rhs.fpp.f90:368
m_rhs::s_initialize_rhs_module
impure subroutine, public s_initialize_rhs_module
Initialize the RHS module.
Definition m_rhs.fpp.f90:709
m_riemann_solvers
Approximate and exact Riemann solvers (HLL, HLLC, HLLD, exact) for the multicomponent Navier–Stokes e...
Definition m_riemann_solvers.fpp.f90:368
m_riemann_solvers::s_initialize_riemann_solvers_module
impure subroutine, public s_initialize_riemann_solvers_module
Initialize the Riemann solvers module.
Definition m_riemann_solvers.fpp.f90:460
m_sim_helpers
Simulation helper routines for cell state, CFL calculation, and stability checks.
Definition m_sim_helpers.fpp.f90:367
m_sim_helpers::dt_limiter
character(len=4), public dt_limiter
Criterion currently limiting the adaptive time step (ICFL, VCFL, CCFL, TCFL, the collision cap,...
Definition m_sim_helpers.fpp.f90:378
m_start_up
Reads input files, loads initial conditions and grid data, and orchestrates solver initialization and...
Definition m_start_up.fpp.f90:367
m_start_up::s_read_ib_restart_data
impure subroutine s_read_ib_restart_data(t_step)
Loads the IBs this rank owns from the IB state file for t_step into patch_ib(1:num_ibs),...
Definition m_start_up.fpp.f90:2135
m_start_up::s_read_serial_data_files
impure subroutine, public s_read_serial_data_files(q_cons_vf)
Read serial initial condition and grid data files and compute cell-width distributions.
Definition m_start_up.fpp.f90:545
m_start_up::s_initialize_modules
impure subroutine, public s_initialize_modules
Initialize all simulation sub-modules in the required dependency order.
Definition m_start_up.fpp.f90:1380
m_start_up::s_read_data_files
impure subroutine, public s_read_data_files(q_cons_vf)
Read data files. Dispatch subroutine that replaces procedure pointer.
Definition m_start_up.fpp.f90:437
m_start_up::s_assign_particle_cloud_ib_defaults
subroutine s_assign_particle_cloud_ib_defaults(cloud_ib_idx, ib_patch)
Fills the properties of a generated particle-cloud IB that the IB state file does not carry (geometry...
Definition m_start_up.fpp.f90:2084
m_start_up::s_read_parallel_data_files
impure subroutine, public s_read_parallel_data_files(q_cons_vf)
Read parallel initial condition and grid data files via MPI I/O.
Definition m_start_up.fpp.f90:662
m_start_up::s_initialize_internal_energy_equations
subroutine, public s_initialize_internal_energy_equations(v_vf)
Initialize internal-energy equations from phase mass, mixture momentum, and total energy.
Definition m_start_up.fpp.f90:937
m_start_up::s_save_performance_metrics
impure subroutine, public s_save_performance_metrics(time_avg, time_final, io_time_avg, io_time_final, proc_time, io_proc_time, file_exists)
Collect per-process wall-clock times and write aggregate performance metrics to file.
Definition m_start_up.fpp.f90:1125
m_start_up::s_compute_ib_neighbor_ranks
subroutine s_compute_ib_neighbor_ranks()
Build ib_neighbor_ranks(-1:1,-1:1,-1:1): MPI ranks of all neighbor domains. Uses two rounds of MPI_SE...
Definition m_start_up.fpp.f90:2284
m_start_up::s_save_data
impure subroutine, public s_save_data(t_step, start, finish, io_time_avg, nt)
Save conservative variable data to disk at the current time step.
Definition m_start_up.fpp.f90:1186
m_start_up::s_get_neighbor_bounds
subroutine s_get_neighbor_bounds()
Definition m_start_up.fpp.f90:2495
m_start_up::q_cons_temp
type(scalar_field), dimension(:), allocatable q_cons_temp
Definition m_start_up.fpp.f90:430
m_start_up::dt_init
real(wp) dt_init
Definition m_start_up.fpp.f90:431
m_start_up::s_initialize_gpu_vars
subroutine, public s_initialize_gpu_vars
Transfer initial conservative variable and model parameter data to the GPU device.
Definition m_start_up.fpp.f90:1610
m_start_up::s_check_restart_file_size
impure subroutine s_check_restart_file_size(ifile, file_loc, expected_bytes)
Abort if an open restart file holds fewer bytes than are about to be read from it,...
Definition m_start_up.fpp.f90:916
m_start_up::s_initialize_mpi_domain
impure subroutine, public s_initialize_mpi_domain
Set up the MPI execution environment, bind GPUs, and decompose the computational domain.
Definition m_start_up.fpp.f90:1535
m_start_up::s_finalize_modules
impure subroutine, public s_finalize_modules
Finalize and deallocate all simulation sub-modules in reverse initialization order.
Definition m_start_up.fpp.f90:2035
m_start_up::s_read_input_file
impure subroutine, public s_read_input_file
Verify the input file exists and read it.
Definition m_start_up.fpp.f90:450
m_start_up::s_check_input_file
impure subroutine, public s_check_input_file
Validate that all user-provided inputs form a consistent simulation configuration.
Definition m_start_up.fpp.f90:526
m_start_up::s_build_ib_neighborhood
subroutine s_build_ib_neighborhood()
Completes this rank's IB neighborhood once s_read_ib_restart_data has loaded only the IBs it owns: ev...
Definition m_start_up.fpp.f90:2214
m_start_up::s_perform_time_step
impure subroutine, public s_perform_time_step(t_step, time_avg)
Advance the simulation by one time step, handling CFL-based dt and time-stepper dispatch.
Definition m_start_up.fpp.f90:997
m_surface_tension
Computes capillary source fluxes and color-function gradients for the diffuse-interface surface tensi...
Definition m_surface_tension.fpp.f90:367
m_surface_tension::s_initialize_surface_tension_module
impure subroutine, public s_initialize_surface_tension_module
Allocate and initialize surface tension module arrays.
Definition m_surface_tension.fpp.f90:435
m_thinc
THINC and MTHINC interface compression for volume fraction sharpening. THINC (int_comp=1): 1D directi...
Definition m_thinc.fpp.f90:362
m_thinc::s_initialize_thinc_module
subroutine, public s_initialize_thinc_module()
Definition m_thinc.fpp.f90:657
m_thinc::position
real(wp), dimension(:,:,:), allocatable position
Definition m_thinc.fpp.f90:396
m_time_steppers
Total-variation-diminishing (TVD) Runge–Kutta time integrators (1st-, 2nd-, and 3rd-order SSP).
Definition m_time_steppers.fpp.f90:367
m_time_steppers::q_t_sf
type(scalar_field) q_t_sf
Cell-average temperature variables at the current time-stage.
Definition m_time_steppers.fpp.f90:402
m_time_steppers::bc_type
type(integer_field), dimension(:,:), allocatable bc_type
Boundary condition identifiers.
Definition m_time_steppers.fpp.f90:398
m_time_steppers::s_initialize_time_steppers_module
impure subroutine s_initialize_time_steppers_module
Initialize the time steppers module.
Definition m_time_steppers.fpp.f90:437
m_time_steppers::q_cons_ts
type(vector_field), dimension(:), allocatable q_cons_ts
Cell-average conservative variables at each time-stage (TS).
Definition m_time_steppers.fpp.f90:395
m_time_steppers::q_prim_vf
type(scalar_field), dimension(:), allocatable q_prim_vf
Cell-average primitive variables at the current time-stage.
Definition m_time_steppers.fpp.f90:396
m_time_steppers::s_compute_dt
impure subroutine s_compute_dt()
Compute the global time step size from CFL stability constraints across all cells.
Definition m_time_steppers.fpp.f90:5146
m_time_steppers::s_tvd_rk
impure subroutine s_tvd_rk(t_step, time_avg, nstage)
Advance the solution one full step using a TVD Runge-Kutta time integrator.
Definition m_time_steppers.fpp.f90:4830
m_time_steppers::stor
integer stor
storage index
Definition m_time_steppers.fpp.f90:405
m_variables_conversion
Conservative-to-primitive variable conversion, mixture property evaluation, and pressure computation.
Definition m_variables_conversion.fpp.f90:367
m_variables_conversion::s_compute_pressure
subroutine, public s_compute_pressure(energy, alf, dyn_p, pi_inf, gamma, rho, qv, rhoyks, pres, t, e_e_in, pres_mag)
Compute the pressure from the appropriate equation of state.
Definition m_variables_conversion.fpp.f90:464
m_variables_conversion::s_initialize_variables_conversion_module
impure subroutine, public s_initialize_variables_conversion_module(store_mixture_fields, enforce_density_floor, preserve_qbmm_number, lagrange_beta_index)
Initialize the variables conversion module.
Definition m_variables_conversion.fpp.f90:697
m_variables_conversion::s_convert_to_mixture_variables
subroutine, public s_convert_to_mixture_variables(q_vf, i, j, k, rho, gamma, pi_inf, qv, re_k, g_k, g)
Dispatch to the s_convert_mixture_to_mixture_variables and s_convert_species_to_mixture_variables sub...
Definition m_variables_conversion.fpp.f90:446
m_viscous
Computes viscous stress tensors and diffusive flux contributions for the Navier–Stokes equations.
Definition m_viscous.fpp.f90:368
m_viscous::s_initialize_viscous_module
impure subroutine, public s_initialize_viscous_module
Initialize the viscous module.
Definition m_viscous.fpp.f90:416
m_weno
WENO/WENO-Z/TENO reconstructions with optional monotonicity-preserving bounds and mapped weights.
Definition m_weno.fpp.f90:366
m_weno::s_initialize_weno_module
impure subroutine, public s_initialize_weno_module
Initialize the WENO module.
Definition m_weno.fpp.f90:531
m_derived_types::int_bounds_info
Integer bounds for variables.
Definition m_derived_types.fpp.f90:450
m_derived_types::scalar_field
Derived type annexing a scalar field (SF).
Definition m_derived_types.fpp.f90:370