MFC
Exascale flow solver
Loading...
Searching...
No Matches
m_start_up.fpp.f90
Go to the documentation of this file.
1# 1 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
2!>
3!! @file
4!! @brief Contains module m_start_up
5
6# 1 "/home/runner/work/MFC/MFC/src/common/include/case.fpp" 1
7! This file exists so that Fypp can be run without generating case.fpp files for
8! each target. This is useful when generating documentation, for example. This
9! should also let MFC be built with CMake directly, without invoking mfc.sh.
10
11! For pre-process.
12# 9 "/home/runner/work/MFC/MFC/src/common/include/case.fpp"
13
14! For moving immersed boundaries in simulation
15# 14 "/home/runner/work/MFC/MFC/src/common/include/case.fpp"
16# 6 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp" 2
17# 1 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp" 1
18# 1 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 1
19# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
20# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
21# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
22# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
23# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
24# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
25
26# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
27# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
28# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
29
30# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
31
32# 46 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
33
34# 58 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
35
36# 68 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
37
38# 98 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
39
40# 110 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
41
42# 120 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
43! New line at end of file is required for FYPP
44# 2 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
45# 1 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp" 1
46# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
47# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
48# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
49# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
50# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
51# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
52
53# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
54# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
55# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
56
57# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
58
59# 46 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
60
61# 58 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
62
63# 68 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
64
65# 98 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
66
67# 110 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
68
69# 120 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
70! New line at end of file is required for FYPP
71# 2 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp" 2
72
73# 4 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
74# 5 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
75# 6 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
76# 7 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
77# 8 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
78
79# 20 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
80
81# 43 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
82
83# 48 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
84
85# 53 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
86
87# 58 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
88
89# 63 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
90
91# 68 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
92
93# 76 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
94
95# 81 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
96
97# 86 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
98
99# 91 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
100
101# 96 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
102
103# 101 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
104
105# 106 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
106
107# 111 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
108
109# 116 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
110
111# 121 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
112
113# 151 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
114
115# 192 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
116
117# 206 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
118
119# 231 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
120
121# 242 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
122
123# 244 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
124# 255 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
125
126# 284 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
127
128# 294 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
129
130# 304 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
131
132# 313 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
133
134# 330 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
135
136# 340 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
137
138# 347 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
139
140# 353 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
141
142# 359 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
143
144# 365 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
145
146# 371 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
147
148# 377 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
149! New line at end of file is required for FYPP
150# 3 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
151# 1 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp" 1
152# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
153# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
154# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
155# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
156# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
157# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
158
159# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
160# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
161# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
162
163# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
164
165# 46 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
166
167# 58 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
168
169# 68 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
170
171# 98 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
172
173# 110 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
174
175# 120 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
176! New line at end of file is required for FYPP
177# 2 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp" 2
178
179# 7 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
180
181# 17 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
182
183# 22 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
184
185# 27 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
186
187# 32 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
188
189# 37 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
190
191# 42 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
192
193# 47 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
194
195# 52 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
196
197# 57 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
198
199# 62 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
200
201# 73 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
202
203# 78 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
204
205# 83 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
206
207# 88 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
208
209# 103 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
210
211# 131 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
212
213# 160 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
214
215# 175 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
216
217# 193 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
218
219# 215 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
220
221# 244 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
222
223# 259 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
224
225# 269 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
226
227# 278 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
228
229# 294 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
230
231# 304 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
232
233# 311 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
234! New line at end of file is required for FYPP
235# 4 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
236
237! GPU parallel region (scalar reductions, maxval/minval)
238# 23 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
239
240! GPU parallel loop over threads (most common GPU macro)
241# 43 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
242
243! Required closing for GPU_PARALLEL_LOOP
244# 55 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
245
246! Mark routine for device compilation
247# 112 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
248
249! Declare device-resident data
250# 130 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
251
252! Inner loop within a GPU parallel region
253# 145 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
254
255! Scoped GPU data region
256# 164 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
257
258! Host code with device pointers (for MPI with GPU buffers)
259# 193 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
260
261! Allocate device memory (unscoped)
262# 207 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
263
264! Free device memory
265# 219 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
266
267! Atomic operation on device
268# 231 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
269
270! End atomic capture block
271# 242 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
272
273! Copy data between host and device
274# 254 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
275
276! Synchronization barrier
277# 266 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
278
279! Import GPU library module (openacc or omp_lib)
280# 275 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
281
282! Emit code only for AMD compiler
283# 282 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
284
285! Emit code for non-Cray compilers
286# 289 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
287
288! Emit code only for Cray compiler
289# 296 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
290
291! Emit code for non-NVIDIA compilers
292# 303 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
293
294# 305 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
295# 306 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
296! New line at end of file is required for FYPP
297# 2 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp" 2
298
299# 14 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
300
301! Caution: This macro requires the use of a binding script to set CUDA_VISIBLE_DEVICES, such that we have one GPU device per MPI
302! rank. That's because for both cudaMemAdvise (preferred location) and cudaMemPrefetchAsync we use location = device_id = 0. For an
303! example see misc/nvidia_uvm/bind.sh. NVIDIA unified memory page placement hint
304# 57 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
305
306! Allocate and create GPU device memory
307# 77 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
308
309! Free GPU device memory and deallocate
310# 85 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
311
312! Cray-specific GPU pointer setup for vector fields
313# 109 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
314
315! Cray-specific GPU pointer setup for scalar fields
316# 125 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
317
318! Cray-specific GPU pointer setup for acoustic source spatials
319# 150 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
320
321# 156 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
322
323# 163 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
324! New line at end of file is required for FYPP
325# 7 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp" 2
326
327!> @brief Reads input files, loads initial conditions and grid data, and orchestrates solver initialization and finalization
329
332 use m_mpi_proxy
333 use m_mpi_common
335 use m_weno
336 use m_muscl
338 use m_cbc
341 use m_rhs
342 use m_chemistry
343 use m_data_output
345 use m_qbmm
347 use m_hypoelastic
350 use m_viscous
351 use m_bubbles_ee
352 use m_bubbles_el
353 use ieee_arithmetic
355 use m_helper
356
357#if defined(MFC_OpenACC)
358# 38 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
359 use openacc
360# 38 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
361#elif defined(MFC_OpenMP)
362# 38 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
363 use omp_lib
364# 38 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
365#endif
366
367 use m_nvtx
368 use m_ibm
371 use m_checker
373 use m_body_forces
374 use m_sim_helpers
375 use m_igr
376
377 implicit none
378
382
383 type(scalar_field), allocatable, dimension(:) :: q_cons_temp
384 real(wp) :: dt_init
385
386contains
387
388 !> Read data files. Dispatch subroutine that replaces procedure pointer.
389 impure subroutine s_read_data_files(q_cons_vf)
390
391 type(scalar_field), dimension(sys_size), intent(inout) :: q_cons_vf
392
393 if (.not. parallel_io) then
395 else
397 end if
398
399 end subroutine s_read_data_files
400
401 !> Verify the input file exists and read it
402 impure subroutine s_read_input_file
403
404 character(LEN=name_len), parameter :: file_path = './simulation.inp'
405 logical :: file_exist !< Logical used to check the existence of the input file
406 integer :: iostatus
407 ! Integer to check iostat of file read
408
409 character(len=1000) :: line
410
411 namelist /user_inputs/ case_dir, run_time_info, m, n, p, dt, &
416 bc_x, bc_y, bc_z, &
417 x_a, y_a, z_a, x_b, y_b, z_b, &
420 ib, num_ibs, patch_ib, &
429# 103 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
434# 108 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
442
443 inquire (file=trim(file_path), exist=file_exist)
444
445 if (file_exist) then
446 open (1, file=trim(file_path), form='formatted', action='read', status='old')
447 read (1, nml=user_inputs, iostat=iostatus)
448
449 if (iostatus /= 0) then
450 backspace(1)
451 read (1, fmt='(A)') line
452 print *, 'Invalid line in namelist: ' // trim(line)
453 call s_mpi_abort('Invalid line in simulation.inp. It is ' // 'likely due to a datatype mismatch. Exiting.')
454 end if
455
456 close (1)
457
458 if ((bf_x) .or. (bf_y) .or. (bf_z)) then
459 bodyforces = .true.
460 end if
461
462 m_glb = m
463 n_glb = n
464 p_glb = p
465
467
468 if (cfl_adap_dt .or. cfl_const_dt) cfl_dt = .true.
469
470 if (any((/bc_x%beg, bc_x%end, bc_y%beg, bc_y%end, bc_z%beg, bc_z%end/) == -17) .or. num_bc_patches > 0) then
471 bc_io = .true.
472 end if
473 else
474 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
475 end if
476
477 end subroutine s_read_input_file
478
479 !> Validate that all user-provided inputs form a consistent simulation configuration
480 impure subroutine s_check_input_file
481
482 character(LEN=path_len) :: file_path
483 logical :: file_exist
484
485 file_path = trim(case_dir) // '/.'
486
487 call my_inquire(file_path, file_exist)
488
489 if (file_exist .neqv. .true.) then
490 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
491 end if
492
494 call s_check_inputs()
495
496 end subroutine s_check_input_file
497
498 !> Read serial initial condition and grid data files and compute cell-width distributions
499 impure subroutine s_read_serial_data_files(q_cons_vf)
500
501 type(scalar_field), dimension(sys_size), intent(inout) :: q_cons_vf
502 character(LEN=path_len + 2*name_len) :: t_step_dir !< Relative path to the starting time-step directory
503 character(LEN=path_len + 3*name_len) :: file_path !< Relative path to the grid and conservative variables data files
504 logical :: file_exist
505 integer :: i, r
506
507 if (cfl_dt) then
508 write (t_step_dir, '(A,I0,A,I0)') trim(case_dir) // '/p_all/p', proc_rank, '/', n_start
509 else
510 write (t_step_dir, '(A,I0,A,I0)') trim(case_dir) // '/p_all/p', proc_rank, '/', t_step_start
511 end if
512
513 file_path = trim(t_step_dir) // '/.'
514 call my_inquire(file_path, file_exist)
515
516 if (file_exist .neqv. .true.) then
517 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
518 end if
519
520 if (bc_io) then
522 else
524 end if
525
526 file_path = trim(t_step_dir) // '/x_cb.dat'
527
528 inquire (file=trim(file_path), exist=file_exist)
529
530 if (file_exist) then
531 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
532 read (2) x_cb(-1:m); close (2)
533 else
534 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
535 end if
536
537 dx(0:m) = x_cb(0:m) - x_cb(-1:m - 1)
538 x_cc(0:m) = x_cb(-1:m - 1) + dx(0:m)/2._wp
539
540 if (ib) then
541 do i = 1, num_ibs
542 if (patch_ib(i)%c > 0) then
543 np = int((patch_ib(i)%p*patch_ib(i)%c/dx(0))*20) + int(((patch_ib(i)%c - patch_ib(i)%p*patch_ib(i)%c)/dx(0)) &
544 & *20) + 1
545 end if
546 end do
547 end if
548
549 if (n > 0) then
550 file_path = trim(t_step_dir) // '/y_cb.dat'
551
552 inquire (file=trim(file_path), exist=file_exist)
553
554 if (file_exist) then
555 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
556 read (2) y_cb(-1:n); close (2)
557 else
558 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
559 end if
560
561 dy(0:n) = y_cb(0:n) - y_cb(-1:n - 1)
562 y_cc(0:n) = y_cb(-1:n - 1) + dy(0:n)/2._wp
563 end if
564
565 if (p > 0) then
566 file_path = trim(t_step_dir) // '/z_cb.dat'
567
568 inquire (file=trim(file_path), exist=file_exist)
569
570 if (file_exist) then
571 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
572 read (2) z_cb(-1:p); close (2)
573 else
574 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
575 end if
576
577 dz(0:p) = z_cb(0:p) - z_cb(-1:p - 1)
578 z_cc(0:p) = z_cb(-1:p - 1) + dz(0:p)/2._wp
579 end if
580
581 do i = 1, sys_size
582 write (file_path, '(A,I0,A)') trim(t_step_dir) // '/q_cons_vf', i, '.dat'
583 inquire (file=trim(file_path), exist=file_exist)
584 if (file_exist) then
585 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
586 read (2) q_cons_vf(i)%sf(0:m,0:n,0:p); close (2)
587 else
588 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
589 end if
590 end do
591
592 if (bubbles_euler .or. elasticity) then
593 ! Read pb and mv for non-polytropic qbmm
594 if (qbmm .and. .not. polytropic) then
595 do i = 1, nb
596 do r = 1, nnode
597 write (file_path, '(A,I0,A)') trim(t_step_dir) // '/pb', sys_size + (i - 1)*nnode + r, '.dat'
598 inquire (file=trim(file_path), exist=file_exist)
599 if (file_exist) then
600 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
601 read (2) pb_ts(1)%sf(0:m,0:n,0:p,r, i); close (2)
602 else
603 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
604 end if
605 end do
606 end do
607 do i = 1, nb
608 do r = 1, nnode
609 write (file_path, '(A,I0,A)') trim(t_step_dir) // '/mv', sys_size + (i - 1)*nnode + r, '.dat'
610 inquire (file=trim(file_path), exist=file_exist)
611 if (file_exist) then
612 open (2, file=trim(file_path), form='unformatted', action='read', status='old')
613 read (2) mv_ts(1)%sf(0:m,0:n,0:p,r, i); close (2)
614 else
615 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
616 end if
617 end do
618 end do
619 end if
620 end if
621
622 end subroutine s_read_serial_data_files
623
624 !> Read parallel initial condition and grid data files via MPI I/O
625 impure subroutine s_read_parallel_data_files(q_cons_vf)
626
627 type(scalar_field), dimension(sys_size), intent(inout) :: q_cons_vf
628
629#ifdef MFC_MPI
630 real(wp), allocatable, dimension(:) :: x_cb_glb, y_cb_glb, z_cb_glb
631 integer :: ifile, ierr, data_size
632 integer, dimension(MPI_STATUS_SIZE) :: status
633 integer(KIND=MPI_OFFSET_KIND) :: disp
634 integer(KIND=MPI_OFFSET_KIND) :: m_mok, n_mok, p_mok
635 integer(KIND=MPI_OFFSET_KIND) :: wp_mok, var_mok
636 integer(KIND=MPI_OFFSET_KIND) :: mok
637 character(LEN=path_len + 2*name_len) :: file_loc
638 logical :: file_exist
639 character(len=10) :: t_step_start_string
640 integer :: i, j
641
642 ! Downsampled data variables
643 integer :: m_ds, n_ds, p_ds
644 integer :: m_glb_ds, n_glb_ds, p_glb_ds
645 integer :: m_glb_read, n_glb_read, p_glb_read !< data size of read
646
647 allocate (x_cb_glb(-1:m_glb))
648 allocate (y_cb_glb(-1:n_glb))
649 allocate (z_cb_glb(-1:p_glb))
650
651 file_loc = trim(case_dir) // '/restart_data' // trim(mpiiofs) // 'x_cb.dat'
652 inquire (file=trim(file_loc), exist=file_exist)
653
654 if (down_sample) then
655 m_ds = int((m + 1)/3) - 1
656 n_ds = int((n + 1)/3) - 1
657 p_ds = int((p + 1)/3) - 1
658
659 m_glb_ds = int((m_glb + 1)/3) - 1
660 n_glb_ds = int((n_glb + 1)/3) - 1
661 p_glb_ds = int((p_glb + 1)/3) - 1
662 end if
663
664 if (file_exist) then
665 data_size = m_glb + 2
666 call mpi_file_open(mpi_comm_world, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
667 call mpi_file_read(ifile, x_cb_glb, data_size, mpi_p, status, ierr)
668 call mpi_file_close(ifile, ierr)
669 else
670 call s_mpi_abort('File ' // trim(file_loc) // ' is missing. Exiting.')
671 end if
672
673 x_cb(-1:m) = x_cb_glb((start_idx(1) - 1):(start_idx(1) + m))
674 dx(0:m) = x_cb(0:m) - x_cb(-1:m - 1)
675 x_cc(0:m) = x_cb(-1:m - 1) + dx(0:m)/2._wp
676
677 if (ib) then
678 do i = 1, num_ibs
679 if (patch_ib(i)%c > 0) then
680 np = int((patch_ib(i)%p*patch_ib(i)%c/dx(0))*20) + int(((patch_ib(i)%c - patch_ib(i)%p*patch_ib(i)%c)/dx(0)) &
681 & *20) + 1
682 allocate (mpi_io_airfoil_ib_data%var(1:2*np))
683 end if
684 end do
685 end if
686
687 if (n > 0) then
688 file_loc = trim(case_dir) // '/restart_data' // trim(mpiiofs) // 'y_cb.dat'
689 inquire (file=trim(file_loc), exist=file_exist)
690
691 if (file_exist) then
692 data_size = n_glb + 2
693 call mpi_file_open(mpi_comm_world, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
694 call mpi_file_read(ifile, y_cb_glb, data_size, mpi_p, status, ierr)
695 call mpi_file_close(ifile, ierr)
696 else
697 call s_mpi_abort('File ' // trim(file_loc) // ' is missing. Exiting.')
698 end if
699
700 y_cb(-1:n) = y_cb_glb((start_idx(2) - 1):(start_idx(2) + n))
701 dy(0:n) = y_cb(0:n) - y_cb(-1:n - 1)
702 y_cc(0:n) = y_cb(-1:n - 1) + dy(0:n)/2._wp
703
704 if (p > 0) then
705 file_loc = trim(case_dir) // '/restart_data' // trim(mpiiofs) // 'z_cb.dat'
706 inquire (file=trim(file_loc), exist=file_exist)
707
708 if (file_exist) then
709 data_size = p_glb + 2
710 call mpi_file_open(mpi_comm_world, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
711 call mpi_file_read(ifile, z_cb_glb, data_size, mpi_p, status, ierr)
712 call mpi_file_close(ifile, ierr)
713 else
714 call s_mpi_abort('File ' // trim(file_loc) // 'is missing. Exiting.')
715 end if
716
717 z_cb(-1:p) = z_cb_glb((start_idx(3) - 1):(start_idx(3) + p))
718 dz(0:p) = z_cb(0:p) - z_cb(-1:p - 1)
719 z_cc(0:p) = z_cb(-1:p - 1) + dz(0:p)/2._wp
720 end if
721 end if
722
723 if (file_per_process) then
724 if (cfl_dt) then
725 call s_int_to_str(n_start, t_step_start_string)
726 write (file_loc, '(I0,A1,I7.7,A)') n_start, '_', proc_rank, '.dat'
727 else
728 call s_int_to_str(t_step_start, t_step_start_string)
729 write (file_loc, '(I0,A1,I7.7,A)') t_step_start, '_', proc_rank, '.dat'
730 end if
731 file_loc = trim(case_dir) // '/restart_data/lustre_' // trim(t_step_start_string) // trim(mpiiofs) // trim(file_loc)
732 inquire (file=trim(file_loc), exist=file_exist)
733
734 if (file_exist) then
735 call mpi_file_open(mpi_comm_self, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
736
737 if (down_sample) then
739 else
740 if (ib) then
742 else
744 end if
745 end if
746
747 if (down_sample) then
748 data_size = (m_ds + 3)*(n_ds + 3)*(p_ds + 3)
749 m_glb_read = m_glb_ds + 1
750 n_glb_read = n_glb_ds + 1
751 p_glb_read = p_glb_ds + 1
752 else
753 data_size = (m + 1)*(n + 1)*(p + 1)
754 m_glb_read = m_glb + 1
755 n_glb_read = n_glb + 1
756 p_glb_read = p_glb + 1
757 end if
758
759 m_mok = int(m_glb_read + 1, mpi_offset_kind)
760 n_mok = int(m_glb_read + 1, mpi_offset_kind)
761 p_mok = int(m_glb_read + 1, mpi_offset_kind)
762 wp_mok = int(storage_size(0._stp)/8, mpi_offset_kind)
763 mok = int(1._wp, mpi_offset_kind)
764
765 if (bubbles_euler .or. elasticity) then
766 do i = 1, sys_size
767 var_mok = int(i, mpi_offset_kind)
768
769 call mpi_file_read(ifile, mpi_io_data%var(i)%sf, data_size*mpi_io_type, mpi_io_p, status, ierr)
770 end do
771 ! Read pb and mv for non-polytropic qbmm
772 if (qbmm .and. .not. polytropic) then
773 do i = sys_size + 1, sys_size + 2*nb*nnode
774 var_mok = int(i, mpi_offset_kind)
775
776 call mpi_file_read(ifile, mpi_io_data%var(i)%sf, data_size*mpi_io_type, mpi_io_p, status, ierr)
777 end do
778 end if
779 else
780 if (down_sample) then
781 do i = 1, sys_size
782 var_mok = int(i, mpi_offset_kind)
783
784 call mpi_file_read(ifile, q_cons_temp(i)%sf, data_size*mpi_io_type, mpi_io_p, status, ierr)
785 end do
786 else
787 do i = 1, sys_size
788 var_mok = int(i, mpi_offset_kind)
789
790 call mpi_file_read(ifile, mpi_io_data%var(i)%sf, data_size*mpi_io_type, mpi_io_p, status, ierr)
791 end do
792 end if
793 end if
794
795 call s_mpi_barrier()
796
797 call mpi_file_close(ifile, ierr)
798 else
799 call s_mpi_abort('File ' // trim(file_loc) // ' is missing. Exiting.')
800 end if
801 else
802 if (cfl_dt) then
803 write (file_loc, '(I0,A)') n_start, '.dat'
804 else
805 write (file_loc, '(I0,A)') t_step_start, '.dat'
806 end if
807 file_loc = trim(case_dir) // '/restart_data' // trim(mpiiofs) // trim(file_loc)
808 inquire (file=trim(file_loc), exist=file_exist)
809
810 if (file_exist) then
811 call mpi_file_open(mpi_comm_world, file_loc, mpi_mode_rdonly, mpi_info_int, ifile, ierr)
812
813 if (ib) then
815 else
817 end if
818
819 data_size = (m + 1)*(n + 1)*(p + 1)
820
821 m_mok = int(m_glb + 1, mpi_offset_kind)
822 n_mok = int(n_glb + 1, mpi_offset_kind)
823 p_mok = int(p_glb + 1, mpi_offset_kind)
824 wp_mok = int(storage_size(0._stp)/8, mpi_offset_kind)
825 mok = int(1._wp, mpi_offset_kind)
826
827 if (bubbles_euler .or. elasticity) then
828 do i = 1, sys_size
829 var_mok = int(i, mpi_offset_kind)
830 disp = m_mok*max(mok, n_mok)*max(mok, p_mok)*wp_mok*(var_mok - 1)
831
832 call mpi_file_set_view(ifile, disp, mpi_io_p, mpi_io_data%view(i), 'native', mpi_info_int, ierr)
833 call mpi_file_read(ifile, mpi_io_data%var(i)%sf, data_size*mpi_io_type, mpi_io_p, status, ierr)
834 end do
835 ! Read pb and mv for non-polytropic qbmm
836 if (qbmm .and. .not. polytropic) then
837 do i = sys_size + 1, sys_size + 2*nb*nnode
838 var_mok = int(i, mpi_offset_kind)
839 disp = m_mok*max(mok, n_mok)*max(mok, p_mok)*wp_mok*(var_mok - 1)
840
841 call mpi_file_set_view(ifile, disp, mpi_io_p, mpi_io_data%view(i), 'native', mpi_info_int, ierr)
842 call mpi_file_read(ifile, mpi_io_data%var(i)%sf, data_size*mpi_io_type, mpi_io_p, status, ierr)
843 end do
844 end if
845 else
846 do i = 1, sys_size
847 var_mok = int(i, mpi_offset_kind)
848
849 disp = m_mok*max(mok, n_mok)*max(mok, p_mok)*wp_mok*(var_mok - 1)
850
851 call mpi_file_set_view(ifile, disp, mpi_io_p, mpi_io_data%view(i), 'native', mpi_info_int, ierr)
852 call mpi_file_read_all(ifile, mpi_io_data%var(i)%sf, data_size*mpi_io_type, mpi_io_p, status, ierr)
853 end do
854 end if
855
856 call s_mpi_barrier()
857
858 call mpi_file_close(ifile, ierr)
859 else
860 call s_mpi_abort('File ' // trim(file_loc) // ' is missing. Exiting.')
861 end if
862 end if
863
864 deallocate (x_cb_glb, y_cb_glb, z_cb_glb)
865
866 if (bc_io) then
868 else
870 end if
871#endif
872
873 end subroutine s_read_parallel_data_files
874
875 !> Initialize internal-energy equations from phase mass, mixture momentum, and total energy
877
878 type(scalar_field), dimension(sys_size), intent(inout) :: v_vf
879 real(wp) :: rho
880 real(wp) :: dyn_pres
881 real(wp) :: gamma
882 real(wp) :: pi_inf
883 real(wp) :: qv
884 real(wp), dimension(2) :: re
885 real(wp) :: pres, t
886 integer :: i, j, k, l, c
887 real(wp), dimension(num_species) :: rhoyks
888 real(wp) :: pres_mag
889
890 pres_mag = 0._wp
891
892 t = dflt_t_guess
893
894 do j = 0, m
895 do k = 0, n
896 do l = 0, p
897 call s_convert_to_mixture_variables(v_vf, j, k, l, rho, gamma, pi_inf, qv, re)
898
899 dyn_pres = 0._wp
900 do i = eqn_idx%mom%beg, eqn_idx%mom%end
901 dyn_pres = dyn_pres + 5.e-1_wp*v_vf(i)%sf(j, k, l)*v_vf(i)%sf(j, k, l)/max(rho, sgm_eps)
902 end do
903
904 if (chemistry) then
905 do c = 1, num_species
906 rhoyks(c) = v_vf(eqn_idx%species%beg + c - 1)%sf(j, k, l)
907 end do
908 end if
909
910 if (mhd) then
911 if (n == 0) then
912 pres_mag = 0.5_wp*(bx0**2 + v_vf(eqn_idx%B%beg)%sf(j, k, l)**2 + v_vf(eqn_idx%B%beg + 1)%sf(j, k, l)**2)
913 else
914 pres_mag = 0.5_wp*(v_vf(eqn_idx%B%beg)%sf(j, k, l)**2 + v_vf(eqn_idx%B%beg + 1)%sf(j, k, &
915 & l)**2 + v_vf(eqn_idx%B%beg + 2)%sf(j, k, l)**2)
916 end if
917 end if
918
919 call s_compute_pressure(v_vf(eqn_idx%E)%sf(j, k, l), 0._stp, dyn_pres, pi_inf, gamma, rho, qv, rhoyks, pres, &
920 & t, pres_mag=pres_mag)
921
922 do i = 1, num_fluids
923 v_vf(i + eqn_idx%int_en%beg - 1)%sf(j, k, l) = v_vf(i + eqn_idx%adv%beg - 1)%sf(j, k, &
924 & l)*(gammas(i)*pres + pi_infs(i)) + v_vf(i + eqn_idx%cont%beg - 1)%sf(j, k, l)*qvs(i)
925 end do
926 end do
927 end do
928 end do
929
931
932 !> Advance the simulation by one time step, handling CFL-based dt and time-stepper dispatch
933 impure subroutine s_perform_time_step(t_step, time_avg)
934
935 integer, intent(inout) :: t_step
936 real(wp), intent(inout) :: time_avg
937 integer :: i, eta_hh, eta_mm, eta_ss
938 real(wp) :: eta_sec
939
940 if (cfl_dt) then
941 if (cfl_const_dt .and. t_step == 0) call s_compute_dt()
942
943 if (cfl_adap_dt) call s_compute_dt()
944
945 if (t_step == 0) dt_init = dt
946
947 if (dt < 1.e-3_wp*dt_init .and. cfl_adap_dt .and. proc_rank == 0) then
948 print *, "Delta t = ", dt
949 call s_mpi_abort("Delta t has become too small")
950 end if
951 end if
952
953 if (cfl_dt) then
954 if ((mytime + dt) >= t_stop) then
955 dt = t_stop - mytime
956
957# 629 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
958#if defined(MFC_OpenACC)
959# 629 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
960!$acc update device(dt)
961# 629 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
962#elif defined(MFC_OpenMP)
963# 629 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
964!$omp target update to(dt)
965# 629 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
966#endif
967 end if
968 else
969 if ((mytime + dt) >= finaltime) then
971
972# 634 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
973#if defined(MFC_OpenACC)
974# 634 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
975!$acc update device(dt)
976# 634 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
977#elif defined(MFC_OpenMP)
978# 634 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
979!$omp target update to(dt)
980# 634 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
981#endif
982 end if
983 end if
984
985 if (cfl_dt) then
986 if (proc_rank == 0 .and. mod(t_step - t_step_start, t_step_print) == 0) then
987 eta_sec = wall_time_avg*(t_stop - mytime)/max(dt, tiny(dt))
988 eta_hh = int(eta_sec)/3600
989 eta_mm = mod(int(eta_sec), 3600)/60
990 eta_ss = mod(int(eta_sec), 60)
991 print '(" [", I3, "%] Time ", ES16.6, " dt = ", ES16.6, " @ Time Step = ", I8, " Time Avg = ", ES16.6, " Time/step = ", ES12.6, " ETA (HH:MM:SS) = ", I0, ":", I2.2, ":", I2.2)', &
992 & int(ceiling(100._wp*(mytime/t_stop))), mytime, dt, t_step, wall_time_avg, wall_time, eta_hh, eta_mm, eta_ss
993 end if
994 else
995 if (proc_rank == 0 .and. mod(t_step - t_step_start, t_step_print) == 0) then
996 eta_sec = wall_time_avg*real(t_step_stop - t_step, wp)
997 eta_hh = int(eta_sec)/3600
998 eta_mm = mod(int(eta_sec), 3600)/60
999 eta_ss = mod(int(eta_sec), 60)
1000 print '(" [", I3, "%] Time step ", I8, " of ", I0, " @ t_step = ", I8, " Time Avg = ", ES12.6, " Time/step= ", ES12.6, " ETA (HH:MM:SS) = ", I0, ":", I2.2, ":", I2.2)', &
1001 & int(ceiling(100._wp*(real(t_step - t_step_start)/(t_step_stop - t_step_start + 1)))), &
1002 & t_step - t_step_start + 1, t_step_stop - t_step_start + 1, t_step, wall_time_avg, wall_time, eta_hh, &
1003 & eta_mm, eta_ss
1004 end if
1005 end if
1006
1007 if (probe_wrt) then
1008 do i = 1, sys_size
1009
1010# 662 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1011#if defined(MFC_OpenACC)
1012# 662 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1013!$acc update host(q_cons_ts(1)%vf(i)%sf)
1014# 662 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1015#elif defined(MFC_OpenMP)
1016# 662 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1017!$omp target update from(q_cons_ts(1)%vf(i)%sf)
1018# 662 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1019#endif
1020 end do
1021 end if
1022
1023 ! Total-variation-diminishing (TVD) Runge-Kutta (RK) time-steppers
1024 if (any(time_stepper == (/1, 2, 3/))) then
1025 call s_tvd_rk(t_step, time_avg, time_stepper)
1026 end if
1027
1028 ! Advance time after RK so source terms see current-step time
1029 mytime = mytime + dt
1030
1031 if (relax) call s_infinite_relaxation_k(q_cons_ts(1)%vf)
1032
1033 ! Time-stepping loop controls
1034 t_step = t_step + 1
1035
1036 end subroutine s_perform_time_step
1037
1038 !> Collect per-process wall-clock times and write aggregate performance metrics to file
1039 impure subroutine s_save_performance_metrics(time_avg, time_final, io_time_avg, io_time_final, proc_time, io_proc_time, &
1040
1041 & file_exists)
1042
1043 real(wp), intent(inout) :: time_avg, time_final
1044 real(wp), intent(inout) :: io_time_avg, io_time_final
1045 real(wp), dimension(:), intent(inout) :: proc_time
1046 real(wp), dimension(:), intent(inout) :: io_proc_time
1047 logical, intent(inout) :: file_exists
1048 real(wp) :: grind_time
1049
1050 call s_mpi_barrier()
1051
1052 if (num_procs > 1) then
1053 call mpi_bcast_time_step_values(proc_time, time_avg)
1054
1055 call mpi_bcast_time_step_values(io_proc_time, io_time_avg)
1056 end if
1057
1058 if (proc_rank == 0) then
1059 time_final = 0._wp
1060 io_time_final = 0._wp
1061 if (num_procs == 1) then
1062 time_final = time_avg
1063 io_time_final = io_time_avg
1064 else
1065 time_final = maxval(proc_time)
1066 io_time_final = maxval(io_proc_time)
1067 end if
1068
1069 grind_time = time_final*1.0e9_wp/(real(sys_size, wp)*real(maxval((/1, m_glb/)), wp)*real(maxval((/1, n_glb/)), &
1070 & wp)*real(maxval((/1, p_glb/)), wp))
1071
1072 print *, "Performance:", grind_time, "ns/gp/eq/rhs"
1073 inquire (file='time_data.dat', exist=file_exists)
1074 if (file_exists) then
1075 open (1, file='time_data.dat', position='append', status='old')
1076 else
1077 open (1, file='time_data.dat', status='new')
1078 write (1, '(A10, A15, A15)') "Ranks", "s/step", "ns/gp/eq/rhs"
1079 end if
1080
1081 write (1, '(I10, 2(F15.8))') num_procs, time_final, grind_time
1082
1083 close (1)
1084
1085 inquire (file='io_time_data.dat', exist=file_exists)
1086 if (file_exists) then
1087 open (1, file='io_time_data.dat', position='append', status='old')
1088 else
1089 open (1, file='io_time_data.dat', status='new')
1090 write (1, '(A10, A15)') "Ranks", "s/step"
1091 end if
1092
1093 write (1, '(I10, F15.8)') num_procs, io_time_final
1094 close (1)
1095 end if
1096
1097 end subroutine s_save_performance_metrics
1098
1099 !> Save conservative variable data to disk at the current time step
1100 impure subroutine s_save_data(t_step, start, finish, io_time_avg, nt)
1101
1102 integer, intent(inout) :: t_step
1103 real(wp), intent(inout) :: start, finish, io_time_avg
1104 integer, intent(inout) :: nt
1105 integer(kind=8) :: i, j, k, l
1106 integer :: stor
1107 integer :: save_count
1108
1109 if (down_sample) then
1111 end if
1112
1113 stor = 1
1114
1115 if (time_stepper /= 1) then
1116
1117# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1118
1119# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1120#if defined(MFC_OpenACC)
1121# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1122!$acc parallel loop collapse(4) gang vector default(present) copyin(idwbuff)
1123# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1124#elif defined(MFC_OpenMP)
1125# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1126
1127# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1128
1129# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1130
1131# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1132!$omp target teams loop defaultmap(firstprivate:scalar) bind(teams,parallel) collapse(4) defaultmap(tofrom:aggregate) defaultmap(tofrom:allocatable) defaultmap(tofrom:pointer) map(to:idwbuff)
1133# 759 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1134#endif
1135 do i = 1, sys_size
1136 do l = idwbuff(3)%beg, idwbuff(3)%end
1137 do k = idwbuff(2)%beg, idwbuff(2)%end
1138 do j = idwbuff(1)%beg, idwbuff(1)%end
1139 q_cons_ts(2)%vf(i)%sf(j, k, l) = q_cons_ts(1)%vf(i)%sf(j, k, l)
1140 end do
1141 end do
1142 end do
1143 end do
1144
1145# 769 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1146#if defined(MFC_OpenACC)
1147# 769 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1148!$acc end parallel loop
1149# 769 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1150#elif defined(MFC_OpenMP)
1151# 769 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1152
1153# 769 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1154!$omp end target teams loop
1155# 769 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1156#endif
1157 stor = 2
1158 end if
1159
1160 call cpu_time(start)
1161 call nvtxstartrange("SAVE-DATA")
1162 do i = 1, sys_size
1163#ifndef FRONTIER_UNIFIED
1164
1165# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1166#if defined(MFC_OpenACC)
1167# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1168!$acc update host(q_cons_ts(stor)%vf(i)%sf)
1169# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1170#elif defined(MFC_OpenMP)
1171# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1172!$omp target update from(q_cons_ts(stor)%vf(i)%sf)
1173# 777 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1174#endif
1175#endif
1176 do l = 0, p
1177 do k = 0, n
1178 do j = 0, m
1179 if (ieee_is_nan(real(q_cons_ts(stor)%vf(i)%sf(j, k, l), kind=wp))) then
1180 print *, "NaN(s) in timestep output.", j, k, l, i, proc_rank, t_step, m, n, p
1181 call s_mpi_abort("NaN(s) in timestep output.")
1182 end if
1183 end do
1184 end do
1185 end do
1186 end do
1187
1188 if (qbmm .and. .not. polytropic) then
1189
1190# 792 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1191#if defined(MFC_OpenACC)
1192# 792 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1193!$acc update host(pb_ts(1)%sf)
1194# 792 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1195#elif defined(MFC_OpenMP)
1196# 792 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1197!$omp target update from(pb_ts(1)%sf)
1198# 792 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1199#endif
1200
1201# 793 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1202#if defined(MFC_OpenACC)
1203# 793 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1204!$acc update host(mv_ts(1)%sf)
1205# 793 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1206#elif defined(MFC_OpenMP)
1207# 793 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1208!$omp target update from(mv_ts(1)%sf)
1209# 793 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1210#endif
1211 end if
1212
1213 if (cfl_dt) then
1214 save_count = int(mytime/t_save)
1215 else
1216 save_count = t_step
1217 end if
1218
1219 if (bubbles_lagrange) then
1220
1221# 803 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1222#if defined(MFC_OpenACC)
1223# 803 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1224!$acc update host(lag_id, mtn_pos, mtn_posPrev, mtn_vel, intfc_rad, intfc_vel, bub_R0, Rmax_stats, Rmin_stats, bub_dphidt, gas_p, gas_mv, gas_mg, gas_betaT, gas_betaC)
1225# 803 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1226#elif defined(MFC_OpenMP)
1227# 803 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1228!$omp target update from(lag_id, mtn_pos, mtn_posPrev, mtn_vel, intfc_rad, intfc_vel, bub_R0, Rmax_stats, Rmin_stats, bub_dphidt, gas_p, gas_mv, gas_mg, gas_betaT, gas_betaC)
1229# 803 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1230#endif
1231# 805 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1232 do i = 1, nbubs
1233 if (ieee_is_nan(intfc_rad(i, 1)) .or. intfc_rad(i, 1) <= 0._wp) then
1234 call s_mpi_abort("Bubble radius is negative or NaN, please reduce dt.")
1235 end if
1236 end do
1237
1238
1239# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1240#if defined(MFC_OpenACC)
1241# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1242!$acc update host(q_beta(1)%sf)
1243# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1244#elif defined(MFC_OpenMP)
1245# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1246!$omp target update from(q_beta(1)%sf)
1247# 811 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1248#endif
1249 call s_write_data_files(q_cons_ts(stor)%vf, q_t_sf, q_prim_vf, save_count, bc_type, q_beta(1))
1250
1251# 813 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1252#if defined(MFC_OpenACC)
1253# 813 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1254!$acc update host(Rmax_stats, Rmin_stats, gas_p, gas_mv, intfc_vel)
1255# 813 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1256#elif defined(MFC_OpenMP)
1257# 813 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1258!$omp target update from(Rmax_stats, Rmin_stats, gas_p, gas_mv, intfc_vel)
1259# 813 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1260#endif
1261 call s_write_restart_lag_bubbles(save_count) ! parallel
1262 if (lag_params%write_bubbles_stats) call s_write_lag_bubble_stats()
1263 else
1264 call s_write_data_files(q_cons_ts(stor)%vf, q_t_sf, q_prim_vf, save_count, bc_type)
1265 end if
1266
1267 ! Write IB kinematic state for restart
1268 if (ib) call s_write_ib_state_file(t_step)
1269
1270 call nvtxendrange
1271 call cpu_time(finish)
1272 if (cfl_dt) then
1273 nt = mytime/t_save
1274 else
1275 nt = int((t_step - t_step_start)/(t_step_save))
1276 end if
1277
1278 if (nt == 1) then
1279 io_time_avg = abs(finish - start)
1280 else
1281 io_time_avg = (abs(finish - start) + io_time_avg*(nt - 1))/nt
1282 end if
1283
1284 end subroutine s_save_data
1285
1286 !> Initialize all simulation sub-modules in the required dependency order
1287 impure subroutine s_initialize_modules
1288
1289 integer :: m_ds, n_ds, p_ds
1290 integer :: i
1291
1293# 856 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1294 if (bubbles_euler .or. bubbles_lagrange) then
1296 end if
1300 if (grid_geometry == 3) call s_initialize_fftw_module()
1301
1303 if (ib) call s_initialize_ibm_module()
1304 if (qbmm) call s_initialize_qbmm_module()
1305
1306 if (acoustic_source) then
1308 end if
1309
1310 if (viscous .and. (.not. igr)) then
1312 end if
1313
1315
1317
1319
1323
1325
1326 if (down_sample) then
1327 m_ds = int((m + 1)/3) - 1
1328 n_ds = int((n + 1)/3) - 1
1329 p_ds = int((p + 1)/3) - 1
1330
1331 allocate (q_cons_temp(1:sys_size))
1332 do i = 1, sys_size
1333 allocate (q_cons_temp(i)%sf(-1:m_ds + 1,-1:n_ds + 1,-1:p_ds + 1))
1334 end do
1335 end if
1336
1337 if (down_sample) then
1340 do i = 1, sys_size
1341
1342# 903 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1343#if defined(MFC_OpenACC)
1344# 903 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1345!$acc update device(q_cons_ts(1)%vf(i)%sf)
1346# 903 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1347#elif defined(MFC_OpenMP)
1348# 903 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1349!$omp target update to(q_cons_ts(1)%vf(i)%sf)
1350# 903 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1351#endif
1352 end do
1353 do i = 1, sys_size
1354 deallocate (q_cons_temp(i)%sf)
1355 end do
1356 deallocate (q_cons_temp)
1357 else
1358 call s_read_data_files(q_cons_ts(1)%vf)
1359 end if
1360
1362
1364 if (ib) then
1366 call s_ibm_setup()
1367 if (t_step_start == 0) then
1368 call s_write_ib_data_file(0)
1369 call s_write_ib_state_file(0)
1370 end if
1371 end if
1374
1375 ! Initialize the Temperature cache.
1377
1378 ! Computation of parameters, allocation of memory, association of pointers, and/or execution of any other tasks that are
1379 ! needed to properly configure the modules. The preparations below DO DEPEND on the grid being complete.
1380 if (igr .or. dummy) then
1382 end if
1383 if (.not. igr .or. dummy) then
1384 if (recon_type == weno_type) then
1386 else if (recon_type == muscl_type) then
1388 end if
1391 end if
1392
1395
1398
1399 end subroutine s_initialize_modules
1400
1401 !> Set up the MPI execution environment, bind GPUs, and decompose the computational domain
1402 impure subroutine s_initialize_mpi_domain
1403
1404 integer :: ierr
1405
1406#ifdef MFC_GPU
1407 real(wp) :: starttime, endtime
1408 integer :: num_devices, local_size, num_nodes, ppn, my_device_num
1409 integer :: dev, devnum, local_rank
1410#ifdef MFC_MPI
1411 integer :: local_comm
1412#endif
1413#if defined(MFC_OpenACC)
1414 integer(acc_device_kind) :: devtype
1415#endif
1416#endif
1417
1418 call s_mpi_initialize()
1419
1420#ifdef MFC_GPU
1421#ifndef MFC_MPI
1422 local_size = 1
1423 local_rank = 0
1424#else
1425 call mpi_comm_split_type(mpi_comm_world, mpi_comm_type_shared, 0, mpi_info_null, local_comm, ierr)
1426 call mpi_comm_size(local_comm, local_size, ierr)
1427 call mpi_comm_rank(local_comm, local_rank, ierr)
1428#endif
1429#if defined(MFC_OpenACC)
1430 devtype = acc_get_device_type()
1431 devnum = acc_get_num_devices(devtype)
1432 dev = mod(local_rank, devnum)
1433
1434 call acc_set_device_num(dev, devtype)
1435#elif defined(MFC_OpenMP)
1436 devnum = omp_get_num_devices()
1437 dev = mod(local_rank, devnum)
1438 call omp_set_default_device(dev)
1439#endif
1440#endif
1441
1442 if (proc_rank == 0) then
1444 call s_read_input_file()
1445 call s_check_input_file()
1446
1447 print '(" Simulating a ", A, " ", I0, "x", I0, "x", I0, " case on ", I0, " rank(s) ", A, ".")', &
1448# 1001 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1449 "regular", &
1450# 1005 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1451 m, n, p, num_procs, &
1452#if defined(MFC_OpenACC)
1453 "with OpenACC offloading"
1454#elif defined(MFC_OpenMP)
1455 "with OpenMP offloading"
1456#else
1457 "on CPUs"
1458#endif
1459 end if
1460
1462
1463 ! Save original BCs before decomposition overwrites them with MPI neighbor ranks
1464 ib_bc_x = bc_x
1465 ib_bc_y = bc_y
1466 ib_bc_z = bc_z
1467
1469
1471
1472 end subroutine s_initialize_mpi_domain
1473
1474 !> Transfer initial conservative variable and model parameter data to the GPU device
1476
1477 integer :: i
1478
1479 if (.not. down_sample) then
1480 do i = 1, sys_size
1481
1482# 1035 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1483#if defined(MFC_OpenACC)
1484# 1035 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1485!$acc update device(q_cons_ts(1)%vf(i)%sf)
1486# 1035 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1487#elif defined(MFC_OpenMP)
1488# 1035 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1489!$omp target update to(q_cons_ts(1)%vf(i)%sf)
1490# 1035 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1491#endif
1492 end do
1493 end if
1494
1495 if (qbmm .and. .not. polytropic) then
1496
1497# 1040 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1498#if defined(MFC_OpenACC)
1499# 1040 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1500!$acc update device(pb_ts(1)%sf, mv_ts(1)%sf)
1501# 1040 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1502#elif defined(MFC_OpenMP)
1503# 1040 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1504!$omp target update to(pb_ts(1)%sf, mv_ts(1)%sf)
1505# 1040 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1506#endif
1507 end if
1508 if (chemistry) then
1509
1510# 1043 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1511#if defined(MFC_OpenACC)
1512# 1043 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1513!$acc update device(q_T_sf%sf)
1514# 1043 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1515#elif defined(MFC_OpenMP)
1516# 1043 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1517!$omp target update to(q_T_sf%sf)
1518# 1043 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1519#endif
1520 end if
1521
1522
1523# 1046 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1524#if defined(MFC_OpenACC)
1525# 1046 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1526!$acc update device(chem_params)
1527# 1046 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1528#elif defined(MFC_OpenMP)
1529# 1046 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1530!$omp target update to(chem_params)
1531# 1046 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1532#endif
1533
1534
1535# 1048 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1536#if defined(MFC_OpenACC)
1537# 1048 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1538!$acc update device(R0ref, p0ref, rho0ref, ss, pv, vd, mu_l, mu_v, mu_g, gam_v, gam_g, M_v, M_g, R_v, R_g, Tw, cp_v, cp_g, k_vl, k_gl, gam, gam_m, Eu, Ca, Web, Re_inv, Pe_c, phi_vg, phi_gv, omegaN, bubbles_euler, polytropic, polydisperse, qbmm, ptil, bubble_model, thermal, poly_sigma, adv_n, adap_dt, adap_dt_tol, adap_dt_max_iters, eqn_idx%n, pi_fac, low_Mach)
1539# 1048 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1540#elif defined(MFC_OpenMP)
1541# 1048 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1542!$omp target update to(R0ref, p0ref, rho0ref, ss, pv, vd, mu_l, mu_v, mu_g, gam_v, gam_g, M_v, M_g, R_v, R_g, Tw, cp_v, cp_g, k_vl, k_gl, gam, gam_m, Eu, Ca, Web, Re_inv, Pe_c, phi_vg, phi_gv, omegaN, bubbles_euler, polytropic, polydisperse, qbmm, ptil, bubble_model, thermal, poly_sigma, adv_n, adap_dt, adap_dt_tol, adap_dt_max_iters, eqn_idx%n, pi_fac, low_Mach)
1543# 1048 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1544#endif
1545# 1052 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1546
1547 if (bubbles_euler) then
1548
1549# 1054 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1550#if defined(MFC_OpenACC)
1551# 1054 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1552!$acc update device(weight, R0)
1553# 1054 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1554#elif defined(MFC_OpenMP)
1555# 1054 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1556!$omp target update to(weight, R0)
1557# 1054 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1558#endif
1559 if (.not. polytropic) then
1560
1561# 1056 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1562#if defined(MFC_OpenACC)
1563# 1056 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1564!$acc update device(pb0, Pe_T, k_g, k_v, mass_g0, mass_v0, Re_trans_T, Re_trans_c, Im_trans_T, Im_trans_c)
1565# 1056 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1566#elif defined(MFC_OpenMP)
1567# 1056 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1568!$omp target update to(pb0, Pe_T, k_g, k_v, mass_g0, mass_v0, Re_trans_T, Re_trans_c, Im_trans_T, Im_trans_c)
1569# 1056 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1570#endif
1571 else if (qbmm) then
1572
1573# 1058 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1574#if defined(MFC_OpenACC)
1575# 1058 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1576!$acc update device(pb0)
1577# 1058 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1578#elif defined(MFC_OpenMP)
1579# 1058 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1580!$omp target update to(pb0)
1581# 1058 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1582#endif
1583 end if
1584 end if
1585
1586
1587# 1062 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1588#if defined(MFC_OpenACC)
1589# 1062 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1590!$acc update device(adv_n, adap_dt, adap_dt_tol, adap_dt_max_iters, pi_fac, low_Mach)
1591# 1062 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1592#elif defined(MFC_OpenMP)
1593# 1062 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1594!$omp target update to(adv_n, adap_dt, adap_dt_tol, adap_dt_max_iters, pi_fac, low_Mach)
1595# 1062 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1596#endif
1597
1598
1599# 1064 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1600#if defined(MFC_OpenACC)
1601# 1064 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1602!$acc update device(acoustic_source, num_source)
1603# 1064 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1604#elif defined(MFC_OpenMP)
1605# 1064 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1606!$omp target update to(acoustic_source, num_source)
1607# 1064 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1608#endif
1609
1610# 1065 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1611#if defined(MFC_OpenACC)
1612# 1065 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1613!$acc update device(sigma, surface_tension)
1614# 1065 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1615#elif defined(MFC_OpenMP)
1616# 1065 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1617!$omp target update to(sigma, surface_tension)
1618# 1065 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1619#endif
1620
1621
1622# 1067 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1623#if defined(MFC_OpenACC)
1624# 1067 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1625!$acc update device(dx, dy, dz, x_cb, x_cc, y_cb, y_cc, z_cb, z_cc)
1626# 1067 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1627#elif defined(MFC_OpenMP)
1628# 1067 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1629!$omp target update to(dx, dy, dz, x_cb, x_cc, y_cb, y_cc, z_cb, z_cc)
1630# 1067 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1631#endif
1632
1633# 1068 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1634#if defined(MFC_OpenACC)
1635# 1068 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1636!$acc update device(bc_x%beg, bc_x%end, bc_y%beg, bc_y%end, bc_z%beg, bc_z%end)
1637# 1068 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1638#elif defined(MFC_OpenMP)
1639# 1068 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1640!$omp target update to(bc_x%beg, bc_x%end, bc_y%beg, bc_y%end, bc_z%beg, bc_z%end)
1641# 1068 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1642#endif
1643
1644# 1069 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1645#if defined(MFC_OpenACC)
1646# 1069 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1647!$acc update device(bc_x%vb1, bc_x%vb2, bc_x%vb3, bc_x%ve1, bc_x%ve2, bc_x%ve3)
1648# 1069 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1649#elif defined(MFC_OpenMP)
1650# 1069 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1651!$omp target update to(bc_x%vb1, bc_x%vb2, bc_x%vb3, bc_x%ve1, bc_x%ve2, bc_x%ve3)
1652# 1069 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1653#endif
1654
1655# 1070 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1656#if defined(MFC_OpenACC)
1657# 1070 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1658!$acc update device(bc_y%vb1, bc_y%vb2, bc_y%vb3, bc_y%ve1, bc_y%ve2, bc_y%ve3)
1659# 1070 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1660#elif defined(MFC_OpenMP)
1661# 1070 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1662!$omp target update to(bc_y%vb1, bc_y%vb2, bc_y%vb3, bc_y%ve1, bc_y%ve2, bc_y%ve3)
1663# 1070 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1664#endif
1665
1666# 1071 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1667#if defined(MFC_OpenACC)
1668# 1071 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1669!$acc update device(bc_z%vb1, bc_z%vb2, bc_z%vb3, bc_z%ve1, bc_z%ve2, bc_z%ve3)
1670# 1071 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1671#elif defined(MFC_OpenMP)
1672# 1071 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1673!$omp target update to(bc_z%vb1, bc_z%vb2, bc_z%vb3, bc_z%ve1, bc_z%ve2, bc_z%ve3)
1674# 1071 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1675#endif
1676
1677
1678# 1073 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1679#if defined(MFC_OpenACC)
1680# 1073 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1681!$acc update device(bc_x%grcbc_in, bc_x%grcbc_out, bc_x%grcbc_vel_out)
1682# 1073 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1683#elif defined(MFC_OpenMP)
1684# 1073 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1685!$omp target update to(bc_x%grcbc_in, bc_x%grcbc_out, bc_x%grcbc_vel_out)
1686# 1073 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1687#endif
1688
1689# 1074 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1690#if defined(MFC_OpenACC)
1691# 1074 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1692!$acc update device(bc_y%grcbc_in, bc_y%grcbc_out, bc_y%grcbc_vel_out)
1693# 1074 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1694#elif defined(MFC_OpenMP)
1695# 1074 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1696!$omp target update to(bc_y%grcbc_in, bc_y%grcbc_out, bc_y%grcbc_vel_out)
1697# 1074 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1698#endif
1699
1700# 1075 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1701#if defined(MFC_OpenACC)
1702# 1075 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1703!$acc update device(bc_z%grcbc_in, bc_z%grcbc_out, bc_z%grcbc_vel_out)
1704# 1075 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1705#elif defined(MFC_OpenMP)
1706# 1075 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1707!$omp target update to(bc_z%grcbc_in, bc_z%grcbc_out, bc_z%grcbc_vel_out)
1708# 1075 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1709#endif
1710
1711
1712# 1077 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1713#if defined(MFC_OpenACC)
1714# 1077 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1715!$acc update device(bc_x%isothermal_in, bc_x%isothermal_out)
1716# 1077 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1717#elif defined(MFC_OpenMP)
1718# 1077 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1719!$omp target update to(bc_x%isothermal_in, bc_x%isothermal_out)
1720# 1077 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1721#endif
1722
1723# 1078 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1724#if defined(MFC_OpenACC)
1725# 1078 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1726!$acc update device(bc_y%isothermal_in, bc_y%isothermal_out)
1727# 1078 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1728#elif defined(MFC_OpenMP)
1729# 1078 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1730!$omp target update to(bc_y%isothermal_in, bc_y%isothermal_out)
1731# 1078 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1732#endif
1733
1734# 1079 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1735#if defined(MFC_OpenACC)
1736# 1079 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1737!$acc update device(bc_z%isothermal_in, bc_z%isothermal_out)
1738# 1079 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1739#elif defined(MFC_OpenMP)
1740# 1079 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1741!$omp target update to(bc_z%isothermal_in, bc_z%isothermal_out)
1742# 1079 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1743#endif
1744
1745# 1080 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1746#if defined(MFC_OpenACC)
1747# 1080 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1748!$acc update device(bc_x%Twall_in, bc_x%Twall_out, bc_y%Twall_in, bc_y%Twall_out, bc_z%Twall_in, bc_z%Twall_out)
1749# 1080 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1750#elif defined(MFC_OpenMP)
1751# 1080 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1752!$omp target update to(bc_x%Twall_in, bc_x%Twall_out, bc_y%Twall_in, bc_y%Twall_out, bc_z%Twall_in, bc_z%Twall_out)
1753# 1080 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1754#endif
1755
1756
1757# 1082 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1758#if defined(MFC_OpenACC)
1759# 1082 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1760!$acc update device(relax, relax_model)
1761# 1082 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1762#elif defined(MFC_OpenMP)
1763# 1082 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1764!$omp target update to(relax, relax_model)
1765# 1082 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1766#endif
1767 if (relax) then
1768
1769# 1084 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1770#if defined(MFC_OpenACC)
1771# 1084 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1772!$acc update device(palpha_eps, ptgalpha_eps)
1773# 1084 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1774#elif defined(MFC_OpenMP)
1775# 1084 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1776!$omp target update to(palpha_eps, ptgalpha_eps)
1777# 1084 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1778#endif
1779 end if
1780
1781 if (ib) then
1782
1783# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1784#if defined(MFC_OpenACC)
1785# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1786!$acc update device(ib_markers%sf)
1787# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1788#elif defined(MFC_OpenMP)
1789# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1790!$omp target update to(ib_markers%sf)
1791# 1088 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1792#endif
1793 end if
1794# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1795
1796# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1797#if defined(MFC_OpenACC)
1798# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1799!$acc update device(igr, nb, igr_order)
1800# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1801#elif defined(MFC_OpenMP)
1802# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1803!$omp target update to(igr, nb, igr_order)
1804# 1091 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1805#endif
1806# 1093 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1807# 1101 "/home/runner/work/MFC/MFC/src/simulation/m_start_up.fpp"
1808
1809 end subroutine s_initialize_gpu_vars
1810
1811 !> Finalize and deallocate all simulation sub-modules in reverse initialization order
1812 impure subroutine s_finalize_modules
1813
1820 if (igr) then
1822 else
1825 if (recon_type == weno_type) then
1827 else if (recon_type == muscl_type) then
1829 end if
1830 end if
1832 if (grid_geometry == 3) call s_finalize_fftw_module
1838 if (viscous .and. (.not. igr)) then
1840 end if
1842
1845 if (ib) call s_finalize_ibm_module()
1846
1847 call s_mpi_finalize()
1848
1849 end subroutine s_finalize_modules
1850
1851 !> @brief Reads IB kinematic state from restart_data/ib_state.dat on restart. Rank 0 reads the last num_ibs records and
1852 !! broadcasts to all ranks. Overwrites patch_ib vel, angular_vel, angles, and centroid.
1853 impure subroutine s_read_ib_restart_data(t_step)
1854
1855 integer, intent(in) :: t_step
1856 character(len=path_len + 2*name_len) :: file_loc
1857 integer :: i, ios, file_unit, ierr
1858 integer, parameter :: nfields_per_ib = 20
1859 real(wp) :: ib_buf(nfields_per_ib)
1860 logical :: file_exist
1861
1862 write (file_loc, '(A,I0,A)') '/restart_data/ib_state_', t_step, '.dat'
1863 file_loc = trim(case_dir) // trim(file_loc)
1864
1865 if (proc_rank == 0) then
1866 inquire (file=trim(file_loc), exist=file_exist)
1867 if (.not. file_exist) then
1868 call s_mpi_abort('Cannot open IB state file for restart: ' // trim(file_loc))
1869 end if
1870
1871 open (newunit=file_unit, file=trim(file_loc), form='unformatted', access='stream', status='old', iostat=ios)
1872 if (ios /= 0) call s_mpi_abort('Error opening IB state restart file: ' // trim(file_loc))
1873
1874 do i = 1, num_ibs
1875 read (file_unit, iostat=ios) ib_buf
1876 if (ios /= 0) call s_mpi_abort('Error reading IB state restart file')
1877
1878 patch_ib(i)%vel = ib_buf(8:10)
1879 patch_ib(i)%angular_vel = ib_buf(11:13)
1880 patch_ib(i)%angles = ib_buf(14:16)
1881 patch_ib(i)%x_centroid = ib_buf(17)
1882 patch_ib(i)%y_centroid = ib_buf(18)
1883 patch_ib(i)%z_centroid = ib_buf(19)
1884 end do
1885
1886 close (file_unit)
1887 end if
1888
1889#ifdef MFC_MPI
1890 do i = 1, num_ibs
1891 call mpi_bcast(patch_ib(i)%vel, 3, mpi_p, 0, mpi_comm_world, ierr)
1892 call mpi_bcast(patch_ib(i)%angular_vel, 3, mpi_p, 0, mpi_comm_world, ierr)
1893 call mpi_bcast(patch_ib(i)%angles, 3, mpi_p, 0, mpi_comm_world, ierr)
1894 call mpi_bcast(patch_ib(i)%x_centroid, 1, mpi_p, 0, mpi_comm_world, ierr)
1895 call mpi_bcast(patch_ib(i)%y_centroid, 1, mpi_p, 0, mpi_comm_world, ierr)
1896 call mpi_bcast(patch_ib(i)%z_centroid, 1, mpi_p, 0, mpi_comm_world, ierr)
1897 end do
1898#endif
1899
1900 end subroutine s_read_ib_restart_data
1901
1902end module m_start_up
type(scalar_field), dimension(sys_size), intent(inout) q_cons_vf
One-way acoustic source injection, Maeda and Colonius JCP (2017).
impure subroutine, public s_precalculate_acoustic_spatial_sources
Pre-compute non-zero spatial source weights before time-stepping.
impure subroutine, public s_initialize_acoustic_src
Initialize the acoustic source module.
Computes gravitational and user-defined body force source terms for the momentum equations.
impure subroutine, public s_initialize_body_forces_module
Initialize the body forces module.
impure subroutine, public s_finalize_body_forces_module
Finalize the body forces module.
Noncharacteristic and processor boundary condition application for ghost cells and buffer regions.
impure subroutine, public s_initialize_boundary_common_module()
Allocate and set up boundary condition buffer arrays for all coordinate directions.
subroutine, public s_finalize_boundary_common_module()
Deallocate boundary condition buffer arrays allocated during module initialization.
subroutine, public s_read_parallel_boundary_condition_files(bc_type)
Read boundary condition type and buffer data from per-rank parallel files using MPI I/O.
subroutine, public s_populate_grid_variables_buffers
Populate the buffers of the grid variables, which are constituted of the cell-boundary locations and ...
impure subroutine, public s_populate_variables_buffers(bc_type, q_prim_vf, pb_in, mv_in, q_t_sf)
Populate the buffers of the primitive variables based on the selected boundary conditions.
subroutine, public s_read_serial_boundary_condition_files(step_dirpath, bc_type)
Read boundary condition type and buffer data from serial (unformatted) restart files.
subroutine, public s_assign_default_bc_type(bc_type)
Initialize the per-cell boundary condition type arrays with the global default BC values.
Computes ensemble-averaged (Euler–Euler) bubble source terms for radius, velocity,...
impure subroutine s_initialize_bubbles_ee_module
Initialize the Euler-Euler bubble module.
Tracks Lagrangian bubbles and couples their dynamics to the Eulerian flow via volume averaging.
impure subroutine s_write_lag_bubble_stats()
Write the maximum and minimum radius statistics for each bubble.
impure subroutine s_write_restart_lag_bubbles(t_step)
Write restart files for the Lagrangian bubble solver.
integer nbubs
Number of bubbles in the local domain.
impure subroutine s_initialize_bubbles_el_module(q_cons_vf)
Initializes the lagrangian subgrid bubble solver.
type(scalar_field), dimension(:), allocatable q_beta
Projection of the lagrangian particles in the Eulerian framework.
real(wp), dimension(:,:), allocatable intfc_rad
Bubble radius.
impure subroutine s_finalize_lagrangian_solver()
Finalize the Lagrangian bubble solver.
Characteristic boundary conditions (CBC) for slip walls, non-reflecting subsonic inflow/outflow,...
impure subroutine, public s_initialize_cbc_module
Initialize the CBC module.
impure subroutine, public s_finalize_cbc_module
Module deallocation and/or disassociation procedures.
Shared input validation checks for grid dimensions and AMD GPU compiler limits.
impure subroutine, public s_check_inputs_common
Checks compatibility of parameters in the input file. Used by all three stages.
Validates simulation input parameters for consistency and supported configurations.
impure subroutine, public s_check_inputs
Checks compatibility of parameters in the input file. Used by the simulation stage.
Multi-species chemistry interface for thermodynamic properties, reaction rates, and transport coeffic...
subroutine s_compute_q_t_sf(q_t_sf, q_cons_vf, bounds)
Initialize the temperature field from conservative variables by inverting the energy equation.
Platform-specific file and directory operations: create, delete, inquire, getcwd, and basename.
impure subroutine my_inquire(fileloc, dircheck)
Inquires on the existence of a directory.
Writes solution data, run-time stability diagnostics (ICFL, VCFL, CCFL, Rc), and probe/center-of-mass...
impure subroutine, public s_initialize_data_output_module
Initialize the data output module.
impure subroutine, public s_finalize_data_output_module
Module deallocation and/or disassociation procedures.
impure subroutine, public s_write_data_files(q_cons_vf, q_t_sf, q_prim_vf, t_step, bc_type, beta)
Write data files. Dispatch subroutine that replaces procedure pointer.
impure subroutine, public s_write_ib_state_file(time_step)
Writes IB state records to restart_data/ib_state.dat. Must be called only on rank 0.
subroutine, public s_write_ib_data_file(time_step)
Dispatch immersed boundary data output to the serial or parallel writer.
Shared derived types for field data, patch geometry, bubble dynamics, and MPI I/O structures.
Derives diagnostic flow quantities (vorticity, speed of sound, numerical Schlieren,...
impure subroutine, public s_initialize_derived_variables
Allocate and open derived variables. Computing FD coefficients.
impure subroutine, public s_initialize_derived_variables_module
Computation of parameters, allocation procedures, and/or any other tasks needed to properly setup the...
impure subroutine, public s_finalize_derived_variables_module
Deallocation procedures for the module.
Global parameters for the computational domain, fluid properties, and simulation algorithm configurat...
type(int_bounds_info) ib_bc_z
real(wp) mytime
Current simulation time.
real(wp) w_x
amplitude, frequency, and phase shift sinusoid in each direction
logical null_weights
Null undesired WENO weights.
logical bubbles_euler
Bubbles euler on/off.
real(wp) cont_damage_s
Exponent s for continuum damage modeling.
integer wave_speeds
Wave speeds estimation method.
logical cont_damage
Continuum damage modeling.
logical igr
Use information geometric regularization.
logical hypoelasticity
hypoelasticity modeling
impure subroutine s_assign_default_values_to_user_inputs
Assigns default values to the user inputs before reading them in. This enables for an easier consiste...
impure subroutine s_finalize_global_parameters_module
Module deallocation and/or disassociation procedures.
integer thermal
Thermal behavior. 1 = adiabatic, 2 = isotherm, 3 = transfer.
integer avg_state
Average state evaluation method.
logical weno_avg
Average left/right cell-boundary states.
type(int_bounds_info), dimension(1:3) idwint
real(wp), dimension(:), allocatable, target z_cb
integer recon_type
Reconstruction Type.
logical igr_pres_lim
Limit to positive pressures for IGR.
logical, parameter chemistry
Chemistry modeling.
type(int_bounds_info) ib_bc_y
logical acoustic_source
Acoustic source switch.
type(int_bounds_info) ib_bc_x
integer num_fluids
number of fluids in the simulation
logical polydisperse
Polydisperse bubbles.
type(bubbles_lagrange_parameters) lag_params
Lagrange bubbles' parameters.
integer proc_rank
Rank of the local processor.
real(wp) hyper_cleaning_tau
Hyperbolic cleaning tau.
logical mixture_err
Mixture properties correction.
logical adap_dt
Adaptive step size control.
logical weno_re_flux
WENO reconstruct velocity gradients for viscous stress tensor.
real(wp) hyper_cleaning_speed
Hyperbolic cleaning wave speed (c_h).
integer t_step_print
Number of time-steps between printouts.
logical dummy
AMDFlang workaround for case-optimization + GPU-kernel bug.
real(wp) poly_sigma
log normal sigma for polydisperse PDF
character(len=name_len) mpiiofs
integer, dimension(:), allocatable start_idx
Starting cell-center index of local processor in global grid.
type(vec3_dt), dimension(num_probes_max) probe
integer sys_size
Number of unknowns in system of eqns.
integer muscl_order
Order of the MUSCL reconstruction.
logical alt_soundspeed
Alternate mixture sound speed.
real(wp) ptgalpha_eps
trigger parameter for the pTg relaxation procedure, phase change model
integer relax_model
Relaxation model.
integer t_step_old
Existing IC/grid folder.
logical viscous
Viscous effects.
logical run_time_info
Run-time output flag.
real(wp) wenoz_q
Power constant for WENO-Z.
integer riemann_solver
Riemann solver algorithm.
type(int_bounds_info), dimension(1:3) idwbuff
logical int_comp
THINC interface compression.
real(wp) re_inv
Inverse Reynolds number.
real(wp) ic_eps
THINC Epsilon to compress on surface cells.
integer model_eqns
Multicomponent flow model.
integer precision
Precision of output files.
logical hyperelasticity
hyperelasticity modeling
type(physical_parameters), dimension(num_fluids_max) fluid_pp
Stiffened gas EOS parameters and Reynolds numbers per fluid.
real(wp) alf_factor
alpha factor for IGR
impure subroutine s_initialize_global_parameters_module
Initialize the global parameters module.
real(wp), dimension(:), allocatable, target y_cc
type(pres_field), dimension(:), allocatable pb_ts
type(pres_field), dimension(:), allocatable mv_ts
type(chemistry_parameters) chem_params
integer fd_order
Finite-difference order for CoM and flow probe derivatives.
logical bubbles_lagrange
Lagrangian subgrid bubble model switch.
real(wp) ca
Cavitation number.
real(wp) alpha_bar
Damage rate factor for continuum damage modeling.
logical polytropic
Polytropic switch.
logical bf_z
body force toggle in three directions
logical mp_weno
Monotonicity preserving (MP) WENO.
real(wp), dimension(:), allocatable, target z_cc
logical relax
activate phase change
logical qbmm
Quadrature moment method.
real(wp) pi_fac
Factor for artificial pi_inf.
logical hyper_cleaning
Hyperbolic cleaning for MHD for divB=0.
integer adap_dt_max_iters
Maximum number of iterations.
real(wp) ic_beta
THINC Sharpness Parameter.
real(wp) bx0
Constant magnetic field in the x-direction (1D).
integer num_source
Number of acoustic sources.
real(wp), dimension(:), allocatable qvs
real(wp), dimension(:), allocatable pi_infs
logical adv_n
Solve the number density equation and compute alpha from number density.
integer num_procs
Number of processors.
character(len=path_len) case_dir
Case folder location.
real(wp) weno_eps
Binding for the WENO nonlinear weights.
integer weno_order
Order of the WENO reconstruction.
logical mhd
Magnetohydrodynamics.
real(wp), dimension(:), allocatable, target x_cc
type(acoustic_parameters), dimension(num_probes_max) acoustic
Acoustic source parameters.
real(wp) tau_star
Stress threshold for continuum damage modeling.
logical parallel_io
Format of the data files.
type(integral_parameters), dimension(num_probes_max) integral
real(wp), dimension(:), allocatable, target y_cb
real(wp) adap_dt_tol
Tolerance to control adaptive step size.
type(cell_num_bounds) cells_bounds
logical down_sample
down sample the output files
type(mpi_io_airfoil_ib_var), public mpi_io_airfoil_ib_data
logical nv_uvm_out_of_core
Enable out-of-core storage of q_cons_ts(2) in timestepping (default FALSE).
logical file_per_process
shared file or not when using parallel io
real(wp) palpha_eps
trigger parameter for the p relaxation procedure, phase change model
integer num_igr_warm_start_iters
number of warm start iterations for elliptic solve
integer nv_uvm_igr_temps_on_gpu
0 => jac, jac_rhs, and jac_old on CPU
logical elasticity
elasticity modeling, true for hyper or hypo
integer nb
Number of eq. bubble sizes.
type(mpi_io_var), public mpi_io_data
impure subroutine s_initialize_parallel_io
Initializes parallel infrastructure.
logical mpp_lim
Mixture physical parameters (MPP) limits.
integer num_igr_iters
number of iterations for elliptic solve
real(wp) teno_ct
Smoothness threshold for TENO.
integer low_mach
Low Mach number fix to HLLC Riemann solver.
integer igr_order
Reconstruction order for IGR.
logical teno
TENO (Targeted ENO).
real(wp), dimension(:), allocatable, target dy
real(wp) dt
Size of the time-step.
type(subgrid_bubble_physical_parameters) bub_pp
integer time_stepper
Time-stepper algorithm.
logical relativity
Relativity (only for MHD).
logical nv_uvm_pref_gpu
Enable explicit gpu memory hints (default FALSE).
real(wp), dimension(:), allocatable gammas
integer bubble_model
Gilmore or Keller–Miksis bubble model.
real(wp) finaltime
Final simulation time.
real(wp), dimension(:), allocatable, target dz
real(wp), dimension(:), allocatable, target dx
logical mapped_weno
WENO-M (WENO with mapping of nonlinear weights).
type(eqn_idx_info) eqn_idx
All conserved-variable equation index ranges and scalars.
integer igr_iter_solver
IGR elliptic solver.
integer muscl_lim
MUSCL Limiter.
type(ib_patch_parameters), dimension(num_ib_patches_max) patch_ib
Immersed boundary patch parameters.
real(wp), dimension(:), allocatable, target x_cb
Basic floating-point utilities: approximate equality, default detection, and coordinate bounds.
elemental subroutine, public s_update_cell_bounds(bounds, m, n, p)
Updates the min and max number of cells in each set of axes.
Utility routines for bubble model setup, coordinate transforms, array sampling, and special functions...
subroutine, public s_upsample_data(q_cons_vf, q_cons_temp)
Upsample conservative variable fields from a coarsened grid back to the original resolution using int...
impure subroutine, public s_initialize_bubbles_model()
Initialize bubble model arrays for Euler or Lagrangian bubbles with polytropic or non-polytropic gas.
elemental subroutine, public s_int_to_str(i, res)
Convert an integer to its trimmed string representation.
Computes the left Cauchy–Green deformation tensor and hyperelastic stress source terms.
impure subroutine, public s_finalize_hyperelastic_module()
Finalize the hyperelastic module.
impure subroutine, public s_initialize_hyperelastic_module
Initialize the hyperelastic module.
Computes hypoelastic stress-rate source terms and damage-state evolution.
impure subroutine, public s_initialize_hypoelastic_module
Initialize the hypoelastic module.
impure subroutine, public s_finalize_hypoelastic_module()
Finalize the hypoelastic module.
Ghost-node immersed boundary method: locates ghost/image points, computes interpolation coefficients,...
impure subroutine, public s_ibm_setup()
Initializes the values of various IBM variables, such as ghost points and image points.
impure subroutine, public s_finalize_ibm_module()
Finalize the IBM module.
type(integer_field), public ib_markers
impure subroutine, public s_initialize_ibm_module()
Allocates memory for the variables in the IBM module.
Iterative ghost rasterization (IGR) for sharp immersed boundary treatment.
subroutine, public s_initialize_igr_module()
Initialize the IGR module.
integer(kind=8) j
integer(kind=8) i
integer(kind=8) l
integer(kind=8) r
integer(kind=8) k
subroutine, public s_finalize_igr_module()
Finalize the IGR module.
MPI communication layer: domain decomposition, halo exchange, reductions, and parallel I/O setup.
impure subroutine s_mpi_abort(prnt, code)
The subroutine terminates the MPI execution environment.
impure subroutine s_initialize_mpi_common_module
Initialize the module.
impure subroutine s_mpi_barrier
Halts all processes until all have reached barrier.
impure subroutine s_mpi_initialize
Initialize the MPI execution environment and query the number of processors and local rank.
impure subroutine s_initialize_mpi_data(q_cons_vf, ib_markers, beta)
Set up MPI I/O data views and variable pointers for parallel file output.
impure subroutine s_mpi_finalize
The subroutine finalizes the MPI execution environment.
subroutine s_initialize_mpi_data_ds(q_cons_vf)
Set up MPI I/O data views for downsampled (coarsened) parallel file output.
impure subroutine mpi_bcast_time_step_values(proc_time, time_avg)
Gather per-rank time step wall-clock times onto rank 0 for performance reporting.
impure subroutine s_finalize_mpi_common_module
Module deallocation and/or disassociation procedures.
subroutine s_mpi_decompose_computational_domain
Decompose the computational domain among processors by balancing cells per rank in each coordinate di...
MPI halo exchange, domain decomposition, and buffer packing/unpacking for the simulation solver.
subroutine s_initialize_mpi_proxy_module()
Initialize the MPI proxy module.
subroutine s_finalize_mpi_proxy_module()
Finalize the MPI proxy module.
impure subroutine s_mpi_bcast_user_inputs()
Since only the processor with rank 0 reads and verifies the consistency of user inputs,...
MUSCL reconstruction with interface sharpening for contact-preserving advection.
subroutine, public s_initialize_muscl_module()
Allocate and initialize MUSCL reconstruction working arrays.
subroutine, public s_finalize_muscl_module()
Finalize the MUSCL module.
NVIDIA NVTX profiling API bindings for GPU performance instrumentation.
Definition m_nvtx.f90:6
subroutine nvtxstartrange(name, id)
Push a named NVTX range for GPU profiling, optionally with a color based on the given identifier.
Definition m_nvtx.f90:62
subroutine nvtxendrange
Pop the current NVTX range to end the GPU profiling region.
Definition m_nvtx.f90:83
Phase transition relaxation solvers for liquid-vapor flows with cavitation and boiling.
impure subroutine, public s_finalize_relaxation_solver_module
Finalize the phase change module.
subroutine, public s_infinite_relaxation_k(q_cons_vf)
Apply pT- or pTg-equilibrium relaxation with mass depletion based on the incoming state conditions.
impure subroutine, public s_initialize_phasechange_module
Initialize the phase change module by setting saturation curve coefficients for pT- or pTg-equilibriu...
Quadrature-based moment methods (QBMM) for polydisperse bubble moment inversion and transport.
impure subroutine, public s_initialize_qbmm_module
Initialize the QBMM module.
Assembles the right-hand side of the governing equations using finite-volume flux differencing,...
impure subroutine, public s_initialize_rhs_module
Initialize the RHS module.
impure subroutine, public s_finalize_rhs_module
Module deallocation and/or disassociation procedures.
Approximate and exact Riemann solvers (HLL, HLLC, HLLD, exact) for the multicomponent Navier–Stokes e...
impure subroutine, public s_finalize_riemann_solvers_module
Module deallocation and/or disassociation procedures.
impure subroutine, public s_initialize_riemann_solvers_module
Initialize the Riemann solvers module.
Simulation helper routines for enthalpy computation, CFL calculation, and stability checks.
Reads input files, loads initial conditions and grid data, and orchestrates solver initialization and...
impure subroutine s_read_ib_restart_data(t_step)
Reads IB kinematic state from restart_data/ib_state.dat on restart. Rank 0 reads the last num_ibs rec...
impure subroutine, public s_read_serial_data_files(q_cons_vf)
Read serial initial condition and grid data files and compute cell-width distributions.
impure subroutine, public s_initialize_modules
Initialize all simulation sub-modules in the required dependency order.
impure subroutine, public s_read_data_files(q_cons_vf)
Read data files. Dispatch subroutine that replaces procedure pointer.
impure subroutine, public s_read_parallel_data_files(q_cons_vf)
Read parallel initial condition and grid data files via MPI I/O.
subroutine, public s_initialize_internal_energy_equations(v_vf)
Initialize internal-energy equations from phase mass, mixture momentum, and total energy.
impure subroutine, public s_save_performance_metrics(time_avg, time_final, io_time_avg, io_time_final, proc_time, io_proc_time, file_exists)
Collect per-process wall-clock times and write aggregate performance metrics to file.
impure subroutine, public s_save_data(t_step, start, finish, io_time_avg, nt)
Save conservative variable data to disk at the current time step.
type(scalar_field), dimension(:), allocatable q_cons_temp
subroutine, public s_initialize_gpu_vars
Transfer initial conservative variable and model parameter data to the GPU device.
impure subroutine, public s_initialize_mpi_domain
Set up the MPI execution environment, bind GPUs, and decompose the computational domain.
impure subroutine, public s_finalize_modules
Finalize and deallocate all simulation sub-modules in reverse initialization order.
impure subroutine, public s_read_input_file
Verify the input file exists and read it.
impure subroutine, public s_check_input_file
Validate that all user-provided inputs form a consistent simulation configuration.
impure subroutine, public s_perform_time_step(t_step, time_avg)
Advance the simulation by one time step, handling CFL-based dt and time-stepper dispatch.
Computes capillary source fluxes and color-function gradients for the diffuse-interface surface tensi...
impure subroutine, public s_initialize_surface_tension_module
Allocate and initialize surface tension module arrays.
impure subroutine, public s_finalize_surface_tension_module
Finalize the surface tension module.
Total-variation-diminishing (TVD) Runge–Kutta time integrators (1st-, 2nd-, and 3rd-order SSP).
type(scalar_field) q_t_sf
Cell-average temperature variables at the current time-stage.
type(integer_field), dimension(:,:), allocatable bc_type
Boundary condition identifiers.
impure subroutine s_initialize_time_steppers_module
Initialize the time steppers module.
type(vector_field), dimension(:), allocatable q_cons_ts
Cell-average conservative variables at each time-stage (TS).
type(scalar_field), dimension(:), allocatable q_prim_vf
Cell-average primitive variables at the current time-stage.
impure subroutine s_finalize_time_steppers_module
Module deallocation and/or disassociation procedures.
impure subroutine s_compute_dt()
Compute the global time step size from CFL stability constraints across all cells.
impure subroutine s_tvd_rk(t_step, time_avg, nstage)
Advance the solution one full step using a TVD Runge-Kutta time integrator.
integer stor
storage index
Conservative-to-primitive variable conversion, mixture property evaluation, and pressure computation.
subroutine, public s_compute_pressure(energy, alf, dyn_p, pi_inf, gamma, rho, qv, rhoyks, pres, t, stress, mom, g, pres_mag)
Compute the pressure from the appropriate equation of state.
impure subroutine, public s_initialize_variables_conversion_module
Initialize the variables conversion module.
impure subroutine s_finalize_variables_conversion_module()
Deallocate fluid property arrays and post-processing fields allocated during module initialization.
subroutine, public s_convert_to_mixture_variables(q_vf, i, j, k, rho, gamma, pi_inf, qv, re_k, g_k, g)
Dispatch to the s_convert_mixture_to_mixture_variables and s_convert_species_to_mixture_variables sub...
Computes viscous stress tensors and diffusive flux contributions for the Navier–Stokes equations.
impure subroutine, public s_initialize_viscous_module
Initialize the viscous module.
impure subroutine, public s_finalize_viscous_module()
Finalize the viscous module.
WENO/WENO-Z/TENO reconstruction with optional monotonicity-preserving bounds and mapped weights.
impure subroutine, public s_initialize_weno_module
Initialize the WENO module.
impure subroutine, public s_finalize_weno_module()
Module deallocation and/or disassociation procedures.
Derived type annexing a scalar field (SF).