MFC
Exascale flow solver
Loading...
Searching...
No Matches
m_boundary_io.fpp.f90
Go to the documentation of this file.
1# 1 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
2!>
3!! @file
4!! @brief Contains module m_boundary_io
5
6!> @brief Boundary condition restart I/O, capillary/IGR buffer population, and grid-variable buffers
7# 1 "/home/runner/work/MFC/MFC/src/common/include/case.fpp" 1
8! This file exists so that Fypp can be run without generating case.fpp files for
9! each target. This is useful when generating documentation, for example. This
10! should also let MFC be built with CMake directly, without invoking mfc.sh.
11
12! For pre-process.
13# 8 "/home/runner/work/MFC/MFC/src/common/include/case.fpp"
14
15! For moving immersed boundaries in simulation
16# 12 "/home/runner/work/MFC/MFC/src/common/include/case.fpp"
17# 7 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp" 2
18# 1 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp" 1
19# 1 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 1
20# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
21# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
22# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
23# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
24# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
25# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
26
27# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
28# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
29# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
30
31# 15 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
32# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
33# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
34
35# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
36
37# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
38
39# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
40
41# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
42
43# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
44
45# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
46
47# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
48
49# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
50! New line at end of file is required for FYPP
51# 2 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
52# 1 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp" 1
53# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
54# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
55# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
56# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
57# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
58# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
59
60# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
61# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
62# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
63
64# 15 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
65# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
66# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
67
68# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
69
70# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
71
72# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
73
74# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
75
76# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
77
78# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
79
80# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
81
82# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
83! New line at end of file is required for FYPP
84# 2 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp" 2
85
86# 4 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
87# 5 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
88# 6 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
89# 7 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
90# 8 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
91
92# 20 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
93
94# 43 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
95
96# 48 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
97
98# 53 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
99
100# 58 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
101
102# 63 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
103
104# 68 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
105
106# 81 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
107
108# 86 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
109
110# 91 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
111
112# 96 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
113
114# 101 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
115
116# 106 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
117
118# 111 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
119
120# 116 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
121
122# 121 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
123
124# 126 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
125
126# 156 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
127
128# 197 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
129
130# 211 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
131
132# 236 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
133
134# 247 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
135
136# 249 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
137# 260 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
138
139# 310 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
140
141# 320 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
142
143# 330 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
144
145# 339 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
146
147# 356 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
148
149# 366 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
150
151# 373 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
152
153# 379 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
154
155# 385 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
156
157# 391 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
158
159# 397 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
160
161# 403 "/home/runner/work/MFC/MFC/src/common/include/omp_macros.fpp"
162! New line at end of file is required for FYPP
163# 3 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
164# 1 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp" 1
165# 1 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp" 1
166# 2 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
167# 3 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
168# 4 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
169# 5 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
170# 6 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
171
172# 8 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
173# 9 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
174# 10 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
175
176# 15 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
177# 16 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
178# 17 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
179
180# 24 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
181
182# 53 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
183
184# 65 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
185
186# 75 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
187
188# 105 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
189
190# 117 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
191
192# 127 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
193
194# 174 "/home/runner/work/MFC/MFC/src/common/include/shared_parallel_macros.fpp"
195! New line at end of file is required for FYPP
196# 2 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp" 2
197
198# 7 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
199
200# 17 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
201
202# 22 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
203
204# 27 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
205
206# 32 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
207
208# 37 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
209
210# 42 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
211
212# 47 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
213
214# 52 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
215
216# 57 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
217
218# 62 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
219
220# 73 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
221
222# 78 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
223
224# 83 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
225
226# 88 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
227
228# 103 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
229
230# 131 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
231
232# 160 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
233
234# 175 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
235
236# 193 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
237
238# 215 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
239
240# 244 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
241
242# 259 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
243
244# 269 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
245
246# 278 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
247
248# 294 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
249
250# 304 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
251
252# 311 "/home/runner/work/MFC/MFC/src/common/include/acc_macros.fpp"
253! New line at end of file is required for FYPP
254# 4 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp" 2
255
256! GPU parallel region (scalar reductions, maxval/minval)
257# 23 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
258
259! GPU parallel loop over threads (most common GPU macro)
260# 43 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
261
262! Required closing for GPU_PARALLEL_LOOP
263# 55 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
264
265! Mark routine for device compilation
266# 112 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
267
268! Declare device-resident data
269# 130 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
270
271! Inner loop within a GPU parallel region
272# 145 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
273
274! Scoped GPU data region
275# 164 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
276
277! Host code with device pointers (for MPI with GPU buffers)
278# 193 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
279
280! Allocate device memory (unscoped)
281# 207 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
282
283! Free device memory
284# 219 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
285
286! Atomic operation on device
287# 231 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
288
289! End atomic capture block
290# 242 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
291
292! Copy data between host and device
293# 254 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
294
295! Synchronization barrier
296# 266 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
297
298! Import GPU library module (openacc or omp_lib)
299# 275 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
300
301! Emit code only for AMD compiler
302# 282 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
303
304! Emit code for non-Cray compilers
305# 289 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
306
307! Emit code only for Cray compiler
308# 296 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
309
310! Emit code for non-NVIDIA compilers
311# 303 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
312
313# 305 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
314# 306 "/home/runner/work/MFC/MFC/src/common/include/parallel_macros.fpp"
315! New line at end of file is required for FYPP
316# 2 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp" 2
317
318# 14 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
319
320! Caution: This macro requires the use of a binding script to set CUDA_VISIBLE_DEVICES, such that we have one GPU device per MPI
321! rank. That's because for both cudaMemAdvise (preferred location) and cudaMemPrefetchAsync we use location = device_id = 0. For an
322! example see misc/nvidia_uvm/bind.sh.
323# 52 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
324
325! Allocate and create GPU device memory
326# 72 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
327
328! Free GPU device memory and deallocate
329# 80 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
330
331! Cray-specific GPU pointer setup for vector fields
332# 104 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
333
334! Cray-specific GPU pointer setup for scalar fields
335# 120 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
336
337! Cray-specific GPU pointer setup for acoustic source spatials
338# 145 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
339
340# 151 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
341
342# 158 "/home/runner/work/MFC/MFC/src/common/include/macros.fpp"
343! New line at end of file is required for FYPP
344# 8 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp" 2
345
346module m_boundary_io
347
348 use m_derived_types
349 use m_global_parameters
350 use m_mpi_proxy
351 use m_constants
352 use m_delay_file_access
353 use m_compile_specific
354 use m_boundary_primitives
355
356 implicit none
357
358#ifdef MFC_MPI
359 integer, dimension(1:3,1:2) :: mpi_bc_type_type
360 integer, dimension(1:3,1:2) :: mpi_bc_buffer_type
361#endif
362
363contains
364
365 !> Create MPI derived datatypes for boundary condition type arrays and buffer arrays used in parallel I/O.
366 impure subroutine s_create_mpi_types(bc_type)
367
368 type(integer_field), dimension(1:num_dims,1:2), intent(in) :: bc_type
369
370#ifdef MFC_MPI
371 integer :: dir, loc
372 integer, dimension(3) :: sf_start_idx, sf_extents_loc
373 integer :: ierr
374
375 do dir = 1, num_dims
376 do loc = 1, 2
377 sf_start_idx = (/0, 0, 0/)
378 sf_extents_loc = shape(bc_type(dir, loc)%sf)
379
380 call mpi_type_create_subarray(num_dims, sf_extents_loc, sf_extents_loc, sf_start_idx, mpi_order_fortran, &
381 & mpi_integer, mpi_bc_type_type(dir, loc), ierr)
382 call mpi_type_commit(mpi_bc_type_type(dir, loc), ierr)
383 end do
384 end do
385
386 do dir = 1, num_dims
387 do loc = 1, 2
388 sf_start_idx = (/0, 0, 0/)
389 sf_extents_loc = shape(bc_buffers(dir, loc)%sf)
390
391 call mpi_type_create_subarray(num_dims, sf_extents_loc*mpi_io_type, sf_extents_loc*mpi_io_type, sf_start_idx, &
392 & mpi_order_fortran, mpi_io_p, mpi_bc_buffer_type(dir, loc), ierr)
393 call mpi_type_commit(mpi_bc_buffer_type(dir, loc), ierr)
394 end do
395 end do
396#endif
397
398 end subroutine s_create_mpi_types
399
400 !> Write boundary condition type and buffer data to serial (unformatted) restart files.
401 subroutine s_write_serial_boundary_condition_files(q_prim_vf, bc_type, step_dirpath, old_grid_in, q_T_sf)
402
403 type(scalar_field), dimension(sys_size), intent(in) :: q_prim_vf
404 type(integer_field), dimension(1:num_dims,1:2), intent(in) :: bc_type
405 logical, intent(in) :: old_grid_in
406 character(LEN=*), intent(in) :: step_dirpath
407 integer :: dir, loc
408 character(len=path_len) :: file_path
409 character(len=10) :: status
410 type(scalar_field), optional, intent(in) :: q_T_sf
411
412 if (old_grid_in) then
413 status = 'old'
414 else
415 status = 'new'
416 end if
417
418 call s_pack_boundary_condition_buffers(q_prim_vf, q_t_sf)
419
420 file_path = trim(step_dirpath) // '/bc_type.dat'
421 open (1, file=trim(file_path), form='unformatted', status=status)
422 do dir = 1, num_dims
423 do loc = 1, 2
424 write (1) bc_type(dir, loc)%sf
425 end do
426 end do
427 close (1)
428
429 file_path = trim(step_dirpath) // '/bc_buffers.dat'
430 open (1, file=trim(file_path), form='unformatted', status=status)
431 do dir = 1, num_dims
432 do loc = 1, 2
433 write (1) bc_buffers(dir, loc)%sf
434 end do
435 end do
436 close (1)
437
438 end subroutine s_write_serial_boundary_condition_files
439
440 !> Write boundary condition type and buffer data to per-rank parallel files using MPI I/O.
441 subroutine s_write_parallel_boundary_condition_files(q_prim_vf, bc_type, q_T_sf)
442
443 type(scalar_field), dimension(sys_size), intent(in) :: q_prim_vf
444 type(integer_field), dimension(1:num_dims,1:2), intent(in) :: bc_type
445 integer :: dir, loc
446 character(len=path_len) :: file_loc, file_path
447 type(scalar_field), intent(in), optional :: q_T_sf
448
449#ifdef MFC_MPI
450 integer :: ierr
451 integer :: file_id
452 character(len=7) :: proc_rank_str
453 logical :: dir_check
454 integer :: nelements
455
456 call s_pack_boundary_condition_buffers(q_prim_vf, q_t_sf)
457
458 file_loc = trim(case_dir) // '/restart_data/boundary_conditions'
459 if (proc_rank == 0) then
460 call my_inquire(file_loc, dir_check)
461 if (dir_check .neqv. .true.) then
462 call s_create_directory(trim(file_loc))
463 end if
464 end if
465
466 call s_create_mpi_types(bc_type)
467
468 call s_mpi_barrier()
469
470 call s_delay_file_access(proc_rank)
471
472 write (proc_rank_str, '(I7.7)') proc_rank
473 file_path = trim(file_loc) // '/bc_' // trim(proc_rank_str) // '.dat'
474 call mpi_file_open(mpi_comm_self, trim(file_path), mpi_mode_create + mpi_mode_wronly, mpi_info_null, file_id, ierr)
475
476 ! Write bc_types
477 do dir = 1, num_dims
478 do loc = 1, 2
479#ifdef MFC_MIXED_PRECISION
480 nelements = sizeof(bc_type(dir, loc)%sf)
481 call mpi_file_write_all(file_id, bc_type(dir, loc)%sf, nelements, mpi_byte, mpi_status_ignore, ierr)
482#else
483 nelements = sizeof(bc_type(dir, loc)%sf)/4
484 call mpi_file_write_all(file_id, bc_type(dir, loc)%sf, nelements, mpi_integer, mpi_status_ignore, ierr)
485#endif
486 end do
487 end do
488
489 ! Write bc_buffers
490 do dir = 1, num_dims
491 do loc = 1, 2
492 nelements = sizeof(bc_buffers(dir, loc)%sf)*mpi_io_type/stp
493 call mpi_file_write_all(file_id, bc_buffers(dir, loc)%sf, nelements, mpi_io_p, mpi_status_ignore, ierr)
494 end do
495 end do
496
497 call mpi_file_close(file_id, ierr)
498#endif
499
500 end subroutine s_write_parallel_boundary_condition_files
501
502 !> Read boundary condition type and buffer data from serial (unformatted) restart files.
503 subroutine s_read_serial_boundary_condition_files(step_dirpath, bc_type)
504
505 character(LEN=*), intent(in) :: step_dirpath
506 type(integer_field), dimension(1:num_dims,1:2), intent(inout) :: bc_type
507 integer :: dir, loc
508 logical :: file_exist
509 character(len=path_len) :: file_path
510
511 ! Read bc_types
512
513 file_path = trim(step_dirpath) // '/bc_type.dat'
514 inquire (file=trim(file_path), exist=file_exist)
515 if (.not. file_exist) then
516 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
517 end if
518
519 open (1, file=trim(file_path), form='unformatted', status='unknown')
520 do dir = 1, num_dims
521 do loc = 1, 2
522 read (1) bc_type(dir, loc)%sf
523
524# 186 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
525#if defined(MFC_OpenACC)
526# 186 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
527!$acc update device(bc_type(dir, loc)%sf)
528# 186 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
529#elif defined(MFC_OpenMP)
530# 186 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
531!$omp target update to(bc_type(dir, loc)%sf)
532# 186 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
533#endif
534 end do
535 end do
536 close (1)
537
538 ! Read bc_buffers
539 file_path = trim(step_dirpath) // '/bc_buffers.dat'
540 inquire (file=trim(file_path), exist=file_exist)
541 if (.not. file_exist) then
542 call s_mpi_abort(trim(file_path) // ' is missing. Exiting.')
543 end if
544
545 open (1, file=trim(file_path), form='unformatted', status='unknown')
546 do dir = 1, num_dims
547 do loc = 1, 2
548 read (1) bc_buffers(dir, loc)%sf
549
550# 202 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
551#if defined(MFC_OpenACC)
552# 202 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
553!$acc update device(bc_buffers(dir, loc)%sf)
554# 202 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
555#elif defined(MFC_OpenMP)
556# 202 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
557!$omp target update to(bc_buffers(dir, loc)%sf)
558# 202 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
559#endif
560 end do
561 end do
562 close (1)
563
564 end subroutine s_read_serial_boundary_condition_files
565
566 !> Read boundary condition type and buffer data from per-rank parallel files using MPI I/O.
567 subroutine s_read_parallel_boundary_condition_files(bc_type)
568
569 type(integer_field), dimension(1:num_dims,1:2), intent(inout) :: bc_type
570 integer :: dir, loc
571 character(len=path_len) :: file_loc, file_path
572
573#ifdef MFC_MPI
574 integer :: ierr
575 integer :: file_id
576 character(len=7) :: proc_rank_str
577 logical :: dir_check
578 integer :: nelements
579
580 file_loc = trim(case_dir) // '/restart_data/boundary_conditions'
581
582 if (proc_rank == 0) then
583 call my_inquire(file_loc, dir_check)
584 if (dir_check .neqv. .true.) then
585 call s_mpi_abort(trim(file_loc) // ' is missing. Exiting.')
586 end if
587 end if
588
589 call s_create_mpi_types(bc_type)
590
591 call s_mpi_barrier()
592
593 call s_delay_file_access(proc_rank)
594
595 write (proc_rank_str, '(I7.7)') proc_rank
596 file_path = trim(file_loc) // '/bc_' // trim(proc_rank_str) // '.dat'
597 call mpi_file_open(mpi_comm_self, trim(file_path), mpi_mode_rdonly, mpi_info_null, file_id, ierr)
598
599 ! Read bc_types
600 do dir = 1, num_dims
601 do loc = 1, 2
602#ifdef MFC_MIXED_PRECISION
603 nelements = sizeof(bc_type(dir, loc)%sf)
604 call mpi_file_read_all(file_id, bc_type(dir, loc)%sf, nelements, mpi_byte, mpi_status_ignore, ierr)
605#else
606 nelements = sizeof(bc_type(dir, loc)%sf)/4
607 call mpi_file_read_all(file_id, bc_type(dir, loc)%sf, nelements, mpi_integer, mpi_status_ignore, ierr)
608#endif
609
610# 252 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
611#if defined(MFC_OpenACC)
612# 252 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
613!$acc update device(bc_type(dir, loc)%sf)
614# 252 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
615#elif defined(MFC_OpenMP)
616# 252 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
617!$omp target update to(bc_type(dir, loc)%sf)
618# 252 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
619#endif
620 end do
621 end do
622
623 ! Read bc_buffers
624 do dir = 1, num_dims
625 do loc = 1, 2
626 nelements = sizeof(bc_buffers(dir, loc)%sf)*mpi_io_type/stp
627 call mpi_file_read_all(file_id, bc_buffers(dir, loc)%sf, nelements, mpi_io_p, mpi_status_ignore, ierr)
628
629# 261 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
630#if defined(MFC_OpenACC)
631# 261 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
632!$acc update device(bc_buffers(dir, loc)%sf)
633# 261 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
634#elif defined(MFC_OpenMP)
635# 261 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
636!$omp target update to(bc_buffers(dir, loc)%sf)
637# 261 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
638#endif
639 end do
640 end do
641
642 call mpi_file_close(file_id, ierr)
643#endif
644
645 end subroutine s_read_parallel_boundary_condition_files
646
647 !> Pack primitive variable boundary slices into bc_buffers arrays for serialization.
648 subroutine s_pack_boundary_condition_buffers(q_prim_vf, q_T_sf)
649
650 type(scalar_field), dimension(sys_size), intent(in) :: q_prim_vf
651 integer :: i, j, k
652 type(scalar_field), intent(in), optional :: q_T_sf
653
654 do k = 0, p
655 do j = 0, n
656 do i = 1, sys_size
657 bc_buffers(1, 1)%sf(i, j, k) = q_prim_vf(i)%sf(0, j, k)
658 bc_buffers(1, 2)%sf(i, j, k) = q_prim_vf(i)%sf(m, j, k)
659 end do
660 if ((chemistry .or. heat_conduction) .and. present(q_t_sf)) then
661 bc_buffers(1, 1)%sf(sys_size + 1, j, k) = q_t_sf%sf(0, j, k)
662 bc_buffers(1, 2)%sf(sys_size + 1, j, k) = q_t_sf%sf(m, j, k)
663 end if
664 end do
665 end do
666
667# 291 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
668 if (n > 0) then
669 do k = 0, p
670 do j = 1, sys_size
671 do i = 0, m
672 bc_buffers(2, 1)%sf(i, j, k) = q_prim_vf(j)%sf(i, 0, k)
673 bc_buffers(2, 2)%sf(i, j, k) = q_prim_vf(j)%sf(i, n, k)
674 end do
675 end do
676 if ((chemistry .or. heat_conduction) .and. present(q_t_sf)) then
677 do i = 0, m
678 bc_buffers(2, 1)%sf(i, sys_size + 1, k) = q_t_sf%sf(i, 0, k)
679 bc_buffers(2, 2)%sf(i, sys_size + 1, k) = q_t_sf%sf(i, n, k)
680 end do
681 end if
682 end do
683
684# 308 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
685 if (p > 0) then
686 do k = 1, sys_size
687 do j = 0, n
688 do i = 0, m
689 bc_buffers(3, 1)%sf(i, j, k) = q_prim_vf(k)%sf(i, j, 0)
690 bc_buffers(3, 2)%sf(i, j, k) = q_prim_vf(k)%sf(i, j, p)
691 end do
692 end do
693 end do
694 if ((chemistry .or. heat_conduction) .and. present(q_t_sf)) then
695 do j = 0, n
696 do i = 0, m
697 bc_buffers(3, 1)%sf(i, j, sys_size + 1) = q_t_sf%sf(i, j, 0)
698 bc_buffers(3, 2)%sf(i, j, sys_size + 1) = q_t_sf%sf(i, j, p)
699 end do
700 end do
701 end if
702 end if
703# 327 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
704 end if
705# 329 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
706
707 end subroutine s_pack_boundary_condition_buffers
708
709 !> Initialize the per-cell boundary condition type arrays with the global default BC values.
710 subroutine s_assign_default_bc_type(bc_type)
711
712 type(integer_field), dimension(1:num_dims,1:2), intent(in) :: bc_type
713
714 bc_type(1, 1)%sf(:,:,:) = int(min(bc_x%beg, 0), kind=1)
715 bc_type(1, 2)%sf(:,:,:) = int(min(bc_x%end, 0), kind=1)
716
717# 339 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
718#if defined(MFC_OpenACC)
719# 339 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
720!$acc update device(bc_type(1, 1)%sf, bc_type(1, 2)%sf)
721# 339 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
722#elif defined(MFC_OpenMP)
723# 339 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
724!$omp target update to(bc_type(1, 1)%sf, bc_type(1, 2)%sf)
725# 339 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
726#endif
727
728# 342 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
729 if (n > 0) then
730 bc_type(2, 1)%sf(:,:,:) = int(min(bc_y%beg, 0), kind=1)
731 bc_type(2, 2)%sf(:,:,:) = int(min(bc_y%end, 0), kind=1)
732
733# 345 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
734#if defined(MFC_OpenACC)
735# 345 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
736!$acc update device(bc_type(2, 1)%sf, bc_type(2, 2)%sf)
737# 345 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
738#elif defined(MFC_OpenMP)
739# 345 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
740!$omp target update to(bc_type(2, 1)%sf, bc_type(2, 2)%sf)
741# 345 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
742#endif
743# 347 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
744 if (p > 0) then
745 bc_type(3, 1)%sf(:,:,:) = int(min(bc_z%beg, 0), kind=1)
746 bc_type(3, 2)%sf(:,:,:) = int(min(bc_z%end, 0), kind=1)
747
748# 350 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
749#if defined(MFC_OpenACC)
750# 350 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
751!$acc update device(bc_type(3, 1)%sf, bc_type(3, 2)%sf)
752# 350 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
753#elif defined(MFC_OpenMP)
754# 350 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
755!$omp target update to(bc_type(3, 1)%sf, bc_type(3, 2)%sf)
756# 350 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
757#endif
758 end if
759# 353 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
760 end if
761# 355 "/home/runner/work/MFC/MFC/src/common/m_boundary_io.fpp"
762
763 end subroutine s_assign_default_bc_type
764
765end module m_boundary_io
m_boundary_io
Boundary condition restart I/O, capillary/IGR buffer population, and grid-variable buffers.
Definition m_boundary_io.fpp.f90:346
m_boundary_io::mpi_bc_type_type
integer, dimension(1:3, 1:2) mpi_bc_type_type
Definition m_boundary_io.fpp.f90:359
m_boundary_io::s_write_parallel_boundary_condition_files
subroutine s_write_parallel_boundary_condition_files(q_prim_vf, bc_type, q_t_sf)
Write boundary condition type and buffer data to per-rank parallel files using MPI I/O.
Definition m_boundary_io.fpp.f90:442
m_boundary_io::mpi_bc_buffer_type
integer, dimension(1:3, 1:2) mpi_bc_buffer_type
Definition m_boundary_io.fpp.f90:360
m_boundary_io::s_assign_default_bc_type
subroutine s_assign_default_bc_type(bc_type)
Initialize the per-cell boundary condition type arrays with the global default BC values.
Definition m_boundary_io.fpp.f90:711
m_boundary_io::s_read_serial_boundary_condition_files
subroutine s_read_serial_boundary_condition_files(step_dirpath, bc_type)
Read boundary condition type and buffer data from serial (unformatted) restart files.
Definition m_boundary_io.fpp.f90:504
m_boundary_io::s_create_mpi_types
impure subroutine s_create_mpi_types(bc_type)
Create MPI derived datatypes for boundary condition type arrays and buffer arrays used in parallel I/...
Definition m_boundary_io.fpp.f90:367
m_boundary_io::s_pack_boundary_condition_buffers
subroutine s_pack_boundary_condition_buffers(q_prim_vf, q_t_sf)
Pack primitive variable boundary slices into bc_buffers arrays for serialization.
Definition m_boundary_io.fpp.f90:649
m_boundary_io::s_read_parallel_boundary_condition_files
subroutine s_read_parallel_boundary_condition_files(bc_type)
Read boundary condition type and buffer data from per-rank parallel files using MPI I/O.
Definition m_boundary_io.fpp.f90:568
m_boundary_io::s_write_serial_boundary_condition_files
subroutine s_write_serial_boundary_condition_files(q_prim_vf, bc_type, step_dirpath, old_grid_in, q_t_sf)
Write boundary condition type and buffer data to serial (unformatted) restart files.
Definition m_boundary_io.fpp.f90:402
m_boundary_primitives
Per-cell noncharacteristic boundary condition primitives applied in the ghost cells.
Definition m_boundary_primitives.fpp.f90:346
m_boundary_primitives::bc_buffers
type(scalar_field), dimension(:,:), allocatable bc_buffers
Definition m_boundary_primitives.fpp.f90:355
m_compile_specific
Platform-specific file and directory operations: create, delete, inquire, getcwd, and basename.
Definition m_compile_specific.f90:6
m_compile_specific::my_inquire
impure subroutine my_inquire(fileloc, dircheck)
Inquire on the existence of a directory or file.
Definition m_compile_specific.f90:56
m_compile_specific::s_create_directory
impure subroutine s_create_directory(dir_name)
Create a directory and all its parents if it does not exist.
Definition m_compile_specific.f90:17
m_constants
Compile-time constant parameters: default values, tolerances, and physical constants.
Definition m_constants.fpp.f90:7
m_delay_file_access
Rank-staggered file access delays to prevent I/O contention on parallel file systems.
Definition m_delay_file_access.f90:6
m_delay_file_access::s_delay_file_access
impure subroutine, public s_delay_file_access(process_rank)
Introduce a rank-dependent busy-wait delay to stagger parallel file access and reduce I/O contention.
Definition m_delay_file_access.f90:22
m_derived_types
Shared derived types for field data, patch geometry, bubble dynamics, and MPI I/O structures.
Definition m_derived_types.fpp.f90:335
m_global_parameters
Defines global parameters for the computational domain, simulation algorithm, and initial conditions.
Definition m_global_parameters.fpp.f90:19
m_global_parameters::bc_z
type(int_bounds_info) bc_z
Boundary conditions in the x-, y- and z-coordinate directions.
Definition m_global_parameters.fpp.f90:65
m_global_parameters::proc_rank
integer proc_rank
Rank of the local processor Number of cells in the x-, y- and z-coordinate directions.
Definition m_global_parameters.fpp.f90:39
m_global_parameters::bc_y
type(int_bounds_info) bc_y
Definition m_global_parameters.fpp.f90:65
m_global_parameters::bc_x
type(int_bounds_info) bc_x
Definition m_global_parameters.fpp.f90:65
m_mpi_proxy
Broadcasts user inputs and decomposes the domain across MPI ranks for pre-processing.
Definition m_mpi_proxy.fpp.f90:7
m_derived_types::integer_field
Derived type annexing an integer scalar field (SF).
Definition m_derived_types.fpp.f90:359
m_derived_types::scalar_field
Derived type annexing a scalar field (SF).
Definition m_derived_types.fpp.f90:349