Numerical Modelling in

Fortran: day 8
Paul Tackley, 2017
 Today’s Goals
1. Introduction to parallel computing
(applicable to Fortran or C; examples are in
Fortran)
2. Finite Prandtl number convection
 Motivation:
To model the Earth,
need a huge number
of grid points / cells
/elements!
• e.g., to fill mantle volume:
– (8 km)3 cells -> 1.9 billion
cells
– (2 km)3 cells -> 123 billion
cells
 Huge problems => huge computer

www.top500.org
 Huge problems => huge computer

www.top500.org
Progress: iPhone > fastest
computer in 1976 (cost: $8
million)

(photo taken at NCAR museum)

 In Switzerland

Each node: 12-core Intel CPU + GPU

Piz Dora

Each node: 2* 18-core Intel CPU

Shared memory: several cpus
(or cores) share the same
memory. Parallelisation can
often be done by the compiler
(sometimes with help, e.g.,
OpenMP instructions in the
code)

Distributed memory:
each cpu has its own
memory. Parallelisation
usually requires
message-passing, e.g.
using MPI (message-
passing interface)
 A brief history of supercomputers

1983-5
4 CPUs,
Shared
memory
1991: 512 CPUs, distributed memory

2010: 224,162 Cores, distributed + shared memory (12 cores per node)
Another possibility: build you own
(“Beowulf” cluster)
Using standard PC cases: or using rack-mounted cases
MPI: message-passing interface
• A standard library for communicating
between different tasks (cpus)
– Pass messages (e.g., arrays)
– Global operations (e.g., sum, maximum)
– Tasks could be on different cpus/cores of the
same node, or on different nodes
• Works with Fortran and C
• Works on everything from a laptop to the
largest supercomputers. 2 versions are:
– http://www.mcs.anl.gov/research/projects/mpic
h2/
– http://www.open-mpi.org/
How to parallelise a code:
worked example
 Example: Scalar Poisson eqn.
∇ u= f
2

Finite-difference approximation:
1
h2 (u i+1 jk )
+ ui−1 jk + uij +1k + uij −1k + uijk +1 + uijk +1 − 6ui, j = f ij

Use iterative approach=>start with u=0, sweep through grid updating

u values according to: 2
n +1 h
u˜ij = u˜ + αRij
n
ij
6
2
Where Rij is the residue (“error”): R = ∇ u˜ − f
Code
Parallelisation: domain decomposition
Single CPU 8 CPUs

CPU 0 CPU 1

CPU 2 CPU 3

CPU 4 CPU 5

CPU 6 CPU 7

Each CPU will do the same operations but on different parts of

the domain
 You need to build parallelization into
the code using MPI
• Any scalar code will run on multiple CPUs, but
will produce the same result on each CPU.
• Code must first setup local grid in relation to
global grid, then handle communication
• Only a few MPI calls needed:
– Init. (MPI_init,MPI_com_size,MPI_com_rank)
– Global combinations (MPI_allreduce)
– CPU-CPU communication (MPI_send,MPI_recv…)
 Boundaries
• When updating points at
edge of subdomain, need
values on neighboring
subdomains
• Hold copies of these locally
using “ghost points”
• This minimizes #of
messages, because they can
be updated all at once
instead of individually
=ghost points
Scalar Grid

Red=boundary points (=0)

Yellow=iterated/solved
(1…n-1)
Parallel grids

Red=ext. boundaries
Green=int. boundaries
Yellow=iterated/solved
 First things the code has to
do:
• Call MPI_init(ierr)
• Find #CPUs using MPI_com_size
• Find which CPU it is, using MPI_com_rank
(returns a number from 0…#CPUs-1)
• Calculate which part of the global grid it is
dealing with, and which other CPUs are
handling neighboring subdomains.
Example: “Hello world” program
 Moving forward
• Update values in subdomain using
‘ghost points’ as boundary condition,
i.e.,
– Timestep (explicit), or
– Iteration (implicit)
• Update ghost points by communicating
with other CPUs
• Works well for explicit or iterative
approaches
 Boundary communication

Step 1: x-faces

Step 2: y-faces (including

corner values from step 1)

[Step 3: z-faces (including corner

values from steps 1 & 2)]

Doing the 3 directions sequentially avoids the need for

additional messages to do edges & corners (=>in 3D, 6
messages instead of 26)
 Main changes
• Parallelisation hidden in
set_up_parallelisation and update_sides
• Many new variables to store
parallelisation information
• Loop limits depend on whether global
domain boundary or local subdomain
 Simplest communication

Not optimal – uses blocking send/receive

Better: using non-blocking (isend/irecv)
Performance: theoretical
analysis
 How much time is spent
communicating?
• Computation time ∝ volume (Nx^3)
• Communication time ∝ surf. area (Nx^2)
• =>Communication/Computation ∝ 1/Nx
• =>Have as many points/cpu as possible!
Is it better to split 1D, 2D or 3D?
• E.g., 256x256x256 points on 64 CPUs
• 1D split: 256x256x4 points/cpu
– Area=2x(256x256)=131,072
• 2D split: 256x32x32 points/cpu
– Area=4x(256x32)=32,768
• 3D split:64x64x64 points/cpu
– Area=6x(64x64)=24,576
• =>3D best but more messages needed
 Model(Time
code performance
per step or iteration)

Computation : t=aN3
Communication : t=nL+bN2 /B
(L=Latency, B=bandwidth)

TOTAL : t= aN3 +nL+bN2 /B

Example: Scalar Poisson equation
2
∇ u= f
t= aN3 +nL+bN2/B
Assume 15 operations/point/iteration & 1 Gflop performance
Þa=15/1e9=1.5e-8
If 3D decomposition, n=6, b=6*4 (single precision)

Gigabit ethernet: L=40e-6 s, B=100 MB/s

Quadrics: L=2e-6 s, B=875 MB/s
Time/iteration vs. #cpus
Quadrics, Gonzales-size cluster
Up to 2e5 CPUs (Quadrics communication)
Efficiency
 Now multigrid V cycles

Smooth 32x32x32

Smooth 16x16x16

Residues (=error)

corrections
Smooth 8x8x8

Exact solution 4x4x4

Application to StagYY
Cartesian or spherical
 StagYY
iterations:
3D Cartesian
Change in scaling from
same-node to cross-node
communication

Simple-minded multigrid:
Very inefficient coarse levels!
Exact coarse solution can take
long time!
New treatment:
follow minima
• Keep #points/core >
minimum (tuned for
system)
• Different for on-
node and cross-node
communication
Multigrid – now (& before): yin-yang

1.8 billion
 Summary
• For very large-scale problems, need to
parallelise code using MPI
• For finite-difference codes, the best method is
to assign different parts of the domain to
different CPUs (“domain decomposition”)
• The code looks similar to before, but with some
added routines to take care of communication
• Multigrid scales fine on 1000s CPUs if:
– Treat coarse grids on subsets of CPUs
– Large enough total problem size
 For more information
• https://computing.llnl.gov/tutorials/parall
el_comp/
• http://en.wikipedia.org/wiki/Parallel_com
puting
• http://www.mcs.anl.gov/~itf/dbpp/
• http://en.wikipedia.org/wiki/Message_Pa
ssing_Interface
 Programming:
Finite Prandtl number convection
(i.e., almost any fluid)

Ludwig Prandtl (1875-1953)

Values of the Prandt number Pr
ν Viscous diffusivity
Pr =
κ Thermal diffusivity

• Liquid metals: 0.004-0.03

• Air: 0.7
• Water: 1.7-12
• Rock: ~1024 !!! (effectively infinite)
Finite-Prandtl number convection

• Existing code assumes infinite Prandtl

number
– also known as Stokes flow
– appropriate for highly-viscous fluids like
rock, honey etc.
• Fluids like water, air, liquid metal have a
lower Prandtl number so equations
must be modified
 Applications for finite Pr
• Outer core (geodynamo)
• Atmosphere
• Ocean
• Anything that’s not solid like the mantle
 Equations
• Conservation of mass (=‘continuity’)
• Conservation of momentum (‘Navier-
Stokes’ equation: F=ma for a fluid)
• Conservation of energy

Claude Navier Sir George Stokes

(1785-1836) (1819-1903)
 Finite Pr Equations
Navier-Stokes equation: F=ma for a fluid Coriolis force
  
⎛ ∂ v  ⎞ 2
ρ⎜ + v ⋅∇v ⎟ = −∇P + ρν∇ v + 2 ρΩ × v + g ρα Tŷ
⎝ ∂t ⎠

Valid for constant viscosity only

“ma”

continuity and energy equations same as before

∂T  
+ v ⋅ ∇T = κ∇ T + Q
2
∇⋅v =0
∂t
ρ=density, ν=kinematic viscosity, g=gravity,
α=thermal expansivity
Non-dimensionalise the equations

• Reduces the number of parameters

• Makes it easier to identify the dynamical
regime
• Facilitates comparison of systems with
different scales but similar dynamics (e.g.,
analogue laboratory experiments
compared to core or mantle)
Non-dimensionalise to thermal
diffusion scales
• Lengthscale D (depth of domain)
• Temperature scale (T drop over domain)
Time to D / κ
2
•
• Velocity to κ / D
Stress to ρνκ / D
2
•
 Nondimensional equations
 ∂T 
∇⋅ v = 0 + v ⋅∇T = ∇ T
2

∂t


1 ⎛ ∂v   ⎞ 2 1  
⎜ + v ⋅ ∇v ⎟ = −∇P + ∇ v + Ω × v + Ra.Tyˆ
Pr ⎝ ∂t ⎠ Ek

ν ν gα∇TD 3
Pr = Ek = 2 Ra =
κ 2ΩD νκ
Prandtl number Ekman number Rayleigh number
 As before, use streamfunction

∂ψ ∂ψ
vx = vy = −
∂y ∂x

Also simplify by assuming 1/Ek=0

 Eliminating pressure
• Take curl of 2D momentum equation: curl
of grad=0, so pressure disappears
 
• Replace velocity by vorticity: ω = ∇ × v
• in 2D only one component of vorticity is
needed (the one perpendicular to the 2D
plane), ∇ ψ = ωz
2

1 ⎛ ∂ω ∂ω ∂ω ⎞ ∂T
⎜ + vx + vy ⎟ = ∇ ω − Ra
2

Pr ⎝ ∂ t ∂x ∂y⎠ ∂x
 => the streamfunction-vorticity
formulation
1 ⎛ ∂ω ∂ω ∂ω ⎞ ∂T
⎜ + vx + vy ⎟ = ∇ ω − Ra
2

Pr ⎝ ∂ t ∂x ∂y⎠ ∂x
⎛ ∂ψ ∂ψ ⎞
∇ ψ = −ω
2
( )
vx ,vy = ⎜ ,− ⎟
⎝ ∂y ∂x ⎠
∂T 
+ v ⋅ ∇T = ∇ 2T + Q
∂t
 Note: Effect of high Pr
1 ⎛ ∂ω ∂ω ∂ω ⎞ ∂T
⎜ + vx + vy ⎟ = ∇ ω − Ra
2

Pr ⎝ ∂ t ∂x ∂y⎠ ∂x
If Pr->infinity, left-hand-side=>0 so equation becomes Poisson
like before:

∂T
∇ ω = Ra
2

∂x
 Taking a timestep
(i) Calculate ψ from ω using: ∇ ψ =ω
2

⎛ ∂ψ ∂ψ ⎞
(ii) Calculate v from ψ ( )
vx ,vy = ⎜ ,− ⎟
⎝ ∂y ∂x ⎠
(iii) Time-step ω and T using explicit finite differences:

∂T ∂T ∂T
= −vx − vy +∇ T
2

∂t ∂x ∂y
∂ω ∂ω ∂ω ∂T
= −vx − vy + Pr ∇ ω − Ra Pr
2

∂t ∂x ∂y ∂x
 T time step is the same as before

Tnew − Told ∂ Told ∂ Told

= −vx − vy + ∇ 2Told
Δt ∂x ∂y
⎛ 2 ∂ Told ∂ Told ⎞
Tnew = Told + Δt ⎜ ∇ Told − vx − vy
⎝ ∂x ∂ y ⎟⎠

w must now be time stepped in a similar way

ω new − ω old ∂ω old ∂ω old ∂ Told

= −vx − vy + Pr ∇ ω old − Ra Pr
2

Δt ∂x ∂y ∂x

⎛ ∂ω old ∂ω old ∂ Told ⎞

ω new = ω old + Δt ⎜ Pr ∇ ω old − vx
2
− vy − Ra Pr
⎝ ∂x ∂y ∂ x ⎟⎠
 Stability condition

h2
Diffusion: dtdiff = adiff
max(Pr,1)

⎛ h h ⎞
Advection: dt adv = aadv min ⎜ ,
⎝ max val(abs(vx)) max val(abs(vy)) ⎟⎠

Combined: dt = min(dtdiff , dtadv )

 Modification of previous
convection program
• Replace Poisson calculation of w with time-
step, done at the same time as T time-step
• Get a compiling code!
• Make sure it is stable and convergent for
values of Pr between 0.01 and 1e2
• Hand in your code, and your solutions to the
test cases in the following slides
• Due date: 18 December (2 weeks from
today)
 Test cases
• All have nx=257, ny=65, Ra=1e5,
total_time=0.1, and random initial T and w
fields, unless otherwise stated

• Due to random start, results will not look

exactly as these, but they should look
similar (i.e.. width of upwellings &
downwellings & boundary layers similar, but
number and placement of
upwellings/downwellings different).
Pr=10
Pr=1
Pr=0.1
Pr=0.01
Pr=0.01, time=1.0
Pr=0.1, Ra=1e7