Active learning of data-assimilation closures using graph neural networks

Abstract

The spread of machine learning techniques coupled with the availability of high-quality experimental and numerical data has significantly advanced numerous applications in fluid mechanics. Notable among these are the development of data assimilation and closure models for unsteady and turbulent flows employing neural networks (NN). Despite their widespread use, these methods often suffer from overfitting and typically require extensive datasets, particularly when not incorporating physical constraints. This becomes compelling in the context of numerical simulations, where, given the high computational costs, it is crucial to establish learning procedures that are effective even with a limited dataset. Here, we tackle those limitations by developing NN models capable of generalizing over unseen data in low-data limit by: (i) incorporating invariances into the NN model using a Graph Neural Networks (GNNs) architecture; and (ii) devising an adaptive strategy for the selection of the data utilized in the learning process. GNNs are particularly well-suited for numerical simulations involving unstructured domain discretization and we demonstrate their use by interfacing them with a Finite Elements (FEM) solver for the supervised learning of Reynolds-averaged Navier–Stokes equations. We consider as a test-case the data-assimilation of meanflows past generic bluff bodies, at different Reynolds numbers $50 \le Re \le 150$, characterized by an unsteady dynamics. We show that the GNN models successfully predict the closure term; remarkably, these performances are achieved using a very limited dataset selected through an active learning process ensuring the generalization properties of the RANS closure term. The results suggest that GNN models trained through active learning procedures are a valid alternative to less flexible techniques such as convolutional NN.

Appendices

Appendix A: Numerical simulations details

Time-resolved numerical simulations were performed using a python code based on the FEniCS [2] library. The inputs and the outputs of the Graph Neural Network model were obtained by averaging these data on-the-fly. More details on the numerical scheme can be found in [60]; the main script is based on the code described in [25], where an extensive discussion is reported. Here we summarize the essential aspects and provides details on the validation.

From the numerical viewpoint, the spatial discretization is obtained by introducing the weak formulation based on the finite element method (FEM). In particular, the finite element used is the Taylor-Hood element, with second order elements P2 for velocity and first order elements P1 for pressure. The implemented spatial integration scheme reads as

$$\begin{aligned} {\left\{ \begin{array}{ll} \begin{array}{r l} \left( \dfrac{3\textbf{u}^n-4\textbf{u}^{n-1} +\textbf{u}^{n-2}}{2\Delta t}\right) +(\textbf{u}^{n-1}\cdot \nabla )\textbf{u}^n+(\textbf{u}^n\cdot \nabla )\textbf{u}^{n-1} & \\ -(\textbf{u}^{n-1}\cdot \nabla )\textbf{u}^{n-1}-\dfrac{1}{Re}\Delta \textbf{u}^{n}+\nabla p^{n}& = \textbf{0} \\ \nabla \cdot \textbf{u}^{n} & =0, \end{array} \end{array}\right. } \end{aligned}$$

(A1)

where $\textbf{u}=(u,v)^T$ represents the velocity vector, p the pressure and Re the Reynolds number. The time marching is performed by second order backward differentiation formula (BFD): the n apex indicates the values of a quantity at the current time, with $n-1$ the values at the previous time step and $n-2$ its values two time steps before. The time step – denoted as $\Delta t$ – it is chosen for granting the Courant-Friedrichs-Lewy condition, $CFL \le 0.5$ for each of the random generated shapes.

Fig. 14

Drag coefficient $C_d$ and lift coefficient $C_l$ of flow past a cylinder

The convective term is treated using the Newton and Picard methods, such that a linear system of equations is solved in each step of the temporal iteration. The basic mesh is based on the one used for the cylinder flow, our reference case; the mesh is refined in the most sensitive regions around the obstacle and in the wake past the cylinder [23]: these are the areas where high refinement is required for increasing the accuracy of the computations in terms of correct frequencies and growth rate of the instabilities. Beside the well known dynamics, the rich literature allows a detailed comparison for validation purposes. Thus, numerical solver and meshes were validated using the cylinder flow, in particular by comparing the drag coefficient $C_d$ and the lift coefficient $C_l$. Figure 14 shows the evolution in time of $C_d$ and $C_l$, while the table Table 2 shows the numerical comparison of (time-averaged) $C_d$ and $C_l$ (amplitude) with respect to values in literature. The first coefficient compares remarkably well with the values in literature, while a slightly higher $C_l$ is found: this can be related to the chosen numerical box that is slightly small along the y direction, thus producing some numerical blockage effects; nonetheless, for the nature of the present work, the results are satisfactory. Finally, the chosen bluff bodies are characterized by mechanisms resembling the phenomena observed in the cylinder flow, thus characterized by comparable time and space scales; the corresponding meshes are characterized by a refinement similar to the one adopted for the reference case.

Table 2 Comparison of $C_d$ average and $C_l$ amplitude

Appendix B: Hyperparameters optimization

Numerous parameters define the structure of a neural network, usually denoted with the term hyperparameters. We can distinguish between model hyperparameters and process hyperparameters.

• A model hyperparameter defines the capacity of the neural network, i.e. the ability of the model to represent functions of high complexity. Thus, the capacity is directly related to the possibility of approximating a large variety of nonlinear functions.

• A process hyperparameter defines the training phase. Tuning these hyperparameters deeply modify the duration of the training, its computational costs and the way the weights are adjusted while the model evolves.

These terms are defined a-priori, before training the model, thus they need to be manually tuned as they can’t be adjusted or learnt during the training process. In general, this does not pose problems in deep learning when the expressivity of the NN (e.g. the number of neurons, the number of layers, etc.) is enough to represent the complexity of the problem under investigation. On the other hand, since the GNN is trained to fulfil a specific task and because the computational cost of the GNN inference has to be affordable compared to the most sophisticated turbulence models available today, we tried to keep the GNN as parsimonious as possible.

To this end, the hyperparameters defining the architecture require optimization. Standard gradient based optimizers cannot be employed when dealing with integer numbers, like the number of neurons or layers. For this purpose gradient-free algorithms can be used. There exists dedicated libraries that can automate the tuning process through all the possible sets of hyperparameters by trying and appropriately pruning the unpromising sets of them. In this work we apply the library Optuna [1], an open-source package that combines efficiently searching and pruning algorithms. By exploring the complex solution hyperspace, a number of combinations of hyperparameters is found, among which the one outperforming the others in terms of monitored validation metrics is given by the following set

• 1. Embedded dimension, 35

• 2. Number of GNN layers, $k = 40$

• 3. Update relaxation weight, $\alpha = 6\times 10^{-1}$

• 4. Loss function weight, $\gamma = 0.1$

• 5. Learning rate, $LR = 3\times 10^{-3}$, as maximal/starting value.

Appendix C: Similarity criteria algorithm details

The similarity criteria algorithm designed to compare different data from the neural network perspective is based on the analysis of the vector gradients of a metric function (Eq. C3) with respect to the $\mathbf {\theta }$ parameters of the neural network. In particular, the similarity comparison between two generic m-dimensional vectors ${\textbf {a}} \in \mathbb {R}^m$ and ${\textbf {b}} \in \mathbb {R}^m$ is computed using the cosine similarity, defined as

$$\begin{aligned} \cos (\beta ) = \dfrac{\textbf{a} \cdot \textbf{b}}{\left\| \textbf{a}\right\| \left\| \textbf{b}\right\| }, \end{aligned}$$

(C2)

where $\beta$ is the angle between the two vectors. In this context the metric we use is the Mean Absolute Error (MAE), a piecewise linear function defined as

$$\begin{aligned} MAE = \sum _{i=1}^{n_i}|x_i-y_i|, \end{aligned}$$

(C3)

in which $x_i$ is the NN prediction on the node i, $y_i$ the ground truth and $n_i$ the number of nodes.

Fig. 15

Visualization of gradient auto-similarity convergence over multiple training epochs using MAE for 9 distinct cases within the training data set

The choice of MAE is crucial for our analysis. In scenarios where the Mean Squared Error (MSE) is used, we empirically observe marked oscillations in the direction of the auto-similarity of the vector gradient throughout successive training epochs. Conversely, when employing MAE, the auto-similarity of the vector gradient tends to approach unity, suggesting a stable direction in the solution space, for the data under analysis. A visual representation of the consistent convergence of the vector gradient auto-similarity as function of the epochs is shown in Fig. 15, for a training process involving 9 cases in the training dataset. Training begins with a specific initial dataset; when every data point in the training dataset achieves an auto-similarity convergence exceeding a predefined threshold of 0.99, the training is stopped and we can assume that the vector gradient’s direction for each instance in the training dataset has stabilized.

The following step is to assess the similarity between cases within the training dataset and those outside it. This aims at identifying the most diverse cases among those not included in the training set, which will then be added to the training dataset to enhance diversity. Firstly, the GNN runs for additional 10 epochs to obtain the vector gradients of each out-of-training dataset instance. Then, a similarity matrix is computed by cross calculating the similarity between each in-training instance and each out-of-training instance. The case that shows the lowest similarity score is also the most diverse one and enables to promote diversification in the training dataset based on available data. Note that the values are normalized using a z-score value

$$\begin{aligned} z = \frac{S - \mu }{\sigma }, \end{aligned}$$

(C4)

where S represents the similarity score for a specific data point, $\mu$ is the mean of all similarity scores, and $\sigma$ their standard deviation. In summary, the approach outlined here serves as a robust method for comparing and evaluating the similarities in data behaviour leveraging the neural network model, thereby enhancing the efficacy of the training process.