Can you determine how many ways you can multiply numbers to get a product of 12? For example, 1 * 12, 2 * 6, and so on. This simple example relates to the main question of this challenge.

The problem of finding non-unique solutions is a common issue in various fields, including mathematics and geosciences. Generative algorithms offer a new approach to understanding nonuniqueness by generating multiple scenarios that can help uncover fundamental principles. In this challenge, you will work with a conditional variational autoencoder (CVAE) algorithm. Initially designed by McAliley and Li (2021) to explore the nonuniqueness of gravity fields, your goal is to fine-tune this algorithm's hyperparameters to generate outputs that thoroughly explore the possibilities of a simple mathematical function.

Background

The data for this challenge starts with only three lines: a decreasing straight line (X0), an increasing straight line (X1), and a line that is their product (Y). The big idea here is that we want an algorithm to tell us all the different values of X0 and X1 to get a given Y (i.e., the product of 12 can be found by 1 * 12, 2 * 6, etc.). Figure 1 shows an example of what we are after, with many different cases of X0 and X1 that multiply to give the same Y. 

Figure 1: Example of some possibilities X0 and X1 to get the product of Y.  The goal of this challenge is get the C-VAE algorithm to produce a wide range of possibilities.

Although this problem could be solved fairly straightforwardly for the specific case where Y is the product of X0 and X1, we are curious to find a neural network architecture that could generalize to the case where Y may be any arbitrary function of X0 and X1. This is why we would like you to work with the CVAE architecture (Figure 2), where Y is the condition imposed on the generative model.  

We have provided an example C-VAE model in the starter notebook of the challenge to serve as a starting point. This initial model does not perform well because the generated outputs (X0 and X1) do not cover the full solution space. Coming back to our “product of 12” example, this is equivalent to the CVAE suggest 1.015 * 11.8226601, 1.0155 * 11.816839, etc. Those results are correct but they are sampling a small space of the potential results. This gets us to your primary task in this challenge: customize the CVAE parameters to produce a more diverse range of X0, X1 solutions.

Figure 2: Simplified model architecture for the C-VAE model

In the starter notebook, you can see more detailed specifications of the C-VAE. At a high level, the CVAE architecture has three main components: the Encoder, the latent space, and the Decoder (Figure 2). Please note that the submission file for this challenge is a tensor of pairs (X0, X1). This tensor is passed through the Encoder considered to be a continuous label. The Decoder is conditioned on the target Y.  

To help you experiment and build your solution, we've provided a model parameters dictionary which can be found in the "config_model.yaml." This dictionary lists some hyperparameters that you can tune, and is used to initialize the model.

The following hyperparameters are to be tuned in this challenge:

• learning_rate: the learning rate for the optimizer used during training

• batch_size: the batch size used during training

• beta: regularization parameter for loss function

• wx, wy: weights for reconstruction losses for x, y respectively

• num_epochs: the number of epochs to train for

• latent_space_dim: the dimensionality of the latent space used by the CVAE

If you wish to change the amount and type of layers, you can edit the “src/CVAE_architecture.py” module.   

To quickly get started with this problem check out the provided ‘’’src/CVAE_function.py’’’ module. 

Data and Labels

The dataset provided consists of multidimensional tensors. Inside each first-dimension slice are:  

• Pairs of X0 and X1 tensors 

• Corresponding Y tensor  computed from the pair (X0, X1)) 

 The values of X0 and X1 are limited between a range of 0 to 1 and represent the ordinate values of a straight line for a specified range of points. The range includes 50 points starting from zero. 

 The second tensor represents the corresponding Y values, which are obtained by applying a specified function to the X0 and X1 pairs. These Y values also serve as the ordinate values for the same specified range of points. 

Evaluation

To explain the evaluation metrics, let's reiterate that the primary objective of the challenge is to design a CVAE neural network that can generate a diverse set of paired straight lines that yield the desired expected result of applying a custom function. For example, let's denote generated pairs as {x0', x1'} and Y as a given result where Y = function(x0, x1) and Y' as result of the function (x0',x1'). The submission file is a tensor file (pt) and an example can be found on the Data Tab. 

The Live Scoring algorithm is based on three criteria, two of which are threshold criteria: 

• Pearson coefficient >0.9. This function assesses the straightness of the generated pairs {x0', x1'}. To determine if the generated pairs are straight enough, a threshold of 0.9 is used. This means that the Pearson coefficient must be higher than this value. 

• RMSE < 0.05. To determine the accuracy of the function's output, we calculate the Root Mean Square Error (RMSE) by comparing the given result Y with the generated result Y'. The RMSE threshold required for acceptance is less than 0.05. 

• Coverage: The generated lines x0' and x1' are evaluated based on their coverage of the specified area [0, 1]. In other words, their spread or diversity should be maximized as much as possible. 

  The Coverage assessment is the primary metric for the Live scoring process and it is this number that will be reported on the Predicted and Final Leaderboards.  Coverage Assessment will only be performed if both the Pearson coefficient and RMSE threshold are met. This means that the generated pairs must exhibit sufficient straightness (Pearson coefficient above 0.9), and the function result accuracy must be within the specified threshold (RMSE below 0.05). Only when these conditions are satisfied will the coverage be evaluated. 

Figure 3: Examples of low and high scores for the three criteria that make up scoring in this challenge.

For the Final Evaluation, we will only consider submissions that employ neural network model architectures that can still be considered to belong to the CVAE family. Also, for final evaluation, you will need to submit: 

• Your modified neural net architecture in a Python file. 

• A Model Parameter yaml file. 

• A Hyperparameters yaml file. 

To help with the final submissions, the Starter Notebook has a function that will generate these files for you automatically.  

In scoring final submissions, 95% of a finalist’s score will be based on the same scoring as the Live Scoring algorithm mentioned above. The remaining 5% of the score will assess submissions on the interpretability of their submitted code. The interpretability criterion focuses on the degree of documentation (i.e., docstrings and markdown), clearly stating variables, and reasonably following standard Python style guidelines.  

References

W. Anderson McAliley and Yaoguo Li, (2021), "Machine learning inversion of geophysical data by a conditional variational autoencoder," SEG Technical Program Expanded Abstracts : 1460-1464.