Open Catalyst Challenge

The Open Catalyst Challenge 2022 invites participants to help in addressing the pressing challenges faced by the world due to energy scarcity and climate change. In this area, a critical problem is the discovery of new catalysts for driving efficient and carbon neutral means for energy storage and generation. A common approach in discovering high performance catalysts is using molecular simulations, where simpler surrogate descriptors are generated to correlate with experimental measurements of catalyst activity and selectivity. The task for this year's challenge is to design new machine learning models to predict the outcome of catalyst simulations used to understand activity.

Specifically, each simulation models the interaction of a catalyst surface with adsorbates that are commonly seen in electrochemical reactions. The simulations correspond to local relaxations of the atomic positions to identify local minima. ML models are trained to predict the energies of the adsorbate-catalyst system at the local minima, known as the “relaxed state”, starting from a provided initial state. By predicting these interactions accurately, the catalyst's impact on the overall rate of a chemical reaction may be estimated; a key factor in filtering potential electrocatalysis materials and addressing the world's energy needs.

This is the 2nd edition of the Open Catalyst Challenge. The 1st edition was held last year, and the results announced at NeurIPS 2021. More details about last year's challenge can be found here.

This year's challenge focuses on the same task -- Initial Structure to Relaxed Energy (IS2RE) -- as last year (details here). The primary differences are: 1) instead of two tracks, we will have a single track where using the IS2RE data and/or the Structure-to-Energy-Forces (S2EF) 2M training data is allowed (details here). 2) A new test-challenge split will be released in September specifically for this year's challenge.

The Open Catalyst Challenge session at NeurIPS 2022 was held on Thursday, December 08, 2022.

The entire session was recorded and is viewable here.

Schedule

15:00 - 15:05 CST	Buffer for attendees to join Zoom [room link]	-
15:05 - 15:20 CST	Challenge overview, results and analysis [video]	Abhishek Das (Open Catalyst Project team)
15:20 - 15:50 CST	Invited talk + Q&A [video]	Gábor Csányi
15:50 - 16:10 CST	Runner-up talk + Q&A [video]	Yi-Lun Liao, Tess Smidt (Atomic Architects, MIT)
16:10 - 16:30 CST	Winner talk + Q&A [video]	Jiaqi Han, Tian Bian, Geyan Ye, Kaili Ma, Yuduo Zhi, Kangfei Zhao, Tingyang Xu, Wenbing Huang, Yu Rong (Tencent AI Lab, Tsinghua University, Renmin University of China, The Chinese University of Hong Kong)
16:30 - 17:00 CST	Invited talk + Q&A [video]	Yoshua Bengio
17:00 - 17:30 CST	Discussion [video]	Attendees, participants, organizers

The Open Catalyst Challenge received 25 submissions in total from 6 teams. All submissions were evaluated on the test-challenge-2022 dataset split consisting of the following 4 subsplits:

• test-like: similar to OC20 test and used to pick winners
• rotated: used to evaluate rotational invariance
• anomalous: structures with desorptions and dissociations
• dense: dense sampling of adsorbate placements for evaluating recall of lowest energy site

The challenge will consist of one primary task -- Initial Structure to Relaxed Energy (IS2RE) [1]. Here the input consists of the atomic positions for an initial structure, and the goal is to predict the energy of the structure's relaxed state.

Relaxed energies are a critical indicator in determining the reaction rate resulting from the use of a catalyst. By placing an adsorbate in multiple locations above a catalyst's surface and relaxing the structure, the binding site between the adsorbate and catalyst with the lowest relaxed energy can be determined. This lowest energy binding site is likely to be the one realized in practice under experimental conditions. The relaxed energy of the lowest energy binding site is also highly correlated with the reaction rates or selectivity of the chemical reaction. If successful, these techniques could be used to screen millions or even billions of potential catalyst materials for the chemical reactions involved in renewable energy storage and solar fuel generation.

Traditionally, relaxed energies are found by first performing structure relaxations through an iterative local optimization process that estimates the gradients (atomic forces) using Density Functional Theory (DFT), which are in turn used to update atom positions until convergence. This very computationally expensive process typically requires hundreds of DFT calculations to converge (hours or days of compute per relaxation) and forms the basis of most computational catalysis efforts.

One approach to the IS2RE task is relaxation-based, i.e. using ML to approximate DFT relaxations. These models iteratively estimate atomic forces and update atomic positions until a relaxed state is reached and finally predict the energy of that state. Evaluation of the IS2RE task on models built for approximating DFT relaxations will help determine whether this approach is sufficiently accurate and fast for practical applications. These models have the additional benefit of predicting the relaxed structure and accelerating future DFT calculations.

Alternatively, it may be possible to develop direct approaches that predict the relaxed energy directly, without estimating intermediate relaxation states, as many of the changes during a relaxation (say due to particular initial guess strategies) are systematic. These direct IS2RE approaches may lead to even greater improvements in computational efficiency.

As such, we place no restrictions on the possible ML approaches to solve this task and used to participate in this challenge. We encourage submissions that are significantly more computationally efficient than DFT. For example, a standard relaxation using DFT takes 8-10 hours, while ML approaches are desired that can bring this down to < 10 seconds per relaxation or < 1 second per direct prediction, at least a 1000x improvement!

To ensure consistent and fair evaluation, we use a public evaluation server hosted on EvalAI.

The challenge will be conducted on the Open Catalyst Dataset (OC20). OC20 training and validation data are available here. A new test-challenge-2022 split has been released here specifically for this challenge. This is to ensure there is no overfitting on the test data through repeated submissions.

OC20 contains approximately ~1.2M DFT relaxations. Due to its significant scale, the dataset required over ~200M hours of compute to generate. Computation was performed on servers Meta has committed to be 100% supported by renewable energy since 2020. Each relaxation contains a series of structures as the atoms move from an initial structure to a relaxed structure obtained through a standard local minimizer built in the computational chemistry code. Structures contain the atoms corresponding to the adsorbate and catalyst. The initial structures are heuristically determined and the relaxed structures correspond to a state in which the atoms are at a local energy minima. Since each step in the relaxation may be used for training / evaluation, the total number of simulation points is over ~264M! The largest training split has ~134M simulation points. However, for this challenge, only the 2M dataset may be used for training. For each structure, DFT computed system energies, per-atom forces and per-atom positions are available as annotations.

The OC20 validation and test splits have several subsplits to help evaluate a model's performance on interpolative and extrapolative tasks. A model's interpolative ability is evaluated on samples from the same distribution as the training dataset (In Domain). Extrapolation is evaluated on two dimensions -- new adsorbates and new catalyst compositions. Subsplits are created by considering all combinations of potential extrapolations -- Out-of-Domain Adsorbate (OOD Adsorbate), OOD Catalyst, and OOD Both (both unseen adsorbate and unseen catalyst compositions).

Summary of all evaluation splits

All submissions to the Open Catalyst Challenge will be made to the EvalAI server and evaluated on the following metrics:

• Energy MAE: mean absolute error between the predicted relaxed energy and the DFT-computed ground-truth relaxed energy.
• Energy within Threshold (EwT): the percentage of predicted relaxed energies within 0.02 eV of the DFT-computed ground-truth relaxed energy.

Challenge winners will be decided based on the Energy MAE metric.

The challenge will have a single track, wherein participants are allowed to train on the IS2RE dataset (size 460k) and/or the S2EF 2M dataset. We are expanding the scope from last year to include and encourage training on intermediate trajectory data from S2EF (in addition to IS2RE direct data) because we have seen that to consistently improve performance. We acknowledge that resource availability may become a bottleneck for some participants and hence will not be accepting entries that train on S2EF splits larger than 2M.

The table below summarizes training compute costs and accuracies of various direct and relaxation-based IS2RE baseline approaches. Training a relaxation-based GemNet-OC S2EF-2M model is about twice as expensive as training a direct Graphormer model, but improves energy MAE by ~20%! Training on S2EF-All (in red) improves accuracies further but gets prohibitively expensive, and hence training on S2EF-All is not allowed for challenge entrants. Our non-challenge test evaluation server and leaderboards are open all year round for these larger trained models. The North star of this challenge will be to marry the accuracy of relaxation-based models with the efficiency of direct models.

Participants will be prompted while making submissions to EvalAI to confirm that they didn't use any training data outside of the IS2RE and the S2EF-2M datasets. Data augmentation and pretraining is permitted as long as it comes only from the IS2RE or S2EF-2M datasets. Using DFT is not allowed.

We will be inviting the winner and runner-up teams, and optionally teams with interesting entries (e.g. best direct approach) for oral presentations at NeurIPS 2022.

To participate in the Open Catalyst Challenge, create a team on EvalAI and upload submissions to the "Predicting relaxed state energy from initial structure (IS2RE) -- Test-challenge-2022" phase:

Submissions must be an `.npz` numpy binary file in the following format:

          {
            "challenge_ids": array(['0', '1', ...]),
            "challenge_energy": array([-3.63920, -1.08237, 12.92103, ...,])
          }
                      

          >>> data["challenge_ids"].dtype, data["challenge_ids"].shape
          (dtype('<U6'), (100010,))
          >>> data["challenge_energy"].dtype, data["challenge_energy"].shape
          (dtype('float64'), (100010,))
                      

The Open Catalyst Challenge invites participants to help in addressing the pressing challenges faced by the world due to energy scarcity and climate change. In this area, a critical problem is the discovery of new catalysts for driving efficient and carbon neutral means for energy storage and generation. A common approach in discovering high performance catalysts is using molecular simulations, where simpler surrogate descriptors are generated to correlate with experimental measurements of catalyst activity and selectivity. The task for this year’s challenge is to design new machine learning models to predict the outcome of catalyst simulations used to understand activity.

Specifically, each simulation models the interaction of a catalyst surface with adsorbates that are commonly seen in electrochemical reactions. The simulations correspond to local relaxations of the atomic positions to identify local minima. ML models are trained to predict the energies of the adsorbate-catalyst system at the local minima, known as the “relaxed state”, starting from a provided initial state. By predicting these interactions accurately, the catalyst's impact on the overall rate of a chemical reaction may be estimated; a key factor in filtering potential electrocatalysis materials and addressing the world's energy needs.

The Open Catalyst Challenge session at NeurIPS 2021 is scheduled for Tuesday, December 07, 2021, 18:05 GMT (or 10:05 AM PST) onwards. Note that it is mandatory to register for NeurIPS to attend the session.

The session starts with a broadcast of the challenge overview and results at 18:05 GMT on the NeurIPS website here, followed by a breakout session on Zoom.

Schedule

The Open Catalyst Challenge received 30 submissions in total from 7 teams. All submissions were to the IS2RE-only track, and were evaluated on the test-challenge dataset split consisting of the following 4 subsplits:

• test-like: similar to OC20 test and used to pick winners
• rotated: used to evaluate rotational invariance
• anomalous: structures with desorptions and dissociations
• dense: dense sampling of adsorbate placements for evaluating recall of lowest energy site

The challenge will consist of one primary task -- Initial Structure to Relaxed Energy (IS2RE) [1]. Here the input consists of the atomic positions for an initial structure, and the goal is to predict the energy of the structure’s relaxed state.

Relaxed energies are a critical indicator in determining the reaction rate resulting from the use of a catalyst. By placing an adsorbate in multiple locations above a catalyst's surface and relaxing the structure, the binding site between the adsorbate and catalyst with the lowest relaxed energy can be determined. This lowest energy binding site is likely to be the one realized in practice under experimental conditions. The relaxed energy of the lowest energy binding site is also highly correlated with the reaction rates or selectivity of the chemical reaction. If successful, these techniques could be used to screen millions or even billions of potential catalyst materials for the chemical reactions involved in renewable energy storage and solar fuel generation.

Traditionally, relaxed energies are found by first performing structure relaxations through an iterative local optimization process that estimates the gradients (atomic forces) using Density Functional Theory (DFT), which are in turn used to update atom positions until convergence. This very computationally expensive process typically requires hundreds of DFT calculations to converge (hours or days of compute per relaxation) and forms the basis of most computational catalysis efforts.

One approach to the IS2RE task is using ML to approximate DFT relaxations i.e. iteratively estimate atomic forces and update atomic positions until a relaxed state is reached and finally predict the energy of that state. Evaluation of the IS2RE task on models built for approximating DFT relaxations will help determine whether this approach is sufficiently accurate and fast for practical applications. These models have the additional benefit of predicting the relaxed structure and accelerating future DFT calculations. Alternatively, it may be possible to predict the relaxed energy directly, without estimating intermediate relaxation states, as many of the changes during a relaxation (say due to particular initial guess strategies) are systematic. These direct IS2RE approaches may lead to even greater improvements in computational efficiency. As such, we place no restrictions on the possible ML approaches to solve this task and used to participate in this challenge. We encourage submissions that are significantly more computationally efficient than DFT. For example, a standard relaxation using DFT takes 8-10 hours, while ML approaches are desired that can bring this down to < 10 seconds per relaxation or < 1 second per direct prediction, at least a 1000x improvement!

To ensure consistent and fair evaluation, we use a public evaluation server hosted on EvalAI.

The challenge will be conducted on the Open Catalyst Dataset (OC20). OC20 training and validation data are available here. A new test-challenge split has been released here specifically for this challenge. This is to ensure there is no overfitting on the test data through repeated submissions.

OC20 contains approximately ~1.2M DFT relaxations. Due to its significant scale, the dataset required over ~70M hours of compute to generate. Computation was performed on servers Meta has committed to be 100% supported by renewable energy since 2020. Each relaxation contains a series of structures as the atoms move from an initial structure to a relaxed structure obtained through a standard local minimizer built in the computational chemistry code. Structures contain the atoms corresponding to the adsorbate and catalyst. The initial structures are heuristically determined and the relaxed structures correspond to a state in which the atoms are at a local energy minima. Since each step in the relaxation may be used for training / evaluation, the total number of simulation points is over ~264M! The largest training split has ~134M simulation points. For each structure, DFT computed system energies, per-atom forces and per-atom positions are available as annotations.

The OC20 validation and test splits have several subsplits to help evaluate a model's performance on interpolative and extrapolative tasks. A model's interpolative ability is evaluated on samples from the same distribution as the training dataset (In Domain). Extrapolation is evaluated on two dimensions -- new adsorbates and new catalyst compositions. Subsplits are created by considering all combinations of potential extrapolations -- Out-of-Domain Adsorbate (OOD Adsorbate), OOD Catalyst, and OOD Both (both unseen adsorbate and unseen catalyst compositions).

Summary of all evaluation splits

All submissions to the Open Catalyst Challenge will be made to the EvalAI server and evaluated on the following metrics:

• Energy MAE: mean absolute error between the predicted relaxed energy and the DFT-computed ground-truth relaxed energy.
• Energy within Threshold (EwT): the percentage of predicted relaxed energies within 0.02 eV of the DFT-computed ground-truth relaxed energy.

Challenge winners will be decided based on the Energy MAE metric.

We acknowledge that resource availability may become a bottleneck for some participants given the large size of the OC20 trajectory data (~134M training points). Thus, as detailed on the discussion forum, we will be recognizing 2 winners for the challenge based on:

1. The best overall performance with no constraints on data used
2. The best performance using ONLY the IS2RE dataset (size 460,328)

To participate in the Open Catalyst Challenge, create a team on EvalAI and upload submissions to the "Predicting relaxed state energy from initial structure (IS2RE) -- Test-challenge" phase:

Submissions must be an `.npz` numpy binary file in the following format:

          {
            "challenge_ids": array(['0', '1', ...]),
            "challenge_energy": array([-3.63920, -1.08237, 12.92103, ...,])
          }
                      

          >>> data["challenge_ids"].dtype, data["challenge_ids"].shape
          (dtype('<U6'), (120000,))
          >>> data["challenge_energy"].dtype, data["challenge_energy"].shape
          (dtype('float64'), (120000,))