Model Card for CrystaLLM-pi_density

Model Details

Model Description

CrystaLLM-pi_density is a conditional generative model designed for the inverse design of inorganic crystalline materials. It is a fine-tuned version of the CrystaLLM-pi framework, based on a GPT-2 decoder-only architecture. This specific variant employs the Property-Key-Value (PKV) attention mechanism (referred to as "Prefix attention" in the associated preprint) to condition the generation of Crystallographic Information Files (CIFs) on physical density and thermodynamic stability.

The model generates crystal structures (cell parameters and atomic positions) based on two target scalar properties:

1. Density ($g/cm^3$)
2. Energy above convex hull ($E_{hull}$, eV/atom) - a proxy for thermodynamic stability.

• Developed by: Bone et al. (University College London)
• Model type: Autoregressive Transformer with Prefix Attention Conditioning
• Language(s): CIF (Crystallographic Information File) syntax
• License: MIT
• Finetuned from model: c-bone/CrystaLLM-pi_base

Model Sources

• Repository: GitHub: CrystaLLM-pi
• Paper: Discovery and recovery of crystalline materials with property-conditioned transformers (arXiv:2511.21299)
• Dataset: HuggingFace: c-bone/mattergen_den_ehull

Uses

Direct Use

The model is intended for research in materials science, specifically for generating structures with targeted densities, such as porous materials (low density) or radiation shielding candidates (high density). Users can input a desired density (e.g., 2.0 $g/cm^3$) and a stability criterion to generate candidate crystal structures.

Out-of-Scope Use

• Organic Materials: The model was trained exclusively on inorganic crystal structures.
• Large Unit Cells: Due to the context window limit of 1024 tokens, the model cannot reliably generate unit cells containing more than approximately 20 atoms.
• Disordered Systems: The model currently generates ordered structures and does not natively handle partial occupancies.
• Production Deployment: This is a research artifact. Generated structures must be validated via Density Functional Theory (DFT) or other simulation methods before synthesis attempts.

Bias, Risks, and Limitations

• Training Distribution Bias: The model is fine-tuned on the MatterGen Density dataset. Performance may degrade for density targets that are outliers in the training distribution (e.g., extremely dense or extremely porous materials).
• Validity: As an autoregressive language model, it may generate syntactically incorrect CIFs or chemically implausible structures. Post-processing validation is required.

How to Get Started with the Model

For instructions on how to load and run generation with this model, please refer to the _load_and_generate.py script in the CrystaLLM-pi GitHub Repository. This script handles the necessary tokenization, property normalization, and prompt construction required to properly condition the model.

Training Details

Training Data

The model was fine-tuned on the MatterGen Density dataset, containing inorganic structures labeled with density and $E_{hull}$ values.

• Source: Materials Project (via c-bone/mattergen_den_ehull)
• Preprocessing: CIFs are augmented, tokenized, and property values are normalized before injection into the attention mechanism.

Training Procedure

• Architecture: GPT-2 with additional Property-Key-Value (PKV) encoder layers. (~61.6M parameters)
• Mechanism: Continuous property values are projected into the attention mechanism's key-value space (Prefix Tuning), allowing the model to attend to the target properties at every generation step.

Evaluation

Metrics

The model is evaluated based on:

1. Validity: Percentage of generated files that are valid CIFs.
2. MAE: Mean Absolute Error between the target density and the density of the generated structures.
3. VSUN: A composite metric ensuring structures are Valid, Stable, Unique, and Novel.

Results

As reported in the associated preprint, the Prefix attention mechanism demonstrated superior performance in adhering to density targets compared to sequence-level conditioning baselines.

Citation

@misc{bone2025discoveryrecoverycrystallinematerials,
      title={Discovery and recovery of crystalline materials with property-conditioned transformers}, 
      author={Cyprien Bone and Matthew Walker and Kuangdai Leng and Luis M. Antunes and Ricardo Grau-Crespo and Amil Aligayev and Javier Dominguez and Keith T. Butler},
      year={2025},
      eprint={2511.21299},
      archivePrefix={arXiv},
      primaryClass={cond-mat.mtrl-sci},
      url={[https://arxiv.org/abs/2511.21299](https://arxiv.org/abs/2511.21299)}, 
}