GPT-2
Use-cases
Transformer-based language model for text generation.
Description
GPT-2 is a large transformer-based language model with a simple objective: predict the next word, given all of the previous words within some text.
Model
| Model | Download | Download (with sample test data) | ONNX version | Opset version | Accuracy |
|---|---|---|---|---|---|
| GPT-2 | 522.81 MB | 438.3 MB | 1.6 | 10 | mAP of 0.024 |
| GPT-2-LM-HEAD | 664.87 MB | 607 MB | 1.6 | 10 | mAP of 0.024 |
Source
PyTorch GPT-2 ==> ONNX GPT-2 PyTorch GPT-2 + script changes ==> ONNX GPT-2-LM-HEAD
Inference
The script for ONNX model conversion and ONNX Runtime inference is here.
Input to model
Sequence of words as a string. Example: "Here is some text to encode : Hello World", tokenized by Byte-Pair-Encoding. input_ids: Indices of input tokens in the vocabulary. It's a long tensor of dynamic shape (batch_size, sequence_length).
Preprocessing steps
Use tokenizer.encode() to encode the input text:
text = "Here is some text to encode : Hello World"
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokens_tensor = torch.tensor([torch.tensor(tokenizer.encode(text))])
Output of model
For GPT-2 model:
last_hidden_state: Sequence of hidden-states at the last layer of the model. It's a float tensor of size (batch_size, sequence_length, hidden_size). past: pre-computed hidden-states. It's a list of tensors (key and values in the attention blocks) of size (batch_size, num_heads, sequence_length, sequence_length), one per each layer.
Output of this model is the tuple (last_hidden_state, past)
For GPT-2-LM-HEAD model:
prediction_scores: Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). It's a float tensor of size (batch_size, sequence_length, vocab_size). past: pre-computed hidden-states. It's a list of tensors (key and values in the attention blocks) of size (batch_size, num_heads, sequence_length, sequence_length), one per each layer.
Output of this model is the tuple (prediction_scores, past)
Note that output_hidden_states=False and output_attentions=False in the PretrainedConfig configs.
Postprocessing steps
For GPT-2 model:
outputs = model(input_ids)
last_hidden_states = outputs[0]
For GPT-2-LM-HEAD model, to generate next 10 words:
import numpy as np
import torch
import torch.nn.functional as F
from transformers import GPT2Tokenizer
batch_size = 1
length = 10
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
text = "Here is some text to encode : Hello World!"
tokens = np.array(tokenizer.encode(text))
context = torch.tensor(tokens, device=device, dtype=torch.long).unsqueeze(0).repeat(batch_size, 1)
prev = context
output = context
for i in range(length):
outputs = model(prev)
logits = outputs[0]
logits = logits[:, -1, :]
log_probs = F.softmax(logits, dim=-1)
_, prev = torch.topk(log_probs, k=1, dim=-1)
output = torch.cat((output, prev), dim=1)
output = output[:, len(tokens):].tolist()
generated = 0
for i in range(batch_size):
generated += 1
text = tokenizer.decode(output[i])
print(text)
Dataset (Train and validation)
The original model from OpenAI is pretrained on a dataset of 8 million web pages. The pretrained model is referenced in huggingface/transformers repository as a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus of ~40 GB of text data. https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin
Validation accuracy
Metric and benchmarking details are provided by HuggingFace in this post.
Publication/Attribution
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, andIlya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
References
This model is converted directly from huggingface/transformers.
Contributors
Negin Raoof Joddiy Zhang
License
Apache 2.0 License