by trl-internal-testing
Open source · 168k downloads · 0 likes
This model, called "dummy GPT2 correct vocab," is a simplified version of GPT-2 designed for testing and evaluation. It generates text in a coherent and natural manner, though its capabilities are more limited compared to more advanced models. Its primary use cases include generating automated responses, simulating dialogues, or experimenting with language architectures. What sets it apart is its lightweight and simplicity, making it ideal for quick tests without requiring significant computational resources. It is particularly suitable for developers looking to validate pipelines or training methods.
from transformers import AutoTokenizer, GPT2Config, GPT2LMHeadModel
config = GPT2Config(n_positions=512, n_embd=32, n_layer=5, n_head=4, n_inner=37, pad_token_id=1023, is_decoder=True)
model = GPT2LMHeadModel(config)
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
model_id = "trl-internal-testing/dummy-GPT2-correct-vocab"
model.push_to_hub(model_id)
tokenizer.chat_template = "{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ ' ' }}{% endif %}{% endfor %}{{ eos_token }}"
tokenizer.push_to_hub(model_id)
config.push_to_hub(model_id)