shashankyadav03/Llama-3-8B-Instruct-Finance-Asset-Strategy

Model Card for Llama 3.1

Model Details

Model Developer: Shashank
Model Architecture: Llama 3.1, an auto-regressive language model utilizing an optimized transformer architecture. The instruction-tuned versions leverage supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) for improved alignment with human preferences.

Supported Languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

Model Release Date: August 10, 2024
Model Sizes: 8B, 70B, 405B parameters

Model Description

Llama 3.1 is a collection of multilingual large language models designed for a variety of natural language generation tasks. These models are optimized for multilingual dialogue use cases and excel in various industry benchmarks. The Llama 3.1 collection includes both pretrained and instruction-tuned versions.

How to Get Started with the Model

Use with Transformers

To use the model with the Transformers library:

import transformers

model_id = "shashankyadav03/Llama-3-8B-Instruct-Finance-Asset-Strategy"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are an expert finance asset advice expert"},
    {"role": "user", "content": "Should i invest in crypto in 2024"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Tool Use with Transformers

Llama 3.1 supports tool use formats through chat templates in Transformers. You can define custom tools and integrate them into chat sessions.

Use with Llama Codebase

For usage with the original Llama codebase, follow the instructions provided in the repository.

Training Details

• Training Data: Trained on ~15 trillion tokens from publicly available data, with fine-tuning on a mixture of human-generated and synthetic data.
• Training Infrastructure: Utilized custom GPU clusters. The total computation involved 39.3M GPU hours on H100-80GB hardware.

Benchmark Scores

Llama 3.1 models excel across various benchmarks. For instance:

• MMLU (5-shot): 69.4 (8B Instruct), 83.6 (70B Instruct), 87.3 (405B Instruct)
• HumanEval (pass@1): 72.6 (8B Instruct), 80.5 (70B Instruct), 89.0 (405B Instruct)
• ARC-Challenge: 83.4 (8B Instruct), 94.8 (70B Instruct), 96.9 (405B Instruct)

How to Use

Llama 3.1 is intended for commercial and research use in supported languages. It can be adapted for various NLP tasks, including text generation and assistant-like chat interfaces. The models also support the generation of synthetic data for improving other models.

Training and Evaluation Data

The pretraining data was sourced from publicly available datasets, with a cutoff in December 2023. Fine-tuning involved over 25 million synthetic examples.

Hardware and Software Requirements

The model was trained using high-performance GPUs, specifically H100-80GB units, and is compatible with the Transformers library starting from version 4.43.0.

Contact

For any questions or issues related to the model, contact shashankyadav03