HenryShan/Llama-3.2-3B-GSM8K-CoT

Llama 3.2 3B GSM8K - Mathematical Reasoning Model

A fine-tuned Llama 3.2 3B parameter language model specialized for mathematical reasoning tasks, trained on the GSM8K (Grade School Math 8K) dataset. This model excels at solving arithmetic, algebra, and word problems typically found in grade school mathematics.

Model Description

This model is based on Meta's Llama 3.2 3B Instruct model and has been fine-tuned specifically for mathematical problem-solving. The GSM8K dataset consists of 8,500 high-quality grade school math problems created by human problem writers, making it an excellent benchmark for evaluating mathematical reasoning capabilities in language models.

Key Features

• Mathematical Reasoning: Specialized for solving arithmetic and algebraic problems
• Step-by-Step Solutions: Generates detailed explanations for mathematical problems
• Chain-of-Thought: Capable of breaking down complex problems into manageable steps
• Grade School Focus: Optimized for K-8 level mathematics problems

Model Architecture

Parameter	Value
Base Model	Meta Llama 3.2 3B Instruct
Parameters	3,212,749,824
Layers	28
Attention Heads	24
Key-Value Heads	8
Hidden Size	3,072
Context Length	131,072 tokens
Vocabulary Size	128,256
Total Model Size	~6.4 GB

Training Details

Dataset

• Training Data: GSM8K (Grade School Math 8K) dataset
• Dataset Size: 8,500 carefully curated math problems
• Problem Types: Arithmetic, algebra, geometry, word problems, and logic puzzles

Fine-tuning Configuration

The model was fine-tuned using parameter-efficient techniques to adapt the base Llama 3.2 model for mathematical reasoning while preserving its general language capabilities.

Performance

While specific benchmarks may vary, this model demonstrates strong performance on mathematical reasoning tasks:

• GSM8K Accuracy: Competitive performance on the GSM8K test set
• Problem-Solving: Excels at multi-step mathematical reasoning
• Explanation Quality: Generates clear, step-by-step solutions

Installation

Requirements

pip install torch transformers accelerate

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = "path/to/your/model"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto"
)

Usage

Basic Mathematical Problem Solving

def solve_math_problem(problem):
    messages = [
        {"role": "user", "content": f"Solve this math problem step by step: {problem}"}
    ]

    inputs = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        return_tensors="pt"
    ).to(model.device)

    outputs = model.generate(
        inputs,
        max_new_tokens=512,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

    response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
    return response

# Example usage
problem = "A bakery sells 3 croissants for $5. How much would it cost to buy 7 croissants?"
solution = solve_math_problem(problem)
print(solution)

Interactive Chat

def chat_with_model():
    print("Math Tutor: Hello! I can help you solve math problems. What would you like to work on?")

    while True:
        user_input = input("You: ")

        if user_input.lower() in ['quit', 'exit', 'bye']:
            print("Math Tutor: Goodbye!")
            break

        messages = [{"role": "user", "content": user_input}]

        inputs = tokenizer.apply_chat_template(
            messages,
            add_generation_prompt=True,
            return_tensors="pt"
        ).to(model.device)

        outputs = model.generate(
            inputs,
            max_new_tokens=256,
            temperature=0.7,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )

        response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
        print(f"Math Tutor: {response}")

# Start chatting
chat_with_model()

Example Interactions

Example 1: Basic Arithmetic

User: What is 23 × 47?

Assistant: Let me calculate that step by step:

23 × 47

First, 20 × 47 = 940
3 × 47 = 141
940 + 141 = 1,081

So, 23 × 47 = 1,081

Example 2: Word Problem

User: Sarah has 3 apples. She buys 2 more apples at the store and then gives 1 apple to her friend. How many apples does Sarah have now?

Assistant: Let's solve this step by step:

1. Sarah starts with 3 apples
2. She buys 2 more apples: 3 + 2 = 5 apples
3. She gives 1 apple to her friend: 5 - 1 = 4 apples

Sarah has 4 apples now.

Example 3: Multi-Step Problem

User: A rectangle has a length of 12 cm and a width of 8 cm. What is the perimeter of the rectangle?

Assistant: To find the perimeter of a rectangle, I use the formula:

Perimeter = 2 × (length + width)

Length = 12 cm
Width = 8 cm

Length + width = 12 + 8 = 20 cm
Perimeter = 2 × 20 = 40 cm

The perimeter is 40 cm.

Model Capabilities

Strengths

• ✅ Arithmetic operations (addition, subtraction, multiplication, division)
• ✅ Basic algebra and equation solving
• ✅ Word problem interpretation
• ✅ Step-by-step explanations
• ✅ Multi-step reasoning
• ✅ Mathematical formatting

Limitations

• ⚠️ May struggle with very complex mathematical concepts beyond grade school level
• ⚠️ Performance may vary on non-standard problem formats
• ⚠️ Like all LLMs, it may occasionally make calculation errors

Ethical Considerations

This model is designed for educational purposes and mathematical problem-solving. Users should:

• Verify all mathematical solutions independently
• Use it as a learning tool rather than a definitive source of truth
• Be aware that it may occasionally produce incorrect answers

License

This model is released under the Llama 3.2 Community License. By using this model, you agree to comply with the license terms and Meta's Acceptable Use Policy.

Key License Requirements:

• Attribution: Any redistributed versions must include "Built with Llama"
• Compliance: Must adhere to applicable laws and regulations
• Restrictions: Cannot be used for prohibited activities as defined in the Acceptable Use Policy

Citation

If you use this model in your research or applications, please cite it as:

@misc{llama3.2-gsm8k,
  title={Llama 3.2 3B GSM8K - Mathematical Reasoning Model},
  author={Model Creator},
  year={2024},
  note={Fine-tuned Llama 3.2 model on GSM8K dataset for mathematical reasoning}
}

Acknowledgments

• Base Model: Meta Llama 3.2 3B Instruct
• Training Dataset: GSM8K (Grade School Math 8K)
• Framework: Built using the transformers library and PyTorch

Contact

For questions, issues, or contributions related to this model, please refer to the appropriate channels based on your needs.

---