I always assumed higher temperature equals more hallucination in LLMs. Seems obvious. But I’ve learned that running on assumptions in data science is a fast way to get embarrassed in production.
So I tested it.
The setup
Task: Generate Etsy tags given a niche market and product specs — a structured, precision task with clear right/wrong answers.
Models: GPT-4.1-mini vs an open-source model via OpenRouter
Evaluation: LLM-as-Judge scoring hallucination (0 to 5) across a temperature sweep from 0 to 2.0, logged in MLflow.
What I expected : A clean, monotonically increasing curve. Increase in temperature up will increases hallucination.
What I actually saw : The open-source model was chaotic, spiking to 4/5 at temp 0.5, crashing to 0 at temp 1.0, peaking at 5 again by 1.5. No stable pattern. The proprietary model (GPT-4.1-mini) was more contained hovering between 1–3 for most of the range, only drifting higher after 1.5.
Same task. Very different stability profiles.
Key learnings
- My assumption was directionally right but the signal is noisy especially with open-source models.
- Proprietary models show more contained chaos, but “contained” doesn’t mean safe at high temperatures.
- This is a small experiment. Next step: scale across more models, vary top-p and other sampling settings, and find the optimal configuration per task type, not just temperature.
- The bigger lesson: don’t deploy at a “feels right” temperature. Measure it.
- The hardest truth is that establishing the right criteria is challenging, yet it is often the most valuable work in production.
This was one experiment on one task. I’m not claiming a universal truth rather I’m sharing the habit of testing assumptions before shipping.
What temperature do you default to for structured generation tasks? Curious what others are seeing.
#LLM #PromptEngineering #GenerativeAI #AIEvaluation #Hallucination
