Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts

Summary

Large language models frequently exhibit human-like behaviors such as expressing emotions, building relationships, and setting boundaries, yet empirical guidance on appropriate usage remains limited. To address this gap, researchers analyzed 21,000 multi-turn conversations across four models: gpt-4o, gpt-4.1-mini, claude-sonnet-4.6, and gemini-2.5-flash. The study combined LLM-as-a-judge techniques with human evaluation to measure behavior prevalence, perceived appropriateness, and prompt controllability across diverse user goals and profiles. Human evaluators judged LLM self-referential and relationship-building behaviors as less appropriate than identical human actions, whereas boundary-maintaining behaviors were rated as more appropriate for models. System prompting effectively steered these behaviors, though the authors emphasize that rigorous evaluation remains necessary to prevent unintended side effects.

Context

Researchers and practitioners lack empirical methods and insights to make informed decisions about when and what types of human-like behaviors large language models should exhibit.

Approach / What changed

Researchers conducted a multi-dimensional analysis of behavior prevalence, effects, and controllability using LLM-as-a-judge and human evaluations across 21,000 multi-turn conversations from gpt-4o, gpt-4.1-mini, claude-sonnet-4.6, and gemini-2.5-flash, evaluating system prompt interventions across diverse user factors.

Takeaways

• Human-like behaviors are pervasive across multi-turn LLM conversations but vary depending on the specific model, conversation goals, and user profiles.
• Human evaluators rate self-referential and relationship-building behaviors as less appropriate from LLMs than from humans, but judge boundary-maintaining behaviors as more appropriate from LLMs.
• System prompts can control human-like behaviors in LLMs, though careful evaluation is necessary to avoid unintended effects.

Related reading

• Grab · Jul 15, 2024

  LLM-powered data classification for data entities at scale

  Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.

  Hualin Liu

• Two Sigma · Mar 21, 2024

  NeurIPS 2023: Our Favorite Papers on LLMs, Statistical Learning, and More