---
title: "Scientific Discovery"
description: "1 posts about Scientific Discovery, summarised, each linking to the original."
---

# Scientific Discovery
> 1 posts about Scientific Discovery, summarised, each linking to the original.

## Articles

### [Can we trust LLMs to judge whether a research idea is novel?](https://yomu.fyi/post/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Kyle Wiggers
- Published: Oct 27, 2026
- Type: Problem & solution

The post presents a study by Ai2 on the reliability of LLMs as novelty judges for scientific research ideas, using six models including GPT-5.4 and Claude Opus 4.6. The researchers mined ICLR 2026 peer reviews to create a validated pool of 154 high-novelty and 145 lower-novelty human ideas, then paired them with 154 simple AI-generated ideas. Testing revealed that small prompt changes could shift accuracy by over 50 percentage points on AI-generated ideas, such as GPT-5.4 dropping from 93.5% to 40.9% with a simpler prompt. Additional search and reasoning did not consistently improve judgments; a simple Claude Opus 4.6 baseline achieved 0.854 macro-F1 at $2.59, while AI Scientist scored 0.772 at $82.05. The findings indicate that LLM-based novelty evaluation remains unreliable, particularly for AI-generated ideas, and the authors release benchmark data and an automated pipeline for future research.
