#55 of 100 · AI
LLM tests on real work
Tests of models on real work, tool-use reliability, destructive actions, censorship and human-sounding writing, plus side-by-side runs of their own prompts.
Asks per year
What the askers want
- Would pay0%
- Want it local or private12%
- Object to subscriptions0%
- Want it open source or self-hosted12%
- Want AI to do it62%
Share of the 8 asks that say so, as judged by Jev.
What exists on Show HN
Built, still asked for7 Show HN launches do this job, and 26 more do part of it. People still ask, so the launches are missed, unknown, or not what they want. Ask what they’re missing before you build.
- I benchmarked LLM agents on fixing real-world security vulnerabilities2026 · 4 pts
- Benchmarking LLM Agents on Consequential Real World Tasks2025 · 3 pts
- Relia – Build your own LLM benchmark2024 · 3 pts
- CATArena – Evaluating LLM agents via dynamic enviroment interactions2025 · 3 pts
- Mandoline – Custom LLM Evaluations for Real-World Use Cases2024 · 2 pts
- Lone Arena – Self-hosted LLM human evaluation, you be the judge2024 · 1 pts
- LLM Litmus Test – compare multiple LLMs for coding tasks with context2024 · 1 pts
Instead of: Claude, Gemini, ChatGPT
The idea, as matched
A benchmark or comparison service that tests LLMs on real-world work rather than toy puzzles: tool-use reliability, long agentic tasks, safety under pressure, censorship, human-sounding writing, with side-by-side outputs for your own prompts.
Jev first matched 12 requests to this idea; a second reader kept 8 of 12.
Every ask, newest first
Each line is an AI paraphrase of what the writer wanted. The link goes to their own words on Hacker News.
Reader’s note. 48545466 duplicates 48545459, excluded.
- Jun 2026Wants a real-use evaluation where developers build with an LLM and share their experience, not one-shot tests. Read it: a comment in “GLM 5.2 vs. Opus”
- Jun 2026Wants a competition benchmarking local models on a single consumer GPU, scored on accuracy and time. Read it: a comment in “Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?”
- Feb 2026Wants a test of how reliably models obey tool-calling instructions, to judge production fit. Read it: a comment in “Gemini 3.1 Pro”
- Dec 2025Wants a benchmark checking whether coding assistants avoid destructive actions like dropping databases or deleting files. Read it: a comment in “AI agents break rules under everyday pressure”
- Aug 2025Wants a ranking of how human-sounding different AI models' writing is. Read it: a post: “Ask HN: Why isn't there a ranking for how human-like AI writing sounds?”
- Mar 2024Wants an off-the-shelf tool to run prompt banks, with right-answer and graded items, across multiple LLMs. Read it: a post: “Ask HN: Is there a tool for testing the performance of multiple LLMs / LLM APIs?”
- Feb 2024Wants a benchmark measuring censorship and alignment across LLMs. Read it: a comment in “Bard's latest updates: Access Gemini Pro globally and generate images”
- Jul 2023Wants a site comparing outputs for fixed inputs across model versions, like a GPU benchmark site. Read it: a comment in “Experiencing decreased performance with ChatGPT-4”
Building this?
Read ten of the asks above first, because they say what the existing launches miss. If you ship it, post it as a Show HN so the next run of this list finds it.
Ship it like Vercel, on a server you own
This site runs on ShipTiffin. A push to GitHub deploys it, the way it would on Vercel, but the server is a Hetzner box in your own account. Postgres, sign-in, email, jobs and analytics are already on it, every project gets a hard limit, and nothing is billed by usage.
$19 a month per box, plus about $10 for the server at Hetzner. Open source, so you can also run it yourself for free.