Interview prep · AI companies
AI system design interview: the prompts AI companies actually ask
An AI system design interview is a design round where the system has a large model in it. AI labs and AI product companies use it to see whether you can build around a model: serve it within a latency and cost budget, feed it the right documents, check what it says, and let it act through tools safely. This page collects the prompts reported at ten of them, grouped by kind, with a link to each company's full guide.
Updated 12 October 2026 · every prompt is in a company guide, with its sources
The short version
- The prompts fall into four kinds: serving a model, retrieval and RAG, evaluation, and agents with tools.
- Retrieval is the most reported kind, from Perplexity's web-scale index to Harvey's cited answers over a law firm's files.
- The model-specific parts are graded in detail: batching, a latency budget per stage, token cost, grounding.
- Not every AI company asks it. OpenAI's reported design rounds are general system design.
- Prepare one design of each kind out loud, and general system design as well.
The prompts, by kind
Each prompt below was reported by candidates or interview-prep sites for that company; the company guide gives the source, the date and how to take it. Prompts a company never confirmed are marked.
Serving a model
What gets probed: Batching requests of different lengths, GPU memory and utilisation, streaming tokens, multi-tenant capacity, and what you autoscale on.
Serve a large language model efficiently through an API; design a Claude chat service; let a model handle several questions in one thread
A multi-tenant, low-latency inference service on shared GPUs
Serve 100+ open-source models on shared GPUs; multi-tenant fine-tuning with LoRA adapters; a KV cache for many chats (prep-site prompts, never confirmed by the company)
Retrieval, search and RAG
What gets probed: Indexing at scale, hybrid keyword and vector search, reranking, a latency budget per stage, freshness, and answers grounded in sources with citations that are checked.
Retrieval over 100 billion pages with sub-second latency; a ranking stack combining BM25, dense retrieval and model reranking; citation verification
Distributed search over a billion documents at a million queries a second, with LLM inference; hybrid search; an indexing pipeline with concurrent crawling
Retrieval-augmented answers with citations over a document vault; a memo Q&A agent for a large law firm
The backend for an insurance claims agent, with retrieval over emails and PDFs and token cost kept down; an LLM API pipeline
Search across a company's apps that stays fresh and returns only what the person asking may see
In its six-hour build: retrieval over legal documents with exact citations, and keeping the model from inventing a case
Evaluation and quality
What gets probed: A labelled set from real queries, the right metrics, a baseline, regression checks when a prompt or model changes, and catching hallucinations.
Agents and tools
What gets probed: The reasoning loop, tool calls with scoped permissions, state across turns, stop conditions, and the path to a human when a step returns nonsense.
A system design screen in place of the coding screen, on agent-shaped systems: a reasoning loop, tool calls and state across turns
The insurance claims agent above, with its retrieval and storage
A clinical tool execution service, with part of the code handed over to complete and defend
Two more rounds sit next to design at AI product companies. ElevenLabs runs a Product Decomposition round where you design a tool, such as one for reviewing and correcting AI-dubbed video, from the screens down to the data model; see the ElevenLabs guide. Cognition asks engineers to build their own coding agent in about eight hours, the same ground covered by building it; see the Cognition guide.
Not every AI company asks AI system design
Candidates report OpenAI's two design rounds as general system design: Yelp, Twitter, a notifications system, a job scheduler, in Excalidraw, with every tool you name open to challenge. Anthropic has been reported asking for a banking app, where consistency is the point. Mistral lists a 45-minute system design round in every engineering ad and says nothing about its content. So prepare general system design too, and ask the recruiter what the round covers.
What interviewers listen for
- A latency budget per stage. When retrieval, reranking and the model share one request, say how much time each gets, so the model cannot eat the whole budget. It matters most in prompts like Anthropic's reported search over a billion documents with LLM inference.
- Batching and GPU memory. Continuous against fixed batching, keeping short requests fast when long ones share the batch, and what the KV cache costs per request.
- Token cost as a constraint. Scale AI's claims agent prompt asks for it outright. Say what you send to the model and why, and what you cache.
- Grounding you can check. Answer from retrieved sources, cite them, and verify each citation against the index after generation. At Harvey and Legora, a fabricated citation is the failure the reported questions keep coming back to.
- Tenant isolation. One customer's documents never appear in another's results, enforced on every query path, including the cache and the embeddings.
- Measure before you change. Build a labelled set from real queries and pick the metric before tuning prompts, models or chunking.
How to prepare
- Read the guide for the company you are interviewing with, and note which kind of prompt it reports.
- Practise one design of each kind out loud: a model-serving API, a RAG system with citations, an evaluation pipeline, and an agent with one dangerous tool.
- For every design, put rough numbers on it: requests per second, tokens per request, GPU memory, the latency budget.
- Say where each design fails: a slow model, a stale index, a wrong citation, a tool call with bad arguments.
- Prepare general system design as well, since several AI companies ask it.
- Ask the recruiter what the design round covers and which tool you will draw in.
The rounds around design are in the AI engineer, ML engineer and research engineer guides, and the questions for AI engineering roles are in the AI engineer question bank.
Questions
What is an AI system design interview?
A system design round where the system has a model in it: serving a large language model, retrieval-augmented answers over documents, an evaluation pipeline, or an agent that calls tools. AI labs and AI product companies such as Anthropic, Cohere, Perplexity, Harvey and Scale AI run it, usually for about an hour, and grade the model-specific parts as hard as the distributed-systems parts.
What questions are asked in AI system design interviews?
Reported prompts include serving a large language model efficiently through an API (Anthropic), multi-tenant low-latency inference on shared GPUs (Cohere), retrieval over 100 billion pages with sub-second latency (Perplexity), answers with citations over a law firm's document vault (Harvey) and a retrieval pipeline over emails and PDFs with token cost kept down (Scale AI).
How is AI system design different from ML system design?
ML system design usually means a recommender, ranking or fraud model and the data and training pipeline around it. AI system design, as AI companies ask it in 2026, is about building a product around a large model: serving it, feeding it the right context, checking its answers and keeping cost and latency in budget. Some loops, such as Scale AI's machine learning track, ask both.
Do all AI companies ask AI-specific system design?
No. OpenAI's reported design rounds are general system design, such as a notifications system or a job scheduler, and Anthropic has also been reported asking for a banking app. Mistral lists a 45-minute design round in every engineering ad without describing it. Prepare general system design as well.
What do interviewers score in an AI system design interview?
The same fundamentals as any design round, plus the parts specific to models: where the latency goes and a budget per stage, GPU batching and memory, token cost, grounding answers in retrieved sources and verifying citations, keeping one customer's data out of another's results, and how you would measure quality before changing anything.
Your AI role
See the rounds and the design question for your posting
Paste the posting you are applying to. Calibrd predicts the rounds and the questions for that role at that company, and you can practise the design round out loud in a mock interview before the real one.
Free to start · No card · Encrypted at rest, never used to train AI, remove anytime
Sources
Every prompt on this page comes from a Calibrd company guide, which cites the candidate accounts and prep sites behind it with dates: the Anthropic, Cohere, Perplexity, Harvey, Scale AI, Glean, Legora, Sierra, Abridge, Together AI and OpenAI guides.