Few-Shot Example Selection

Category: Foundational patterns

Which examples you include in a few-shot prompt — and their order — strongly affects results. Choosing relevant, representative, well-balanced examples (often ones similar to the input) outperforms picking them at random.

What it is

Few-shot example selection is the practice of deliberately choosing the demonstrations placed in a prompt rather than grabbing arbitrary ones. Research shows in-context learning is sensitive to which examples are used, how they are ordered, and even their label distribution — poor choices can hurt performance while good ones lift it substantially.

A widely used strategy is to retrieve examples semantically similar to the current input (dynamic, input-specific selection), and to balance label coverage and ordering so the model is not biased toward a majority class or a position effect. It turns few-shot prompting from a static block into a tuned, sometimes retrieval-driven, component.

When to use

Few-shot results are unstable or sensitive to which examples you happen to pick.
You have a pool of examples and can select the most relevant per input.
Label balance or ordering effects are skewing predictions.
You can afford a retrieval step to pick input-similar demonstrations dynamically.

How it works

Curate a diverse, correct pool of candidate examples.
For each input, select the most relevant examples (e.g. by semantic similarity).
Balance label coverage and vary or fix ordering deliberately to limit bias.
Evaluate selection choices empirically rather than assuming they help.

Illustrative structure

The structure adds a selection step before assembling the few-shot prompt: for a new input, retrieve the K most similar solved examples from a pool, order them consistently, then build the standard few-shot block from those K rather than from fixed or random examples.

Pitfalls

Retrieved examples too similar to each other reduce useful coverage.
Ordering and majority-label bias can dominate; test different orders.
Dynamic selection adds a retrieval dependency and latency.
Leaking near-duplicates of the test input as examples inflates apparent quality.

Frequently asked questions

What is Few-Shot Example Selection?

When should you use Few-Shot Example Selection?

Few-shot results are unstable or sensitive to which examples you happen to pick. You have a pool of examples and can select the most relevant per input. Label balance or ordering effects are skewing predictions. You can afford a retrieval step to pick input-similar demonstrations dynamically.

What are common pitfalls of Few-Shot Example Selection?

Retrieved examples too similar to each other reduce useful coverage. Ordering and majority-label bias can dominate; test different orders. Dynamic selection adds a retrieval dependency and latency. Leaking near-duplicates of the test input as examples inflates apparent quality.

Sources

Related techniques

← All prompt engineering techniques

Apply Few-Shot Example Selection with Prompeteer, the Agentic Contextual AI Platform →