Why subjective categories are hard to label
Machine-learning models learn to recognize the world through examples labeled by people. This works well for concepts with clear, widely accepted boundaries, such as a stop sign or a golden retriever. But in tasks such as identifying healthy food, selecting inspiring works of art or moderating content, a category’s boundaries can vary with culture, personal views and workplace rules.
People also rarely start with a complete, precise definition of a subjective concept. Usually, they begin with a general idea, and discover exceptions and nuances when they encounter specific examples. Conventional labeling processes do not account for this: they ask people to set the rules in advance, even when their own understanding of the category is still changing.
Agile Deliberation treats the creation of a classifier as an iterative dialogue. The system helps users refine their own definition before building a classifier, turning them from sources of fixed labels into collaborators. The authors have also released the project’s source code and algorithms on GitHub.
The researchers tested the system in live sessions: static benchmarks cannot assess subjective concepts or track how a person’s understanding of them changes. This format made it possible to observe how users refined their intent while working with the system. The authors found that the approach reduced fatigue and anxiety: instead of manually adjusting prompts, participants gave feedback on borderline images. As a result, the classifiers became more consistent and outperformed standard baselines.
Two stages for refining concepts
Agile Deliberation has two stages: defining a concept’s structure and refining it step by step. Together, they turn a general idea into precise instructions for a model.
Source: research.google
1. Defining the structure
At the first stage, the system helps break a broad category into subcategories and explicit exclusions. For example, “healthy food” can be divided into positive subcategories such as “salads,” “grain bowls” and “lean protein.” The user separately specifies exclusions, such as “processed foods” or “raw, uncooked ingredients.” This creates an initial structure for the model to work from.
2. Refining through examples
Next, the system moves from a preliminary plan to testing and revision. Traditional machine learning assumes that annotators know all the rules in advance. Agile Deliberation instead offers a way to refine them with the system.
The approach draws on active learning, a method that selects informative data to train a model more efficiently. But the goal here is not simply to reduce the model’s uncertainty. Vision-language models search for images that sit close to the boundaries of the user-defined categories and could reveal weaknesses in their definitions.
For example, the system might show a fruit smoothie and ask whether it counts as healthy because it contains fruit, or should be excluded as a processed product high in sugar. Such cases help people notice unstated assumptions, express their intent more precisely and update the rules. User feedback feeds into an automated prompt-optimization loop, which then changes the classification boundaries.
How the study was conducted
The study involved 18 people. Each spent an hour and a half working with the system and alternative approaches; the order of the options was randomized so participants would not gain an advantage from learning during earlier tasks.
The researchers chose two challenging subjective areas:
Agile Deliberation was compared with three approaches:
Results and participant workload
Participants using Agile Deliberation labeled data more consistently over time. The classifiers they created outperformed all standard baselines. The authors saw the clearest gains on difficult subjective images, where conventional classifiers are more likely to rely on typical AI assumptions than on the user’s precise intent.
Source: research.google
Experts also noted the system’s diagnostic value: by surfacing confusing borderline examples, it helped reveal gaps in participants’ initial definitions — situations they had not considered before encountering ambiguous images from the real world.
Manually adjusting prompts required substantial mental effort. Participants found it difficult to come up with search queries in advance that would help them find borderline cases. They also worried that fixing one prompt might make the classifier perform worse on other cases they had already tuned.
Compared with manual prompt adjustment, participants using Agile Deliberation reported a statistically significant reduction in required effort and stress, lower mental workload, and higher ratings for success and ease of use. The system handled the computation, leaving users to review selected borderline images and give feedback in plain language.
Source: research.google
Participants using Agile Deliberation also reported less frustration and mental exhaustion. Every participant preferred this approach to manual methods.
Where the approach could be useful
Agile Deliberation offers a shift from one-way learning based on human labels to people and AI refining concepts together. The authors say it could be useful for trust and safety teams building content-moderation filters, curators of digital collections and scientists who need to classify complex visual data.
The publication also discusses active-learning methods and the technical implementation of algorithms for defining and grouping concepts; details are available in the research paper and open-source code on GitHub. To try the system, the authors offer an interactive example in Colab or Jupyter focused on healthy food. It can be adapted to other areas, such as identifying misleading clickbait or selecting works of art according to personal criteria.
The work was prepared by Google Research research scientist Otilia Stretcu and research intern Leijie Wang. Other project contributors include Wei Qiao, Thomas Denby, Krishnamurthy Viswanathan, Enming Luo, Chun-Ta Lu, Tushar Dogra, Ranjay Krishna and Ariel Fuxman. The research was a collaboration between Google Research and Google Ads.
Dataist’s take: the approach matters for tasks where the rules cannot be fully defined before labeling begins. The system helps surface disagreements through concrete examples and turn them into refined criteria. However, the results cover two subject areas and 18 sessions, so the publication does not show how well the method will transfer to other tasks.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X