Tutorial · Review analysis

Review analysis API: themes and sentiment

A star rating tells you the average. An aspect taxonomy tells you which part of the product or service produced the average, and which quote proves it.

Short answer. Define an aspect taxonomy before you prompt, deduplicate the corpus, extract aspect-level sentiment with an exact evidence quote, reject quotes that are not in the review, and aggregate with sample sizes. Measure agreement with human labels before trusting a dashboard.

Start with a taxonomy

“Positive” and “negative” are not analysis. A review can be positive about the product, negative about delivery and neutral about price in the same paragraph. Aspect-based sentiment analysis splits the text into the things being evaluated and scores each one.

AspectDefinitionExample phrase
product_qualityBuild, materials, durability, defects“The hinge broke after two weeks.”
ease_of_useSetup, learning curve, interface“I was running it in ten minutes.”
deliveryShipping speed, packaging, accuracy“Arrived a day late, box was crushed.”
price_valueWorth the price, subscription value“Good, but not at this price.”
supportResponse time, resolution, courtesy“Support replied twice and never fixed it.”

Keep the taxonomy small enough to review. Ten to twenty aspects is usually more useful than a hundred tags nobody reads. Add an other bucket and review it monthly; it is the source for the next version. Version the taxonomy and store the version with every analysis row.

Ingest and deduplicate

Normalize the text: trim whitespace, normalize Unicode, remove boilerplate signatures and decode HTML entities. Keep the original for the evidence quote and audit trail. Do not translate before extraction unless the model is stronger in the target language; translation can change the sentiment of an idiomatic phrase.

Deduplicate in two passes. An exact hash catches copy-paste. A near-duplicate pass using shingles, MinHash or embedding similarity catches small edits. Cluster near-duplicates and keep one representative for counting, but keep every source row linked to the cluster. Do not delete automatically; a cluster may be a legitimate template or a coordinated campaign, and that distinction needs a human.

Store source, date, rating, locale and product variant with every row so a percentage can be read with its sample size.

Extract aspects and sentiment

Ask for one row per aspect mention, not one label per review. A mixed review should produce a positive product row and a negative delivery row. Each row needs the aspect, polarity, intensity, an exact quote and the supporting span.

import json
from openai import OpenAI

client = OpenAI(api_key="YOUR_KEY", base_url="https://api.plugsky.com/v1")
taxonomy = {"product_quality": "build, durability, defects",
            "ease_of_use": "setup, interface", "delivery": "shipping, packaging",
            "price_value": "worth the price", "support": "response and resolution",
            "other": "a real theme outside the list"}
schema = {"aspects": [{"aspect": "taxonomy key", "polarity": "positive|negative|neutral|mixed",
                       "intensity": "1-3", "quote": "exact substring"}],
          "overall": "positive|negative|neutral|mixed"}
review = "Fast delivery and the app is simple. Support never answered my email."
response = client.chat.completions.create(
    model="plugsky-lite",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": "Extract every aspect mention. Copy the quote exactly. "
         "Use only taxonomy keys. Do not infer an aspect that is not stated. Return JSON, or [] if none."},
        {"role": "user", "content": json.dumps({"schema": schema, "taxonomy": taxonomy, "review": review})},
    ],
)
result = json.loads(response.choices[0].message.content)
for item in result["aspects"]:
    assert item["aspect"] in taxonomy and item["quote"] in review
    assert item["polarity"] in {"positive", "negative", "neutral", "mixed"}

The check rejects a paraphrased quote. It does not prove the polarity. Sample the rows and read the quote in context.

Guard against hallucination

Three failures matter most: the model invents a theme, assigns sentiment to the wrong aspect, or overstates intensity. Guard each one.

  • Controlled vocabulary: validate the aspect against the taxonomy. Put new themes in other and review them later.
  • Quote grounding: require an exact substring. If it is not found, retry once with a narrower instruction, then mark the row failed.
  • Abstention: allow an empty aspect list. “Fine” may have no actionable aspect; forcing a label creates noise.
  • Language: keep the original text and quote. Translate only for display and label the translation.
  • Adversarial text: reviews can contain instructions or URLs. Treat them as data, strip markup, and never follow them.

Aggregate and export

Report counts, not just percentages. “Delivery negative: 42 of 310 delivery mentions (13.5%)” is more useful than “delivery sentiment: 13.5%.” Include the collection window, channel mix, taxonomy version and model version. For a trend, compare equal windows and show the sample size; a jump from two mentions to four is not a trend.

Export fieldTypeUse
review_idstringJoin back to the source
aspectenumGroup and filter
polarityenumSentiment split
quotestringEvidence panel
taxonomy_versionstringReproducibility

Export JSON Lines for the warehouse and flat CSV for the dashboard. Keep one row per aspect mention and a separate review-level table for rating, date and source, so a dashboard can count mentions without double-counting reviews.

Cost at 10,000 reviews

Cost is driven by input tokens, output tokens, retries and the share of reviews needing a second pass. Measure your own distribution rather than assuming an average.

reviews = 10_000
input_tokens = reviews * measured_input_per_review * (1 + retry_rate)
output_tokens = reviews * measured_output_per_review * (1 + retry_rate)
token_cost = input_tokens * input_rate + output_tokens * output_rate
review_cost = reviews * human_sample_rate * minutes_per_sample / 60 * hourly_rate
total_cost = token_cost + review_cost + storage + engineering

At 150 input and 50 output tokens per review, 10,000 reviews is 1.5 million input and 500,000 output tokens, about 2 million before retries. That is the Hobby daily allowance, with no headroom for retries or a second pass. Split the job across days or use a larger plan. See pricing; the arithmetic is illustrative, not a benchmark.

Evaluate

Sample 200 to 500 reviews, stratified by rating, channel and language, and have a human label aspects and polarity. Measure aspect recall, aspect precision, sentiment agreement, quote validity and abstention rate. Report Cohen's kappa alongside raw agreement when the label distribution is skewed; 90% agreement sounds strong until 90% of the sample is positive.

Re-run the evaluation when the taxonomy, model or prompt changes. Accuracy last quarter does not guarantee accuracy now.

FAQ

What is aspect-based sentiment analysis?

It extracts the specific thing being evaluated, such as delivery or support, and assigns sentiment to that aspect rather than to the review as a whole. A mixed review can produce several aspect rows with different polarities.

How do I stop the model inventing review quotes?

Require the quote to be an exact substring of the review, validate it in code, and reject the row if it is not found. Allow an empty aspect list, and sample the accepted rows to check that the quote supports the label.

How many reviews can I analyze per day?

It depends on review length and your plan's daily token allowance. At 150 input and 50 output tokens per review, 10,000 reviews is about 2 million tokens before retries. Measure your own token distribution and leave headroom for retries and a second pass.

How do I measure sentiment accuracy?

Label a stratified sample of 200 to 500 reviews by hand, then measure aspect recall, precision, sentiment agreement and quote validity. Report Cohen's kappa when the label distribution is skewed, and re-run the evaluation after any prompt or model change.

Should I delete duplicate reviews?

Keep the source rows and cluster duplicates rather than deleting them. Exact hashes catch copy-paste; near-duplicate clustering catches small edits. A cluster may be legitimate or coordinated, so route suspicious clusters to a human.

Sources: SemEval-2014 Task 4 on aspect-based sentiment analysis and the scikit-learn documentation for Cohen's kappa. Both checked 20 November 2026. Reviewed 20 November 2026. Cost figures are illustrative arithmetic from stated token assumptions, not a benchmark.