AI song generator benchmark: public 2026 test protocol

Reproducible three-product AI song generator benchmark with fixed prompts, 90 required outputs, 270 blind rating rows, failure logs, unedited samples, and downloadable raw data.

Status: The three-product cohort and complete evidence package are published. Collection has not started, so no score, ranking, or winner is claimed.

Evidence plan: 90 product-specific runs and 270 independent blind rating rows.

Compiled state: not_started. Terminal runs: 0/90. Completed ratings: 0/270. Published samples: 0. Scores published: no.

Declared comparison cohort

Melody Genie: Benchmark publisher and product under test. Its scores receive no special weighting.
Suno: Independent product under test. No affiliation, sponsorship, or vendor-supplied score.
Udio: Independent product under test. No affiliation, sponsorship, or vendor-supplied score.

Two separate tracks

Track A: controlled brief: Every product receives the same original brief. Presets, personas, uploaded reference audio, and artist names are disabled. This isolates instruction following and output quality.
Track B: native workflow: Each product may use its strongest first-party controls, including presets or personas. Reviewers must record every setting, click, retry, and elapsed second. This measures the real user workflow, not a raw model.

L1: Fixed-concept hook

Write a four-line hook about missing the last train and choosing to walk home because the silence feels safer. Use one repeated anchor phrase, one concrete city detail, and no words related to luxury, fire, shadows, or destiny.

Checks: Exactly four lyric lines; One repeated anchor phrase; One concrete city detail; No banned filler words.

L2: Four-bar rhyme architecture

Write four connected bars about deleting a voice note at 2:17 a.m. Use one coherent AAAA rhyme family, create an emotional turn in line three, keep each line between 8 and 13 syllables, and do not mention brands.

Checks: Exactly four bars; Coherent AAAA end-rhyme family; Line-three turn; 8-13 syllables per line.

L3: Meaning-preserving rewrite

Rewrite this original line without changing its meaning: "I kept working when nobody believed in me." Add one physical detail, keep it under 12 syllables, and avoid the words dream, grind, pain, and prove.

Checks: Meaning preserved; One physical detail; Under 12 syllables; No banned filler words.

S1: Brief-to-song adherence

Create a 90-120 second original song at 92 BPM: intimate alternative R&B, dry close vocal, restrained drums, warm electric piano, no rap verse, chorus before 45 seconds, and a quiet final chorus with one instrument removed.

Checks: Requested duration and tempo range; Chorus arrives before 45 seconds; No rap verse; Final chorus has a clearly reduced arrangement.

S2: Section edit continuity

Replace only the second chorus. Keep the melody, vocalist identity, key, tempo, mix perspective, and surrounding transitions consistent. Change the lyric so the narrator chooses to leave instead of waiting.

Checks: Only target section changes; Musical identity remains consistent; New lyric changes the decision; Transitions are not audibly broken.

Run procedure

Freeze the product list, test date, paid tier, model, language, region, and every optional setting before the first run.
Use a fresh project for each product and task. Do not carry conversation history or hidden examples between runs.
Capture the first three valid outputs. A transport or service failure may be retried, but a weak valid output may not be replaced.
Log latency, displayed credit use, estimated USD cost when available, retries, moderation blocks, truncation, and export availability.
Remove product names and identifying metadata. Randomize output order independently for every reviewer.
Use at least three human reviewers. Each scores 0-4 independently before discussion; report the median and inter-rater spread.
Publish complete original prompts, unedited outputs or audio, raw score rows, exclusions, disclosures, and the protocol version.

Evidence required before publication

One run-log row for every track, task, and attempt, including failures and retries.
One independently completed rating row per reviewer and output before product identity is revealed.
A stable evidence URL and SHA-256 hash for every unedited text or audio output.
The exact product tier, model, region, language, settings, test time, credit use, and access disclosure.
A dated limitations section and changelog whenever the protocol, product behavior, or scoring rubric changes.

Download the complete evidence kit

Compiled results JSON · Versioned protocol JSON · Operational run log CSV · Failure ledger CSV · Sample registry CSV · Blind reviewer scorecard CSV · Sealed blind assignments CSV · Evidence manifest JSON · Replication README

Interpretation limits

The protocol measures the listed tasks; it cannot prove that one product is best for every genre, language, creator, or workflow.
Human preference remains subjective even with blinding, multiple reviewers, medians, and inter-rater spread.
AI products change frequently. Results are comparable only within the same protocol version and a clearly stated test window.
Cost, latency, availability, and moderation behavior can vary by plan, region, traffic, and account history.
Melody Genie designed and publishes the protocol, so complete raw evidence and external replication matter more than the publisher name.

Suggested citation

Melody Genie Music Team (2026). AI Song Generator Benchmark: Protocol, Results and Raw Data, version MG-ASB-1.2. https://melodygenie.app/ai-song-generator-benchmark

Method references

Google Search Central: write high-quality reviews: Evidence, quantitative measurements, benefits, drawbacks, and original research.
Stanford CRFM: HELM: Transparent, standardized, and reproducible evaluation across scenarios and metrics.
ITU-R BS.1534: subjective audio assessment: Established experimental design, listener, test material, statistical analysis, and reporting principles for subjective audio evaluation.
Zheng et al.: Judging LLM-as-a-Judge: Documents position, verbosity, and self-enhancement bias in automated judging.

Melody Genie has not assigned itself a winning score. Valid results require three attempts per task, at least three blinded human reviewers, unedited evidence, and full access disclosure.