How We Test Quiz Makers: Four Scenarios, Evidence and Scoring Rules

• Publish the scenario, requirements and weights before scoring.
• Use the same input and task for every included product.
• Keep dated screenshots, official-source records and exported artifacts.
• Separate documented capability from first-hand observation.
• State exclusions, failed requirements and unresolved limitations.
• Allow factual corrections without allowing verdict negotiation.
• Exclude products covered by the publication's conflict register.

The method is designed to make a conclusion reproducible. A vendor can challenge a plan limit or missing feature with evidence, but cannot buy inclusion, placement or a higher score.

Each product receives the same four inputs: a 12-page textbook chapter, a five-page workplace policy with exceptions, a 20-minute instructional-video transcript and a scanned worksheet requiring OCR. The task is to produce 15 mixed-format questions with stated difficulty and answer explanations.

Reviewers record source support, factual correctness, coverage, traceability, distractor quality, ambiguity, cognitive level, explanation quality, editing time, export options, file-handling claims and accessibility. A subject-matter reviewer checks correctness. Generation speed is recorded, but it is not treated as evidence of quality.

The build contains eight questions, four outcomes, weighted answers, one disqualifying condition, a possible tie and a default result. Synthetic response patterns test whether every result is reachable and whether ties behave consistently.

The evidence record covers one-to-one and many-to-many answer mapping, tie handling, result balance, branching, result-page personalization, lead-gate placement, mobile delivery, question-level analytics and deletion controls.

A presenter runs a 30-person quiz on mobile devices. Two participants deliberately disconnect and rejoin. The test records joining requirements, participant caps, latency, reconnect behavior, late joining, team and individual modes, pace controls, leaderboard logic, moderation, accessibility, report export and weak-connection behavior.

A participant limit is counted only when it applies to the tested plan and user-created content. Promotional templates or differently licensed modes are documented separately.

The reviewer builds a 40-item bank and a randomized 20-question assessment with a pass score, two attempts, item feedback, a certificate and an export or LMS route.

The record covers randomization, partial credit, pass and retake rules, time limits, accommodations, audit controls, certificates, item analysis, QTI or LMS interoperability, roles, retention and cost at three learner volumes. Export success is not enough: the target-system import must preserve the required question and scoring behavior.

Each comparison publishes its job, requirements, weights, test date, results and limitations. Raw evidence is retained with a versioned codebook. A correction changes the affected fact and is logged publicly. A retest receives a new date and preserves the earlier test boundary.

No single overall winner is calculated across the four scenarios. Live games, marketing outcomes and certification assessments solve different jobs, so averaging their scores would create false precision.