Test an assistant on questions a calculation settles.

The Zodiacs sky-fact benchmark is 300 questions about the sky: which sign a body was in, whether a planet was retrograde, and on what date a body entered a sign or the Moon reached a phase. Each asks for its answer in a fixed form, so a reply can be scored without a person reading it. Version 0 was drawn on 5 October 2026, from dates between 1900 and 2049. Its files will not change: a test holds them to the bytes they were published with, and the generator will not draw them again with another engine, so a new engine or a new rule makes a new version.

The answer key is the engine's: @zodiacs/engine 0.1.1-rc.16, its longitudes, speeds and Moon phase sampled every 10 minutes to 6 hours, depending on the body and the span searched, and narrowed to the second. Every answer in the key also agrees with NASA JPL Horizons. Beside the key are the replies of check_sky_fact, the MCP adapter's tool for these questions, which runs the compute API's sky-fact calculation, unchanged. No assistant has been scored on it yet.

One from each family, with its answer:

• In the tropical zodiac, which sign was the Sun in at 23:26 UTC on 15 February 1985? Answer with only the name of the sign. Aquarius
• In the tropical zodiac, which sign was the Moon in on 6 January 2024? If the answer depends on the time of day or the time zone, answer DEPENDS; otherwise answer with only the name of the sign. DEPENDS
• Will Mercury be retrograde on 23 April 2044? If the answer depends on the time of day or the time zone, answer DEPENDS; otherwise answer YES or NO. NO
• In the tropical zodiac, on what date in 2041 will the Sun enter Aries? Answer with only the date, as YYYY-MM-DD. 2041-03-19 or 2041-03-20
• On what date was the new moon in June 1937? Answer with only the date, as YYYY-MM-DD. 1937-06-08 or 1937-06-09

A date with no time or zone is read in every UTC offset in use today at once, from −12:00 to +14:00: from 14 hours before its midnight UTC to 36 hours after. When a body changes sign or stations within those hours, the answer turns on the time of day or the zone, and the right reply is DEPENDS. For a question whose answer is a date, every date that holds the event in one of those offsets is right: two dates for 114 of the 120 such questions, and three for the rest. Such a question names a year, or for the Moon a month, in which the body enters the sign once, moving forward, or which has one lunation of that phase, read in every offset.

A question is left out when its answer would turn on less than a margin: a body within 1′ of a sign boundary at the instant asked about, or the Moon within 30′; an event within 10 minutes of the edge of a date, or within the time the body takes to move 30″, more than the engine's largest measured error for any planet; a station within 6 hours of one. Where the answer is that a planet stays in one sign on a date, or enters a sign once in a period, a station within 30″ of that sign's boundaries also leaves the question out. So does another entry of the body into the sign within those margins outside the period, for the Sun and the Moon too, and for a lunation, another of the same phase within 10 minutes outside it. ΔT is observed to 24 September 2026, predicted to 2 October 2027 and extrapolated after that. The engine's estimate of its uncertainty reaches about 12 seconds by 2049, far inside these margins.

The scorer reads each reply strictly and, only when that fails, leniently. Strictly, the reply's last line that is not empty, without quotes, backticks or emphasis around it and without full stops or exclamation marks at its end, must be one answer in the asked-for form: a sign's name or symbol, YES, NO or DEPENDS where the question allows them, or a date written YYYY-MM-DD. Failing that, the whole reply, with any copy of the question or its instruction taken out, and with "Gemini" ignored where it names Google's assistant in the forms the scorer lists, must name exactly one such answer. A date may then also be written as 7 March 2023, as March 7, 2023 or as 2023/03/07, and a reply that gives two days, as in "18 or 19 March 2041", names two dates. YES or NO counts only as a word on its own, so "no idea", "no-one" and "no (direct) way" are not NO; "yes and no" names both, and "yes or no" and "yes/no" name neither. Letter case never matters, and a reply that neither reading parses counts as wrong, as does a question with no reply.

The strict score is the benchmark's score. The lenient one counts a reply right under either reading, so it is a rough guide to how much of a gap is the form of the replies: a reply that opens with a stray "Yes," still reads as YES, and one that names two signs, with neither alone on its last line, reads as nothing.

The scorer needs Node.js 18 or later and nothing else, and reads items.json and key.json from its own folder. The replies go in a file, one per line, as JSON:

Then:

It prints the strict and lenient scores, overall and for each family, how many questions had a reply, and how each reply was read.

• items.json: the questions, word for word, and the form each answer takes.
• key.json: the engine's answer to each, every answer accepted, and the values that decide it.
• tool-answers.json: what check_sky_fact returned for each question when the version was drawn, with its receipt.
• scorer.mjs: the scorer.

The generator draws the questions from fixed rules with a seeded random number generator, so with @zodiacs/engine 0.1.1-rc.16 it draws the same questions and key on every run. The site's tests fail if they differ from the published files, or if check_sky_fact no longer gives the published replies' answers, or the facts behind them; the receipts beside the replies may change. No licence has been chosen for these files yet; until one is, they fall under the site's all-rights-reserved notice.

check_sky_fact finds sign changes, stations and lunations with the compute API's search, which samples every 5 days, or every day for the Moon; the key samples every 10 minutes to 6 hours. On every question the tool gives the key's answer, and a test holds it to the key on 2,022 facts: for a question about a sign, each of the twelve signs; for a question about whether a planet was retrograde, that question; for a question whose answer is a date, every date the key accepts and the date either side; and, for each of the 18 entries into a sign while retrograde in the periods the ingress questions name, every date that holds it, 36 in all.

Both read the same engine, so this checks the compute API's search and its reading of a date against a dense scan of that engine on these questions. It does not prove the search misses nothing, and it says nothing about whether the engine is right.

For that, every answer in the key was checked against NASA JPL Horizons' apparent longitudes of date, sampled around each question, and all 300 agree. The evidence record gives the differences. The engine's accuracy over many more instants is measured on the conformance page.

• The tropical zodiac only, and questions in English only.
• The questions ask for an answer in a fixed form, which is not how people usually ask.
• The key is public, so a model may have seen it, and a later model may have been trained on it.
• Three DEPENDS answers turn only on offsets more than 10 hours from UTC, and the questions do not say which offsets count: sd-001 on those east of +11:20, sd-037 on UTC+14 alone, and sd-049 on UTC−12:00 alone, an offset no inhabited place keeps.
• check_sky_fact ships only in the MCP adapter's candidate 0.1.0-rc.16.2, which is not on npm.