[davd-gzl]

Before: the monthly Wikidata fetch on 1 November ran for 351 of its 355 minutes, then stopped without publishing anything. The queries for the towns of Italy, Japan and the United States, and for Japan's stations, kept hitting the query service's 60-second limit, and three countries with no towns is more than the fetch allows.

Splitting a country into five parts by the last digit of each item's id does not save the towns. The digit filter has to read the id of every item with a population worldwide, 147,222 of them, so it costs 27 seconds before the details are read, and a fifth of the United States still timed out.

After: when a country's whole town query fails, the fetch reads its town ids alone, then their details 500 towns at a time. Stations come from one query for the whole world instead of one per country, and a failure there stops the run before the sights. A country whose towns still can't be read keeps what the last committed fetch had for it, so a bad day at the query service leaves last month's data in place instead of failing the run. More than 10 countries kept fails the run, since a broken query would otherwise publish last month's data for every country under today's date. The town query checks whether a class is a kind of town once per class instead of once per item.

How:

• townIdsQuery returns a country's town ids and citiesByIdQuery the details of a list of ids. citiesQuery builds the same text as before from the same two pieces.
• STATIONS_QUERY reads every open railway station on Wikidata. COUNTRY_CODES_QUERY and labelsQuery add their countries and their names, 5,000 stations at a time, since reading either in the same query went past the one-minute limit.
• fetch.ts tries each whole town query twice, it used to try three times, then falls back to the batches, then to the previous data/raw/wikidata.json through keepPrevious.
• FETCH_IN_PARTS=1 skips the whole queries, and fewer than 5,000 United States towns read that way now fail the live Wikidata check.

Live check on 2 and 3 November, with no previous file:

• United States: 7,811 towns from 7,958 ids in 16 batches, read in 6 minutes with no batch failing.
• Italy: 2,760 towns. Japan: 1,314 towns.

The stations took 151 of the 351 minutes on 1 November, in 255 queries, one per country. The sights of the towns that now come back add about 30 minutes.