Adding crawl URLs to sub-yaps

November 4, 2026 · Andrew Arrow

I wanted a sub-yap (a topic board on Yapern) for finding new short fiction without having to copy links from literary magazines by hand. The magazines already have editors choosing what to publish. Yapern can collect references to that work and give people somewhere to discuss it, with a link back to the original. The reading list is at y/fiction.

So I added a Crawler section beside Basics in a sub-yap’s settings. Paste one URL, add it to the list, and press Crawl beside the site. Each new article becomes an ordinary post with a title, source URL, and short description. The first version is deliberately small: it looks at a recent page of results and leaves the archive alone.

The screenshot is a local preview using live publisher requests. CRAFT and X-R-A-Y each produced ten posts. Getting those rows to say “Added 10 articles” took more than downloading the two homepages.

Starting with nine magazines

The reading list started with Electric Literature, The Offing, and SmokeLong, then expanded to smaller magazines: CRAFT, X-R-A-Y, Fractured Lit, Pithead Chapel, Fictive Dream, and The Citron Review. I’d asked for rough audience tiers, but I don’t have comparable traffic measurements for these sites. I’m treating the nine as a list of places to find fiction, without pretending the tiers establish a ranking.

The earlier research couldn’t reliably retrieve their robots files. Direct HTTPS requests worked when I checked on November 4, 2026, using yapern.com crawler v1.0.0. This is what those files said about the root path for this crawler. Other paths can have different rules.

These are crawl directives, not a license to republish stories. CRAFT and Fractured Lit also have terms restricting competitive uses of their sites. A successful robots check doesn’t settle those separate terms. The feature stores references and brief descriptions; it doesn’t create a full-text fiction archive.

The table is a direct read of robots files, separate from the application’s network checks. Our shared DNS guard currently rejects WordPress.com’s 192.0.78.* hosting addresses before fetching robots.txt. That needs a narrower address check; these robots results don’t establish that all eight sites can complete a crawl in the current app.

The Worker upgrades source URLs to HTTPS and checks /robots.txt at the origin, even if the saved URL points to a feed or a category. It applies the matching agent rules before fetching pages and checks redirects too. A 404 or 410 robots response permits the attempt; a robots access error stops it. That’s a conservative policy within the Robots Exclusion Protocol’s handling of access results. We also reject private network destinations and skip article pages marked noindex.

RSS is still doing useful work

I did wonder whether it was worth supporting RSS. It’s XML, it’s old, and every magazine already has a website. But a homepage makes me infer which link is a story, which date belongs to it, and whether a block of text is a description or a subscription pitch. A feed often hands me those fields together.

That saves work even when the story still needs an HTML request. The feed finds candidates; the article page supplies its title, description, publication date when available, and canonical URL. If the publisher redesigns its homepage, the feed can keep delivering the same structure. WordPress includes RSS and Atom feeds, which makes them worth looking for on this particular list of sites.

The discovery code tries a supplied feed directly, then feed links advertised in the page’s HTML and a few conventional locations. If that produces no candidates, it tries robots-declared sitemaps, a root sitemap, and a bounded number of child sitemaps. Article links on the source’s first page are the final fallback.

I’m keeping all three. Replacing a usable feed with layout-dependent selectors would give me more code to maintain for the same links. When a magazine has a fiction-specific feed, that’s a better starting URL than its homepage. The current extractor doesn’t retain author or genre fields, and it doesn’t classify a piece as fiction just because a literary magazine published it.

The button starts a job

A crawl can outlast the page that started it. The API endpoint checks that you own the sub-yap, saves a job in D1, and sends its ID to the existing Cloudflare queue. It returns HTTP 202. The browser polls for status, and reopening Crawler picks up the latest job for each site.

POST /boards/:boardId/crawler-urls/:urlId/crawl
GET  /boards/:boardId/crawler-urls/:urlId/crawl

The first version considers at most 20 recent candidates. It prefers available dates, follows no pagination, and bounds discovery to 45 HTTP requests and 90 seconds. Those bounds keep a click from turning into an archive crawl. They also expose a limit: SmokeLong’s 20-second delay won’t let us fetch twenty article pages in one run. Slow sites need work split across smaller, resumable jobs. CRAFT’s ungrouped delay is another rough edge: the current parser doesn’t attach that line to a user-agent group.

Starting an already active source returns its existing job. The queue worker claims it with a lease, and a maintenance task can recover expired claims. That maintenance task repairs interrupted jobs; it doesn’t periodically crawl every saved site. For now, you press Crawl.

The next crawl must remember the last one

The same story can turn up in a homepage feed, a fiction feed, and a sitemap. It can also arrive again because the queue delivered the job twice. Checking only the list of sources would do nothing to stop duplicate posts.

We keep a record of imported URLs for each sub-yap. Tracking parameters and fragments are removed, and the extractor uses an acceptable same-site canonical URL when the page supplies one. A D1 batch saves the message, its import record, and the job count together. Another source in the same sub-yap then finds that URL already recorded.

The import record survives deletion of the message. Otherwise deleting a post would only hide it until the next crawl recreated it. In this sequence, a repeated discovery returns to the saved URL record, even after the post is gone.

Then every crawl failed

The first production attempts against CRAFT and X-R-A-Y all ended with crawl_failed. The queue logs said the deliveries were OK. The job had caught an exception, stored a failure state, and acknowledged delivery, so “OK” described the queue handler finishing. It said nothing about whether we’d fetched an article.

That exception wasn’t even a publisher blocking us. Running the crawler inside Cloudflare’s local runtime exposed TypeError: Illegal invocation before the first robots request. I’d kept a reference to the global fetch function on the crawler instance:

// The native function becomes a method of this instance.
this.fetcher = options.fetcher ?? fetch;
await this.fetcher(url, options);

// Keep the global receiver when using native fetch.
this.fetcher = options.fetcher ?? globalThis.fetch.bind(globalThis);

Calling it as this.fetcher(...) supplied the crawler as its receiver. Cloudflare requires the global receiver for that native function; its illegal-invocation documentation describes this class of error. The Node tests and injected fetch functions hadn’t reproduced it. There’s now a regression test that runs discovery inside the Worker runtime.

After that fix, local Worker runs against the live CRAFT and X-R-A-Y sites each found ten recent articles. Failure logs now include the source URL, request stage, HTTP status when one exists, and the exception with its stack and cause. “Crawl failed” is a useful status for a reader; it was a lousy debugging log.

A reading list still needs editorial choices

The imported posts use the existing message renderer, votes, replies, and search. They’re posted under the sub-yap owner’s account, with the publisher’s URL in the body. This local preview shows the result of the live crawl, including an interview. That’s evidence of the remaining fiction-filtering problem, right in the reading list.

I also haven’t added conditional feed requests with ETags or Last-Modified dates, a fiction classifier, or a scheduled subscription for each source. Those are separate pieces of work. A feed describes what was recently published, and an HTML extractor can recover a title, but neither gives the crawler an editor’s judgment.

The useful next step is to choose fiction-specific sources and see whether the resulting list is worth reading. I can already collect references without copying whole stories or reposting the same URL on each run. Now I need to make the selection good enough that opening the sub-yap helps someone find a story they want to read at its original magazine.