fedi-map

A map of the fediverse you can sweep through time.

Instances are nodes; edge weight is the share of an instance's outward attention that goes to one peer — mentions and boosts together, over everything it directed outward, rather than a raw count. So a 200-person server that talks constantly to its neighbours registers as strongly as a 200,000-person one, and an instance whose culture is mostly local conversation is not penalised for it. A time slider then sweeps across history: instances appear and disappear, communities form, merge and split.

This is a rewrite of the late fediverse.space, whose good idea was buried under four runtimes (Elixir + Postgres + Elasticsearch + a Java/Gephi layout job + a React/Redux frontend) and which architecturally could not do the time sweep: it pruned its crawl history after a month and rebuilt its edge table by truncation every night.

fedi-map is one Go binary, one SQLite file, and a vanilla-JS frontend with no build step. History is first-class from the first commit.

Status

Early. Phases 0 and 1 are done and verified against live instances:

• Phase 0 — schema, seed loader
• Phase 1 — HTTP client, software adapters, probe
• Phase 1.5 — HTTP cache, per-host gate, request audit log
• Phase 1.6 — synthetic dataset generator with ground truth
• Phase 3 — ForceAtlas2 layout, frame builder
• Phase 4 — HTTP API, frontend, the map on screen
• Phase 5 — the time slider
• Consent layer — /about, robots.txt, opt-out (brought forward out of Phase 7: it is what has to exist before a crawler may run at all)
• Phase 2 — crawl scheduler, queue leases, ingestion, adaptive intervals. crawl is a bounded command rather than a daemon: the first real crawl should be something an operator watches.
• Phase 6 — community detection
• Phase 7 — opt-out by direct message, via an ordinary account on an ordinary server rather than an actor of our own (instance panel and search still to do)
• Phase 8 — live relay layer: a minimal actor, relay subscription, liveness and discovery from the firehose, /api/live for the frontend
• Phase 9 — authenticated instances

Building

Go 1.26+ and nothing else. Anything from 1.21 works too, since it fetches the toolchain go.mod asks for; older than that cannot, and the build simply refuses. The only direct dependency is a cgo-free SQLite driver, so go build produces a static binary you can copy to a server.

go build -o fedi-map .
go test ./...

Running

cp config.example.toml config.toml   # then edit public_url and contact
$EDITOR config.toml

# Seed from a fedilist export (~30,600 instances, under a second).
curl -o fedilist.csv 'https://fedilist.com/instance/csv?onion=&fam=&nsfw=&sort=domain'
./fedi-map seed -csv fedilist.csv

# Identify a single instance and read one timeline page.
./fedi-map probe fosstodon.org
./fedi-map probe -json mastodon.social
./fedi-map probe -offline mastodon.social    # cache only, no network

# Crawl. Refuses to run until `contact` is set.
./fedi-map crawl -n 20        # 20 due instances
./fedi-map crawl -n 0         # until nothing is due (plain `crawl` defaults to -n 20)
./fedi-map crawl -n-dry -n 5  # read exactly as a real crawl would, write nothing

./fedi-map stats
./fedi-map checkpoint       # reclaim the write-ahead log's space
./fedi-map software         # what the fediverse runs, and what we cannot read
./fedi-map software -write  # record software recovered from cache (no network)
./fedi-map cache            # what is cached, and what actually left the machine
./fedi-map cache -prune

# Honour an opt-out request. See "Consent" below.
./fedi-map optout -n mastodon.example    # dry run: what would be deleted
./fedi-map optout mastodon.example

# Domains not worth spending requests on. Not the same thing as opt-out:
# see "Blocklist" below.
./fedi-map block -suggest             # what looks like wildcard DNS
./fedi-map block -n troll.example     # what blocking it would remove
./fedi-map block troll.example        # block it, and delete what is there
./fedi-map block -list

# The same requests, arriving by themselves. See "Consent" below.
./fedi-map inbox -check     # is the token good, and whose account is it
./fedi-map inbox -dry-run   # what is waiting, and what would be done about it
./fedi-map inbox            # action it

Seeing the map

./fedi-map synth                       # generate data
./fedi-map layout -score               # lay it out, scored against ground truth
./fedi-map frames                      # materialise the per-day payloads
./fedi-map serve                       # http://127.0.0.1:8099

layout -score reports how well the layout separates known communities. Anything near 1.0x means the map carries no structure; a healthy run on synthetic data scores 3–5x with a positive silhouette. layout -gexf out.gexf writes a file you can open in Gephi for the things a number does not catch.

Keyboard: space plays and pauses, arrows step a day, shift+arrows a week, / focuses search, Escape closes it.

Watching it grow

Playback interpolates between chronological layouts computed on the server. Each day starts from the previous day's positions, then settles against that day’s membership and edge weights. The shared result is deterministic across browsers and immediately available for backward scrubs and repeat plays; the browser does no force simulation of its own.

The colors popover can switch between reported software and experimental relationship communities. Community detection runs deterministic weighted Louvain on each day's displayed 28-day attention graph; software, language, geography and layout coordinates are descriptions of its output, never inputs. Daily partitions are matched by member overlap so colors remain stable through playback and splits or merges can be recorded without preventing them. The default modularity resolution is 0.5; frames -community-resolution N and the matching serve flag exist for experiments.

The display hides union-degree-zero discovery records by default; they have no relationship evidence and otherwise form a structureless repulsive halo. Real small components remain because every member has an edge. The closed loforo.com root/subdomain component is explicitly omitted because it has no edge to a domain outside that family.

Debugging the frontend

/_probe.html is a dev-only page that drives the render pipeline through five sampled days and reports decoded/positioned node counts through document.title. Native headless screenshots may omit the WebGL canvas, so numbers in the title or an Xvfb framebuffer capture are the reliable checks:

chrome --headless --use-gl=angle --use-angle=swiftshader \
  --enable-unsafe-swiftshader --virtual-time-budget=25000 \
  --dump-dom http://127.0.0.1:8099/_probe.html | grep -o '<title>[^<]*</title>'

probe is the tool for checking adapter behaviour against a real server. It makes at most a handful of requests and writes nothing.

Deploying

Deployment is a file copy: one static binary and one SQLite file. The frontend is embedded in the binary, so there is nothing to serve from disk and no build step on the host.

GOOS=linux GOARCH=amd64 go build -o fedi-map .
scp fedi-map user@host:

deploy/fedi-map.service is a systemd unit for the map server. It assumes the checkout on the host is the installation: binary, config.toml and the database all live in one directory, nothing is installed to /etc or /usr, and updating is a pull and a rebuild in place.

sudo install -d -o fedi-map -g fedi-map -m 2775 /srv/fedi-map
sudo -u fedi-map git clone <this repo> /srv/fedi-map
sudo usermod -aG fedi-map "$USER"     # log out and back in to edit the checkout

cd /srv/fedi-map && go build -o fedi-map .
cp config.example.toml config.toml && $EDITOR config.toml   # public_url, contact

sudo install -m644 deploy/fedi-map.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/fedi-map.service
sudo systemctl enable --now fedi-map

The setgid bit (mode 2775) is what makes the shared checkout work: files you create stay group-owned by fedi-map, so the service can read a config you edited and write a database you created.

The unit runs unprivileged with an empty capability set, a read-only root, and /srv/fedi-map as the only writable path. Note that this puts the binary somewhere the service can write, which is the price of building in place; the unit's header comment says how to close that if you would rather.

Keeping it crawling

deploy/fedi-map-crawl.service and its timer run the crawler on a repeating bounded run: 55 minutes of work, then five minutes off, then again from a fresh view of what is due.

sudo install -m644 deploy/fedi-map-crawl.service /etc/systemd/system/
sudo install -m644 deploy/fedi-map-crawl.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now fedi-map-crawl.timer

systemctl list-timers fedi-map-crawl.timer   # when it next runs
journalctl -u fedi-map-crawl -f              # what it is doing

Run one crawler at a time. The per-host gate — one request in flight per host, one second between them, Crawl-delay obeyed — lives inside the process. Two crawlers are safe for the database, because queue leases stop them claiming the same instance, but they are not safe for the promise on /about: nothing coordinates them, so an instance can be hit by both at once. Stop the one in your terminal before enabling the timer.

A repeating bounded run rather than a daemon is not squeamishness about long-lived processes. crawl -n 0 freezes what counts as due at the moment it starts, so a long run does not chase its own tail through the 30-minute failure backoff — which means a run lasting a week would never see the instance that became due an hour into it, and the adaptive re-crawl intervals that keep the whole-fediverse load at well under a request per instance per day would stall for as long as the run lasted. Ending the run every hour is what makes those intervals real, and it leaves the crawler holding no state that a deploy or a reboot can lose.

-for bounds the run by wall clock rather than -n by count because the two ends of the queue differ by two orders of magnitude in cost per instance: a name that no longer resolves is one failed DNS lookup, a live server is several paged requests with a second of spacing between them. -n 50000 is twenty minutes or ten hours depending on where the queue has got to.

Keeping it current

deploy/fedi-map-refresh.service and its timer run layout then frames daily at 04:00, so the map reflects what the crawler has been collecting without anyone running anything.

sudo install -m644 deploy/fedi-map-refresh.service /etc/systemd/system/
sudo install -m644 deploy/fedi-map-refresh.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now fedi-map-refresh.timer
sudo systemctl start fedi-map-refresh          # build one now

The third step is not decoration. WAL space is reused but never returned, so the file sits at whatever high-water mark it has ever reached: one VACUUM on a 2 GB database left a 20 GB WAL behind on a disk with 40 GB free. checkpoint folds it back in and truncates it, and a rebuild is exactly the thing that makes it necessary. It is also available on its own — ./fedi-map checkpoint — for the same situation after a bulk delete.

Daily rather than hourly because a layout is minutes of CPU and node positions are stable by design: a day's crawling moves edge weights rather than the shape of the thing. It passes no -top-k or -window for the same reason the server unit passes none — all three default to the same values, and a value given to one and not the others produces a frame version the server looks for and cannot find.

Caddyfile.example terminates TLS in front of it. Prefer a bare hostname there so Caddy provisions a certificate: public_url becomes the ActivityPub actor URI once the opt-out inbox lands, and Mastodon will not federate with an http:// actor, so HTTPS is a prerequisite for that rather than a nice-to-have.

Because the database is one file, upgrading data and upgrading code are separate operations. A hot backup is VACUUM INTO, which needs no downtime.

Rebuilding under the live service is cache-safe. Every payload URL is keyed to the content it was built from: /api/frames/v{fver}/{day}.bin folds the layout version into fver, and /api/atlas/v{ver}.bin folds the frame version plus the last built day into ver. A rebuild mints new URLs rather than rewriting old ones, so the immutable, max-age=31536000 they are served with is a promise the bytes never change — and it is kept. Each frames run also GCs every superseded row, so the database does not grow a dead timeline per rebuild.

Two caveats remain. A tab that loaded the map before a rebuild holds the old URLs and will get 404s until it reloads — no corruption, just a stale tab. And frames -rebuild refuses to overwrite days already built under the current version; run layout first, which mints a fresh version cheaply by carrying node positions forward.

fedi-map serve drains in-flight requests on SIGTERM, so systemctl restart does not truncate a payload mid-transfer.

Developing without crawling

The real crawl runs from the deployment host, where /about and the opt-out endpoint exist. Everything downstream of the crawler is built against a generated dataset instead, so no development work touches the network:

./fedi-map synth                       # 600 instances, 540 days, ~0.3s
./fedi-map synth -instances 4000 -days 1100 -communities 20   # ~5s, 27 MB
./fedi-map synth -stats

Generation is deterministic: the same -seed reproduces the dataset exactly, so a layout regression can be told apart from a different random draw.

Synthetic data buys something live data cannot — a known answer. synth_truth records which community each instance genuinely belonged to over which span, and synth_event records the structural changes the generator deliberately staged: one community splits, two merge, instances are born and die throughout. ForceAtlas2 and Louvain can therefore be scored rather than eyeballed, and the time slider has explicit acceptance criteria.

Distributions are fitted to the live measurements. At 4,000 instances over 1,100 days:

	Synthetic	Measured live
Median instance size	3 users	2 users
p90 instance size	71 users	49 users
Above the 10-user threshold	~30%	14.5%
Distinct mention targets per active instance-day	5.5 (max 172)	~7 (pleroma.envs.net), ~24 (fosstodon)
28-day window graph	1,039 nodes, 18,911 edges	fediverse.space had ~4K nodes

Domains are under .synth.invalid — reserved by RFC 2606 and permanently unresolvable, so a synthetic dataset can never cause a real request.

Politeness

Every request goes through three stages before it can reach a server: the cache, the per-host gate, and only then the network.

The cache is there for politeness more than speed. Re-running a probe while developing, or restarting the crawler, must not produce a fresh burst of traffic. Defaults:

Response	Cached for
nodeinfo, /api/v1/instance, /api/meta	24 h
peer lists, robots.txt	7 days
timelines	30 min
401 / 403 / 404 / 410 / 422	7 days
429, 5xx	15 min, or Retry-After if longer

Denials are cached deliberately and for a long time. A 422 from Mastodon's DISALLOW_UNAUTHENTICATED_API_ACCESS is a durable fact about how an instance is configured, not a transient failure. Repeatedly asking an endpoint that keeps answering 401 is indistinguishable from probing for a way in, and is the single most reliable way to get an IP banned.

A server's own Cache-Control is honoured when it asks for longer than our floor, never shorter — the floor bounds how often we can possibly ask.

The per-host gate enforces one in-flight request per host, a minimum delay between requests, and any server-requested backoff (Retry-After, 429, or an x-ratelimit-remaining budget nearing exhaustion) before the next request is sent rather than after it fails.

fedi-map cache reports what is stored and how much traffic actually left the machine. Four passes over six instances costs 28 requests on the first pass and zero on the rest.

While developing, set cache_ttl_override high, or use probe -offline to work entirely from cache. probe -refresh is the only thing that forces real traffic.

What it crawls, and what it does not

• Metadata for everything discovered — nodeinfo and peers, a few KB each.
• Timelines only for instances with 10 or more users. Of ~30,600 live instances, 12,883 are single-user. The project maps communities, not individuals; this is the same line fediverse.space drew.
• No statuses, media, or account names are ever stored. The adapter layer converts mentions to hostnames and discards everything else before any data reaches the database. A configured known-network reader keeps one opaque status id as its pagination cursor, not a history of posts or ids.
• Steady state is roughly 0.68 requests per instance per day — less load than one logged-in user's client refreshing their timeline.

Live activity sources

An instance that will not serve us its timeline still pushes its public posts through relays. That is the only route to a large part of the network — mastodon.social answers 422 to every timeline request — so fedi-map can subscribe to one.

./fedi-map relay -actor                              # our actor, key and inbox
./fedi-map relay follow https://relay.example/actor  # ask to be sent its firehose
./fedi-map relay -list                               # asked, accepted, refused, and how much has arrived
./fedi-map relay unfollow https://relay.example/actor

This is the one place fedi-map is an ActivityPub actor, and it is as small as that can be. A keypair, an actor document, a signed Follow, and an inbox that verifies signatures. No outbox, no delivery, no followers, nothing it will ever post. The consent inbox deliberately borrows an ordinary account on an ordinary server to avoid exactly this — but a relay will not be followed by an account. ActivityRelay refused one, and Mitra has no instance-level relay feature to fall back on, so the narrowest possible actor is what remains.

Mitra's authenticated known-network timeline can complement those relays. Set [known_network] enabled = true to poll the public timeline using the existing [consent] account credentials. The cursor is persisted, pages are processed oldest first, and consecutive browser flashes are spread across a configurable 50 ms–1 s jitter window. A missing cursor is advanced without replaying old posts as present-tense activity. This is still a biased local view — public activity Mitra actually received — and is never used for edge weights.

The ActivityPub endpoints (/actor, /inbox, /.well-known/webfinger) exist only once there is a subscription to serve. /api/live also exists when the known-network reader is enabled. An inbox nobody needs is still an inbox anyone can post to.

robots.txt and opting out are not the same refusal, and the relay path is where the difference shows. robots.txt governs the requests we make to a server — it is checked before any other request, and an instance that disallows us is never fetched, relay or no relay. It says nothing about what a third party sends us, and an instance publishing into an open relay is broadcasting to every subscriber by choice; recording that it posted is not a request to it, and its queue entry stays blocked. Opting out is different: that is somebody asking to be removed, so relay traffic never re-adds them and never lights them up. /about states the distinction rather than leaving an admin to assume the stricter one.

What arrives is used for liveness and discovery, never for edge weights. An Announce usually carries only the post's URI, and that URI's host is already proof that the instance exists, federates and is posting now — worth more than any peer list, and free. Reading its mentions would mean fetching every post, which is a request per activity and is deliberately not done. Relay rows are stored as source = 1 and every weight query filters source = 0, because a relay shows a biased sample of an instance's output, biased differently per instance.

/api/live streams those hosts as server-sent events, and the map consumes them: an instance lights up as its posts reach us, and the edges it sits on brighten with it. SSE rather than a WebSocket because it is one-directional text that reconnects by itself, so a map left open overnight survives a deploy without anybody writing reconnection logic.

Sparkles are painted only while the timeline is showing the present. Scrubbed back to last Tuesday, a node flashing because it posted a second ago would be claiming something about last Tuesday that is not true — so the badge in the header says the feed is live and the map is not, rather than the map quietly lying. Only instances already placed can light up immediately. Live-source liveness itself makes an unplaced origin a node, so after the next layout/frame rebuild it has a dot and subsequent messages light it up; neither relay nor known-network traffic contributes connection weights.

Reading the map

Node radius runs 2–16 px from a log-scaled user count, and now responds to zoom: sub-linearly, clamped to 0.45–3.2×, so zooming out thins the map to its large instances and zooming in separates things that overlap. A node below 1.8 px fades rather than being clipped, because a hard cutoff makes nodes pop in and out while panning, which reads as flicker rather than as detail arriving. At the default view nothing is dimmer than it was: the fade threshold sits just under the minimum radius, so it starts on the way out.

The gear button opens three multipliers — node size, link brightness, link width — kept in localStorage and never sent anywhere. They are preferences about legibility on somebody's screen rather than claims about the data: a map too faint on a bright monitor is a different problem from a map that is wrong, and only one of them is fixed by changing what is drawn. The tuned appearance is the 1× baseline, so resetting all three controls to 1× preserves it.

Blocklist

block DOMAIN covers that domain and everything under it. Blocked names are refused at discovery, so a peer list full of them puts nothing in the database, and whatever is already there is deleted.

It is not opt-out, and is deliberately a separate command with a separate table. Opting out is a promise kept to somebody about their own instance; blocking is housekeeping about our own attention, and the two should not be reachable by the same typo. Blocking refuses outright if the domain has any collected history, so a mistyped block cannot quietly delete an instance that is on the map — optout is the command that deletes real data, and it says so.

block -suggest finds candidates the way the first one was found: parent domains with many subdomains and not one answer between them. Read the output before acting on it. The signal is the ratio, not the count — masto.host and wordpress.com have thousands of subdomains and thousands of live instances, and blocking either would delete a real part of the fediverse.

Consent

Nothing may be crawled before an admin has a way to say no, so this exists before the crawler does.

/about is served by the binary at exactly the path the User-Agent points at — fedi-map/0.1 (+https://your.host/about). A contact URL that 404s is worse than no contact URL, so TestUserAgentContactURLResolves reads the URL out of the User-Agent and asserts that path serves HTML. The page shows the real configured User-Agent, states precisely what is and is not collected, and carries the opt-out instructions. Set contact in config.toml; serve warns on startup while it is empty.

The consent inbox turns "DM a human" into something that happens whether or not the human is reading. Point [consent] at an ordinary fediverse account — one on the server you already run — and fedi-map inbox reads its mentions, works out what each one asked for, and actions it:

./fedi-map inbox -check      # confirm the token, and whose account it is
./fedi-map inbox -dry-run    # decide, but change nothing
./fedi-map inbox             # action what is waiting
./fedi-map inbox -review     # older provisional opt-outs awaiting a human

serve runs the same loop on poll_interval when one is set.

fedi-map is deliberately not an ActivityPub actor. The account's own server does delivery, signatures, spam handling and moderation, all of which an admin's ability to say no would otherwise depend on this crawler implementing correctly.

Who counts as an admin is decided from what the instance itself publishes — staffAccounts in nodeinfo (Mitra, Pleroma, Akkoma) or the contact account in /api/v1/instance (the Mastodon family) — never from what the message claims. Both directions require a verified admin. An account merely being hosted on an instance is not authority to remove that instance, delete its history, or grant permission to crawl it:

• Opt-out and opt-in from unverified accounts are refused. The reply names the published staff account that can make the request, when one exists.
• robots.txt outranks both. It is the instruction we can be certain came from whoever controls the server.

Opting in also does something opting-out-in-reverse does not: it lifts the ten-user floor for that instance, which is what instance.opt_in is for. A small instance that wants to be on the map can say so.

While every domain in the database ends in .synth.invalid, the page says so in a banner. That is derived from the data rather than from a flag, so it cannot be left switched on after real data arrives, or off before it.

robots.txt is the opt-out that costs an admin nothing and requires them to know nothing about this project:

User-agent: fedi-map
Disallow: /

It is checked ahead of even the HTTP cache, so an instance that disallows us reports the same way whether or not we spoke to it last week. The parser implements RFC 9309 properly rather than approximately — longest-pattern-wins with Allow breaking ties, * and $, an empty Disallow meaning no restriction — because Disallow: /api/ with Allow: /api/v1/instance is a real configuration and inverting it would silently ignore the admin either way. A robots.txt that returns 5xx is read as a refusal, per §2.3.1.4, not as permission. Crawl-delay raises the per-host spacing and can never lower it.

Direct requests are actioned with optout:

./fedi-map optout -n mastodon.example              # what would be deleted
./fedi-map optout -actor @admin@mastodon.example mastodon.example
./fedi-map optout -undo mastodon.example
./fedi-map optout -list                            # the audit trail

The argument can be a bare domain, a profile URL, or a handle — an admin who wants to be left alone should not have to get a format right first.

Opting out deletes rather than hides. Hiding would be easier, since every read path already filters opt_out = 0, but a database that still holds the data has not honoured the request. It removes daily activity counts, mentions in both directions, edges, software history, crawl records, the cached HTTP responses from that host — which is where anything fetched actually lived — and the metadata on the instance row. What is kept is the domain and the flag, which is the minimum that can stop a re-crawl the next time somebody's peer list mentions them; the instance id must not be freed for reuse, because edge ids and layout indices are keyed to it.

-n reports by doing the whole thing in a transaction and rolling it back, so what it promises is what the real run deletes rather than a separate estimate that can drift.

Notes on the ecosystem

Why weight is a share of outward attention

w(A->B) = (mentions + boosts A->B) / (everything A directed outward + 20)

Folded to the stronger direction, because relationships are routinely asymmetric and the strong direction is the one that says something: a small instance may send a third of its attention to a large one that returns a thousandth of its own.

Three properties, each of which the obvious alternative gets wrong:

• Only the source needs a denominator. The first version divided by both endpoints' status counts, which meant an instance that will not let us read it could never have an edge at all. mastodon.social answers 422 to every timeline request and had 891 mentions from instances we can read — every one discarded, leaving the most-mentioned instance in the fediverse off the map. It is now placed by the attention directed at it, which is the truest thing this data can say about a server that will not let us look.
• Boosts count, their contents do not. A boost is an editorial act about another community and one of the strongest available signals that two of them read each other. The mentions inside a boosted post belong to whoever wrote it; counting those would attribute one person's conversation to everyone who passed it on, which is why reposts were originally dropped wholesale.
• Thin evidence is shrunk, not cut. Dividing by observed activity hands an instance whose entire outward history is three mentions of one peer a perfect 1.0. Adding a constant to the denominator pulls small numbers down and leaves large ones alone — a floor on confidence rather than on volume, so a small instance keeps its only relationships instead of being cut from the map.

Relay-derived rows (source = 1) are excluded from all of it. A relay shows a biased sample of an instance's output, biased differently per instance, so its numerator and denominator have to stay in their own arithmetic.

Findings from surveying live instances, which shaped the design:

• ~40% of fediverse posting volume is behind authentication. Blocking correlates strongly with instance size. See docs/auth-required-instances.md.
• The Misskey family needs POST /api/notes/local-timeline. Zero of ten probed Misskey/Sharkey/Iceshrimp instances returned anything usable from GET /api/v1/timelines/public. Porting the Mastodon path naively loses ~2,000 instances silently.
• Mitra denies its public timeline to guests by default and advertises the fact in allow_unauthenticated.timeline_local, so we read that instead of spending a request.
• GoToSocial statistics need an explicit trust boundary. Its instance-stats-mode=baffle publishes randomized, preposterous user and post totals and disallows nodeinfo in robots.txt. fedi-map therefore does not import GoToSocial user/post counts from the fedilist seed; it only stores counts observed during a permitted live crawl. Unknown counts are shown as such rather than turned into node sizes. Forks and mislabeled servers can evade a software-name check, so any still-unverified Fedilist count above 100,000 is also treated as an estimate and given the default node size. The raw hint remains available for crawl priority and is labeled in the detail panel; a successful crawl replaces it with an observed count.
• fedilist's discovered_at spans 2021 to now, so node birth dates backfill on day one and the time slider is useful immediately.
• Peer lists are a public write surface. 98.5% of everything the first real crawl discovered — 11,732,133 of 11,908,227 instance rows — was *.activitypub-troll.cf: one registered domain answering wildcard DNS with endless invented subdomains, arriving through the peer lists of instances that had done nothing wrong (one contributed 202,039 of them by itself). None ever answered. Anyone who can serve wildcard DNS can do this to any crawler that reads peer lists, so block is not an optional nicety; without it the queue is whatever the most motivated troll wants it to be.
• Peer lists are mostly a graveyard. Of the peer-discovered domains probed for the first time, 2.4% answer: a 10-minute sample of 6,131 first contacts gave 148 live instances, 1,614 names DNS has never heard of, and the rest timeouts, TLS failures and hosts that are not fediverse servers. Retry budget is therefore not uniform — an instance crawled before gets eight attempts, a never-confirmed name gets three, and one that returns NXDOMAIN gets two. At eight attempts each, retrying the dead would have cost more crawler-hours than every live instance combined.
• Discovery outruns crawling by three orders of magnitude. The first 3,680 instances identified named 3.4 million distinct peer domains between them, and one Mastodon peer list runs to 221,703 entries on its own. A queue ordered by due time alone therefore works through hostnames nobody has ever posted from for years before it reaches the ~7,000 instances that carry the graph, so queue.prio tiers it: above the personal threshold, below it, and never-probed. Everything is still crawled; the order is what changes.
• Identification and support are different questions. A Lemmy or PeerTube instance answers nodeinfo perfectly and still has no adapter. That is a fact about us, not about them, so it is recorded as no_timeline = unsupported and the instance keeps the slow metadata cadence — rather than being retried with failure backoff and marked dead, which is what happens to a host that is actually gone. fedi-map software reports the resulting distribution; what is missing from it is the list of adapters worth writing next.

How the time sweep works

Positions and topology are static and ship once; a frame is only presence and weights.

Node presence carries forward for a week past the last day an instance was observed. That is not smoothing: re-crawl intervals run from six hours to five days, so on any given morning most instances have not been read yet, and the newest day of the map was a cliff that made the fediverse look like it emptied out overnight. The span already spreads across quiet days in the middle for the same reason — an instance that says nothing for a week has not ceased to exist — and the end of the timeline is where that reasoning could not previously be applied, because a span cannot extend past its last observation.

Node coordinates never change, because the layout is computed over the union of every instance ever seen. The edge universe is enumerable and gets dense, stable integer ids. So a day's graph is a subset of a fixed structure, and the browser uploads topology to the GPU once and then writes preallocated typed arrays per frame. Scrubbing is a buffer write, not a graph rebuild.

• /api/atlas/v{n}.bin — fetched once. Positions, identity, the full edge list, and run-length encoded node presence across the whole timeline.
• /api/frames/v{fver}/{day}.bin — a few KB. Delta-varint edge ids and one log-quantised byte of weight each.

Measured on a 4,000-instance, 3-year synthetic dataset: 1,139 graphed nodes, 64,854 edges, 6.8 MB for the entire timeline at ~6.6 KB per frame, with a 220 KB atlas. All of history fits in browser memory.

Node presence is tracked separately from edge presence on purpose. An instance that goes quiet for a month loses every edge, but it is still alive and stays on the map as an isolated dot rather than blinking out.

License

AGPL-3.0-or-later, matching fediverse.space, whose design this builds on.