Pub Crawler

Pub Crawler is a research project by Evan Prodromou, a master's degree student in Computer Science from Georgia Tech.

Research

Pub Crawler is a crawler for the Fediverse -- a network of independent, interoperable social network platforms, including Mastodon, Meta Threads, Flipboard, and WordPress.

Pub Crawler follows links between actors on the Fediverse -- mostly people, but also bots and groups.

The data collected by Pub Crawler will be used to answer questions like:

• How big is the Fediverse?
• Is it one connected network, or are there sub-networks that don't link to each other?
• What's the average and maximum distance ("degrees of separation") between actors on the network?
• What are mean and median number of followers and following of actors on the network?

Behavior

Pub Crawler only follows social connections through the "followers" and "following" collections. It does not read content posted by the actors. It only uses public connection data.

Pub Crawler tries to minimize impact on running servers by following rate limits used by Mastodon servers: 300 requests per 5 minutes for most data, 300 requests per 15 minutes for collection pages (followers/following).

About Evan Prodromou

Evan is also the research director of the Social Web Foundation, co-author of ActivityPub, and author of the book "ActivityPub: Programming for the Social Web" from O'Reilly Media. He can be reached at evanp@gatech.edu.