Skip to main content

Excited to announce that I will be at #fediforum today speed demo-ing my latest project: an ActivityPub data observatory!

This observatory does not collect any user data or metadata. Instead I am looking at the *shape* (aka schema) of data being sent around the fediverse. This will let software devs ask questions like "How is a Mastodon 4.2.0 image post formatted differently from a Misskey 2024.7.0 image post?"

And we'll get real answers based on data rather than on poor documentation.

#fediforum
in reply to Darius Kazemi

Evan Prodromou
Evan Prodromou
• •
can you compare to browser.pub?
in reply to Evan Prodromou

Darius Kazemi
Darius Kazemi
• •
@evan yeah. on browserpub I can say "hey help me take a look at these particular messages I know about". This observatory will surface information about stuff floating around the fedi that I don't even know about. For example I am already learning about server software I've never even heard of, and I would not have put that into browser.pub because I wouldn't have known it existed
@Evan Prodromou
in reply to Darius Kazemi

Evan Prodromou
Evan Prodromou
• •
Ah, OK, interesting. Where does your network tap plug in?
in reply to Evan Prodromou

Darius Kazemi
Darius Kazemi
• •
@evan still figuring it out. Right now I am subscribing to a public relay as that is the most software-neutral source I could think of, but I am looking at other ingestion methods too. Importantly I want to ingest AP only... I'm not going to hit proprietary API endpoints like most scrapers do
@Evan Prodromou
in reply to Darius Kazemi

Darius Kazemi
Darius Kazemi
• •

I won't be actually LAUNCHING this tool until I've found out how you all would feel about it being opt-out vs opt-in. I will provide a longer blog post for you all to read with details, but in short:

It would be really helpful for general interop on the fedi if this were opt-out. But if people are generally freaked out by having technical details about software data formats being opt-out... I'll make it opt-in.

Quick explanation of the data scrubbing in the attached images

An image of some data labeled "WHAT SCRAPERS COLLECT" that contains things like: atomUri: 'https://friend.camp/users/darius/statuses/113052195027718205', inReplyToAtomUri: null, conversation: 'tag:friend.camp,2024-08-30:objectId=26481883:objectType=Conversation', localOnly: false, content: '<p>On Sep 12 the Applied Social Media Lab (where I work) is hosting an event</p>',
An image of some data labeled "WHAT THIS DATA OBSERVATORY COLLECTS" that contains things like: atomUri: '<uri>', inReplyToAtomUri: null, conversation: '<string>', localOnly: <boolean>, content: '<string>',
in reply to Darius Kazemi

blaine
blaine
• •
nice! The only folks who *I* could imagine insisting on this being opt-in are Oracle's legal team, and they were told in no uncertain terms that this sort of data isn't even *eligible* for opt-out, even in the US of A. 😅
in reply to blaine

Darius Kazemi
Darius Kazemi
• •
@blaine every morning I ask myself: "Am I going to do something today that Oracle's legal team won't like?" and if the answer is no I have already failed
@blaine
This entry was edited (5 days ago)
⇧