Bluesky downtime

Bluesky was down last night and I saw a couple posts questioning how this could happen if it was really decentralized. Worth a few thoughts here.

If mastodon.social went down, what would the user sentiment be? It hosts 2.7 million users, or roughly 15% of the fediverse. (Not counting Threads because Meta doesn’t publish their data.) While down, users on mastodon.social wouldn’t be able to interact with the fediverse or download a copy of their posts.

This is best illustrated with this humorous Mastodon critique from Rob Shearer:

M: You start by choosing an instance. But the important thing is it doesn’t matter because all instances are federated and you can migrate between instances.

U: So if the instance I chose gets shut down I can migrate to another?

M: No.

Even so, if a single very large Mastodon server went down, no one would use it to question whether Mastodon is decentralized, other than as a reminder that smaller communities are often better and make the network more resilient. Likewise, it’s not really fair to frame bsky.social as merely a large server; it’s effectively the only thing right now, which isn’t true for mastodon.social.

With AT Protocol, it’s possible to host your posts outside of Bluesky. Very few people do this, but it will become more common as third-party hosting services (like Micro.blog) support it. In that case, if bsky.social or bsky.app went down, you could still access your posts.

When you call the Bluesky API via bsky.social, it actually proxies your requests to the appropriate backend data server if it’s hosted on Bluesky. This makes the API feel very centralized, but all of the data could still be accessed directly in a more distributed way, like accessing individual websites.

Bluesky is not federated like Mastodon, but the open architecture that Bluesky was designed around is valuable. It is distributed sort of like the web is distributed even though Google exists and dominates search. Eventually, there will be other AppViews that work like Bluesky but run in parallel with their own timeline. In that case, if Bluesky went down, some other apps might still function, and could utilize the same data and social graph.

We need to move beyond a Mastodon vs. Bluesky mindset. As I’ve blogged several times, they are solving slightly different problems. We can draw inspiration from each one to make the web more open. I want both to exist.

Update: Tweaked a couple references to how PDSes work to reflect feedback in comments.

Kuba Suder 🇵🇱🇺🇦

Hey, a few clarifications:

it’s not really fair to frame bsky.app as merely a large server; it’s effectively the only thing right now

Bsky.app isn’t an account server, it’s really just a web client. There’s a fork now at deer.social which can access the same data, if bsky.app site was down.

Kuba Suder 🇵🇱🇺🇦

> When you call the Bluesky API via bsky.social it actually proxies your requests to the appropriate backend data server that might be hosted elsewhere. This makes the API feel very centralized

This is only really true for Bsky-hosted PDSes, I need to call lab.martianbase.net instead of bsky.social

Kuba Suder 🇵🇱🇺🇦

For me, with a self-hosted PDS, the actually centralized parts right now are the AppView, relay and plc.directory. I could still post and talk to some people if bsky.app, bsky.social and Bluesky PDSes were all down (I could post yesterday the whole time).

Manton Reece

@mackuba.eu Whoops, I didn’t realize that. So if you call e.g. createRecord on bsky.social, does it return an error?

Manton Reece

@mackuba.eu Thanks, I’m going to update the post to reference bsky.social in that line instead.

Kuba Suder 🇵🇱🇺🇦

I’m not 100% sure (can’t check now on mobile), but I think trying to log in with createSession to post records would fail for me. bsky.social is an “entryway” server for authentication, but only for *.bsky.network PDSes.

Paul's Dev Notes

1. Bluesky’s about interop and scale, mastos about topological distribution, blockchains are about censorship resistance. Decentralization means different things

  1. All of our PDSes got DDOSed. Any PDS that wasn’t stayed online. If the same happened to AP, they would’ve had the same outage
Manton Reece

@pfrazee.com I like that summary. I’m going to edit my post a little because I think I got a couple things wrong about PDSes, don’t want to add more confusion.

Paul's Dev Notes

ActivityPub doesn’t scale correctly for the social media usecase anyway so this is a pretty pointless debate

🦋 Franklin Seal ✂👑

So you found the mythical edit button, then?

Paul's Dev Notes

I didn’t think you were far off in your writeup. I was just sharing additional thoughts on my mind

🦋 Franklin Seal ✂👑

Source of the attack? I’m sure I’m not the only one curious to know.

Jason Culverhouse

Was there a DDOS? I could see that anonymous requests were going through fine. It was only the session system that seemed down. Any request with an Authorization header was basically returning a 503 NotEnoughResources.

Nemes Content

So does that mean those self hosted pdses could still post to and see each other?

Kyle Drake

DDoS is a part of life unfortunately, of any online service, of any scale. Mastodon is far easier to DDoS because you just winnuke the crappy unpatched VPS it’s being hosted on and then the hosting provider blackholes your server.

DDoS is a universal, mandatory training on resilience improvement.

minbash.bsky.social

Internet based solutions are all prone to DDoS, as the network itself is prone to DDoS. The only thing that we can do is to make DDoS attacks extremely expensive.

ZippyVtuber🔞

Interesting

Everett Bogue

You’re spot on in your write-up on your blog. This is a similar problem to what was described in the initial Nostr readme under “The problem with Mastodon and similar programs” fiatjaf.com/nostr.html – unfortunately ATProto repos cannot exist in many places yet, so “and similar programs” applies.

Gov

#notmypds

Manton Reece

@manton.org Ugh, I hope my blog post didn’t add to the stress. I love what y’all are doing with Bluesky, hopefully that came across a little in the post for other folks.

🧼
Paul's Dev Notes

oh no manton not at all. You helped relieve stress. I just ended up yapping in your replies

Snorlax goes by Stephanie

Sorry manton

Doug Jones

ATProto is interesting and seems like a well thought out protocol. But I do wonder what the uptake will be like on other groups running critical infra at scale like relays and PDSes.

Christine Lemmer-Webber 🌀

hm

Decorum

No, #ActivityPub is more resilient since it doesn’t need millions of dollars to work. The very existence of the fediverse proves you wrong.

Joe Ruelle

I think the point is use case.

Decorum

So?

Joe Ruelle

Not about funding or resiliency. But about use case. It’s microwaves and kettles. You can find both on the kitchen counter, but they do different things.

stellz

mastodon instances regularly shut because they’re too expensive to run

Jon P

hm indeed. it’s not like there’s only one social media use case.

i do think ATProto scales better for all-public “flat” networks – and that’s certainly an interesting use case! But what about networked communities with scoped visibility? That’s not implemented yet, so hard to project scaling.

>.<

yeah i would rephrase that as “ActivityPub doesn’t scale well for global all-public town-square type social media”

Index

- AP has its flaws too (e.g. not being able to move between servers afaik).

  • It is not ATP’s fault that a lot of users don’t self-host their PDS.
  • It is not cost millions of dollars to run the ATP, and many have proven that to not be the case.
Decorum

- Users can move between servers with ActivityPub…

  • Hosting their own PDS is insufficient if there is no alternative to Bluesky’s servers.
  • Running ATProtocol is surely mostly costless if there is not millions of users to connect with it, which is not the case with Bluesky anymore.
Decorum

Yet, setting up one is easier and cheaper than financing an alternative to Bluesky’s infrastructure.

Index

Sorry, I’m not that well-versed when it comes to AP, I just went based off what I’ve heard and Wikipedia.

There are several people hosting their own relays and appviews.

Stove Avoidin' Chuck

And the network doesn’t shut down when one mastodon server goes down. Bluesky doesn’t see this problem yet because it’s too hard right now to run more nodes. (using that term colloquially)

Manton Reece @manton
Lightbox Image