Bryan Newbold Protocol engineering
A protocol engineer at the social-media platform Bluesky describes building an open protocol around a fast-growing social media platform.
Let’s begin with how you introduce yourself. Who are you?
My name is Bryan Newbold. I work at Bluesky on the AT Protocol. I’ve worked here for a couple of years, after previously working at the Internet Archive. I have an ongoing interest in the web as a public commons and cultural artifact. I live in Seattle. That’s my usual intro.
How did the story that brought you to Bluesky begin? How did you start working on this new social network? You can go back as far as you want.
I could go way back to being interested in the web in general, but the more concrete narrative starts with participating in the open source and free software movement. I worked on Open Hardware project. Then I moved cross-country in the early 2010s, which disrupted my career in hardware.
After that move I attended the Recurse Center, a programming workshop, which exposed me to a bunch of new ideas. My project at Recurse was to build a decentralized tool for modeling mathematical equations in the natural sciences. I was trying to build a model of the physical world—one giant equation of how everything works. It was somewhat intentionally over-ambitious, the sort of project that was never going to really work. But it raised questions: How would people collaborate on a huge model? How could different people work on different pieces of it, and then have it all come together?
But when did you first get into building technology? Was it the math? Was it trying to figure out the nature of things? Was it just building things that were useful to you? What was the starting point?
I’m a bit uncomfortable with “it all started I was 7 years old” narratives. But I did grow up in a household where my father worked in computer networking. He worked at 3Com under Bob Metcalfe, the Ethernet inventor. Then he worked at various places—Iris, Lotus, IBM.
He worked on Notes, an enterprise software product which used interesting database replication technology. You could synchronize arbitrary company data down onto your laptop. I could see this in practice because we’d go on a family vacation, and he’d be synchronizing his work database at night. The 56K modems back then were slow. There was a vision where workers could download company information onto personal computers and still collaborate offline. It was a vision of computing technology enabling collaboration.
Like every millennial I grew up with the web. There was constant turn-over on the web—lots of people making their own little websites, participating idealistically. But then later it all centralized into big platforms. When Twitter and Facebook first came out I thought: “These networks are getting big. But presumably they too will churn after a year or two,” because that was the pattern at the time. There used to be a hype cycle, but then all at once one last product generation got fixed in stone and the turnover stopped.
That was the start of the decentralization interest for me—this feeling that it would continue to evolve and end up better. I was interested in alternative and decentralized networking technologies because they felt like the natural progression: water running downhill. It felt like giant corporations wouldn’t innovate, and that the internet would continue to evolve and stay heterogeneous.
Part of that belief was because people on the early web talked about these dynamics a lot. Using the web was a big deal, part of their identity. The experience was new, and people really thought about what the impact on society would be—how the web should be structured, the power dynamics. The fraction of people caring about the network itself was so high in the early days. It really felt like that aspect was going to continue.
But that energy hasn’t stuck around. People got busy, grew up, had kids, had other things to do. They didn’t want to spend ten hours a day talking about the impact of blogging on journalism, or what the internet means for civil liberties, or whether it would be the end of religion. Remember early debates about atheism and the internet? What the impact on broader society was going to be? Would it be the end of war? How could you convince normal people to go to war against strangers when everybody can chat directly on IRC? All that churning idealistic energy.
Folks still discuss social impacts today, but much less, and that is crazy because the Internet has become so much more important. So many more people are employed working on the web. Our reality today is that proclamations from “the king” come down on the king’s own website as shitposts. That would have sounded like a fever dream in the nineties, but now it’s happening. And people are just like, “Oh well, there’s nothing we can do. This is just how the web works.”
But then Elon Musk bought Twitter and kind of set it on fire. There was this incredible feeling of things becoming un-stuck. The impenetrable fortress of modern, multinational corporations—how could you possibly out-compete their engineering resources? Their computers are faster, their bandwidth’s better, they have so many employees on call, they have trust and safety teams. For a while you couldn’t compete with that. But now that is all kind of crumbling.
Pre-Musk, pre-Bluesky, where were the spaces you were finding, the communities that kept that dream of an open, playful, post-war, post-everything internet alive?
The dream never full went away. I worked at the Internet Archive during that era of corporate dominance and it felt like an institution holding on to its original ideals. The European hacker scene is more collectivist and politically thoughtful about the impact of technology on society. For many years I would travel to the Chaos Communication Congress. I’ve probably been to ten or fifteen of them over time. It was always inspiring to see what people there were thinking about, worrying about, working on.
There are role model figures—people like Moxie Marlinspike working on Signal. I have disagreements with some of Signal’s decisions, but I’m very impressed by Moxie’s principles and drive to take the work seriously and make an impact. I was never close personal friends with him, but being able to watch the Signal project and the debates around it—not just the team building Signal, but a community debating and critiquing it. There was a whole group of people saying, “But you failed to do this, or you were doing it in some way that’s problematic.” A critical community like that really raises the bar.
Another space keeping the dream alive is wiki energy, which connects to academia and scientific values. The common thread is people trying to work together and make progress for the benefit of broader society, the wiki folks just have a different model and way of working. You don’t have to go work at Encyclopedia Britannica—anyone can participate and improve Wikipedia. You don’t have to go work at Microsoft on Windows if you want to improve computer security. You can go hack on the Linux kernel or Debian. It’s a more direct way for people to participate.
That was a utopian dream for my generation. In the past to contribute you had to go work in a specific place, in a physical setting. Then we got the internet. People could collaborate at great distances and get really impressive, large projects done with less social structure. But it turns out you still need the social structures. It’s deeply important. Just leaving things unstructured, nothing happens. Successful projects like Wikipedia, Debian, or Linux stand out as being more structured compared to other open online collaborations.
In the process of being part of these communities and working at the Archive, what kinds of skills did you develop?
That’s a great question. I don’t know about “skills” though. Taste, maybe, or practice debating—developing lines of argument, or a value system. Working at the Archive did gave me experience with their technical approach. They are pretty frugal, trying to get as much done with as little money as possible. They do a kind of scrappy engineering, at large scale. They have buildings, racks, computers. They are not trying to get it done at home on a used laptops. They do have resources, but they try to squeeze the most out of them.
There are some skills there—weird tricks with kernel tuning, things you can do with GZIP that you never thought of, systems you can build cheaply with spinning discs. The actual skills aren’t necessarily different, but the line of thinking and debates and background knowledge is different from what you’d get working at a big tech company.
Take us back to that trip across the country.
I moved from the East Coast to the West Coast. I didn’t find a new job when I arrived, so I went to the Recurse Center, which got me thinking more creatively. It shifted me from hardware work—embedded hardware stuff, scientific instrumentation—toward more web, internet, collaborative online stuff.
After Recurse I worked in the tech industry a little, then ended up at the Internet Archive because they had a job opening. There was a career thread where earlier on I had done data collection for scientific instruments—projects doing multiple gigabits of streaming brain data that needed to end up on disk arrays and servers. I was used to working with large amounts of data. The Internet Archive was trying to get a big grant to build a scientific data repository using new dweb tech. The specific protocol was dat://, which was originally pitched as a successor to BitTorrent for doing distribution of research datasets that could be updated. The idea was that scientists would put their data in a folder folder on their laptop, and the system would synchronize out to other server clusters and researchers. Like a Dropbox alternative, but using modern content-addressed data structures and Merkle DAGs.
When I was hired the Archive thought that they were going to get on the bandwagon and work with the dat:// people. I thought that was a good collaboration. The Internet Archive has cheap data storage. Scientists need to store data. If they store it in the commercial cloud, it can be very expensive. All the research data wasn’t ending up in a good home and was difficult to access. Researchers need to track down where datasets are actually stored, which causes friction. If researchers got something like a BitTorrent magnet link and could download it from anywhere, that would work better. But torrents don’t work right because they don’t support updating data. Datasets get corrected, there are errata, things need to be redacted. So you need updates. I thought it was good technology and a good organization. And they needed an engineer to figure this out and make it work.
I was very excited and applied to work at the Archive. But that project fell through immediately. Almost as soon as I started at the Archive, it was like, “Okay, this grant isn’t going to happen. We don’t actually have a big budget for this.” But first I did have a couple of months to dig into implementing the dat:// protocol, and got involved in the community. That was my hook in to the whole DWeb thing.
At the Archive I ended up preserving papers and datasets from the web. I built the scholar.archive.org service and the Fatcat project, neither of which had much to do with peer-to-peer or distributed, decentralized technologies. But the whole time I worked at the Archive I was involved in the dweb scene, because that’s what I’d originally be hired for. Other people in the organization were interested in DWeb and we would talk and hang out.
The Archive had been hosting DWeb Summit events. I didn’t go to the first one, but they became a series that I attended and helped out with. I’d chat with people about how their projects and new tech could be used at the Internet Archive. That was my introduction to the scene socially.
One connection was Paul Frazee, who worked on the dat:// project and now works at Bluesky. We collaborated a bit on written specifications for dat://. And Jay Graber helped organize some of the DWeb Summit events.
Let’s get into Bluesky. Tell us about your relationship to it, how you saw this project developing, the motivation, and how you got involved.
I thought Bluesky was bullshit at first. I thought there were more interesting projects in the DWeb scene, things like Secure Scuttlebutt. I knew Jay a bit socially, and I remember her mentioning it early on: “I’m going to try to get involved with this Bluesky thing.” I thought to myself: “You are working with Twitter and Jack Dorsey—this sounds extremely suspicious. You are probably wasting your time.” I thought she was kind of a hot ticket, and wished she would work on something else. She could have contributed to an existing project or started a non-profit.
Jay is quite smart. She was effective at going around and figuring out what other people were working on. That is an important lesson I’ve learned from European hackers: First go help other people with their projects. Don’t come in with attitude of “I solved the problem”—no one’s going to listen to you. You have to go around, fix some patches, try other people’s stuff, give other folks attention first if you’re going to expect any attention on your own work.
Jay was good at that. She went around, talked to all these other people, got involved in these other protocol discussions, and did write-ups of the differences between protocols. It is so valuable to have someone who is not a founder or competitor do comparisons. Being a credible, neutral party allowed her to do really helpful community work in the DWeb scene.
But with Bluesky I thought, “She’s going to get screwed over somehow. Twitter is going to win in some way.” I was disappointed that she was working with them. I was impressed when I saw that she had hired Paul. But I was still very skeptical of the venture capital model in general—it has not worked out super well for society.
In the summer of 2022 Bluesky put out a whitepaper and I skimmed it. Later that fall, I was trying to figure out what I wanted to do next in my career. I liked the Internet Archive, but didn’t want to work there forever, and was ready to do something new. I loved working in-person at headquarters, but I had moved up to Seattle during COVID. I was thinking about getting back in to hardware, maybe robotics.
I remember around that time traveling to an Internet Archive event back down in San Francisco. While I was there I had a conversation with Brewster Kahle about what was going on the DWeb scene. Bluesky had put out another white paper about AT Proto, and I started reading it. I though to myself, “Yeah, okay, they really nailed it. But I don’t want to work on a DWeb project right now.”
I started implementing it in Rust anyways. I stayed up late at night programming while living in a motel room for a week at the Archive event. I traveled back home to Seattle and continued hacking on it. When I finished I thought it would be out of my system. “It works, it’s cool. They are probably going to win. This is probably going to be a big deal.” But I was done with DWeb.
But then I just couldn’t stop thinking about it. Elon was buying Twitter, and all this other crazy stuff was going on.
I’m just pitching you now, I guess—this is kind of the hype story.
Say more about what attracted you when you were implementing it in Rust. What stood out to you about this approach?
The protocol was simple enough that I could implement it as a solo developer. There are a couple of bigger projects floating around—SOLID or these other efforts—and they are like two hundred pages of specifications. To be fair, AT Proto has gotten more complex. But at the time it was a relatively simple thing I could dive into.
I had been hacking on IPFS, which was less web-y and more about file sharing, a kind of distributed file system. Looking at IPFS, looking at Scuttlebutt, looking back to these ancient things like W.A.S.T.E. and GNUNet, I had seen a lot of projects and knew what might work. AT Proto really nailed this balance of peer-to-peer stuff and web stuff.
The web works, right? The traditional web, HTTP, HTML—that stuff is pretty accessible. Working with it feels like you’re on the right path. The Internet Archive builds on all that. Indexing webpages, dealing with billions of documents, dealing with URLs—everything just kind of works. So if you can make your system function more like the web, things will work. We all know the web works from a scaling and infrastructure perspective.
AT Proto is pretty web-y. It’s not trying to force developers into the pattern of “all the data will be on your phone, you have to index everything on your phone, you need to figure out how to get your phone to talk to everyone else.” Instead it says, “Nope, you’re going to have big servers, and they cost some money, but it works. You can build big indexes.” But at the same time they kept all the user autonomy and control that you get from the more peer-to-peer stuff. The cryptographic ownership aspects—they retained a lot of that.
I take part in puzzle hunts, like the MIT Mystery Hunt. When you start solving a puzzle, you’re like, “How am I going to approach this?” But some point, even if you don’t have the final solution, you can see how you’re going to solve it. And when you see it, you’re always like, “This is going to work.” That it is elegant enough to work. That’s how I felt with AT Proto. It was just like, “This is going to work. I can see how it will all fit together, and it’s actually going to work.”
Did you join pre- or post-Musk’s takeover?
I think he had completely taken over when I started. It had been going on all summer, and I just—
And people were already migrating, right?
At the time people were migrating to ActivityPub. When I started taking Bluesky more seriously, they had a public chatroom, and I joined it. I talked to Paul, and because I had implemented the protocol, he was asked if I was interested in working with them. “Is this something you want to do?” Initially I replied, “No, I don’t want to social media stuff. I want to go back and do robotics.”
But two weeks later, I was replied, “Okay, I can’t stop thinking about this. It’s actually going to work, so I’ll come work for you.” Right around that same time they were getting a prototype of the app working. They hadn’t really announced that they were going to do an app yet. They were making a decision to pivot away from just R&D and to build an app. I got in during that early transition phase.
Was that in 2022?
This was all around November 2022. October was the event at the Internet Archive, and November was when I was hacking. I started work the first week of January. But before starting I went on a good bike trip.
At the same time as all of this, ActivityPub was starting to take off. Some time around that summer or fall, the Internet Archive set up a Mastodon instance, and I was pretty excited about that.
I had never really used Twitter. I had an account for a month, but on principle I was against using corporate social media since I was in high school or college. I’d never really used Twitter and always felt it had a bad impact on people around me.
But there was some appeal, so it was fun to use Mastodon. I got to experiment and experience that kind of social media. The library community is pretty online, and people have relatively healthy, productive conversations on social media around archives. It’s a good way to find out about events and new projects and gossip—all the classic stuff. So it was fun to start using it.
But there are some usability issues with Mastodon, which I still feel conflicted about. I do think they could probably be resolved over time. I do think that ActivityPub will last. It’s going to be like IRC—it will not grow and be mainstream, but it works for a number of communities quite well. The experience works for some folks.
Now you’re at a moment where Elon Musk has bought Twitter, you’re working for a project that had spun out of Twitter and was starting to become a competitor, and it had gone from building a protocol architecture to actually building a product. Talk about what you started getting involved in there. First of all, was your title Protocol Engineer from the beginning?
Yeah, it was. I originally applied for the only open job position, which was DevOps, kind of like being a sysadmin. I had done a lot of sysadmin and scaling stuff at the Internet Archive, and at the job before that I had been doing infrastructure stuff. It was my most marketable skill set, and the role I was most likely to be hired for. So that’s what I applied for. But they someone else for that at the same time as me, and I was put on protocol engineering.
What is it like to build a protocol in the context of a new product that’s growing and starting to be in the political spotlight?
It’s interesting. To be clear, many of the core ideas had already been worked out before I started. They had gone through a couple rounds of design iteration. So I can’t say that I came up with many of the ideas.
My first day, I remember very clearly. My first week was a team offsite which I flew out for, then sat down in the airport. When I landed, I had been granted access to all the internal documentation, which I hadn’t seen, so I was like, “All the secrets! Surely the plan is in here—the rest of the protocol, the federation parts.” Before I applied I had seen that it could all fit together, but the federation details were not public.
So I sat down in the airport reading for two hours, while the team called me: “Where are you? Why haven’t you shown up yet.” I was trying to find the details, but the design wasn’t done. There were still big parts missing. The main pieces were there, but a lot of details were not.
That time was very intense because there was a feeling of a historic window being open. Through my first two years, every time Twitter did something weird that drove everyone nuts, we’d think, “Surely this can’t be real. Elon Musk is weird and eccentric and seems like kind of a jerk, but surely he won’t drive Twitter into the wall repeatedly. He just spent so much money on it.”
We felt we had a short window to get the protocol out before Twitter changed direction again, hired a new CEO, and cleaned up the mess. There was a huge amount of time pressure—just insane pressure to get it all launched, get a product out that both worked and used the protocol, and to do moderation and safety and comms and everything. There was high pressure on all dimensions at the same time.
That was a very intense period, trying to get it launched. To some degree, we were just incredibly lucky and fortunate. By historical accident, because of external events, we had a huge amount of press attention. Anytime we wanted an interview, we just picked up the phone and someone would do the interview. People would give us their time and attention and give the app a chance. If we gave people invite codes, they’d come over, try it out, and write about it. People were endlessly patient—for the most part—about bugs or slowness or these other things. That is pretty incredible in the modern era.
Everyone on the team, and Jay in particular, has an awareness of modern markets and monopoly power and how the whole project could go wrong. She is very attuned to power dynamics between the government and companies and individuals, and extreme wealth. She wants us to build a company that will not enshittify and go wrong, which is hard. It’s hard to come up with a business model that cannot go wrong.
There was a lot of pressure to get these these conceptual long-term things right and to actually make it work at the same time. It was the right kind of pressure, but it was very intense to get everything done. Figuring out how to sequence things was painful and difficult, but also classic for an early startup. “What do we work on this week versus next week,” endlessly.
Can you articulate why the approach of designing around a protocol was so important? I remember that was part of the original idea of what Jack Dorsey wanted to create out of Twitter—to build a protocol around it so that the platform itself wouldn’t be responsible for everything.
To some degree, we are a protocol company. We talk about the protocol a lot. We’re all interested in protocols, we like protocol values. But the protocol for the sake of protocol doesn’t matter. It’s interoperation that matters—working with other projects.
Around the time I started, thinking was solidifying around a core thesis. We focused a lot on “credible exit”—credible exit being that users can walk away. You can’t have decentralization without that. You need that counter-pressure on the operators. There are many ways to define decentralization, and I don’t think credible exit on its own is sufficient for decentralization—they are related but different concepts.
Another phase we used a lot internally: “No one organization should moderate the entire internet.” We say that over and over again.
There were so many times when it would be tempting to just have one organization that operated some aspect of the network. But we would come back to the goals over and over again. We would iterate on architectures and ideas and shift things around. Does this design give us credible exit? Can other people build on top of it? Will it avoid one party moderating the entire internet? We came back to those questions a lot.
Everyone on the team was influenced by Mike Masnick’s essay, “Protocols, Not Platforms,” which was a founding document. The design challenge was to ensure that there was no piece of the system that could not be substituted. That’s a big thing for us—that’s still how we think about it. There are pieces of the protocol we haven’t finished yet, like private data, and we relentlessly ask: How will you be able substitute that? How can a small new company or team—not necessarily a hobbyist, but a group who is really trying to build infrastructure—have a relatively smooth on-ramp into the network?
A small team needs to be able to grow market share. You need to be able to come in and compete and start seeing success, or new providers will never join. The system has to be designed with antitrust in mind, and remove any points where you could squeeze and exploit too much.
Those are the goals, and we try not to violate those goals. There are a bunch of other ideals we haven’t delivered—community stuff, democracy. I’m interested in those. The team isn’t against them, but when we are up at 4 a.m. sweating, the anxiety isn’t “how do we make sure everyone has a vote?” It’s “how do we make sure that no one party is moderating the whole internet? How do we make sure you have credible exit and new people can come in and compete?”
Those are the core values, and the protocol is a means to those ends. How do you have credible exit? You need to be able to take your stuff and import it into another server, and it can’t be hard. It needs to be an easy thing to do. That’s why account migration ended up being such an important thing.
This has been a fun weekend for account migration—the Blacksky PDS is getting up and running, and more people are migrating. By total numbers, it’s not huge, but this is a moment when some of these ideas are being tested.
I’d love to hear about building a protocol while also building a product and trying to solve user problems. ActivityPub was designed in the context of the W3C, kind of in the abstract, and then Mastodon showed up and implemented some of it. But it was a protocol designed not necessarily with a single product in mind. You’ve been designing a protocol while also building a product and trying to solve user problems. What are the advantages and disadvantages of building a protocol that way, while the plane’s flying?
My understanding of the history is that ActivityPub came out of a succession of similar protocols and projects. There was a moment where there were three or four small but growing projects, and they were like, “We should interoperate better.” They were not all building the same thing, but they were all doing somewhat similar things. So they went to the W3C and said, “Let’s do a standard.”
They were trying to get existing projects to interoperate, and they were not people who had standards experience before. That is not normally a thing the W3C does, but somehow they were able to get a working group and make it happen. It was a painful process. I think most people involved have said the ActivityPub working group was a difficult process.
They were a bunch of projects trying to figure out a kind of lowest common denominator. Mastodon already existed and was one of the larger project, and during the W3C process Mastodon continued to grow. But they only updated Mastodon a bit to match ActivityPub. Mastodon doesn’t implement all the details. I’m very sympathetic, it’s a hard situation, and that they tried at all is really admirable.
The situation for Bluesky is very different. We are doing the “cathedral” thing. We have six people sitting together in a conference room. We are all aligned, we’re all working on the same thing. The core ideas came from three or four individuals, all of whom had worked on other projects and protocols and had thrown them away. We started with a blank slate.
The original idea was that Bluesky would build a protocol that Twitter could run on, at scale, but even from the beginning, everyone on the team was more ambitious. We want the protocol to be general purpose. We don’t want it just to be microblogging. It is a good first use-case, but we want it to work for other things.
What was it like doing the product and the protocol together? It worked in the ways you would want it to work. The people designing the protocol were the people who were getting paged on the weekend because the service wasn’t working. So they are really motivated to fix performance issues in the protocol. Enough of us were programmers that we were doing separate implementations and testing to make sure that the implementations all interoperated from the beginning. So the specifications stayed pretty close to implementation and implementation to specification. Not perfect, but we’ve kept it much tighter, and we were able to do that because we were all on the same team. It wasn’t like we were writing the protocol and then we tossed it over the wall to developers.
That is a value people have had in standards for a long time. The IETF embodies it. They have the slogan of “rough consensus and running code,” where the work is not done until someone has implemented it. I suspect that never happened with ActivityPub—no one had built an app that implemented the entire ActivityPub standard completely and correctly before it was ratified. When you do implementation work, it turns up issues. It’s like trying to write a recipe without making the meal.
As a tangential analogy: I worked on LIGO as an intern. LIGO is this giant gravitational observatory. They were working on an iteration of the entire whole system to upgrade the end mirrors. All the work was done in CAD, and they had designed this huge aluminum structure. One of my tasks as an undergrad was spending two or three weeks just putting it all together. They had the structure machined and were like, “We don’t know if a human can actually put it together,” because when you use CAD, there can be problems where you can’t actually get a bolt in to position or you can’t fit a tool.
So it’s worth having a human put together a prototype to make sure it all works. You have to do that with standards. You can’t just specify things and assume it’s going to work. You’ve got to actually implement it and make sure it fits together. Really, you need more than one person to do it.
For all the core parts of AT Proto, we’ve had at least two people implement it independently and make sure the implementations work together before we publish the specs. We had the resources to do it. A lot of that was having funding. We had time and money. People can still do this if they are not in the same company, but it sure helped to do it together.
In retrospect, the protocol-product design loop was tightest—with feedback from the user base or external people critiquing the protocol—back when we were not interoperating at all yet. We were still making major changes to the protocol. That was really only a six-month window.
We made a bunch of changes. The most famous in my mind was getting rid of repo history. By default, we were saving all the old revisions. So if you deleted a post, it would still be there, like in Git. We knew that we needed to be able to purge things, because of experience with Secure Scuttlebutt. So from the start we did build in a special operation that deleted history.
But what we were finding was—literally, there was a situation where a bunch of people were posting nudes, and then an elected congressperson was seeing at the nudes and talking about it. People were like, “Oh my god, I need all my nudes out of here immediately. When I delete the nudes, they need to be gone now, reliably.” Developers were starting to build a service that would purge history every time you deleted a post, and we’re like, “Okay, this is a mess. We need to get rid of history.” Clearly, when confronted with real users, it just wasn’t going to work. That was the biggest change.
There are other problems we still haven’t figured out a solution to. Having blocks be public and enumerable clearly drives everyone insane, and we would really like to get rid of that. We just haven’t quite found a way. We spent a lot of time thinking about that one.
I don’t know how to replicate the opportunity we had. We had fifty thousand, one hundred thousand extremely online, extremely experienced, very engaged, very positive and hopeful people giving us lots of feedback while we were building. I hated that we were not interoperating at the time, and I was super anxious that we weren’t ever going to interoperate properly. But it did let us make major changes to the protocol without having to worry that we were breaking other projects.
Today if we want to make a protocol change, it’s like, “How are we going to roll this out to the network?” We are still the authority. We haven’t completely turned over protocol authority yet, so we don’t have to have a public debate. If we decide we want to make a change, we do have to figure out how it’s going to roll out. We do increasingly need to get out of being the authority though.
But it sounds like there are advantages to retaining that control and to not having really widespread federation in the process when you’re designing and working through some of these issues.
There was a time when it was important for us to move fast and iterate. But the tradeoff is that the network can get too centralized. So there’s a question of when is having control is beneficial, and when is it not. It’s not the only way to do protocol work. I don’t think you need to do it this way.
It’s like Apple coming up with a protocol. You’re like, “Is this going to be the best for everyone? Or is it just good for that company? What were their motivations? What were they trying to do?” I would be very skeptical if Apple put out a protocol. The process we have taken is kind of like Apple coming up with a protocol. The Google Chrome team develops tons of web standards, but is that one team designing everything really in everybody’s best interest?
I think our process worked out—at least it has worked out to come up with the nugget of ideas. The protocol is compact. It clearly fits in one or two people’s brains. It works for one application quite well. We were able to really focus. Especially because the company was still a startup, there weren’t complicated internal politics or anything. No one was trying to get the numbers up this month or other things. We were just trying to make the protocol good. So we were able to do that.
I think the small team model is a good tool in the toolbox, and good option for designing protocols. The analogy I make is public research funding. How should do you give out funding? Should people be accountable and assessed every year? Every quarter? How long is the grant for? Two years? How long of a runway is right? If it’s too long, maybe you just slack off and never get anything done. Or if it is too short, you’re just constantly doing paperwork and going month to month.
There are people who talk about DARPA, or the original ARPA, and they say, “If you could get five people for five years, that’s just a real sweet spot.” For a modern academic, it’s like, “Five years? That’s a very long time! And five people? That’s a whole group!”
It is enough time to breathe. It’s not a whole career, but you can get a lot of work done. I feel like we kind of got that. Because of external events, we didn’t get the full five years. We had to do it in a year or so. But that model, I think, has a lot of appeal. I don’t think all research should be done this way, but this timeframe and size of a team is enough to do good, focused work.
What are you learning from other projects that are starting to use AT Proto to do things other than microblogging, or from people who are starting to build other services on top of Bluesky, like the personal data servers? What is surprising you about how other people are starting to take up the protocol?
The other people have been pretty great. We don’t really have any jerks in the ecosystem. I really enjoy talking to the other people building stuff with AT Proto, and that has not been my experience in free software in general. There are some great people in free software, but it can be pretty mixed overall. Even software projects I really love, often it’s like, “This is fun, but I do want to avoid…” It’s fun with a big asterisk.
But broadly, I can say that the AT Proto development community is just great vibes, and really smart, thoughtful people. I really like that.
I think one big distinction from the ActivityPub community is in being “open for business.” We are intentionally open to people building businesses. There are some startups building on the protocol, and that’s good. There are still open questions about how all the funding will work, how monetization will work. But we are open about it, and we want to figure it out. We want people to experiment with money. Though there are people who are strictly hobbyists as well.
People are coming up with a bunch of technical tricks. It’s really impressive what some people have been able to cram on to teeny devices. Using exotic database engines or weird indexing solutions—I’ve learned some new things about databases I had never seen before. So there’s definitely some technical learning from the ecosystem.
Everything is communications. For the people who have made attempts at moderation or community-building projects, communications is everything. You have to build trust, and whatever the internal processes are or whatever safeguards you build, all that matters is communications.
I’ve seen so many projects flame out and get dragged down by their communities. While other projects have been more successful and have community support and legitimacy. I want to say it’s because they have voting or some form of direct accountability like that, but really so much of it is just being out there and being personally accountable, and apologizing and owning failures. Classic community leadership things.
Especially for social media, what’s most important for success is having leadership and processes, and responding to people out in the open.
Where do those conversations take place? Is it through posts on Bluesky, or are there dedicated spaces where people are debating potential changes to AT Proto?
Mostly posts on Bluesky.
It has been interesting to see who feels empowered to make bold proposals about parts of the protocol and who does not. For a long time we were quite worried that people would rip the protocol out of our hands, or use it in ways that we thought were bad. That made us hesitant at times.
Maybe we did the right thing and it was important to thread the needle the way we did. Or maybe we had too much anxiety and could have let people pull it out of our hands sooner. I don’t think our approach was a big mistake either way.
Where are you now in the process of moving that protocol, the governance of that protocol, outside of the organization, outside of Bluesky?
There are two parts to that. Your question is how are we moving governance of the design of the protocol out of Bluesky. The other part is material decentralization: who else is powerful in the ecosystem? The two parts are entwined—maybe a chicken and the egg thing. We have to do both.
Other people are taking big steps, setting up independent hosting and building other apps, and that helps. To some degree that was a blocker for creating a standard through the IETF—they want to see other implementations by other parties. So we couldn’t move forward on governance until there were other projects, or it didn’t really make sense to try. But on the other hand, not a lot of people want to take the leap of investing time and energy in a protocol if there isn’t clear governance. So the two parts have to go together. Recently we have been focusing on encouraging projects getting built, and being responsive to the frictions that people identify when they do.
I just traveled to an IETF event a couple weeks ago. There is a long history of projects that get taken to the IETF and succeed or fail. We are trying to find the path that will make sense for the IETF folks. They would be investing a fair amount of time in facilitation for ATproto. We want to make sure we are doing the process right. So there is diplomacy there, and it’s a slow process. I think it is going well, and we will be taking more steps in this direction.
But bringing the protocol to the IETF raises new questions. Who gets to participate in governance there? At a minimum, there ends up being a public mailing list and people can bring their concerns. There is an expectation of responses to points raised on the mailing list. It’s a little more formal. It’s different from people complaining in the app, who often have very good points. The set of people who are going to participate in these venues is pretty different.
The goal is to bring the protocol to an existing venue and have Bluesky-the-company transition from being the sole authority to being just a participant with opinions. Because the team does have strong opinions, and we want to be able to express our opinions to peer participants—have a debate, basically. If we set up our own venue and inviting people in, we would have to play the host, and we wouldn’t get to debate. That’s some of our thinking.
We tried to design the protocol so that it does not result in a single community structure. Instead you build multiple communities on top. We do have an ecosystem of many developers working on the protocol itself. But if you need to be a protocol engineer to change how the social structure of the web works, that’s not a great outcome. The IETF should not be the venue for content moderation decisions or a lot of the important questions of how communities function.
We tried to design a infrastructure layer of the protocol that has a couple opinionated values baked in—like credible exit—and then we want to have communities build on top in a way that Bluesky can’t exploit them.
I imagine that the protocol governance will matter less over time. When you take a protocol to the IETF, there’s a period of debate, there are questions that get resolved, and then it gets frozen. So it’s not such a living thing after a certain point.
Say a bit more about your experience hearing about people’s feedback using the app. Bluesky gone through, as any social network has, some crises around moderation and disagreements over what a social network should look and feel like. On the one hand, you have to address those. On the other hand, you have this longer-term vision—that people will build their own moderation systems. But that answer is, in the short term, not satisfying to a frustrated user. Is there something that you hope users in the future, in a more protocolized social network landscape, might think differently about? Are there ways in which we should be shifting our expectations for how these networks should function?
It can be tempting to respond to folks, “Open a pull request, or fork the code,” or something like that. I think that is just being dismissive. Because it’s usually being addressed to an individual who doesn’t necessarily know how to do all those things, and doesn’t have venture capital funding to do it. That is why I want there to be many institutions involved, of different types—some are going to be startup companies, some are going to be tech companies, but I hope some of them will be small community groups or civic groups. It’s groups with funding, who have figured out how to have a bank account. I want more of that.
There are a lot of existing groups and organizations in society that could be participating online in healthier ways. That’s my idealistic, optimistic goal—to make it possible for more people to participate through organized groups. It’s not necessarily going to be an individualist, libertarian thing. Groups, small organizations—that’s my hope.
I think there’s an interesting learning curve. In the context of my university, when they were coming on to Bluesky, I was working with the social media team as they were doing that. The idea of moving from a bsky.social handle to our university’s domain as a handle was a difficult unlock. They said, “We want an account on the platform, and you’re telling us we have to set up our domain name as a handle?” The idea that they were one node in a network was very foreign, as opposed to just relying on somebody else’s infrastructure.
I would love for institutions to manage their social networks more. The trend is in the other direction, unfortunately. It wasn’t that long ago that many universities ran their own email servers, but today they mostly do not. It’s not that the email protocol changed, or that it got bought out. It is just easier to pay someone else to do it.
I think Bluesky set a reasonable baseline for the network. If it was just a hellscape when you first logged on, then no one would want have anything to do with the protocol. The stakes for moderation and accountability and legitimacy were quite high from the beginning, so we had to get those right. We have ever-growing and improving moderation and trust-and-safety resources. More recently we have been able to catch our breath on a lot of fronts, finally, and start to raise the bar and do better.
The rest of the industry has just walked away from talking about trust and safety, so Bluesky has a growing role in that area. But on the flip side, I hope that the importance of our moderation work to the network decreases over time. We have shown that moderation can be done in the network, and it’s not a hellscape. That doing an open protocol doesn’t mean you’re going to be awash with spam.
To be clear, we’ve shown it as a dominant company. The question of whether we can do this with no one company controlling more than ten percent of the network—that remains to be seen. But I think the decision to moderate made the network appealing and approachable enough for other people to get involved. I think that was an important phase to get through.
If we had been really stridently at the start, like “We’re not going to moderate the network at all,” we would never have gotten as far. It has been tempting at a number of points to respond, “If you don’t like the moderation, do it yourself.”
One take is that we should have done moderation even worse. We could have been super evil to encourage people to move off our servers and build their own infrastructure. But I don’t think that would have worked. I think we needed to do a decent job with moderation, and I think we have.
It is tempting to think, “Can we take the weekend off?” When I was in elementary school, the school held a fundraiser, and one of the auction items was “your child can be principal for the day.” When you’re a kid, you think, “Wow, they’re really letting a kid be principal?” I think about that sometimes… what did it feel like for the principal. “Alright, you think it’s so easy?”
What lessons would you share with people building protocols—someone starting from scratch, or someone who’s taking over your job?
I don’t know! A lot of the Bluesky story has been driven by historic events. It’s just been really weird. That is awkward because I would like to be a role model and have a process that is reproducible, but a lot of our story hasn’t gone that way.
I think it important not be totalitarian about protocols. A lot of designers have very grand ambitions. But draw a box around your project and try to make it as small a box as possible. Start by contributing to existing projects and find allies that you can work with, so you’re not alone. Don’t make it all about just your project. Try to make it so your new thing fits into an ecosystem, and spend time making attachment and integration points actually work.
I don’t know if we’ve done that super well, but we tried. Bluesky is not trying to reinvent how TCP/IP works. Sometimes early projects tend to be over-ambitious or take on too much.
The other thing is what Jay did so well early on: go help other people first. Go and learn from them. If you’re a coder, go fix a couple issues in other people’s projects. Even simply install their thing. Set up an instance. Even if you think you’re not going to like it, go participate in what other people have done first and give it a try. You’ll at least get the flavor of it, and a feel for what you could do better. You’re building social capital when you talk to other people about their projects. I think that is important.
As a last thing, you either need to be in very tight contact with a diverse set of users, or you’ve got to be a really active user yourself of whatever you are building. Especially if you have a small group or a small team. Trying to fix a problem that you don’t care about or that you don’t have intimate experience with does not usually work out well.