← Blog

Why we stopped running our own WebSocket servers

October 26, 2026 · Hibira

Our own realtime server jammed when messages spiked, and we chased patches for vulnerabilities like PHP2Shell ourselves. What we ran into, the ongoing work a self-hosted server needs, and what a hosted service takes on.

Before Hibira, we ran our own WebSocket server on past projects: an off-the-shelf open-source PHP server, run as a standalone service on a single node. Realtime was a small part of the system, so as long as it worked, nobody paid it much attention. Then message volume grew, and it started falling behind: the server was busy with the messages it already had, new requests queued up behind them, and latency kept climbing. By the time a message reached the user, it was already out of date. The price had moved, the score had changed, and the "typing…" indicator showed up after the other person had already sent their message.

This post is what we took away from that: why a realtime server can jam at peak load, what you have to add if you run one yourself, and when handing it to a hosted service is worth it. Looking back, open source wasn't the problem; we had made sure it worked, but never planned for monitoring, scaling and maintenance. Self-hosting can be done well; mature open-source servers exist. What it takes is ongoing work, and that work is what this post is about.

How the jam happens

What a realtime server does looks simple: take an event and send it to every subscribed connection. The trouble is that the work gets multiplied, and as soon as one part slows down, everything behind it waits.

Arriving faster than processed: the queue only grows

This is the basic rule of queues: as long as messages arrive faster than the server can process them, the queue keeps growing, and every new message waits for all the ones before it.

Put some numbers on it. Say the server can write 5,000 messages a second to clients (that counts copies actually sent, not publishes; the next section explains the difference), and during a burst it has 6,000 a second to send for 10 seconds. Those 10 seconds leave 10,000 messages unprocessed. Assuming the burst ends and new messages stop entirely, the server still needs another 2 seconds to clear them, so the last message in the queue goes out 2 seconds late. If normal traffic keeps arriving after the burst, it clears even more slowly.

On a web page, 2 seconds late is a worse experience. In a realtime feature, a price, position or game state from 2 seconds ago is often simply wrong.

One event, times every subscriber

Publishing once doesn't mean sending once. On a channel with 2,000 subscribers, every event you publish means 2,000 writes. The real workload is publishes per second × subscribers: 6,000 publishes a second to 2,000 subscribers each would be up to 12 million writes a second. One-to-many cases such as live chat rooms, match scores and operations dashboards are the ones most likely to tip over at the moment the audience grows.

Waiting spreads

Once the server is busy, the HTTP requests that publish events slow down too. Your backend waits for them, so its own requests pile up. If timed-out requests are retried automatically, each retry adds more work to a server that is already overloaded. What started as a slow realtime server ends up as a slow system.

Scaling out was the hard part

Our WebSocket server ran on its own, deployed apart from the application rather than on the same machines. What held us back was scaling. One server can only process so much, and adding a second one means every machine has to share channel subscriptions and presence, so that an event reaches users connected to the other server, with a load balancer in front to spread the connections. Ours was a single node with none of that in place, so there was no way to scale out, and that one server had to take the whole peak. And because there was only one node, every update or restart dropped every WebSocket connection at once, and they all reconnected together.

We couldn't see it until it broke

The other problem was visibility. We ran the server ourselves with no dashboards and no alerts: we couldn't see connection counts, message throughput or server load in real time, let alone queue length, message latency or slowed connections, so we never had a clear picture of how the server was really doing. CPU looking fine didn't mean messages were on time. More often than not we only found out once users were affected and it had dragged our main application down with it: publish requests got stuck, the main app waited on them and slowed down too, just as described in "Waiting spreads" above.

The other reason: security updates for the runtime

Besides performance, the other reason we stopped running our own was security. We also ran into PHP security vulnerabilities back then and had to chase the patches ourselves. When you run your own servers, every vulnerability in the operating system, the language runtime and the libraries is yours to track, and none can slip.

A recent example is the pair of critical PHP vulnerabilities known as "PHP2Shell", fixed by the PHP project in October 2026 (Canadian Centre for Cyber Security advisory AL26-026):

• CVE-2026-109618: memory corruption while handling out-of-memory (OOM) conditions.
• CVE-2026-109619: a use-after-free caused by state left over between requests.

Both score 9.2 on CVSS v4.0 and can let an unauthenticated attacker execute code remotely. They affect PHP serving web requests, such as PHP-FPM, Apache mod_php and php-cgi. The fixed versions are 8.2.35, 8.3.36, 8.4.27 and 8.5.12; 8.1 and older no longer get routine official updates.

To be clear, these two are in PHP serving web requests, so they don't necessarily affect a long-running WebSocket process at all. We cite them as an example of the routine work, not of a WebSocket flaw: whatever language your server is written in, the moment a vulnerability in your stack is public, you have to work out whether you're affected, schedule the upgrade and redeploy, as fast as you can.

If you run it yourself, add these

Each of these problems has a fix, but every fix is ongoing work, not a setting you change once:

• Deploy it apart from your application. Let the realtime server scale and deploy on its own, so a peak can't take the rest of your app down with it.
• A bounded queue per connection. Writes must not wait on each other. Each connection gets its own queue; when it fills up, that client can't keep up and should be disconnected rather than allowed to slow everyone else.
• Separate receiving from sending. The publish API shouldn't wait until every subscriber has the message before it answers, or a slow broadcast slows your backend too.
• Horizontal scaling. A single server always has a ceiling. Adding machines means every machine has to share channels and presence, with a load balancer in front.
• Plan for reconnects and lost messages. Clients reconnect after a drop, a deploy or a network change. Reconnects need back-off so they don't all arrive at once, and events published while a client was away are not replayed, so the app has to fetch the current state again after reconnecting.
• Test capacity against peaks. A load test at average traffic tells you everything is fine. Test several times your peak, and many clients reconnecting at once.
• Monitor queues and latency, not just CPU. The queue can be growing while CPU still looks fine. You need to see how long messages wait and how many connections are dropped for being too slow, and get alerted when it gets worse.
• Someone on call. The peak is usually your most important moment: a ticket sale opening, the last minute of a match, a promotion going live. Someone has to be there when the server has trouble.
• Keep up with security updates. When a vulnerability in the operating system, language runtime or libraries goes public, assess it, upgrade and redeploy.

What a hosted service takes on for you

With a hosted realtime service, the provider does the work above. Taking Hibira as the example:

• Slow clients don't slow anyone else. Each connection has its own bounded queue, and writes never wait on each other. A client whose queue fills up or whose writes time out is closed with the Pusher protocol's 4100 error code, which tells pusher-js to wait a moment and reconnect. The disconnect protects everyone else; it doesn't make that device's network faster, so a client that is still slow can be dropped again.
• Your app restores state after a reconnect. The Pusher protocol does not replay events a client missed while it was disconnected. On paid plans you can turn on channel history in the app settings, and a reconnecting client can then ask for the events of the last few minutes with pusher:rewind. For prices, game state or anything else that must be right, treat events as updates and fetch the current state from your backend after connecting; pusher-js and Laravel Echo tell you when the connection is back.
• You can see what happened. These disconnects show up on the app's errors card as "Slow client disconnected", with the last 24 hours, on every plan including Free, so you don't have to dig through server logs.
• We plan the capacity. The servers run as a cluster that shares channels and presence, behind a load balancer. It adds servers automatically on CPU, memory and the busiest server's connection count. A new server takes a few minutes to start taking connections, so scaling handles sustained growth, not an instant spike; until then a full server turns new connections away with 4100 and the clients retry on another one. Existing connections stay where they are. That is our side of the infrastructure: your usage is still bounded by your plan's hard limits, so there are no surprises on the bill, and the uptime SLA comes with the Enterprise plan.
• It speaks the Pusher protocol. Your existing pusher-js, Laravel Echo and server libraries work as they are, so switching usually needs no rewrite of your app logic: you change the host and credentials, recreate your webhooks, and check the features and limits you rely on. The migration guide lists what is supported and how to switch without downtime.
• We do the patching. Security updates for the WebSocket servers and the runtime under them are ours to handle. You still update your own app, but there is one less thing to watch.

The costs should be said plainly too: past the free tier you pay monthly, your events pass through a third party, and you live with the plan's limits.

When running your own still makes sense

A hosted service isn't the only answer. Running your own may suit you better when:

• Your traffic is small and steady, without sharp peaks.
• Event contents can't leave your infrastructure.
• You already have an operations team, monitoring and an on-call rotation, so one more long-running service is no burden.

When you compare costs, count the whole bill: self-hosting is the servers plus the engineering time spent on monitoring, scaling, patching and on-call; a hosted service is a monthly fee based on usage. With small, steady traffic and someone already looking after it, running your own is often cheaper. When you have to scale for peaks without dedicated ops, the hours usually cost more than the servers.

Using Laravel? Running Reverb yourself versus a hosted service deserves a post of its own, and that's coming next, so stay tuned. Both speak the Pusher protocol, so changing your mind later is only a configuration change.

Our takeaway

Mature hosted services already existed, and adopting one might have been simpler. We rebuilt it ourselves anyway: we wanted to really understand how WebSocket and distributed realtime messaging work underneath, and to measure the performance, resource use and running cost of different architectures, which also showed us how services like this are priced.

We rebuilt realtime messaging as a service that scales out. A load balancer sits in front, each connection has its own bounded queue, and servers are added automatically as load grows, so traffic is spread across more servers instead of one server straining to keep up, with monitoring and alerts on connections, message volume and latency. We used it internally first, then opened it up to everyone else. That is Hibira.

A self-hosted WebSocket server can run very well. The hard part isn't getting it to run; it's the ongoing work of backpressure, capacity planning, isolation, monitoring and security updates that keeps it on time during the busiest few minutes, which are exactly when users care most and when things are most likely to break. Hibira takes on that ongoing work as a hosted service, so your team can spend its time on the product.

Hibira's free plan needs no credit card and is enough for development and small apps. Already on Pusher? You usually only change the connection settings, not your app logic; the migration guide has the steps, and we'll share our own migration story in an upcoming post, so stay tuned.