Keep a Discord bot online through crashes and deploys
By Arthur De WittePublished Updated 13 min read
On this page
- Why sharding alone isn’t enough
- How sharding works
- Three ways to run the shards
- The architecture: a thin gateway, a stream, and workers
- What this looks like in production
- Running the gateway on Kubernetes
- Running it on Docker Swarm
- Setting up the streams and consumers
- When something fails
- What to watch
- Test your reading
- Doing this on A-Line.cloud
- Frequently asked questions
- If you change one thing
Your bot works fine until you deploy. Slash commands stop responding for a minute, a few messages never get moderated, and a user asks in the support channel whether the bot is down. Most guides about scaling Discord bots stop at sharding. Sharding spreads the load, but on its own it doesn’t keep anything online. This guide covers the other parts: a gateway that only forwards events, a durable queue that holds them, and workers that can crash or restart without anything getting lost.
Why sharding alone isn’t enough
Discord requires sharding once a bot is in 2,500 servers, which Discord’s API calls guilds. Each shard is a separate connection to Discord’s gateway, and each connection may send 120 events per 60 seconds (Discord docs). So sharding is about staying within Discord’s limits, and it does that well.
What it doesn’t do is keep the bot working while a process restarts. If the code that receives an event is also the code that handles it, every deploy and every crash drops whatever arrives during the restart. A bot that stays up needs three things working together: shards to spread the connections, an orchestrator to restart what fails, and a durable queue so that no event depends on one process being alive.
How sharding works
Discord decides which shard receives a guild’s events with one formula (Discord docs):
shard_id = (guild_id >> 22) % shard_count
Every guild ID carries a timestamp in its upper bits. Shifting it 22 bits to the right and taking the remainder spreads guilds evenly, at roughly 2,500 per shard at most. Each shard keeps its own WebSocket, its own heartbeat, its own session ID and its own sequence number, so one shard can drop and reconnect while the others carry on.
You don’t have to guess the number of shards. The /gateway/bot endpoint returns a recommended shards count, along with max_concurrency: how many shards may identify at the same time. For almost every bot that number is 1, which means one new connection every 5 seconds. Only bots in more than 250,000 guilds get a higher value.
That 5-second spacing matters later. Starting 10 shards from scratch takes close to a minute, so the part of your bot that holds these connections should restart as rarely as possible.
Three ways to run the shards
Libraries offer three ways to place shards in processes. They differ in memory, in how much can fail at once, and in how quickly they recover.
| Strategy | How it runs | Complexity | When one part fails |
|---|---|---|---|
| Internal | One process holds every connection | Low | Everything stops until it restarts |
| External | One process per shard | Medium | One shard stops; the rest keep going |
| Hybrid | Several shards per process, several processes | Medium | Only the shards in that process stop |
- Internal200 MB
- Hybrid400 MB
- External800 MB
Show the numbers as a table
| Strategy | Memory (approx.) |
|---|---|
| Internal | 200 MB |
| Hybrid | 400 MB |
| External | 800 MB |
Internal sharding uses the least memory, but the whole bot is one process: one crash takes every shard down. External sharding isolates failures, at the cost of loading your code and caches once per shard. Hybrid sits in between, and for most bots past a few thousand guilds it’s the sensible choice. The discord-hybrid-sharding project reports going from about 4 GB to about 700 MB by switching, and has been run with up to 600,000 guilds.
The architecture: a thin gateway, a stream, and workers
The shape that keeps a bot available has three layers, and each layer has one job.
The gateway holds the connections to Discord. It receives events and publishes each one to the stream. It contains no business logic: no database calls, no moderation rules, no image processing. Because identifying is rate-limited, the gateway is the slowest part to restart, so you want it to change rarely and crash never. Little code means little to go wrong.
The stream sits in the middle. NATS JetStream stores every event on disk until a worker confirms it has handled it. If no worker is listening, events wait.
The workers do the actual work. They don’t keep state of their own between events, so you can run as many copies as you need, restart them in any order and deploy them as often as you like.
Press the button to restart the worker while events keep coming in.
Why JetStream
The stream has to keep messages when nobody is listening, and that rules out the fastest options. NATS Core and Redis Pub/Sub are fire-and-forget: a message sent while no worker is connected is gone. Among the brokers that do keep messages, JetStream has the lowest latency in one 2025 benchmark.
Keeps messages when no worker is listening
- JetStream3 ms
- RabbitMQ12.5 ms
- Kafka30 ms
Drops them
- NATS Core0.3 ms
- Redis Pub/Sub1.25 ms
Show the numbers as a table
| Broker | Latency | Throughput |
|---|---|---|
| JetStream | 3 ms | about 300,000 msg/s |
| RabbitMQ | 12.5 ms | about 75,000 msg/s |
| Kafka | 30 ms | about 750,000 msg/s |
| NATS Core | 0.3 ms | about 11,000,000 msg/s |
| Redis Pub/Sub | 1.25 ms | about 500,000 msg/s |
Kafka handles more messages per second, but no Discord bot comes near JetStream’s ceiling, and Kafka brings partitions, consumer group rebalancing and a KRaft quorum to run. JetStream is a single small binary that a bot with a handful of shards can run next to everything else.
What this looks like in production
Hylist.io, a server list I run, uses this setup. The bot is written in Rust on Tokio and runs as 2 replicas holding 4 shards on Docker Swarm. JetStream carries 10 streams that feed more than 11 kinds of worker, on subjects such as votes.cast, servers.check and tasks.media.
A few details that turned out to matter:
- Shards are claimed through Postgres. Each replica writes a heartbeat to a
discord_bot_shardstable every 10 seconds. A row without a heartbeat for 30 seconds, three missed beats, is free to claim.pg_try_advisory_xact_lockmakes sure two replicas never claim the same shard, and on shutdownrelease_all_shards()hands them back at once instead of waiting out the 30 seconds. - Work is split by kind. The TASKS stream has
tasks.process,tasks.mediaandtasks.dead, with separate consumers for CPU-heavy and I/O-heavy work, so a pile of video transcodes never delays a vote. The SERVERS stream has 10 consumer replicas, because checking servers is both frequent and time-sensitive. - Every worker has a health port, from 9001 to 9011, and Swarm checks it every 10 seconds.
- Logging never blocks. Errors go to Sentry at a 20% sample rate, and logs go through Rust’s
tracingwith a non-blocking writer. A log call that blocks would stall the async runtime it runs on.
Running the gateway on Kubernetes
On Kubernetes, the gateway fits a StatefulSet. Its pods get stable names, discord-shard-0, discord-shard-1 and so on, and that number can serve as the shard ID.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: discord-shard
namespace: discord-bot
spec:
serviceName: discord-shard
replicas: 3
selector:
matchLabels:
app: discord-shard
template:
metadata:
labels:
app: discord-shard
spec:
terminationGracePeriodSeconds: 30
containers:
- name: shard
image: registry.example.com/discord-bot:1.4.2
env:
- name: SHARD_ID
valueFrom:
fieldRef:
fieldPath: metadata.labels['apps.kubernetes.io/pod-index']
- name: SHARD_COUNT
value: "3"
- name: NATS_URL
value: nats://nats.discord-bot.svc.cluster.local:4222
- name: DISCORD_TOKEN
valueFrom:
secretKeyRef:
name: discord-secrets
key: bot-token
ports:
- name: health
containerPort: 8080
readinessProbe:
httpGet: { path: /health/ready, port: health }
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet: { path: /health/live, port: health }
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 5
resources:
requests: { memory: 256Mi, cpu: 250m }
limits: { memory: 512Mi }
The parts that do the work:
apps.kubernetes.io/pod-indexis a label Kubernetes puts on every StatefulSet pod since version 1.28. Reading it intoSHARD_IDmeans no pod needs to work out its own number.- Pods start one at a time. A StatefulSet waits for each pod to be ready before starting the next, so if
/health/readyonly answers once the shard has identified, the 5-second spacing Discord asks for happens by itself. - Updates go one pod at a time as well, highest number first. The restarting shard resumes its session while the other two keep running.
terminationGracePeriodSecondshas to be longer than your shutdown takes. More on that under failover.
Changing the number of shards
SHARD_COUNT is part of the formula, so changing it moves guilds between shards, and every shard has to reconnect with the new count. Tools such as Marver and Kubecord help coordinate that.
Don’t let an autoscaler do it. Ten new shards take close to a minute of identifying, during which some guilds get no events at all. Check the recommended count from /gateway/bot once a week and change it by hand, at a quiet hour. Workers are different: scale them as often as you like, because they hold no connection to Discord.
Running it on Docker Swarm
Swarm has no StatefulSets, so shards need another way to decide who owns what: Postgres advisory locks as at Hylist.io, a NATS key-value bucket, or a Redis lease. In return you get a much smaller system to run. Mirantis has committed to supporting Swarm through 2030 (The Decipherist). As a rough line: under 20 shards, Swarm is plenty; past 50, Kubernetes pays for its extra work.
# docker-compose.swarm.yml
services:
discord-bot:
image: registry.example.com/discord-bot:1.4.2
environment:
NATS_URL: nats://nats:4222
DATABASE_URL: postgres://bot:${PG_PASSWORD}@postgres:5432/bot
secrets: [discord_token]
stop_grace_period: 30s
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9001/health"]
interval: 10s
timeout: 5s
retries: 3
deploy:
replicas: 2
update_config:
parallelism: 1
delay: 30s
order: stop-first
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
networks: [bot-internal]
nats:
image: nats:2.11
command: ["--jetstream", "--store_dir", "/data"]
volumes: [nats-data:/data]
deploy:
resources:
limits: { memory: 128M }
networks: [bot-internal]
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: bot
POSTGRES_PASSWORD: ${PG_PASSWORD}
volumes: [pg-data:/var/lib/postgresql/data]
deploy:
placement:
constraints: [node.labels.db == true]
networks: [bot-internal]
networks:
bot-internal:
driver: overlay
volumes:
nats-data:
pg-data:
secrets:
discord_token:
external: true
Some choices worth explaining:
delay: 30swithparallelism: 1gives each replica time to claim its shards and resume before the next one goes down.order: stop-firststops the old replica before starting the new one, so two replicas never hold the same shard.- Pinned versions.
nats:2.11andpostgres:16-alpineinstead oflatest, so a redeploy never upgrades your queue by surprise. - Postgres is pinned to one node. A Swarm volume lives on the node that created it, so a database that moves to another node starts empty.
- The password comes from the environment and the Discord token from a Swarm secret, so neither ends up in the file.
NATS itself is light: the image is under 10 MB, and it runs on 1 vCPU and 32 MiB of memory at the minimum (NATS docs).
Setting up the streams and consumers
With the nats command-line tool, one stream holds every Discord event:
nats stream add DISCORD_EVENTS \
--subjects "discord.events.>" \
--storage file \
--retention limits \
--max-msgs=-1 \
--max-bytes 1GB \
--max-age 24h \
--discard old \
--replicas 1
--replicas 1 is fine on one server. On a cluster of three NATS servers, use 3, so the stream survives losing one of them. --max-age 24h and --max-bytes 1GB put a ceiling on how much can pile up if every worker is down for a long time.
Each kind of work then gets its own consumer, which keeps track of what it has handled:
nats consumer add DISCORD_EVENTS moderation-worker \
--filter "discord.events.message_create" \
--deliver all \
--ack explicit \
--max-deliver 3 \
--wait 30s \
--pull \
--max-pending 100
nats consumer add DISCORD_EVENTS voice-tracker \
--filter "discord.events.voice_state_update" \
--deliver all \
--ack explicit \
--max-deliver 5 \
--wait 10s \
--pull \
--max-pending 50
The ack wait (--wait) is the setting to think about. It is how long JetStream waits for a worker to confirm a message before handing it to someone else. Set it too short and slow handlers get duplicates while they’re still working. Set it too long and a crashed worker’s messages sit untouched for that whole time. Start a little above the slowest normal run of your handler.
When something fails
Recovery happens in two layers, and each covers a different kind of failure.
The connection to Discord. When a shard loses its connection, it reconnects and sends RESUME with its session ID and the last sequence number it saw. Discord replays the events it missed. If the session has expired, the shard falls back to a fresh IDENTIFY. Discord doesn’t document exactly how long a session stays resumable, so treat any gap longer than a few seconds as one where events may be lost, and resync what your bot depends on.
The work itself. A worker that crashes mid-task never confirms its message, so JetStream redelivers it once the ack wait runs out. A durable consumer picks up from the first unconfirmed message after a restart. --max-deliver stops a message that crashes the worker every time from looping forever.
Around both, the orchestrator does its part: Kubernetes probes or Swarm health checks restart what stops answering.
Shutting down cleanly
Most lost events happen during deploys, not crashes. When the orchestrator sends SIGTERM, stop taking new events from Discord first, then drain NATS so that everything already received gets published and everything in hand gets finished. The other way round, events that arrive between the two steps have nowhere to go.
import asyncio
import signal
class GracefulShutdown:
def __init__(self, bot, nats):
self.bot = bot
self.nats = nats
self.shutting_down = False
def register(self):
loop = asyncio.get_running_loop()
for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, lambda: asyncio.create_task(self.shutdown()))
async def shutdown(self):
if self.shutting_down:
return
self.shutting_down = True
# Stop receiving from Discord, then publish and finish what is in hand.
await self.bot.close()
await self.nats.drain()
terminationGracePeriodSeconds on Kubernetes, or stop_grace_period on Swarm, must be longer than this takes. Otherwise the process is killed halfway through draining.
What to watch
Three groups of numbers tell you most of what you need to know.
Per shard: the heartbeat ACK time, normally well under 500 ms; the number of guilds, which should be roughly equal across shards; and events per minute. A shard whose events suddenly drop to zero while its connection looks fine is stuck.
Per consumer: pending messages, ack waits that ran out, and the share of messages delivered more than once.
Per machine: memory and CPU, with alerts on trends instead of single values. A leak of 1 MB an hour takes about three weeks to fill a 512 MB limit, and a trend alert catches it in the first days.
| Metric | Warning | Critical |
|---|---|---|
| Heartbeat ACK time | over 1,000 ms | over 5,000 ms |
| Pending messages | over 1,000 | over 10,000 |
| Redelivered | over 5% | over 15% |
| Time since the last event | over 30 s | over 120 s |
| Memory | over 80% of the limit | over 95% |
Start with five to ten alerts. Every alert that fires without anyone needing to act teaches the team to ignore the next one.
Test your reading
Three dashboards. Decide for yourself whether each one shows a real problem, then open the answer.
The queue that keeps growing
stream DISCORD_EVENTS consumer moderation-worker pending 4,012 (10 minutes ago: 380) redelivered 0.4% workers 3 running, 3 ready
Is this a real problem?
Yes. Pending grew more than tenfold in ten minutes, while every worker reports healthy.
The workers are up but can’t keep up. Few redeliveries means they aren’t crashing or timing out; they’re just too slow for the current load, often because of one slow step such as a database call. Add workers to work through the backlog, then find the slow step.
One slower heartbeat
shard guilds heartbeat ACK events/min 0 2,418 92 ms 3,910 1 2,377 88 ms 3,850 2 2,402 181 ms 3,880 3 2,391 90 ms 3,870
Is this a real problem?
No. Shard 2's heartbeat is twice as slow as the others, but far below the 1,000 ms warning, and its events keep flowing.
Heartbeat times vary with the route to Discord and the load on the machine. What matters is that shard 2 handles as many events as the others. Keep an eye on it if it keeps climbing.
The steady redeliveries
consumer voice-tracker pending 42 redelivered 18% (last hour) ack wait 10s handler p95 9.4s
Is this a real problem?
Yes. Almost one in five messages is delivered twice, because the slowest handler runs come close to the 10-second ack wait.
JetStream assumes the worker has died and hands the message to another one while the first is still busy. Raise the ack wait above the slowest normal run, or make the handler faster. Either way, make sure handling the same event twice does no harm.
Doing this on A-Line.cloud
Everything above is work you can do yourself, and on a few servers it’s a reasonable weekend project. If you’d rather not run the platform underneath, A-Line.cloud runs the same pattern for you:
- Managed NATS with JetStream, so the stream in the middle is there without running it yourself.
- Workers that scale on pending messages through KEDA: more workers when the queue grows, fewer when it’s empty.
- Rolling deploys that only continue once the new version is ready, and roll back when a deploy fails.
- Builds from GitHub and certificates set up for you, with no Kubernetes YAML to write.
A-Line.cloud is invite-only for now. If you run a bot like this and want to move it, request access and tell us what it does today.
Frequently asked questions
How many shards does my bot need?
Ask Discord: the /gateway/bot endpoint returns a recommended number. Below 2,500 guilds one shard works; above it, Discord requires sharding. Many teams start with a little more than recommended, because changing the count later means reconnecting every shard.
Do I need Kubernetes for this?
No. For a bot with up to about 20 shards, Docker Swarm with a Postgres table for shard ownership does the job, and it is far less to learn and run. Kubernetes starts paying off when you run dozens of shards and many kinds of worker, or when you already run it for other things.
Why not use Redis Pub/Sub as the queue?
Redis Pub/Sub delivers a message only to whoever is listening at that moment. If a worker is restarting, its messages are gone. Redis Streams do keep messages and support acknowledgements, so if you already run Redis, they are a reasonable alternative to JetStream.
Why do slash commands fail during a deploy?
Discord gives you 3 seconds to answer an interaction. If the worker that handles it is restarting, that time runs out and the user sees “The application did not respond”. Keep at least one worker for interactions ready during a deploy, or send a deferred response straight away and do the work afterwards: a deferred response gives you up to 15 minutes.
Does RESUME give back every event I missed?
Only while the session is still valid. Discord replays the events after the last sequence number your shard saw. If the session has expired, the shard identifies again and the events in between are lost. After a fresh identify, have the bot reload the state it relies on instead of assuming nothing changed.
If you change one thing
Put JetStream between your gateway and your handlers. With the stream in the middle, the gateway only forwards, and the workers can come and go without losing an event. Then restart a worker at a quiet hour and watch the pending count rise and fall back to zero, just like the demo above.