Yharnam TogetherThe server netcode
Why Yharnam Together replaced peer-to-peer with a server that owns the team's world: the game's native netcode, the UDP protocol, the reliability layer, team worlds and the native team lobby.
Why a server
Bloodborne's own netcode assumes one host's world, a short visit and peer-to-peer links. Seamless co-op breaks every one of those assumptions, and patching around them took about 120 patch sites and a set of side channels. The rewrite replaces it: every game keeps one UDP connection to a server, and nothing goes game to game.
Before: peer to peer
6 links for 4 hunters; the world lives in the host's game
After: one server
4 links for 4 hunters; the world lives on the server
Links per team size
- Peer to peer
- Server
| Question | Decision | Why |
|---|---|---|
| Peer to peer or server? | Server | Players reach each other over a VPN anyway, so P2P's one advantage (NAT traversal) is gone. The world survives the host leaving, one place orders every change, and each game has one link instead of a mesh. The cost is one extra hop between guests: milliseconds on a VPN. |
| Who owns the world? | The server owns its state; the games still play it | The server records and orders kills, flags, doors, lamps, shortcuts, items, enemy HP and map state, keeps them across sessions and crashes, and hands them to joiners. It runs no game. |
| Rewrite movement sync? | No | The game's own player and enemy sync keeps running; its packets just travel through the server. |
| Wire format? | UDP; protobuf for our messages, raw bytes for the game's | A schema ends hand-written frames breaking older readers. Game packets are many per second and the server never reads them. |
What the game does on its own
Reverse engineering of the 1.09 eboot, checked with a packet trace on the test rig:
- Every game packet goes through one send and one receive point. Packets travel on 68 numbered channels (about 60 in use), some unreliable (movement) and some reliable (everything that must arrive). Hooking those two points lets the co-op code see, drop, rewrite or inject any packet.
- The native model is one host's world. The host runs every enemy and sends their positions and HP; a guest's hit on an enemy goes to the host, which applies it. A joining guest gets the host's flags for the host's current map only, once.
- Players sync themselves. Each game sends its own player: position, heading, HP and the animation state. A hit on a player goes to the victim's game, which decides its HP.
When members are on different maps, those assumptions produce double damage, late health bars and joins that only cover one map. That is where the old design's bugs came from.
Architecture
Hunter 1
- Bloodborne PC, the 1.09 game code
- co-op hooks at the packet choke point
Hunter 2
- Bloodborne PC, the 1.09 game code
- co-op hooks at the packet choke point
Co-op server
- accounts, presence, the team lobby
- the session: members, places, one enemy owner per map
- the world log: ordered, saved to disk, snapshots for joiners
- the relay: game packets to the right members
Hunter 3
- Bloodborne PC, the 1.09 game code
- co-op hooks at the packet choke point
Hunter 4
- Bloodborne PC, the 1.09 game code
- co-op hooks at the packet choke point
| Traffic | Server's role |
|---|---|
| Team and session: lobby, members, who runs which map's enemies | owns |
| World state: kills, flags, doors and levers, lamps, shortcuts, items, enemy HP, map state | owns: orders, stores, settles conflicts, builds snapshots for joiners |
| Player movement, animation and HP | relays, unread, to members on the same map |
| Hits, throws, bullets, forced animations, character data | relays; drops what is meant for members on other maps |
| Enemy movement and state from the map's owner | relays to the others on that map |
| Enemy damage and HP | replaced by world-log HP entries, ordered by the server |
| Join snapshots | built from the world log, for any map, not only the host's current one |
Enemies' AI runs in one game per map, chosen by the server: the first member there, handed on when they leave, mid-fight if needed. Each player's character runs in its own game.
The protocol
- 01
Game sends
one send point sees every packet
- 02
Channel policy
pass, same map only, ours, or drop
- 03
Our UDP link
sequence, ack, resend, in order for reliable channels
- 04
Server relays
to the members who need it; it never parses game packets
- 05
Game receives
injected into the receiver's per-member queue
- One UDP socket per game to the server, on a fixed port next to the TCP one. Each datagram holds messages; each message has a one-byte kind and a length.
- Kinds:
Game(a small fixed header{channel, reliable bit, sender member, map, visit}followed by the game's packet untouched),Proto(one protobuf message),AckandPing. - Our own reliability layer, per member and per class: sequence numbers, acks, resends and in-order delivery for reliable traffic (the game's reliable channels, world-log entries, snapshots). Unreliable traffic such as movement is never resent; the newest wins. Who is connected, outages and timeouts come from this layer, not from the game's own timers.
- Protobuf messages (
server/proto/coop.proto, shared by the game and the server):Hello(account, build, protocol), the lobby (TeamCreate,Invite,InviteAnswer,TeamState,Ready,Start), the session (Place,MemberJoined,MemberLeft,EnemyOwner), the world log (Propose,Entries,SnapshotRequest,Snapshot), vitals, party death and portraits. - One build for everyone.
Hellocarries the build, and a different build is refused at the lobby with a clear message. There are no compatibility fallbacks.
What a game sends
Measured on the test rig: two players at 30 FPS on one PC, from the server's per-kind counters and the games' own logs. The game sends each packet once per teammate, and the server forwards it only to teammates on the same map, so a player on another map gets almost nothing from the others.
Packets per second, to each teammate
- Enemy owner, with a teammate
- Teammate on the same map
- Alone on another map
| Kind | Enemy owner, with a teammate | Teammate on the same map | Alone on another map |
|---|---|---|---|
| Event sync | 62 | 0.5 | 0.3 |
| Enemy positions | 9.5 | 0 | 9.1 |
| Enemy HP | 7.5 | 0 | 7.5 |
| Player state | 6 | 6 | 6 |
| Enemy AI records | 1.5 | 0 | 1.6 |
| Forced animations | 0.1 | 0 | 1 |
| Hits on enemies | 0 | 0.2 | 0 |
| Everything else | 0.5 | 1.5 | 0.6 |
| Total | ~87 | ~8 | ~26 |
- Event sync is most of the owner's packets. It is reliable, so every packet is acknowledged. It ran at about 2 a second in an earlier trace, so the current rate is being looked into.
- Frame rate sets most of the rates. Enemy HP goes every 4th frame and player state every 5th, so at the 60 FPS used online both double.
- Bandwidth is not the limit. Under 10 KB/s per game with two players; with three, the host's upload peaks around 13 KB/s. The rig link had 2 ms round trips and no resends.
Upload to the server, KB/s
- Host, enemy owner
- Guest
Where the server sends each kind
| Kind | Route |
|---|---|
| Player state, event sync, enemy AI records, forced animations, script calls | teammates on the same map |
| Enemy positions, HP and catch-up for joiners | from the map's enemy owner to the others on that map |
| A teammate's hits on enemies, enemy near lists | to the map's enemy owner only |
| Player hits, throws, bullets, objects | teammates on the same map |
| Join handshake, character data | every teammate |
| Map items, door corrections, the boss-room kick | never forwarded |
Our own messages
Besides the game's packets, each game and the server exchange protobuf messages on their own reliable stream: about five a second each way when nothing changes.
| Message | Direction | How often |
|---|---|---|
| Ping | game and server | once a second |
| Map and enemy owners board | game to server to team | once a second, and at once on a map change |
| Vitals (HP, downed) | game to server to team | once a second; 4 a second while someone is down |
| HP proposals | game to server | twice a second, and at once on a change |
| Enemy hits | game to server to team | one per hit that takes HP |
| World-log entries | server to team | heartbeat once a second, at once on news |
| Flags (doors, lamps, levers) | game to server | at once, then every half second for 10 seconds |
| Map state report | host to server | about every 6 seconds, ~1.25 KB |
| Portrait | game to server to team | every 15 seconds, 16 KB |
Team worlds
Your save, on your PC
the character
- level and stats
- gear and inventory
- Insight and echoes
- appearance
- NPC gifts you received
The team world, on the server
one per team, saved to disk
- bosses beaten
- lamps, doors, levers, shortcuts
- items picked up
- NPC and quest states
- kills and map state
- The server stores the worlds itself, with atomic writes and backups. Every accepted world-log entry is written at once, so a crash, a quit or a host change loses nothing, and any member can be the game-level host.
- A new team world starts from the leader's progress.
- Every save keeps its personal world. Loading into a team world puts the save's own world aside and applies the team's; playing solo again brings it back. The character is never touched by this.
- Bosses beaten solo stay in that save's personal world; in a team only the team world counts.
- Up to four hunters per team: the game's remote player slots.
The team lobby
Continue, Load Game or New Game opens the team screen, where the server keeps the team: its members, the hunters online and the invitations. When the leader begins the hunt, every member's game loads its own save; the first to load becomes the game-level host (the server still owns the world) and the others join its session as they finish loading. After a crash or a quit, starting the game again rejoins the team. Using it is on Playing together.
- 01 Continue, Load Game or New Game
- 02 Team screen: create a team or accept an invite
- 03 Everyone picks a character and marks Ready
- 04 The leader begins the hunt
- 05 Loading holds until the team is together
- 06 One world, each hunter where they stand
Status
The rewrite was built in phases on a branch, each with tests first and runs on the multi-game rig: a packet trace, a channel policy table, one UDP link to the server, the world log on the server, enemy owners from the server, the native team lobby, the server owning the session (faster joins, host migration, resuming after a drop), one owner per map with a single HP path, server routing with the old peer-to-peer paths deleted, and crash-safe world saves with a server that restarts without ending sessions. It is in main and ships in the pre-releases.
The research and the plan with every phase are in the repository's design notes: NATIVE_NETCODE.md and SERVER_COOP_PLAN.md.