Fail-over and load balancing colyseus Testing

We have successfully configured Colyseus with two nodes in fork mode using PM2, installed latest REDIS (4.0.9), and also added RedisPresence code. It seems to be working.

Snippet to configure Redis

const gameServer = new Server({
    server: createServer(app),
    presence: new RedisPresence(),
});

pm2 configuration

const os = require('os');
module.exports = {
    apps: [{
        port: 2567,
        name: "suicidePoker",
        script: "./index.ts", // your entrypoint file
        watch: true,           // optional
        instances: 2,
        exec_mode: 'fork',         // IMPORTANT: do not use cluster mode.
        exec_interpreter : "ts-node",
        env: {
            DEBUG: "colyseus:errors",
            NODE_ENV: "production",
        }
    }]
}

We now need to test the failover and load balancing.

Will are assuming that all room states will be saved in Redis transparently, without us having to anything.

We then connected to one node (2567 port) and then added two players using different browsers. We saw them appear in the same in the room, using the monitor. We saved some room state e.g. this.player.coins=5 etc etc. Please check screenshot. You can see that on both nodes, we can see the same states. So far, so good. We can see state replication.

We had also created an API that crashes the server.

Something like this:

app.get('/api/ping', function (req, res) {
     setTimeout(function () {
      throw new Error("Let's crash the server for Testing");
         }, 10000);
    res.status(200).json({'status':''})

}); 

Colyseus Monitor

After room state data was saved, we crashed the 2567 node. After a few seconds we checked the Colyseus monitor on both ports i.e. 2567 and 2568 there is no room, and all state is gone.
What is happening here?
Video demo