Launch Post-Mortem

[Hegemon]

Post-Mortem

Hello everyone, these last 2 days we have been working frantically to fix things such as instance crashes and quest-states, we deployed many changes yesterday which did solve things such as:

• Various performance issues
• Various skill rollbacks
• Built more redundancy into quests when server crashes
• Database saving issues

However, there is one persistent issue that it was clear was not resolved based on what happened overnight and that is:

• Objectives saving

What are objectives? Objectives are basically task containers that award you with quest states when you complete the tasks, this is in addition to NPCs awarding quest states on dialogue completes. When we stress-tested this with 200 people, this was fine. However, as you all know we had a peak of basically 2,000 players all saving to objectives.

This firstly overloaded our database on the first day and we made fixes to that which solved that issue and tags, and such were saving properly again. The issue is that tasks and objective containers are serialized to JSON which is then synced to a master instance before writing to database. The issue is the sheer volume of data being saved caused the master instance to fail to serialize, and when saved to Database, the corrupted JSON was not recognized causing Database to not save it.

This means:

Your objective is gone

But your quest state from dialogue prevents you getting the objective