Good morningI just wanted to post an update on the scanner issues from yesterday and the various periods of site down time over the last few weeks. Scanner Issues and Untrackable Onsite StatusYesterday we had to do some server maintenance on our main database server (more in this later), and in order to reduce the load on it while this maintenance took place, we paused the scanners. There’s normally more than 20 TA scanners running constantly that add and update gamer data in the tables we were working on, so pausing the scanners allows this maintenance to happen in theory much faster and without slowing down the rest of the site. However, this maintenance took far longer than expected, which meant we had a huge queue (over 120,000 gamers) when we could finally turn them back on again about 12 hours later.Of course, as luck would have it, that coincided with the Halo Infinite Campaign release, and as sometimes happens when there’s a AAA game release, Xbox Network (formally Xbox Live) started having some major issues.Various gamer accounts could not log in at all, which meant some of the special accounts we use for the scanners couldn’t authorize and therefore couldn’t pull gamer data. Other accounts were shown as having their privacy settings on which means we can’t see their game and achievement data and we flag them as “Untrackable” on TA.The good news is that both of these issues seem to be resolved at the moment.The bad news is that we have an enormous backlog of gamers to scan (over 225,000), and also a large number that are incorrectly set to Untrackable (around 50,000).I’m going to prioritise the queue to put the Untrackable people first as soon as I get to the office this morning so we can get those fixed first – now the Xbox Network issues are resolved they should scan normally and have the Untrackable status removed and any missing achievements picked up the as soon as they are scanned again. If you are seeing the Untrackable status on your profile, don’t worry, we have it in hand and you don’t need to do anything or contact us about it.I’m not sure how long it will take us to clear the scanner queue, but we’ve got through about 4,000 people while I’ve been typing this post, so it will probably take most of the day. However, if you request a scan you will be put ahead of anyone that has been added automatically (although behind anyone else that has requested a scan on your scanner before you).Some of these issues also prevented new users from registering on the website – this should also now be resolved.GTASC Untrackable penaltiesWhen gamers in the GTASC go private, we set their score for the period to -5,000 TAD to discourage people pre-loading thousands of points offline and then suddenly coming online at the end of the period to cheat the leaderboard. Again, as soon as you are scanned again the untrackable penalty should disappear (we’ll keep an eye on this later today to make sure it happens).Achievement StreaksIf your achievement streak has ended due to the issues, as long as you unlocked an achievement and it popped on Xbox Live, do not worry. As soon as those achievements have been scanned into TA, you can rebuild you streak on your achievement streaks page and this will fix the broken streak.Database Server IssuesThe other issue we have been having for a couple of weeks now has been more serious – we seem to have a hardware fault in our main database server that is causing it to throw Fatal Exceptions under certain load conditions. Sometimes this results in a little slow down of the site, sometimes the database itself goes briefly offline, and sometimes it goes offline and can only be fixed with manual intervention. Unfortunately the latter has happened on a couple of occasions in the middle of the night for us, and this has resulted in multiple hours of site down time.Most of the scanner offline time yesterday was due to us running diagnostic jobs on some key tables that we thought might be the cause of the exceptions, but we found nothing wrong and have now been left with just faulty hardware as a source of the issues.Because of that, we are ordering a new replacement server today. We are going to take this opportunity to beef it up a bit and add more storage (24TB!). Once it’s been delivered, we’ll need to set it up and then begin moving the various databases over to the new server. During that time we’ll have to put the site into read-only mode so we don’t lose any data.This will be a big job, so we will post about the exact plan once we have more details.We’re also having the server fully managed, so if there’s anything that happens in the middle of the night there will be a technician there to kick it for us.I’ll post scanner and Untrackable updates in the thread for this news post as I have them this morning.I want to apologise for all this disruption, I realise it’s far from ideal.Rich
Source link
