• 3 December 2013 23:01:14

    Day 1 patches, fixes, cure for the slowness and NMM temporarily down

    posted by Dark0ne Site News
    First of all I want to thank the majority of you for your patience over the past 36 hours. I know that you know that we’re working extra freaking hard right now (9am-5am yesterday, 10am-10pm so far today) to get things not just back to where they were, but better.

    Today has been plagued with bugs and, most of all, slowness. You see, I can deal with being logged out, downloads not working, Premium memberships failing and so on but what I can’t deal with, what really frustrates me, is having to wait minutes for a page to load only for that bug to happen. You find it annoying? Get the hell in line, I was here first.

    First of all the good news. We found the cause of that slowness. Like the plotline to a bad B Horror Movie (which I’ve dubbed NMM 2: Revenge of the Mod Managers), it was Nexus Mod Manager that was inadvertently DDoSing the web servers. Now I know what you might be thinking, “Hey Dark0ne, didn’t this happen before?”. Why yes, it did. This time it’s different, of course, but I’m hoping if and when it comes to finishing off the trilogy we’ll be a bit more well versed in spotting the signs earlier next time, because we’ve spent the better part of the past 36 hours tweaking the hell out of our config wondering why and how our database cluster was serving 15,000 mysql queries a second and our web server was serving 2,700 requests a second without exploding. By the time you’re reading this we’ll have sorted this issue, at least temporarily, by shutting down the NMM web services. You can still use NMM in offline mode, you can still add mods to NMM and manage your mod lists, but you’ll just have to make do with doing it the old fashioned way by manually downloading mods from the site and adding them to NMM after-the-fact. The results, I hope you can tell, are pretty darned obvious. I’m browsing around the various games on the network without a care in the world, split-seconds of loading times between each click. It almost brings a tear to my eye. Queue people commenting on this news post saying that they can’t connect to NMM (one turd sandwich, coming right up!). Tomorrow we’ll investigate why exactly NMM was DDoS’ing our web servers and hopefully by the end of tomorrow it, too, will be back up and operational again.

    Now, we weren’t just working on that all day, we’ve implemented a number of fixes to the code based on your feedback (with my thanks). The list is as follows:

    • Fixed tracked files missing on all games except Skyrim. If you are still experiencing this issue then let me know, and tell me if you’re missing your entire track list or just some of the files are missing.
    • Fixed a link in the tracked files feed.
    • Fixed the issue with the file upload and update dates being wrong.
    • Fixed an issue with the comment pagination not allowing you to go to another page of comments.
    • Fixed an issue with the tracking centre pagination.
    • Fixed an issue where new Premium Members who had signed up after the work started were not being recognised as Premium Members on the site. I will be giving all Premium Members compensation (5 days, probably) once we’re done with the bug fixing to say sorry.
    • Fixed an issue with only a few file servers being operational. More are being brought online as we speak, but there should be several to choose from now. And they should work.
    • Fixed an issue preventing files from being uploaded to the X: Rebirth site. If people are experiencing this anywhere else then please let me know as we do NOT know about it and cannot fix it until you tell us!

    We’re still aware of a few issues that are on our list of things to fix, these include:

    • NMM, obviously.
    • An issue with file uploads for newly added games that is preventing them from propagating to our file servers and, ergo, from being downloaded.
    • Intermittent logouts or the issue where you’re logged in, but in the top-right corner your username is missing. Possibly related to site load which might now be fixed. Time will tell.
    • Issue where sometimes, as a Premium Member, it shows you the “normal” member file server screen thus preventing you from downloading from a Premium-only server. Quick fix is to close the window, reload the page and try again. Will require looking in to for a permanent fix.
    • An issue we’re finding hard to replicate where HTML and ASCII characters are replacing certain elements of the file descriptions you submit for new files. E.g.
      tags or apostrophes being replaced with ASCII equivalent. Would like to hear from anyone who is still experiencing this.
    • An issue where ReadMe uploading isn’t working properly if done outside of the file adding/editing wizard.
    • Probably some more I’ve seen but am too tired to remember right now.

    We’ll be working hard throughout the week to sort out the aftermath of the work done yesterday but I think we’ve finally turned a corner, things are starting to look up. Sleep is no longer a pipe-dream. And little glimmers of hope have emerged, the first pioneering mod authors have uploaded the first mods for some new games in the network including Minecraft, Mass Effect 3 and Knights of the Old Republic 2. I can’t wait to see the results once the network is back on its feet.

Comments (589)