Happily and thankfully a week after I wrote my 2,600 word End of Year review/essay
we managed to stabilize our new server setup and run through some quick but efficient code updates that have enabled us to run the sites almost down-time free over the past week. No database desyncs, no maintenance windows, just a few times where we've taken the pages down for a few minutes to perform code updates. Paint me f'ing happy on that. And vindicated in my own eyes. No doubt we've still got work to do, but the important thing is that things are way better now.
In the same article I also mentioned that during January we would be decommissioning all 21 of our file servers (2 static content servers, 15 file servers, 4 Premium file servers) and replacing them with new servers. The old file servers were coming to the end of their life-line and running out of hard-drive space fast so we needed to upgrade and update the whole line up. We're now replacing those 21 file servers with 26 new file servers (4 static content servers, 20 file servers, 2 Premium file servers with probably more Premium servers to come). As you can imagine that's quite a job; we've got to set up each server with our custom server modules and configurations as well as transfer the 4TB or so of current files there are on to every single one of those servers (so we've got to transfer about 100TB of data over the next month). It's time consuming to be sure. We're currently setting up the first new servers which requires these new servers sync from the old ones, this is causing some issues with the network which is causing some downloads to fail (more than usual) or not start at all. Once we've got these first new servers setup we can sync the files between all the new servers, from the new servers themselves, thus reducing the load on the old setup in the mean-time.
I know there's been some issues with downloading reliably through NMM recently, especially when downloads don't start at all. In those instances it's important (for your own sake, not mine) that you know how to do the simplest of things: how to add mods to NMM without using the "download with manager" button. It's so simple I might as well just write it here:
- Instead of clicking "Download with manager" click "download manually" on the file you want to download.
- A window will pop up with a list of file servers to download from. Pick a file server. Typically the fastest server will be the one with the least users, not the one that's closest to you.
- Your download should start within 1-30 seconds. Let the download finish.
- When your download has finished, open NMM.
- Open the folder where your file downloaded to.
- Drag the file in to the "mods" window in NMM.
- OR: in the "mods" window, click the button that looks like a jigsaw piece with a plus (+) symbol next to it and navigate to your file on your HDD.
- NMM will add your mod just as if you'd used the "download with manager" button.
That's it. If you download from the Nexus and don't change the filename then NMM will recognise the file and the file will work exactly as if you had downloaded it through NMM itself. There seems to be some myth that if you download manually then NMM won't show you any new mod versions. That's wrong/a lie. NMM cannot show you mod versions for mods not downloaded via the Nexus (so if you download a mod from the Workshop and add it to NMM then NMM won't show you any new mod updates/versions for that file), hopefully for very obvious reasons, but it will show you updates for any Nexus mods whether you downloaded the file directly through NMM or manually via the site.
At times like these you will find it easier and quicker to download your files manually and then add them to NMM rather than using the "download with manager" button and have it failing on you a lot.
While we're working on the file servers we're directing a bit of attention to how NMM handles downloads, and especially downloads that fail. Right now NMM will try and find a working server that fits your download location preferences as well as possible. However if that server fails then it doesn't automatically try another server, it just fails. We're trying to add more fail safes in to the program so you're left with no failed connections.
Similarly we're having to implement a download queuing system based on feedback I've personally been getting from a few...interesting individuals. With the ever increasing interest in projects like Skyrim GEMS
there's a lot of users who are downloading large quantities of mods all at once. I'm getting support tickets from users wondering why they're struggling to download 200 mods all at the same time in NMM. Hopefully I don't need to tell you why downloading 200 files at the same time isn't good for the servers (or respectful to the other users of the site). Now this is partially our fault as we haven't hard-coded any limits in to NMM in this regard. NMM will let you download as many files as you want. They probably won't all work, but it'll let you do it. We've got hard-coded limits on our servers (which I believe are 16 connections for normal users and 64 for Premium, with an increase for Premium as they can use 4 threads, 4 multiplied by 16 is 64). With currently 15 file servers that's a theoretical maximum of 240 concurrent downloads, but that works on the assumption that NMM will pick servers that you aren't already maxing out, which it won't. But I digress, trying to download 200 mods all at once is really stupid and us not having hard-coded limits in the program is equally stupid. As such we are going to be introducing a download queuing system. Normal members will be able to download 5 files concurrently, Premium members will be able to download 10 files concurrently. You can add as many files as you want to your download list but only 5 (or 10) will download at any one time. Once a download finishes a new download in the queue will start automatically. Such functionality is commonplace on many sites and will enable us to provide a more stable and fair service to all users.
We hope to get this new NMM version out to coincide with the completion of our transition over to the new file servers. It's likely this version will be another "forced" update, in that you'll need to update to this version in order to be able to use NMM online, because it's updating and fixing things that are the cause of issues on the network. I'll have a word with the programmers to see if we can update the error you receive when trying to login to an unsupported version of NMM rather than the generic error that is currently used.
In the meantime I hope you're enjoying a much smoother site experience, even if the downloads are currently a little iffy. The important thing is: we're getting there. Like I said we would.