Happily and thankfully a week after I wrote my 2,600 word End of Year review/essay we managed to stabilize our new server setup and run through some quick but efficient code updates that have enabled us to run the sites almost down-time free over the past week. No database desyncs, no maintenance windows, just a few times where we've taken the pages down for a few minutes to perform code updates. Paint me f'ing happy on that. And vindicated in my own eyes. No doubt we've still got work to do, but the important thing is that things are way better now.
In the same article I also mentioned that during January we would be decommissioning all 21 of our file servers (2 static content servers, 15 file servers, 4 Premium file servers) and replacing them with new servers. The old file servers were coming to the end of their life-line and running out of hard-drive space fast so we needed to upgrade and update the whole line up. We're now replacing those 21 file servers with 26 new file servers (4 static content servers, 20 file servers, 2 Premium file servers with probably more Premium servers to come). As you can imagine that's quite a job; we've got to set up each server with our custom server modules and configurations as well as transfer the 4TB or so of current files there are on to every single one of those servers (so we've got to transfer about 100TB of data over the next month). It's time consuming to be sure. We're currently setting up the first new servers which requires these new servers sync from the old ones, this is causing some issues with the network which is causing some downloads to fail (more than usual) or not start at all. Once we've got these first new servers setup we can sync the files between all the new servers, from the new servers themselves, thus reducing the load on the old setup in the mean-time.
I know there's been some issues with downloading reliably through NMM recently, especially when downloads don't start at all. In those instances it's important (for your own sake, not mine) that you know how to do the simplest of things: how to add mods to NMM without using the "download with manager" button. It's so simple I might as well just write it here:
- Instead of clicking "Download with manager" click "download manually" on the file you want to download.
- A window will pop up with a list of file servers to download from. Pick a file server. Typically the fastest server will be the one with the least users, not the one that's closest to you.
- Your download should start within 1-30 seconds. Let the download finish.
- When your download has finished, open NMM.
- Open the folder where your file downloaded to.
- Drag the file in to the "mods" window in NMM.
- OR: in the "mods" window, click the button that looks like a jigsaw piece with a plus (+) symbol next to it and navigate to your file on your HDD.
- NMM will add your mod just as if you'd used the "download with manager" button.
That's it. If you download from the Nexus and don't change the filename then NMM will recognise the file and the file will work exactly as if you had downloaded it through NMM itself. There seems to be some myth that if you download manually then NMM won't show you any new mod versions. That's wrong/a lie. NMM cannot show you mod versions for mods not downloaded via the Nexus (so if you download a mod from the Workshop and add it to NMM then NMM won't show you any new mod updates/versions for that file), hopefully for very obvious reasons, but it will show you updates for any Nexus mods whether you downloaded the file directly through NMM or manually via the site.
At times like these you will find it easier and quicker to download your files manually and then add them to NMM rather than using the "download with manager" button and have it failing on you a lot.
While we're working on the file servers we're directing a bit of attention to how NMM handles downloads, and especially downloads that fail. Right now NMM will try and find a working server that fits your download location preferences as well as possible. However if that server fails then it doesn't automatically try another server, it just fails. We're trying to add more fail safes in to the program so you're left with no failed connections.
Similarly we're having to implement a download queuing system based on feedback I've personally been getting from a few...interesting individuals. With the ever increasing interest in projects like Skyrim GEMS and S.T.E.P. there's a lot of users who are downloading large quantities of mods all at once. I'm getting support tickets from users wondering why they're struggling to download 200 mods all at the same time in NMM. Hopefully I don't need to tell you why downloading 200 files at the same time isn't good for the servers (or respectful to the other users of the site). Now this is partially our fault as we haven't hard-coded any limits in to NMM in this regard. NMM will let you download as many files as you want. They probably won't all work, but it'll let you do it. We've got hard-coded limits on our servers (which I believe are 16 connections for normal users and 64 for Premium, with an increase for Premium as they can use 4 threads, 4 multiplied by 16 is 64). With currently 15 file servers that's a theoretical maximum of 240 concurrent downloads, but that works on the assumption that NMM will pick servers that you aren't already maxing out, which it won't. But I digress, trying to download 200 mods all at once is really stupid and us not having hard-coded limits in the program is equally stupid. As such we are going to be introducing a download queuing system. Normal members will be able to download 5 files concurrently, Premium members will be able to download 10 files concurrently. You can add as many files as you want to your download list but only 5 (or 10) will download at any one time. Once a download finishes a new download in the queue will start automatically. Such functionality is commonplace on many sites and will enable us to provide a more stable and fair service to all users.
We hope to get this new NMM version out to coincide with the completion of our transition over to the new file servers. It's likely this version will be another "forced" update, in that you'll need to update to this version in order to be able to use NMM online, because it's updating and fixing things that are the cause of issues on the network. I'll have a word with the programmers to see if we can update the error you receive when trying to login to an unsupported version of NMM rather than the generic error that is currently used.
In the meantime I hope you're enjoying a much smoother site experience, even if the downloads are currently a little iffy. The important thing is: we're getting there. Like I said we would.
The aim of this article is to answer a few regularly brought up questions and concerns in regards to the Nexus Mod Manager, talk about the plan for NMM this year as well as talk about how you yourselves can help us.
I think one of the biggest questions that gets thrown around about the Nexus Mod Manager is why, after more than 2 years, it is still in open beta. Some people seem to find this rather insulting and claim we're using it as a cop out for not following a strict stable/beta release structure. I can tell you right now, without any sense of shame, that's pretty much exactly why NMM is still in open beta.
We don't hide this either, it's not as though we claim NMM is some super stable, magical, bug free piece of software that's going to be the solution to all your modding issues. It's really not and we don't claim it is. When we say that, "We cannot stress enough that we are still in the beta stage of NMM. NMM is in a very good state and will be usable by nearly everyone, but Beta stages are typically used for testing and bug fixing, and some of you will find bugs." on the NMM download page and put "open beta" on our pages and within the program itself, that isn't just some blurb we put up to fill column inches, we actually mean it. Even the versions of NMM that are considered "stable" aren't very stable, and to put a "stable" moniker on any current version of NMM would be misleading for new users. No. When you download NMM, when you use NMM, you do so being fully and properly informed that NMM is still open beta and is still susceptible to all the foibles that comes with that. If you don't like that, if you're not comfortable with that, then absolutely nothing is keeping you using NMM over any other mod manager out there (or, manually installing the mods yourselves!). This is how we've chosen to do this as it's making our jobs a lot easier at this time. Once we get a 1.0 version of NMM out we will then most definitely change our structure to ensure we follow a more Linux oriented style of NMM releases (e.g. stable release/bleeding edge seperate releases). Until then don't act like NMM is anything more than in open beta, because it isn't, and if you have the idea that it is then you didn't get it from us.
The reason why we're not just slapping a 1.0 version on NMM and moving into a more stable release structure is because NMM is not complete. I wouldn't say it's "no where near complete" but I'd say there's still a long way to go. And I think perhaps "complete" is the wrong word to use, I instead mean "complete to the point of being happy to put a 1.0 moniker on the program" because NMM, in all it's cliche glory, will never be complete, as we'll always be adding to it and upgrading it even after our 1.0 feature list is implemented.
When I first contacted kaburke, a developer of the Oblivion Mod Manager (OBMM) and Fallout Mod Manager (FOMM), about coding NMM we wrote out a rather informal design document that contained a number of features that NMM would need to contain before it would be considered 1.0. Over the years there's been a bit of feature creep here and there, but to this day several of these features are yet to be finished. So you're probably asking, what features are these?
- Stability. Naturally I'm not going to release a piece of software that is still struggling to get online half the time.
- Bugs. As bug free as is feasible considering the myriad of hardware and software combinations out there.
- Documentation. Proper, decent, detailed documentation on how to use NMM.
- Modern UI interface. We commissioned work on a new design for NMM towards the beginning of last year and I showed you some of the work that had been done at the time. A 1.0 version of NMM will not look the same as it does now.
- Mod profiling. More on this below.
- Mod packaging. A piece of accompanying software for mod authors that will help them package their files into a single, open, proprietary file (e.g. a .nex file) that will ensure their mod is NMM compatible and also help with setting up custom/scripted installers for use within NMM while also remaining open so anyone with a zip program can open/use/extract the files manually if they so wish.
Some of these features are major additions to NMM that require extensive testing, the sort of extensive testing that can only be achieved if the majority, rather than a small test pool of users, are helping us to test the program. So we rely on you to help us.
It's now been over 2 years since we released NMM and things are going very slowly. It's not hard to see and, once again, I don't hide that fact. So what's taking so long? It's simply the sheer scope of the project at hand. Take mod profiling for example. We started work on mod profiling back in the middle of March 2013. That's 9 months ago, and it's still not done.
Which leads to the next regularly asked question; why is it taking so long when programs like Mod Organizer have had it for yonks now and it was seemingly coded in a shorter time? Simply put and specific to mod profiling, we're having to do it in a completely different way because the scope of NMM, which has to support modding for multiple different game engines and not just GameBryo, is much larger. As such we've had to use different methods that aren't just specific to one engine. The result will be a mod profiling system that will not only work for all games but will also continue to work if game developers make some changes to the way modding in their games is handled. And herein lies the issue. Mod Managers developed for specific games or specific engines are always going to be at an advantage in some regards to NMM which is trying to create a platform from which any and all games can potentially be modded. That doesn't mean that NMM can't have powerful features unique to individual games it supports, but while NMM has a framework from which we work from, engine specific mod managers are more free to go off in any direct and do anything without being confined by an encompassing framework which often requires more time to develop for. The framework is what allows us to support all games in one piece of software but it also presents additional barriers that need to be overcome compared to a piece of software developed specifically for a single game engine.
It's why when certain high-and-mighty users of other mod managers feel the need to come take a dump on the progress of NMM I dismiss the criticism for the ignorance it exudes. Thankfully, and much to the credit of Mod Organizer in particular, I know the author is actually a rather stand-up individual who's done everything in his coding power to ensure that the NMM services he uses aren't placing an undue burden on our servers. He hasn't caused a hassle at all (even when services have been changed or taken down) and has responded to any questions asked in great manner. So when such individuals come and act like self-righteous asses on the topic of NMM I'm able to form a clear dividing line between these minority users of MO and Mod Organizer itself, and its author, which are both great.
Lest people forget, the services used to allow users to check for new mod versions or download straight from the site in to NMM or MO or any other program are currently open and free for anyone to use. We've kept these services open because we didn't want to be those types of developers who make everything open source apart from the one thing other programs might need in order to be on a level footing. And to clarify, NMM is open source, our web services are not (they're completely different entities). Nothing is stopping us from closing the web services to anything other than NMM in the future, and the main reason we haven't is because (a) no one has given us a reason to close them and (b) the people who are using them are using them well and in a morally sound way. In the future we'd like to create an API to be able to better control who can use the services, and extend the services on offer. I know that the S.T.E.P. crew, for example, have been wanting a way to allow more than 1 mod to be downloaded from a single link to make the process simpler, and that's something we want to look into, but considering the time it's taking us to develop things right now I don't think it's going to be happening any time soon.
Which brings me on to my next topic. Help. While we've been going through the recent site issues a lot of people have been asking "How can I help?". On the website end there really isn't much anyone can do except not mash F5 twenty times a second if something goes wrong, and perhaps become a Premium Member. But with NMM things are completely different. You can actually help, and you can pay with time, not money (and maybe even earn some money).
When we released NMM we made it completely open source. That means anyone can go to our SourceForge page for the NMM project, download all the source code and see exactly what NMM is doing and how it's doing it. I wanted to do this for a number of reasons:
- It sets us apart from other modding networks who have mod managers but haven't made them open source to ensure they maintain some form of "competitive edge" in a niche area of the gaming industry that I don't think should be competitive at all.
- It enables us to be completely transparent about what NMM is doing on your PC. If you're worried it's doing anything bad, malicious or nefarious, intentionally or otherwise, then you can either look for this yourself or someone else can find it and let others know. It enables us to remain above-board with you.
- My stance on mod publishing is well known among mod authors. I think all mod authors should release their mods with an open and free license to allow users to fix bugs and add or change features based on personal preference, and that these fixes and changes should be allowed to be shared on the premise that the original mod is still required. That's my feelings on the matter. I couldn't in good conscious release a mod manager that didn't follow that philosophy. NMM is open source which means you can add to it, change it and publish it anywhere and I can't do anything about it.
- It means anyone and everyone can help with the development of NMM.
Unfortunately that last point has never really happened. While a few people have expressed an interest in helping with NMM it never turns into anything. This community has a lot of talent within it across a broad range of skillsets. There are plenty of programmers out there and I would like to harness that potential to help us with NMM.
When you write on the forums that you can't believe NMM doesn't have feature X yet, or that bug Y has not been fixed, you need to realise that you yourself can add feature X, and fix bug Y, and in not doing so you've, at least in my eyes, accepted that we will (or won't) add feature X and fix bug Y in our own time. If you don't like that then crack open your favourite coding program and do it yourself and help out millions of other users in the process.
We recently did a trial run with a developer from the X:Rebirth community who coded the integration of X:Rebirth in to NMM which was recently added. It went brilliantly. So much so I'm now talking with another developer from the Starbound community to get NMM integrated with Starbound modding. I think this is an untapped resource in the community that could be put to good use if done right and with the proper incentive.
I was planning to hire an additional .NET programmer this year to help the 2 we currently have already working on NMM. However what I'd like to do is run a little experiment within the community over the new few months to see if that money could be better spent working with multiple individuals on a project by project basis. Anything from some simple bug fixes to adding functionality we just never got around to, like properly integrating NMM with BOSS. The plan is to talk with would-be developers, discuss what needs to be done (and honestly, you can come to me with ideas for what you think NMM should do and I'll listen), provide rough estimates on how long such functionality would take to code and then provide a respectable financial incentive that both parties agree upon based on how long we think the work will take to complete. I'm not planning to pay a coders hourly wage because the site coffers simply can't afford it (and it would defeat the point of doing this!). Instead, this would be the perfect opportunity for individuals looking for a hobby coding project to earn some money on the side while also getting a rather nice addition to your CV or resumé. If you were looking for a coding job in the web or gaming industry then having Nexus Mods on your CV would definitely help in that regard, and I can, of course, provide references if necessary. Perfect if you've just graduated and you're looking for some names to add to that resumé.
If this idea interests you and you're looking for a hobby coding project right now then get in contact with me by PM or using the site contact form and we'll have a chat. Neither of us are under any obligation and everything will remain informal, such is my way.
With all that said, I will simply end by saying that I believe that the worst of the site issues are now behind us. If you've been using the sites over the past week hopefully you've noticed now that things are a lot more stable. However things within NMM still remain quite unstable, and we're looking in to sorting that out as a matter of urgency at this time.
At the start of the year I outlined my plans for 2013 with our focus this year on improving the stability and reliability of the service on offer. Now, quite clearly, this hasn't exactly gone to plan. We've hit snag after snag along the way that has caused massive delays, dampened spirits and made for a rather sorry year for us. We're having our most fun when we're coding exciting new features and updates to the sites that you can actually use. We do not have fun working on behind the scenes stuff and it's made worse when it doesn't actually work out, but it really needs to be done.
We started the year with a hefty sum of money to invest that had been saved up from before Skyrim was launched, primarily from Premium Membership fees. I'd been saving this money over the course of a few years so we could roll out a whole new server setup that would not only provide us with the power we needed but also the quick expandability necessary to deal with the ever increasing traffic and load placed on the servers. This was most definitely the right course of action and I have no regrets on that decision at all. Look at it like you would the gears on your car. In second gear you can only reach a certain speed before you start redlining, you can't go any faster without switching to another gear. Problem is, when you've only got a 2-speed gearbox with no potential to go to third gear the only thing you can do is rip out the old one and install a new 7-speed gearbox. And that's what we did. We installed our new gear box, switched into third gear and opened the way to fourth, fifth, sixth and seven gears in case we need it in the future. What we didn't take into account was just how hard it would be to switch gears and how many unforeseen circumstances you can actually run in to.
We ordered the database cluster servers at the beginning of February and received them at the beginning of March. We initially thought it would only be a matter of weeks to get things all set up and have the sites moved over to the new database cluster. 3 month's later, at the end of May, we thought we had it but after a straight 48 hours of work we couldn't get it to work properly and had to accept defeat. At which point we paid a rather hefty sum of money for two separate professional consultants to come in and take a look at the setup. It wasn't until the end of July that we finally managed to get our database cluster setup completed and moved to, 5 months after we'd originally started working on a project we thought would take 2-3 weeks. In September I proceeded to buy a further 2 cluster servers, taking the setup to 5 servers, each running 96GB of RAM and dual processors for a combination of 480GB of RAM and 80 CPU cores running at 2.1Ghz (or a hypothetical 160 CPU cores with threading).
The move to our database cluster helped to remove one issue only to highlight another major issue that needed to be rectified. While we could now serve the database requests we were struggling with a bottleneck with the HTTP requests. Thankfully we'd already earmarked funds for a move to a Cloud setup that would form the basis of our expandability in to the future. Towards the end of July I was in discussions with our server provider to have a Cloud setup specially requisitioned for us. They'd need to build it from the ground up for us so we weren't actually given the keys until the middle of October. All the while the sites on the old setup were on their last legs. Well, they weren't really standing up so much as they were spluttering blood all over the place with multiple puncture wounds in this analogy. Once we got the keys we needed to run extensive tests and mock runs and get it ready for an eventual move over, which didn't occur until the start of December to coincide with our centralisation of the sites. By centralising the sites we managed to make our future jobs a lot easier. No longer were we dealing with 20 different databases for 20 different sites on multiple separate servers, we were now dealing with 1 database with 1 site on a distributed, but for all intents and purposes centralised, server setup. This simplifies things drastically with the down-side being that the problems that were originally localized mainly to just Skyrim Nexus were now problems for all the sites,
Traffic is at an all time high, as usual...
If we were still on the old setup then you wouldn't be reading this right now, even if it did take you 5 page reloads and 20 minutes to reach this page. The old setup would simply have been incapable of handling this Christmas/Steam sale traffic. So you can picture this situation like a dramatic Indiana Jones scene if you wish, there's Harrison Ford (the Nexus sites in this analogy), stood on a crumbling platform about to collapse at any second. With a crack of his whip he hooks on to a low dangling tree branch and swings on to a new platform. Albeit this platform still doesn't seem too stable but it's a darn sight better than the previous platform he was stood on which has now fallen into a seemingly bottomless chasm. And this is where we find ourselves at the end of 2013, stood on a precarious platform right now, but the right platform, from which we need to build from and solidify our position.
The situation is extremely infuriating for us. I've said it before and I'll say it again, we know it's frustrating for you when you can't download your mods or update your file pages but my god, try working 18-20 hour days trying to sort it out in the background all the while trying to answer people's ferocious questions and trying to remain calm. I can't do it and I blow my lid sometimes. It is infuriating to have spent this much money and time on an issue that still isn't resolved. But it will be. It's not a matter of if, it's a matter of when.
As users you get understandably upset. Lets do a quick Q&A on the regularly asked questions on this topic:
Q: Isn't it about time you spent more money to fix this issue?
We've spent close to £150,000 ($250,000) on sorting out our stability issues this year. Aside from the fact it's no longer about the amount of money that needs to be spent but what we need to do with what's already been bought, do you have a spare extra £150,000 lying around that we can use? Please? No? Didn't think so (if you do, by all means get in contact. I can offer you a bag of Nik Naks and half of a Panettone as payment).
Q: It was way better before. Why not just go back?
If you're talking from a Skyrim perspective: It wasn't. Your memory has failed you.
If you're talking from any other Nexus site: it probably was, although it wouldn't be right now in the midst of all the Christmas traffic. You've been lumped in with the same lot as Skyrim now. Your fate is linked to Skyrim's as much as Skyrim's is to yours. It's a necessary temporary down-side in the interest of future stability.
Q: Why do you keep performing "maintenance" at peak times?
We put the sites into "maintenance mode" to give the servers time to catch their breath and recover. If we put the sites into maintenance mode they'll recover 90% of the time within 1 minute. If we don't then the problem will remain the same, or get worse, for hours.
Q: Why don't you work on fixing the issues rather than releasing new NMM versions/updating the sites/eating/sleeping/leaving the house?
I think by the 20,000+ words written on this very topic this year alone, and indeed the regular maintenance updates and redirects, you can infact see that work is being done. However of the 5 people (including myself) working on the sites only 2 of us can help in this field.
Telling us to not release an NMM update until the site stability issues are rectified would be like telling a UI designer at Microsoft to not release any fixes or updates to the Windows UI until they've sorted out all their security bugs. The UI designer doesn't work on that aspect of Windows, he doesn't know anything about it and he can't be roped in to help. Telling him to stop working until the bugs on a completely different aspect of the program are sorted would be dumb. And a waste of money.
I'd like to have another full-time person on board to be able to handle this side of things but there's a few barriers in this regard. Money. The time it would take to hire the person. The time it would take to bring that person up to speed. All negatives for issues we need resolved right now, and not in 3-6 months time.
Q: I haven't been able to login to NMM since you updated the sites at the beginning of December, what gives?
So you found the forums and the "new topic" or "reply" buttons but you didn't notice the multiple news posts and indeed the 1,000 other threads and posts about this issue raised in the past week? Sometimes you really need to help yourself. Or do a search! You need to update to version 0.46.0 of NMM. Download it from the site and simply install it over your current NMM installation, making sure your folder locations are the same as your old NMM install. If you do this you won't lose any of your mods.
Having said that we are going to make documenting things in NMM a bit better over the next year.
So what has 2014 got in store? Well hopefully we can resolve these issues once and for all very soon and get back to doing what we enjoy doing. I don't think I need to say anything more on that topic.
At the start of the new year we're going to be decommissioning all our current file servers as they're close to their space capacity, replacing the current 19 file servers: 15 for everyone and 4 for Premium Members with 23 brand new ones, 20 for everyone and 3 for Premium Members. Why a drop for Premium Members? A couple of reasons. We've bought Premium servers in various locations across the globe to act as alternatives to the normal servers. These haven't worked out as well as I'd hoped and they're often unreliable for people. I'll be replacing them with some top quality stuff in Dallas, Washington and Amsterdam that will cost twice as much. I want Premium Members to realise that the real bonus on the downloading front isn't the Premium-only servers but instead is the speed cap being removed and the ability to multi-thread your downloads in NMM. Some Premium Members come undone because they will only use the Premium-only servers when the normal servers are equally or often times better suited for their connection.
We haven't touched the Image Share section in quite some time. I want to overhaul many aspects of the Image Share section and implement some updated features like galleries, and removing those horrible image pages some users make that scroll on and on into infinity as the author has added their entire back catalogue in to the image description. We'll replace it with a more suitable system that hopefully won't diminish what some people try to do with their descriptions but instead simply make it a lot more aesthetically pleasing and manageable for users. I'll begin consulting with certain users from the Image Share soon, likely to be ones that cause as little drama as possible. No need to get in contact with me, I'll get in contact with you if I'm interested. Image Share drama is the worst drama on the Nexus and I try to steer well clear of it as much as possible.
I also want to begin work on a Videos section, much like the Image Share section but for YouTubers to showcase their work on the Nexus. I've no interest in hosting people's videos so this system will link straight into YouTube's API system. I think it'd be great to have all the top Skyrim mod video authors, for example, having their own channels on the Nexus from which they can easily showcase their work and the work of the mod authors and you can easily keep up to date with all of them. It won't take away from their subscriber base, it will simply augment it, and best of all for us it won't cost us any valuable bandwidth or server resources.
On the NMM front we're still working on the 0.50 update with it's profiling features. We want to get it right, and we also want to create a full backup system for users so that they can revert to a legacy system if needed. We've managed to gather valuable feedback from the people using the 0.50 alpha but what we cannot ascertain is how successful it's been for users as a percentage. We don't know how many people it hasn't worked for, e.g. how many people couldn't update to 0.50. We want to create a version of NMM that we'll likely dub NMM Legacy. This will be a version of NMM that we shall feature-freeze at 0.46.0 for ever but that we'll always ensure is able to access the NMM web services (for logging in/downloading/mod version checking). It's our hope that each time we release a major update to NMM we'll ask you if you want to backup your current copy of NMM. If the update fails for any reason then your backup will work with NMM Legacy, so you won't have lost anything.
We've also got that NMM design overhaul to look in to. I'll be writing up a blog post about NMM soon as it seems some people are upset we're still in Beta after a couple of years. I'll set the record straight on that one because we're not coming out of Beta any time soon.
I've absolutely no idea how long the stabilization of our services is going to take. I'm not even going to fathom a guess because each time I do, each time I assume we're close, we hit another snag. We've now got a huge backlog of stuff to do after the centralisation, however, so I hope we can get some updates out thick and fast soon.
I hope your Christmas, New Years and indeed your entire year has not been as stressful (or expensive) as ours has. I hope that 2014 brings us all more success.
Before I get into the fixes we've implemented today on the sites I wanted to add a clarification to my previous news post regarding requiring users update to 0.46.0 of NMM in order to gain online access to NMM and our legacy support. In the post I said, "At this time I'm unsure if we'll ever open up legacy support for the previous versions again". I want to clarify that this doesn't mean we won't continue to provide legacy support at all (e.g. every time we release a new version of NMM you will be required to update otherwise you won't be able to login/download through NMM/check file versions, etc.) but it does mean that, from time to time, we will release a version of NMM that will require you to update in order to gain access to the online functionality. When I said we won't open up legacy support for previous versions again, I meant for versions before 0.46.0. It's my hope that we'll continue to offer legacy support from this point on until the next time we're required to force users to update. I believe the last time we forced an update on users was version 0.33.1 of NMM, well over a year ago.
Legacy support is important to me because I think it's important you have a choice. If you don't like something we do with NMM you should be allowed to roll-back to a version you did like, but in doing so you accept the consequences of that decision. We'll continue to move forward in ways we think are best and it's important you can choose to either continue to move forward with us or stay where you are. We can do this because the structure of our web services (what NMM uses to retrieve the file versions, download information, login details, etc.) stays relatively the same for very long periods of time. However, when we change the structure of our web services (perhaps to something more efficient, perhaps to accommodate changes with the sites) it's important that we force everyone to update, or force those who won't update into offline mode so that we don't end up DDoS'ing our servers, again, or spending extremely long periods of time trying to provide legacy support to the relative minority of users who don't update their NMM to the latest version as and when they come out, which is what happened last week. By not updating your NMM, by sticking with a version of NMM that you like or prefer, you are enacting a choice I've given you, but in doing so you understand and accept the consequences. However, please note, I don't in any way, shape or form blame you folks who don't update NMM for what happened last week. We simply should have put our foots down from the start and said "nope, you've got to update if you want online NMM". But we didn't want to because we didn't want to disappoint you.
Now I know some of you will wonder why offering the choice is quite important to me, and why I don't just force everyone to update NMM in order to gain access to the web services. After all, in doing so not only would we reduce the risk of potential issues but we'd also reduce the number of support tickets from users reporting bugs we fixed many, many versions ago. My wish to offer legacy support actually stems from my extreme dislike for the way in which Steam auto-updates my games without giving me any say in the matter. Sure, I can turn off the auto updating feature for specific games, but once an update is installed it's very, very difficult to roll it back. And I hate that. At the end of the day I bought the game and I think I should have as much control as possible over the updating process. In the same vein when we release patches, new features, UI updates and so on for NMM not everyone is going to like them and I'm willing to give those people as much opportunity as possible to enjoy the version of NMM that they prefer. Because, as I continually say, modding is all about choice, and NMM is built with that in mind. It's why it's open source, it's why we offer legacy support as much as possible.
I haven't received a single message complaining about this and indeed I believe there was only one comment about it in the previous news article but it's actually quite important to me, and I wanted to ensure my words weren't taken out of context. So there you go.
Now, on to the site fixes and updates implemented today.
First of all we've got the Nexus Wiki back up. With all the issues last week we didn't find time to port it over with the other sites but we've managed to get it across today.
We've fixed the issue on user's profiles that meant the pagination on the Images tab wasn't working (preventing you from seeing all a user's images). We've also fixed an issue with users being able to see (but not edit) other people's files in the file management area.
I'm sure many of you have noticed that you now have access to your complete download history on the sites, irrespective of whether you are a Premium Member or not. This isn't a bug, you don't need to pretend like you haven't seen it so that we don't "fix" it and patch it out. This was actually an added bonus of our centralisation process. We've changed the way that we store your download history and file download stats so that it is no longer a burden on our database cluster. I've always said that the complete download history was made a Premium Member feature because if everyone had access to it, it would have brought down our servers (300 million rows in the downloads stats on Skyrim alone, lots of people trying to view their download historys...boom). That's no longer the case and there's no longer a reason to make this a Premium only feature. So, enjoy your complete download history. Note: I'm aware the Premium Membership page still shows this as a feature, it'll be edited out.
In the same vein some people have reported that since the centralisation last week the files they download aren't being recorded in their download history and that as a result they can't endorse these files. We have now fixed this issue but unfortunately we cannot bring back your missing downloads since the centralisation. So it'll work from now on but those downloads you made after the centralisation but before today cannot be retrieved in your history.
Since the centralisation a few people have said they miss the "Nexus Network" drop-down menu not containing links to all the Nexus sites any more. Obviously now we've opened up the network to all games that list will quickly become untenable (before the centralisation we supported 24 games, we now already support 43 games...). That list could potentially grow to hundreds or even thousands of games. What we have done instead is implement a new favourite games system to the sites. Using this system you can select 8 games that you visit the most and these games will display in the Nexus Network drop-down menu for faster access.
We're still working on getting a fix out for the alpha 0.50 version of NMM to those users who upgraded. Hopefully there'll be a download link for that soon.
Update: The NMM 0.50.1 hotfix can now be downloaded. Simply over-write your install with this version. Again; this is only for people who specifically updated to 0.50 to help us test our new profiling system.
I feel some people felt when I asked people to update to version 0.46.0 that this implied that version 0.46.0 was going to fix all the problems for you. When I said "Downloads might work sporadically or just not at all and service might be intermittent but it'll be fixed eventually" this meant "this new version of NMM is going to help out the situation immensely but we've still got work to do to actually fix all the issues, and there will still be issues". I hope this clarifies things for the people going bonkers that they've updated to version 0.46.0 and things aren't perfect.
Since this news post we've now turned off the web service support for versions before 0.46.0. So you now have to update to version 0.46.0 if you want to be able to login to NMM or download files. At this time I'm unsure if we'll ever open up legacy support for the previous versions again. It's presenting serious issues and while we like providing legacy support for people who, for whatever reason, don't want to update NMM to new versions, my main concern is focusing on what's ahead and not on what's behind. I'm sorry if this disappoints you, but that's my mind on the matter. We've spent a considerable amount of time (literally days) trying to sort this issue so that no one would need to update and so that we could maintain legacy support. There comes a point where you need to accept defeat and disappointment and just move on. I think we're at that point.
I'm also aware we've got some folks who moved to our alpha build of 0.50. We'll be releasing an update for the alpha to fix this for you guys, but it probably won't be until Monday. Sorry for that; sorry you're being penalised for actually trying to help, but we don't have the time to do it tomorrow unfortunately so I think you should know it'll be released on Monday.
We've performed more optimisations this evening and right now, 00:46am, things are running very smoothly both on the NMM side and the site side from what we can see (and what we can experience browsing around). I can't guarantee it'll stay this way, but it's a sign we're moving in the right direction.
Thank you for your continued patience. My original news post follows below:
One of the main reasons we've had to disable NMM over the past 5 days is because it directly calls our web services from the sites themselves. When too many requests occur, or those requests are bad for whatever reason, it directly affects the performance of the sites and that's obviously not good.
One of the beauties of our new cloud setup is that we can setup new "servers" instantly (technically they're now "nodes"). We decided it made practical sense to set up a new server that's sole purpose is to handle the requests that NMM makes. The reasoning should be quite obvious: if this happens again, if NMM brings down a server, it won't be a server that handles the sites and the sites will still work fine. We don't want a repeat of the recent situation and I don't think the popularity of NMM should negatively affect the people who don't use NMM.
We've released an update today, 0.46.0 that changes the connection settings in NMM so it no longer calls our web servers but instead calls the dedicated NMM node. To put it simply, the more people who update to 0.46.0, the less load there'll be on the web servers and the faster the sites will load.
On top of that we've launched NMM support for X Rebirth in this latest version. We recruited the help of Azhwkd on the Egosoft forums, who has worked on his own Mod Manager for X Rebirth, to help us get support for X Rebirth in to NMM. I'm very happy about this as it's allowed the two dedicated programmers we have for NMM to get on with their work while Azhwkd worked, with some pointers from us, largely on his own to get this code into NMM. It not only means we can keep our focus on current project but it also means that someone who knows the games well is working directly on that game's support. It makes sense. We're very thankful to him and he's been compensated accordingly.
This is something I'd love to see more of in the community: we've released NMM as open source not just so we're totally transparent about what's being installed on your PC, but also so that developers are able to help us out in a wide range of ways if they so wish. This includes the potential to get more games supported in NMM and I'm happy to compensate those developers for their time. Anyone can download the source code for NMM from our Sourceforge page.
At the moment the X Rebirth support is simple enough, it'll download the files from X Rebirth Nexus and place the files in the correct folder. In the future, if necessary, we'll be happy to work with Azhwkd again to get some more complicated functions into NMM for X Rebirth.
We continue to work on the sites and NMM over the weekend. Downloads might work sporadically or just not at all and service might be intermittent but it'll be fixed eventually. But in the meantime your best bet is to download this latest version of NMM as soon as possible, if you use it.
It's been a few days now since I originally said NMM would hopefully be up by the end of Wednesday and it's the end of Friday now. We continue to work hard to try and resolve the issues with NMM crashing our web servers but are yet to find a definitive solution (indeed, the times we have tried have resulted in the servers going back down again). Simply put now, unless you hear something different from me personally, we're still working on it and NMM will be back as soon as we've fixed things so that it no longer crashes the sites. Please, for the love of all that is good in this world, do not continue to ask me via PMs (I'm getting thousands a day) or support tickets (similarly, thousands) when NMM will be back. If I knew, I'd be telling you. If I had an update on the situation, I'd tell you. Thank you.
On the site front we've managed to implement a number of fixes since my last news post. I'll list them below for easy reference:
- All the file servers are back online and working for manual downloads again.
- Fixed an issue with the servers randomly "fake" logging you out on a page reload.
- Fixed an issue with the notifications area saying you're not tracking any files, and not showing tracked file updates, when you are.
- Fixed an issue with hot file images not working properly on XCOM Nexus.
- Fixed an issue with the readme uploader not working unless you used the file wizard.
- Fixed an issue where you could no longer use letters or non-integer numbers in your file versions (e.g. version "5.1" became "5").
- Deleting comments should now work again.
- Removed the links to Nexus sites in the footer of the site because if we end up supporting 100s of games it will become untenable (will be replaced by a favourites system soon).
First of all I want to thank the majority of you for your patience over the past 36 hours. I know that you know that we’re working extra freaking hard right now (9am-5am yesterday, 10am-10pm so far today) to get things not just back to where they were, but better.
Today has been plagued with bugs and, most of all, slowness. You see, I can deal with being logged out, downloads not working, Premium memberships failing and so on but what I can’t deal with, what really frustrates me, is having to wait minutes for a page to load only for that bug to happen. You find it annoying? Get the hell in line, I was here first.
First of all the good news. We found the cause of that slowness. Like the plotline to a bad B Horror Movie (which I’ve dubbed NMM 2: Revenge of the Mod Managers), it was Nexus Mod Manager that was inadvertently DDoSing the web servers. Now I know what you might be thinking, “Hey Dark0ne, didn’t this happen before?”. Why yes, it did. This time it’s different, of course, but I’m hoping if and when it comes to finishing off the trilogy we’ll be a bit more well versed in spotting the signs earlier next time, because we’ve spent the better part of the past 36 hours tweaking the hell out of our config wondering why and how our database cluster was serving 15,000 mysql queries a second and our web server was serving 2,700 requests a second without exploding. By the time you’re reading this we’ll have sorted this issue, at least temporarily, by shutting down the NMM web services. You can still use NMM in offline mode, you can still add mods to NMM and manage your mod lists, but you’ll just have to make do with doing it the old fashioned way by manually downloading mods from the site and adding them to NMM after-the-fact. The results, I hope you can tell, are pretty darned obvious. I’m browsing around the various games on the network without a care in the world, split-seconds of loading times between each click. It almost brings a tear to my eye. Queue people commenting on this news post saying that they can’t connect to NMM (one turd sandwich, coming right up!). Tomorrow we’ll investigate why exactly NMM was DDoS’ing our web servers and hopefully by the end of tomorrow it, too, will be back up and operational again.
Now, we weren’t just working on that all day, we’ve implemented a number of fixes to the code based on your feedback (with my thanks). The list is as follows:
- Fixed tracked files missing on all games except Skyrim. If you are still experiencing this issue then let me know, and tell me if you’re missing your entire track list or just some of the files are missing.
- Fixed a link in the tracked files feed.
- Fixed the issue with the file upload and update dates being wrong.
- Fixed an issue with the comment pagination not allowing you to go to another page of comments.
- Fixed an issue with the tracking centre pagination.
- Fixed an issue where new Premium Members who had signed up after the work started were not being recognised as Premium Members on the site. I will be giving all Premium Members compensation (5 days, probably) once we’re done with the bug fixing to say sorry.
- Fixed an issue with only a few file servers being operational. More are being brought online as we speak, but there should be several to choose from now. And they should work.
- Fixed an issue preventing files from being uploaded to the X: Rebirth site. If people are experiencing this anywhere else then please let me know as we do NOT know about it and cannot fix it until you tell us!
We’re still aware of a few issues that are on our list of things to fix, these include:
- NMM, obviously.
- An issue with file uploads for newly added games that is preventing them from propagating to our file servers and, ergo, from being downloaded.
- Intermittent logouts or the issue where you’re logged in, but in the top-right corner your username is missing. Possibly related to site load which might now be fixed. Time will tell.
- Issue where sometimes, as a Premium Member, it shows you the “normal” member file server screen thus preventing you from downloading from a Premium-only server. Quick fix is to close the window, reload the page and try again. Will require looking in to for a permanent fix.
- An issue we’re finding hard to replicate where HTML and ASCII characters are replacing certain elements of the file descriptions you submit for new files. E.g.
tags or apostrophes being replaced with ASCII equivalent. Would like to hear from anyone who is still experiencing this.
- An issue where ReadMe uploading isn’t working properly if done outside of the file adding/editing wizard.
- Probably some more I’ve seen but am too tired to remember right now.
We’ll be working hard throughout the week to sort out the aftermath of the work done yesterday but I think we’ve finally turned a corner, things are starting to look up. Sleep is no longer a pipe-dream. And little glimmers of hope have emerged, the first pioneering mod authors have uploaded the first mods for some new games in the network including Minecraft, Mass Effect 3 and Knights of the Old Republic 2. I can’t wait to see the results once the network is back on its feet.
Well hello there. Before we get into the main meat of this news post I’d like to list a few things happening right now on the sites that we are aware of, that we don’t need you to tell us about or confirm, or reconfirm, that will be fixed soon. I say soon, we’ve been working 14 hours straight, from 9am to 11pm and while the main conversion is complete there’s still work to do which will be pushed into Tuesday day (i.e. we’re going to bed soon!). Anyway, here’s the list, and if you comment on this news post bringing up these points it means you didn’t actually read this news post before commenting and that you, sir, deserve a turd sandwich:
- Apparent slowness on all sites or pages not loading at all with 500-504 errors. Sometimes at random, sometimes all the time. This is because our new server setup and configuration needs lots of testing and tweaking with the new load put on it. It should go away over the coming days. Good news for us is it’s not database related. First time this year.
- Not all the file servers are up and operational yet. The ones that are up might be slow, or not working at all, due to this fact.
- Issues with some Premium Memberships not working. Unsure whether this is on the sites (can’t replicate this issue on my end) or in NMM (can replicate this issue on my end, Premium Members not being recognised in NMM). Feel free to share feedback on this one to help me troubleshoot (obligatory turd sandwich is waived in this instance).
- A few broken links. I know about the forum button link and the “More from this uploader” link. Any others feel free to share.
And now, to the meat of the news...
Our maintenance work is now completed on all sites and I can now happily announce the completion of my plans to centralise the Nexus into one database and open up the site to mods for all PC games.
The overall premise of this update is to centralise the management and presentation of game files across all the games we support while still maintaining that drilled-down, focused aspect of browsing individual Nexus sites for individual games. I understand that some users only visit the Nexus to get files for a single game and I also understand that some users use various different Nexus sites and also have files and images on multiple different Nexus sites. I didn’t want to remove the more personal Nexus browsing experience you get when browsing a Nexus site for a specific game but I did want to offer the option to users to very quickly and easily find files for multiple different games very quickly and easily without having to switch between sites, or even pages.
Perhaps more exciting is the fact we have updated our file upload wizard and now accept files for any and all games on the PC platform. As part of my wish to support modding in totality I wanted to open up the Nexus so it could be used by all mod authors out there, irrespective of whether we support the game currently or not. As a mod author it’s actually extremely simple to add a file for a game we don’t currently support and there’s very little extra work involved. The process is as follows:
- Go to our file upload wizard on nexusmods.com
- The very first step, Step 1, is where you choose what game your file is for
- If the game already exists simply select it from the drop-down menu
- If the game does not exist enter the name of the game
- Continue with the file upload wizard as per before
Note: this step is made simpler if you use the file upload wizard from a specific Nexus site, e.g. if you use the file upload wizard on Skyrim Nexus then step 1 is completed for you with Skyrim pre-selected.
The game addition phase is a manual one which requires an administrator to confirm the submittal of your suggested game to the database. This means until an administrator confirms your game suggestion your file won’t be visible on the sites. While adding this bottleneck is a bit counter to how we’ve done things up until now I’m sure you’ll agree it could become an organisational nightmare if the game adding process was automatic. It’s simply there to allow me time to add the game box art and ensure that people haven’t uploaded different files for the same game under different names, e.g. C&C3, Command & Conquer 3, Command And Conquer 3, CNC3 and so on. In such an event I can merge all the files in all those games under one name and ensure everything isn’t a mess before publishing the game on the network. However, you can still create your entire file page including uploading your images and files and submit your file so that as soon as the game is approved by an admin it will be instantly published along with the site. That means you don’t have to suggest a game, wait for it to be approved, and then submit your file. You do it all at once and then simply wait for me to approve the game, at which point your file will become available.
Once a new game is approved a new Nexus site is automatically created and added into the pool of games to browse in the file database and our game list. For all intents and purposes once the game is approved you’ve just created an entirely new Nexus site in seconds. Minecraft Nexus? You can do that. The Sims Nexus? Yep. GTA Nexus? Uh huh. I no longer dictate what games the Nexus supports, the mod authors do.
Any new Nexus site created by this new system will use a generic design under the original Nexus orange and grey colour scheme (the Oblivion colour scheme) with no background image. You can tell what Nexus site you’re currently browsing in many different ways. Established Nexus sites have unique colour schemes and unique background images that really make them stand out, but all Nexus sites have the name of the game in the top-left hand corner of the site below the Nexus logo. It’s my hope and intention to create unique colour schemes and background images for any games that reach a certain level of activity. In my mind that level of activity is around about the 30 files uploaded mark. Once 30 files have been added to a specific Nexus site I’ll create a unique background and colour scheme for it.
With this update we’ve also changed the addresses/URL’s used and we’ve moved away from the subdomain system of separating Nexus sites (e.g. skyrim.nexusmods.com, oblivion.nexusmods.com, etc.). I know this is annoying for some of you who like to ensure you’re using the latest links but we really did think it through and, with this update, the benefits of changing from a subdomain system far outweigh the annoyance of the change. We’ve been careful to ensure any old links to your files will still work with the new system; they should seamlessly redirect to the new URL, but I do understand the frustration of having to change links, just please know it was in our best interests, and ultimately, yours (it’s all to do with search engine optimisation). All new Nexus sites will not have a subdomain attached to them. If you’re noticing that redirects from the old URL to the new URL are not working then please let me know and I’ll get it fixed up as soon as possible.
We’ve updated the NexusMods.com homepage accordingly to become the central hub for finding all the games we support and even accessing all the files across all the games from one single page. From NexusMods.com you can find any file for any game on any Nexus site from the files page. This page is exactly the same as the file page on individual Nexus sites only you can broaden your browsing range from a single game to all games. If you prefer to browse for files from specific Nexus sites then you can still do that (indeed, the Nexus sites remain pretty much unchanged with this update), and you can also access our game list to see a list of all the games the Nexus currently supports.
As part of this work we’ve centralised our database into one single database. Originally each Nexus site used a different database. 18 games, 18 databases. We’ve now merged all the databases into one. This provides practical functionality to the sites, for example if you’ve uploaded files or images across multiple different Nexus sites you can now manage all of them from a single page or, in keeping with my wish to maintain a more personal approach, you can continue to go to each individual Nexus site and maintain your files for individual games from there. Your file management pages have a form field at the top that lets you filter between all the games you’ve uploaded files for which provides quick and easy access. Similarly the centralisation also allows us to present a unified listing of all your files and images on your profile page for everyone to see. Once again, filter options are available that allow users to see what files you’ve uploaded for specific games.
While originally only Skyrim was using our new Cloud setup we’ve now moved every site on to the Cloud. While this might create a few days of instability as we re-tune all the settings and config for the increased load on the nodes it is the next step in our efforts to increase the reliability of the network which I’m sure you’ll agree (and if you don’t you’re being silly) has improved over the past couple of months compared to the past few years. It’s still not perfect by any means but we are getting there and this work is a step towards the ultimate goal of 100% uptime and 100% reliability. Liken it to surgery; relatively brief times of pain for much longer term benefits and gain.
Having said that this centralisation work is the combination of thousands of hours of work and while we’ve tried to test it all as much as possible I think logic (and past experience) dictates that there are going to be niggling issues and, potentially, bugs. We’ll be working overtime to ensure we fix any bugs as soon as we find out about them and, once again, we’re relying on you guys to let us know when you find something amiss or not working properly. We can only fix what we know about.
It’s important to understand that this update is laying the groundwork to future improvement. It’s not the end, it’s simply the beginning of a new era, a new foundation for Nexus Mods from which we’ll be working from. This year has been all about two things. Improving the reliability of the network and opening up the network to files for all PC games. In the new year we’ll build on both of those objectives and we’ll be switching our focus away from these two big projects into a lot of smaller, more functional updates and improvements which we haven’t really focused on this year at all. It’s difficult to say “the best is yet to come” but I honestly believe it. What you see before you right now is a foundation. Next year we want to build on it and improve on it based on your feedback to us and we’ve positioned ourselves accordingly.
Heads up folks, we will be taking all the Nexus sites down one at a time starting on Monday the 2nd of December as we begin our conversion work over to our centralised network. I can’t give you exact times of when each site will be down and how long each site will be down for but you can safely assume that the larger sites (Skyrim, Oblivion, New Vegas, Fallout 3, Dragon Age) will be down for longer than the other sites. Depending on how long things take this might spill over into Tuesday, though we’ll only take sites down when we’re actually working on them so this basically means if we don’t get around to taking down a site on Monday, we’ll be taking it down on Tuesday instead (and not that the site will be down from Monday through to Tuesday...making sense?).
I’ll make a bigger announcement about the work done and the changes made after it’s all been converted over but this is a heads up to say, if you’re planning to release your new mod you’ve worked on for 2 years on Monday or Tuesday, you might want to hold off until Wednesday.
Naturally when problems occur on the Nexus they don’t occur at 9.30am just as we’ve showered, eaten and gotten to our desks. They always occur at 1am just as we’ve gone to bed, leaving a good 8 hours before we know there’s anything wrong.
In true Nexus fashion we hit a rather comical problem in the early hours of this morning whereby we’d reached the maximum number of registered accounts as set by the forum software we use, Invision Board. I’d never thought to check up on whether there was a maximum because, well, why would there be? Oops. So for most of the day our registrations have been either broken, spewing out some ambiguous database error (of course, rather than saying we’d reached the maximum it simply gave a generic error), or deliberately taken offline by us so we could sort it out.
We think we’ve sorted it now. It required a lot of altering of some rather huge database tables, hence why you might have witnessed a few slowdowns or 500/504 errors today, but we should be back on track now.
If you’re one of those people who registered between the hours of 12am and 12pm today please check if the details you registered with work. If they don’t then I’m afraid the only fix will be to make a new account. If you’re one of those users who paid to become a Premium Member as you were registering during that time then please, please get in contact with me via the Contact Us link at the bottom of the site so I can sort it out for you.
Jump to page