• 10 February 2015

    Nexus Mod Manager v0.60 alpha : Upgrading from v0.53

    posted by Gopher Mod News
    Nexus Mod Manager 0.60 is now available in alpha, but it does things a little differently and the update process is more involved. In this video I show you how I updated to 0.60 using Fallout: New Vegas as a test dummy, and I show you how to restore a backup of 0.53 if you need to.

    The basic principals should be the same for games such as Fallout 3 and Skyrim.

    You can download Nexus Mod Manager 0.60 Alpha here.
  • 04 February 2015

    Latest site updates

    posted by Dark0ne Site News
    We've made lots of updates to the sites recently. Because quite a few of the changes we make these days are subtle, rather than obvious, here's a run-down.


    This is the least subtle of the changes we've made recently. We've completely revamped the notifications system on the sites to be much more powerful and useful, especially for mod authors. Notifications are still located in the top bar of the site. If you have notifications you will see a globe symbol with a number next to it which indicates how many new notifications you have.

    The new notification system follows a similar style and system more in-line with the way Facebook notifications work, so users of Facebook should recognise the changes easily.

    Your site preferences page now contains options that will allow you to customise exactly what you'd like the site to notify you about, or even if you'd like to have the system on at all. These options include:

    • Game specific notifications - whether you'd like notifications to only show for the game you're looking at, or whether you'd like to, for example, receive notifications for updates made to Fallout 3 files while you're browsing the Skyrim Nexus section.
    • The ability to track and receive notifications when other users of the site upload new files or images.
    • Notifications for comments on your files, images and videos.
    • Notifications for files you're tracking, including a separate option to enable or disable notifications for comments made on tracked files.
    • Notifications for new images or videos that people add for your files.
    • Notifications of changes made to your files by other members of your team, or by members of staff

    If you would like to track a user you like then there is a new option to do that on user profile pages.

    Notifications will now remain on your account even after you've viewed them (new or unread notifications have a lighter grey background to indicate their unread status). They can be removed by clicking the red cross symbol. You can have a maximum of 50 read notifications at any one time, and new notifications will delete the oldest read notifications in the list if you go over 50.

    We are open to hearing feedback and ways we can improve this new system, including if there should be any more options to track other sorts of things you could be notified about on the site. What we won't be doing is going back to the old system or running both systems side-by-side so if you're resilient to change I'd save yourself the typing time if you want to suggest that.

    Specify your main NMM file

    You've probably noticed during your time using a site that has NMM support that some files have a green "Download (NMM)" button in the top section of the file page while some don't have that button and only have the green "Download (Manual)" button. This doesn't necessarily mean that the file doesn't support NMM, more often than not it means that the author has uploaded more than one "main file" and so the option to download through NMM isn't presented as we don't know which file is the main file NMM is supposed to download, and authors often provide multiple main file options (e.g. texture replacers of varying sizes).

    We've added a new option for mod authors when managing their uploaded files labelled "Set as main NMM file". If you click this NMM will now know which is the main (MAIN!) file and will show the "Download (NMM)" button on your file page.

    VirusTotal improvements

    Over the past couple of months we've had a few glitches with the automated VirusTotal virus scanning system. If it took too long or VirusTotal were doing some maintenance then newly uploaded files wouldn't be downloadable until the issue was resolved.

    We've now changed the system so that you can download all new files straight away, before the virus scan is complete, but you do so completely at your own risk and the system tells you as such.

    Arrow keys on Images

    We've update the image gallery on file pages so you can use the arrow keys again.

    We are currently rolling out 8 new static content servers to try and resolve the issues with images loading slowly for people.

    Spoiler tags in file comments

    A long requested feature, the spoiler bbcode tag will now work like it does in the forums. These tags can be used in file/image descriptions as well as file comments.

    Action log shows user

    File pages contain two action logs. One is a tab, which shows all the actions performed on a file, including things like endorsements. The other is accessible by clicking the little cog icon in the actions bar labelled "logs". This will tell you all the actions performed by just the authors on the file page.

    We've made a slight update to this second log that will also tell you who made this edit. This might be helpful for mod teams who all have access to the same file page. If you don't like an edit or want to keep track of the edits being made you can now see exactly who has edited what and when.

    Bug fixes

    We've also fixed some minor bugs on the site. These include:

    • Fixed several issues with the "Manage other files" system, including the sorting system on that page, issues when you hide the mod, and uploaded images going to the "From users" tab rather than the "From author" tab. Also added a link to "Manage other files" to the "Files" drop-down on the main site navigation.
    • Fixed the pagination on the news archive page and added pagination to the top as well as the bottom of the page.
    • Fixed a few bugs with donating Premium Membership to other users as a gift.
    • Fixed an issue with ordering files by "file size, descending" not working properly.
    • Fixed an issue with certain search terms not returning the correct results when searching in the "mod description" search field e.g. searching for skyrim.ini.

    Current To-Do List

    Work we've earmarked for the immediate future:

    • Roll out 8 new static content servers (mainly images) to replace the 4 current static content servers. Links should be preserved so if you directly link to one of the current static servers they should still work after this change.
    • The ability to block files/images added by specific authors. If there's an author whose work you're really not a fan of and would rather not see, you can block their work from showing while you browse the site, like tag blocking.
    • The ability to disable emoticons from showing in file comments, for those times when you're referencing code or file paths and emoticons annoyingly replace what you've written.
    • A "remove all" option in the notifications drop-down.
    • A complex bug reporting system for file pages to make use of.

  • 23 January 2015

    NMM 0.60 Alpha release with profiling, ready for testing

    posted by Dark0ne Site News
    It's been a year in the making but we're finally ready to release what we think will be our final build for the next major step in NMM functionality, profiling. The download link for this build is at the bottom of this article, however, before you rush off, I highly recommend you read this article as it contains important information about this version of NMM.

    What is profiling?

    Profiling is the ability to create and switch between multiple different mod setups within NMM to facilitate different playthroughs of your game. One profile might be for your axe wielding Nord with mods specific to that character while another might be for your Argonian thief with active mods that you wouldn't need in your Nord playthrough. Or, you might just want to test a new setup without worrying about breaking your stable, current setup. That's fine, just make a new profile and do what you want; if it doesn't work all you have to do is switch to your Stable profile. With profiling, you can pick and choose what mods you have active, in what order, for the specific playthrough of the game you want.

    While this functionality in itself is powerful, it's the added things that the Nexus as a site can do with it that excites me. We want to build on this functionality (note the future tense here, we haven't done this yet...) so that people can share their mod profiles with others, including their entire mod list and load order. Having an issue with your mod build or load order? Publish it via NMM and make it public on the site and others can take a look and see where the issue might be. Have a favourite mod reviewer on YouTube but you're tired of asking what mods they have installed to get your game looking like theirs? Ask them to publish their NMM profile on the site and you can then use this functionality to setup your game just like theirs. Want to back up your profiles so if your computer explodes all your mods and load orders are saved on the site, ready to be used by you should the worst happen? It's all possible. It's quite an exciting prospect, which is why we're putting in the time with this major update to NMM.

    Why is it an Alpha? If it doesn't work for me can I go back easily?

    This version of NMM makes such major changes to the way in which your mods interact with the game folders and we're such a small team here (2 programmers and me testing...) we cannot properly test this version of NMM enough to release it even under the "beta" moniker which we hide behind when there are bugs in the versions we release. As such, we are releasing this version of NMM as an Alpha release. That means it is not an automatic update that everyone can get when they start up NMM and see there's a new version and the only way you can get this version of NMM is by downloading it from the link in this news article. It will remain in alpha until we've had enough feedback from people using the alpha to confidently release it as a beta.

    If you are one of those people who doesn't agree with NMM still being a beta and only wants to update to the latest version when it's confirmed as completely stable then I'd recommend not updating to this version. Additionally, If you're the type to moan about this, please, save yourself the time and don't bother (unless you want to donate a couple of hundred thousand pounds to the Nexus coffers so we can get some full-time bug testers using a myriad of different computer setups).

    If you do upgrade to this version please do it with the knowledge that it's likely to have some bugs, but also with the knowledge that we here on the NMM team are extremely thankful to you for taking the risk and offering to help us out. It's people like you who help us to make NMM better.

    To help you out, and to make sure you don't destroy your carefully tuned current NMM setup, when you install version 0.60 of NMM it will automatically create a backup of your current NMM setup in your NMM folder. If anything at all goes wrong with 0.60 for you and you get so fed up you no longer want to help us out all you need to do is reinstall version 0.53.x to the same folders and use the "Restore backup" functionality. You're then back to exactly where you were before you installed 0.60. No fuss. We've slapped up a quick wiki tutorial on how to do that, just in case.

    With the backup functionality just mentioned it's my hope we can convince plenty of you to help us to test this version. The more people we have testing and reporting bugs the sooner we can get this rolled out and move on to some of that other cool functionality I mentioned.

    Installing NMM 0.60 and the migration procedure

    As per usual, to install this update of NMM simply run the installer executable and ensure you point NMM to the same folders as your current NMM setup. If you install 0.60 and no mods are showing, you've directed NMM to the wrong folders (and yes, that's your fault and not NMM's).

    This version of NMM is a major update which completely changes the way in which NMM installs and interacts with files in your game folders. Because of this, 0.60 has a migration tool that will upgrade you from your current version to 0.60. The migration tool will automatically install any mods you have active in your current version of NMM and set it up as before. This can take a long time, depending on how many mods you have installed. However...

    Unfortunately, because of the way scripted installers work (the mods you download where they have different options you select in NMM while installing the mod), NMM has no way of knowing what options you have picked once you've installed a mod. So while the migration tool can install any mods that don't use scripted installers easily, during the migration process you will have to interact with any and all mods that have scripted installers again. I understand, for some, that's quite a big ask. I will say this though, once 0.60 is out of Alpha, it will become the only version of NMM that continues to be updated. We will not be maintaining a profile version of NMM and a non-profile version of NMM, so the sooner you get it out the way and upgrade to version 0.60 of NMM, the sooner you can get on and continue to enjoy the bug fixes and new functionality we're releasing each month.

    Reporting bugs and hot fixing

    Considering the major changes in this version of NMM it would be unrealistic to assume it's completely bug free. We think it's bug free right now as we can't find any bugs, but 3 people testing something on specific setups can't test every possible variation. Hence why we need you.

    As always, bug reports should be directed to our bug tracker on the forums. We've setup a specific category for this alpha version, away from the normal NMM bug tracker category, to make this easier for everyone. You can find it here.

    We will be fixing bugs as fast as possible as soon as we find out about them and can replicate them. Remember that the more information you can provide us about the bug, what is causing it and how we can replicate it, the faster we can fix it for you. Keep an eye out in the site news or on the Alpha file page for information about bug fixes.

    Please note: version 0.53, which we released recently, added a new "mod activation queue" system. Version 0.60.0a is not using this mod activation queue functionality as we are still considering version 0.53 as "in testing" rather than stable and we didn't want to complicate things by adding two relatively major changes into one big update.

    Downloading 0.60.0a

    We've decided to release this version of NMM via a Skyrim Nexus file page. In doing so, we can actively track the number of users who have downloaded this version of NMM, so we know how much help we're getting, and also quickly release hot fixes via the same mechanism.

    NMM still works on all games listed on the NMM download page. The Nexus does not have cross-game functionality (one file page for multiple games) and we chose Skyrim Nexus as it is the most familiar and most used Nexus site.

    To download this version simply head to the file page and download the file as normal.

    All that's left for me to say is a big thank you, once again, to all those who choose to help us out and run the risk with installing this new version. While the risk is mitigated with the back-up functionality, we still appreciate you taking the plunge early with this alpha.
  • 17 January 2015

    NMM 0.53 release and fixes

    posted by Dark0ne Site News
    We released NMM version 0.53.0 at the beginning of the week which includes a new mod activation queue window. The idea is to allow you to queue up multiple installations or uninstallations of your mods as well as not locking out the other functions of NMM while your mod actions are being processed.

    This update also enables you to select multiple mods and decide whether to install them or uninstall them, potentially saving a lot of clicking if you're doing a fresh install of hundreds of mods. Where before you would have to select each mod individually and either double-click it or press the activate button on the side, you can now select all the mods you want to install at once using standard Windows drag and drop, shift+click or ctrl+click controls.

    Version 0.53.0 and the hotfix version 0.53.1 showed some performance issues for a minority of users and also lacked a progress column that accurately showed how far along the mod installation or uninstallation was for each mod in the queue. We believe we have remedied both these issues in version 0.53.2 and ask that users update to this latest version and let us know if they're getting any more issues in this regard.

    Version 0.60, complete with mod profiling, is just around the corner and version 0.53 is a necessary stepping stone on the way to this release as it also has a "load backup" functionality. When/if you install version 0.60 it will automatically create a backup of your NMM directory (not its entire contents, for the hard-drive space conscious among you). If, for whatever reason, you need to revert back to version 0.53 from 0.60 this backup file can be used, by version 0.53, to quickly restore your NMM to exactly how it was. Such functionality is extremely important as version 0.60 presents a massive overhaul to the way NMM stores and processes mods. Something that is going to be a bit of a pain if you are someone who has hundreds of mods installed as all your scripted mods will need to be reinstalled and interacted with again. We didn't want people to be afraid about trying version 0.60 for fear that it might break their NMM setup, and thus, we needed to release version 0.53 for this purpose to provide a fail safe backup option for those users who are struggling with 0.60 or come across bugs that prevent NMM from working entirely.

    NMM Version 0.53.2 is up now and available for download.
  • 06 January 2015

    The Nexus State of the Union - 2015

    posted by Dark0ne Site News
    As far as I'm concerned I'm dubbing 2014 "The Year of Stability and Sleep" for the Nexus. The first year where we finally grappled with and beat the recurring downtime issues that have plagued the site ever since its inception back in 2001, and, once we finally stabilised our new cloud and cluster setup around about the middle of the year, myself and the staff have had practically 5 months of uninterrupted sleep. No more text messages or phone calls at 3am telling us the sites are down again. That, in itself, makes 2014 a success for me.

    Indeed, the sites have been so stable that it enabled me to take a bit of a hiatus from the Nexus over the final few months of 2014 to focus on some other businesses, to travel more and see more of my friends across the UK. While I've been away the programmers have been working on some long term projects for the Nexus and the moderation staff have continued to do an excellent job. Before I talk about that and what we've got in store for 2015, lets look at the year of 2014 in figures, because statistics tend to speak louder than words.

    New members: 1,788,372 (26% increase)
    New NMM users: 1,392,499 (50% increase)
    New files: 18,640 (20% increase)
    New file authors: 6,912 (18% increase)
    Total downloads: 268,197,960 (39% increase)
    New Image Share images: 126,775 (32% increase)

    The percentage increases are based on statistics from April 2005 up until December 2013.


    In the recent Steam sale we saw traffic reach 8,000 concurrent users at any one time generating 100 page views a second. Both traffic and new registrations exceeded that of last December and during the sale we were gaining a new member to the network every 10 seconds. You wouldn't have known it from using the sites though, where load times remained normal and the sites were fine to browse and download from. The only down-time we've had since July has been caused by issues unrelated to traffic, or us, and have been beyond our control.

    I couldn't be happier. I freaking love sleep.


    Back at the start of September we changed our download serving mechanism away from the original system, which involved running and maintaining 24 individual, dedicated, high-capacity, physical servers located in the USA and Europe to a new CDN system, a distributed system of thousands of shared servers across the globe that we rent from a high profile company.

    This change has cut down maintenance time, increased redundancy, reduced reliability issues and increased our flexibility to allow us to go from serving 4GBit/second of downloads to 10GBit/second of downloads within seconds (which happened in the recent Steam sale...), something that we would never have been able to do before. If we wanted to increase our capacity before we had to buy more servers, set them up and transfer all the files over to them. This takes time. About 2 weeks to be exact, by which point the burst in traffic has past and you're now paying for additional servers that aren't being used. You see the dilemma? Now, if demand increases, we do nothing, the CDN simply allocates more bandwidth to us, logs the amount of bandwidth being used and bills us at the end of the month depending on how much we've used. It couldn't be simpler.

    For the vast majority this change brought improved download speeds (many users are now maxing out their 1MB/second normal member limits where before they were not), yet for a small minority this change was negative, either decreasing speeds or preventing downloads from working altogether. One of our programmers has been working on this issue full-time and the reports from the Download Diagnoser have been invaluable in helping him to better find and fix issues with the downloading system. We've helped out lots of people who were having issues already and we're continue to work on the system to improve it for as many people as possible. We haven't forgotten about it and we are still working on it.

    Nexus Mod Manager

    Feature updates for NMM were relatively mild in 2014. We've continued to squash as many bugs as possible but for most of the second half of 2014 programmer time has been spent on the much overdue profiling system.

    The profiling system completely changes the way NMM installs mods on your system and enables you to choose between different profiles for your games/mods quickly and easily. With one install of NMM you can have different profiles for different users in your house, or different profiles for different playthroughs of your game.

    It's taken a long time to get right because we didn't want to do any game/engine specific .DLL hacks. This profiling system has to work for every game that NMM supports, current and future, and not just for Bethesda games.

    The profiling build, version 0.60.0, is currently in internal testing and due out this month.

    Version 0.60 takes us one step closer to a full 1.0 release for NMM. On my roadmap, the only thing after profiling is a UI overhaul, smartening up, good documentation and a final round of bug squashing, which will likely take us in to 2016.

    Site and file statistics

    We completed a major update to the site back in November that now tracks, stores and presents statistics to mod authors in graph format so they can easily see how their files are doing on the site. We haven't launched this functionality yet as the script we're running to go through all the past 930 million downloads is taking quite a long time, as you can imagine, but once it's ready I'll put a news post up about it.

    For an idea of how it'll work/look you can check out our network statistics page which has been up for a while now. Rather than hide how the network is doing I've chosen to show, in plain sight, how active and busy it is. I don't have any shareholders to impress or reasons to hide the data, so take a look if you're interested. It's also great to direct Nexus haters to when they say, year after year, the Nexus is slowly dying. Sorry guys, but it isn't! ;)

    Mod authors will be able to choose whether to make their file statistics public for all to see or just keep it to themselves.

    New games

    The end of 2014 also marked the end of the first year of the Nexus supporting mods for all games. We now support mods for 147 games, up from 27 at the end of 2013, with 3,400 mods being provided for games we never would have supported had we not moved to our centralised system.


    In March of 2014 I started a thread in the private moderator forums titled "Consistency, transparency and strict guidelines on warnings, restrictions and bans". In it I laid out a set of formal guidelines for moderators, with their consultation, on specific, regularly seen infractions and the actions that all moderators should take in these circumstances.

    While it might sound crazy, up until that point there were no hard set guidelines (there was plenty of informal documentation) on moderating the Nexus that meant there were often inconsistencies in the way the Nexus was moderated depending on the member of the moderation team who was performing the action. Some moderators took a more rehabilitative approach, others a more hard-line approach, compounded by the fact I simply did not have the time to sit down, consult and draw up an extensive and exhaustive moderation document.

    With the introduction of the warning system back in 2013 the moderators were given more tools in tackling infractions of our TOS including, most importantly, the warning system. With this system in place, and with the moderators up to speed with using it, it was possible to formalise documentation to help consolidate and bring all the moderation team under a single doctrine on how to manage the most common infractions on the site.

    I'm very pleased with how this has panned out. Direct reports to me of moderator "abuse" dropped substantially in 2014. And I mean substantially. The warning system is being used for all but the worst of infractions on the site where before users were often simply banned outright.

    I'm aware that there are still some actions that might seem arbitrary and confuse some members as to why one member was warned and another one was outrighted banned. It's often down to the subtle differences. For example, in the case of uploading another mod authors work without permission, if the user seemingly isn't aware they're breaking the rules when doing this, and they still get everything else right (like proper accreditation) they'll receive a warning (despite the glaringly obvious terms they agree to when uploading their files...). If the user has uploaded the file and is knowingly aware they've broken our TOS in doing so (e.g. by saying "I know I'm not meant to do this but if someone doesn't like it then they can just take it down) they'll receive an outright ban. Because doing that is just plain stupid. And it still happens. Really.

    While there's still room for improvement, especially in the transparency department which I can't seem to get through to some moderators about for some reason, I'm pleased with the direction the moderation is going on the sites.

    Plans for 2015

    The major site update being worked on at the moment is a Facebook-style notification system that will completely replace the current notifications system. We want to make it as easy as possible for mod authors to be able to keep up-to-date with what's going on with their files, but it will also be usable by all users of the site, mod author or not, as the system will track a whole range of things on the site including new comments, new images, someone linking a video to your files, comments on your articles, replies to your comments, actions a member of your team performs on your files (someone you've given editing access to), actions moderators perform on your files or images, friendship requests, new uploads, versions or articles for files you're tracking as well as the return of author tracking that notifies you when your favourite authors upload new files, images or videos. Work continues on this functionality.

    I will also be commissioning some web design work on the sites to change certain visual aspects on the Nexus. I am not looking for a major overhaul of the sites like back in 2011, but many areas of the site could do with some smartening up, better UI functionality and sharper visuals. On top of that I would like to increase the usable content width on the sites from the current 960px to 1200px, which should allow more content to fit on to the user's visible page at any one time while increasing spacing. It'd be really nice if I could find a professional from within the community to do the work, as the people who know what the site needs the most are obviously the people who use it the most. When I'm ready I'll get a news post up about it, but if you're one such professional and you're interested in a side-job then by all means, let me know. Only professional web-designers though, please.

    Lastly, it is my intention to hire a full-time community manager for the Nexus. While typically this has been my role over the past 13 years I think anyone who's used this site for any length of time and has read my news posts and/or comments in response to them will know that I have public relation skills not too far off Christian Bale on the set of Terminator Salvation. It's simply not my forté. It'd be great to have someone with a happy-go-lucky attitude fronting the sites, representing the community among official game communities and interacting with mod authors from across the internet, not just on Nexus. Someone who's better than I am. Shouldn't be hard. The community manager will liaise between the users of the site, the mod authors, the site programmers and site moderation team to produce new features and ideas to be built for the site and NMM, in conjunction with me, and talk about issues that are directly affecting the community. On top of that they will also represent the Nexus abroad, among other communities, especially official game communities to discuss their needs and see if we can provide support for more games within the Nexus and help new mod authors get their mods on to the site. Once again, this job will go to someone within the community, who knows the community well. Preferably someone with a bit of experience in such things.

    While it sounds like I am shirking responsibilities I merely see it as a logical progression for the Nexus, one that involves a devolution of power and responsibilities away from me so as to reduce the reliance and bottleneck produced when one person is needed to manage all aspects of a business. I've always said it, I'm a jack of all trades, and most certainly master of none. Much like I began hiring on full-time programmers back in 2011 to take over those responsibilities and improve that aspect of the sites, it's very clear to me that this side of the sites needs to improve too, and I'm not good enough for it. I'll still be around, don't you worry, I'll just be around in the shadows, pulling on strings, doing the accounts, sorting out deals with ISPs and hosts, annoying the staff, confusing mod authors. You know, where I belong. What I'm good at.

    In conclusion

    We're still here. We're still working away. The Nexus is getting bigger. The Nexus pretty much works now. I'm happy with the way things are going. I'm now able to sleep.

    Happy new year.
  • 05 November 2014

    Slow download speeds? Please use our Download Speed Diagnoser

    posted by Dark0ne Site News
    Back in August we moved to a new CDN server setup for serving our downloads. For many, download speeds have stayed the same or improved with this move but for some, download speeds have gotten worse.

    We're continuing to work on the download setup within the CDN as we continue to learn all the little intricacies involved in such a complex setup but we're struggling to troubleshoot a lot of people's download speed issues, a lot of the time, because we're not getting the right sort of feedback. For example, simply saying "I'm getting bad download speeds" isn't useful at all for us!

    As such, we've coded a small piece of stand-alone software which we've dubbed the Nexus Download Diagnoser. It's a very simple piece of software that will run a download speed test against our CDN server and generate a report for you to post for us to see which will provide us with a lot of information. Hopefully useful information that will help us to understand what is happening and why your download speeds are slow. You do not need to install the software in order to run it and you can download it here. Remember, only download it from a domain, never from anywhere else.

    In the interests of security and privacy we've kept things very simple for you. The Nexus Download Diagnoser will obviously need to be able to connect to the internet so it can run the download speed test (so update your firewall as necessary) but it will not send or report any information automatically to us. If you want to share the information, you'll have to willingly post it to us, and you can see all the information in the report so you know exactly what you're sending us. On top of that we've signed the software with our "Black Tree Gaming Ltd." key, just like NMM, so you know the software is legitimately from us and not from someone malicious and we're releasing the source code of the software with the executable just for your own peace of mind. If you really really don't trust us for whatever reason, you can check the source code yourself!

    Hopefully with enough reports we will be able to form a decent picture of what's going on and work towards sorting it out.

    For more information about the Nexus Download Diagnoser and where to post your reports please head to the specially setup forum thread here.
  • 28 August 2014

    August holidays, Fileserver to CDN switchover and WYSIWYG editor for descriptions

    posted by Dark0ne Site News
    Hello, and let me start by apologising for the lack of updates recently. August is typically our holiday month where most of us take at least a week or two off (or 4, in my case) to have our holidays, rest and recuperate away from the sites. I'll doubly apologise to those people who've sent in support tickets via the contact form. As I am the sole point of contact for support tickets if I'm not around they go unanswered, and a lot have gone unanswered this month (though I've tried to at least cover the Premium Members who are having issues with their payments). So, sorry about that. Service should resume as normal come September.

    I'll also do a shout out to a few of you who spotted me at i52 last weekend at the Ricoh Arena. Always slightly odd when randomers shout out "Dark0ne" across a hall, across some tables during board games or drinking with friends at one of the bars but it was great to meet some of you and share some modding stories. Never be afraid to come up and say hello. I'm not a celebrity, just a regular Joe.

    Having said that, it's not like we've done nothing this month.

    File servers and CDN switchover

    During the Steam Summer Sale, and for several weeks afterwards, we experienced issues with our download servers due to the sheer number of people who were coming to the site to get their fill of mods, a problem we didn't originally have due to the fact the sites would crash before the download servers would. We've been working since then on a switch-over to a CDN service used by the likes of Steam and Eve Online so that this will never be an issue again.

    Earlier on in the week we released a new version of NMM that makes use of this CDN service and also switched our manual download mechanism over to the CDN service as well. You no longer choose the file server you want to download from, the CDN should automatically select the nearest server they have to your location and serve your file from there. Even if we wanted to there's no mechanism to manually choose what CDN server you download from; our provider has thousands of servers across the globe.

    A few people have come on to the forums to say that their download speeds have decreased and a few people have come on to the forums to say their download speeds have increased. We want to give the CDN a chance to balance itself out -- it does have a self-learning process that adapts to the way in which we use the CDN (e.g. the demographics and usage during the time of the day) and it can only learn this information with prolonged use. It needs to learn how we use their resources so it can allocate them accordingly. The hope is within a couple of weeks it will have enough information to sort itself out for everybody.

    This update will require another forced update for NMM users which we'll enforce some time next week. This is because we will be decommissioning our complete set of current file servers as they will no longer be used (and frankly, we cannot afford to keep both the old file servers and CDN service running. We'd be paying double for little reason). What we will do, however, is keep the Premium servers. While it's our hope that the CDN will completely fulfill your download needs I've noticed a few Premium Members stating they're getting slower speeds on the CDN (while others are claiming they can now download at 13MB+ a second...) so, out of respect, these will be kept and we'll add the option for Premium Members to use these again at some point soon.

    I'd love to hear your feedback on this but please note if you're getting slower download speeds than before then please be patient and give it some time. If it's still just as bad in 2 weeks time then please come back and let us know. One of the programmers was getting terrible speeds to begin with and now it's maxing out his connection consistently.

    What You See Is What You Get (WYSIWYG) editor in uploads

    Something that's been requested a lot over the years is a decent way to manipulate the BBCode usable in file and image descriptions to make file and image pages look prettier. Lots of mod authors have learnt the code themselves while lots of others have struggled.

    We introduced a WYSIWYG to the description fields to help with this. The editor should provide tools to help manipulate your description and show you exactly what it looks like on the page.

    It's not been without its bugs, and we know there are still a couple left to squash. If you're really struggling with it/hate it then if you click the "BBCode" button then it'll switch to code-mode, which is basically resetting the form field to how it used to be. No WYSIWYG stuff.
  • 21 July 2014

    Downloading issues update

    posted by Dark0ne Site News
    Following on from an update on the downloading situation made at the start of the month I want to write a quick post to say we're still working on the CDN setup I touted in the original update.

    Many people are (wrongly) blaming the new download system we added to the site back in the middle of June, which worked fine up until the Steam sales made everything go a bit crazy. To enable us to fully discount this update as a factor for those of you who are getting slow downloads and downloading issues we've re-enabled the old file server selection download method which we'll likely keep up and running until tomorrow morning for our own piece of mind. This will enable us to monitor the situation and conclude whether it's having any factor on the downloading issues.

    Talking of the Steam Summer sale, which ended at the start of this month, we're still seeing unchanged and unprecedented registration traffic on the network since the end of the sale which is the driver for all the download issues occuring. For example, yesterday we hit 9,500 new registrations and today, a Monday (which are typically our slowest days), we've already had 6,200 new registrations with 7 hours of the day still left to go. We're not entirely sure what's driving this big influx of new users well past the end of the Steam sale, all we know is it's happening and we're working to sort out our file server infrastructure to make sure it's no longer an issue. For a more detailed explanation of what we're doing please refer to the news article linked at the start of this post.
  • 12 July 2014

    Virus scan reports now visible on file pages, more security incoming

    posted by Dark0ne Site News
    After recent events by a malicious user to upload viruses to the Nexus sites and gain access to Nexus accounts (for reasons completely unknown to us as your accounts are worth very little in tangible terms!) we've been stepping up our public-facing security options.

    Thanks to the great support of VirusTotal, who have given us very generous access to their virus API for free, we've been able to integrate their virus scanning functionality into the Nexus file pages. VirusTotal is an online tool that will scan files you send it using over 50 of the most used anti-virus programs and generates a report showing how many of the anti-virus programs have flagged the file as a virus.

    Any new files uploaded to the site will be sent off to VirusTotal to be scanned and will not be downloadable by users until the report has come back all clean. If more than 4 anti-virus programs used by VirusTotal flag the file as suspicious the file will be sent to a quarantine that will require one of the moderation team to verify the file is either safe, or not safe, before it can be downloaded. This process should only take 5 to 10 minutes, but during this time your file will not be downloadable. While the Nexus has always provided instant upload/download functionality I think a wait of 5 to 10 minutes for added security and peace of mind is a worthwhile sacrifice to make. I'm also aware that there are certain types of mods, especially those that make use of TexMod, that get flagged as false positives quite regularly. While this might be frustrating for you we will endeavour to get your file online as soon as possible.

    The VirusTotal report generated for each uploaded file is easy to access by clicking the new icon present on the file tab of file pages. While the file scan report is quite conclusive you should always have your own anti-virus and anti-malware software installed to compliment this service and it should not replace software already on your system.

    We are currently, slowly, scanning through the entire back catalogue of 250,000 uploaded files on the Nexus at a rate of 20 files a minute which is going to take a week or two. But yes, it is our hope that every file, new and old, will be scanned at some point soon.

    Two-Factor Auth

    Our two-factor authentication system is relatively close to being completed as well but has been put on the backburner for the next week or two while we evaluate the CDN situation. This system will work in the same way as Steam and Facebook; if you login from an unrecognised location you will be sent a unique authentication code via email to verify it's actually you. You will be able to turn this system off in your preferences but we'd obviously recommend having it on for maximum security.

    Staff changes

    After the compromise of one of our staff accounts we have removed the ability for staff to upload files to file pages they are not authors of. If you're wondering why they had that functionality to begin with it was a commonly used feature by the staff to help authors who were struggling to upload their files here for one reason or another. The author would upload the file to dropbox or similar, the staff would download the file and then upload it to their page for them. Staff can no longer do this, but it should ensure that any compromises in staff accounts again would have less implications.

    Many of the staff features are hidden behind a second password gateway that is unrelated to the staff member's username and password. For instance, you can't ban someone without being logged in to a staff account and knowing this secondary username and password. All the staff have been told to never save this information in their browser and to simply write it down on a notepad near their PC. This was already present before the compromise and probably helped to limit the compromise substantially.

    Recent outages

    We've had a couple of outages this week. Earlier on in the week we had a couple of hours of down-time because someone who is in the same Cloud as us had some how managed to take our allocated IP addresses. Without any IP addresses you can't access the sites. We managed to sort that one out and our hosts have told us it shouldn't happen again, but it was completely out of our control.

    Last night was a sleepless night for us as we had some extended down-time as well. Our hosts were attempting to install a lot of expensive new hardware under some scheduled maintenance. It didn't go to plan for them and took a lot longer than expected without even being finished. After that our internal network was extremely unstable and has yet to be resolved. We're working with our hosts to get this sorted on their end. However, you might notice things being quite slow, or slower than usual, today. That's because we're only running on 3 of our 5 database nodes. Given how good they've been to the Nexus over the years we won't be kicking up a fuss over a couple of incidents but it is (here's the silver lining) nice to be talking about down-time that isn't actually something to do with our setup for once.

    We'll get there.
  • 02 July 2014

    Steam sale and downloading issues prompts CDN expansion

    posted by Dark0ne Site News
    Over the past 10 days we've had widespread reports of downloading issues on the sites that has gone hand-in-hand with the annual Steam summer sale promotion that sees games getting massively discounted on Valve's gaming platform. These downloading issues were caused, simply put, by the fact every single one of our 20 download servers was filled to capacity with people trying to download.

    If our registration statistics are anything to go by this year's summer sale was the most successful one yet for Steam. Over the past ten days we've averaged 8,200 new registrations a day including a new Nexus record of 14,505 new members in a single day beating the previous registration record set on November 26th 2011 of 13,570 new members just a couple of weeks after Skyrim's launch. Typically the Nexus will average 3,500 - 4,500 new registrations a day when something special isn't going on.

    When you have a huge influx of new members in a short space of time this has quite a detrimental effect on the file servers. While you can typically only browse the site one page/tab at a time, which helps us maintain our resources on the web servers, you can have many downloads running at any one time. The inherent problem with having a huge influx of new people is that their downloading habits are different to "regular" users. As a new user you want to download a lot of mods all at once. You'll go through the top 100 and look up "best mod" lists on the internet and try and download as many as possible. As a "regular" user you've already done this, your mod list is pretty set, and you're now browsing the Nexus to see what's new, perhaps only downloading one or two new files a day to augment your current mod lists. So having a huge influx of 14,000 new users in a day is like adding an extra million regular users to the site over night for a short term period. The result was 20 file servers all serving 400 concurrent downloads each which meant during the Steam sale we were serving 8,000 concurrent file downloads at any given second and maxing out a 10Gbit line. That number would have likely been much higher if it weren't for the hard connection limits we've set on the servers. Hopefully you can appreciate that's a lot and the infrastructure you need to handle that has to be extremely powerful and resolute. While our file server infrastructure is powerful it's typically designed to handle around 6,000 concurrent downloads, and we average around 4,000-5,000 on a normal, non-Steam sale day.

    Question: Why has this only become an issue now?

    Aha, here's a silver lining (ahem). The reason this is the first time we've maxed our file servers is because this is the first time our web servers (the servers we use just to display the sites) have held under all this traffic. Secretly (ahem), we're patting ourselves on the back that the sites themselves were accessible for practically the entire Steam sale week, which means our new Cloud setup and centralised database cluster is finally working. We're obviously not happy about the file server setup so we're working to sort it out.

    Question: Why weren't you more prepared?

    I thought we were :)

    Back in January I posted that we had completely decommissioned our file server setup and we were moving from a 15 standard download server setup to a 20 standard download server setup, an increase in capacity of 33%. The inherent problem was, because our web servers always used to fail before the file servers did it meant we'd never thoroughly tested our file setup under extreme load conditions. Now that the web servers are up to scratch and holding under these conditions the file servers are taking on a lot more load. And so now we can react.

    Question: Why didn't you just buy more servers when the Steam sale started and it became apparent the load was too much?

    The file servers we need can't just be requisitioned overnight. They need to be ordered, delivered, plugged in and have all the firmware and updates applied before we can even get the entire file database copied on to the drives. That takes time, more time that the Steam sale was going to last.

    Picture the situation like a huge rock festival (lets take Glastonbury as it's only just finished) that comes to a very small town (population just under 9,000) in England once a year. 361 days of the year the local road infrastructure is completely fine, but 4 days a year, when the Glastonbury festival sets up in nearby fields, the roads are completely choked full of cars and the local residents can barely get out of their own town. Is it prudent for the local council to build an 8 lane highway to support a 3rd party event that may or may not happen from year to year that will only be used for 4 days of the year? I think not. In a similar vein, we'd be talking an extra $5,000 expense each month, minimum, to accommodate an event that happens once or twice a year.We can't just say to our server provider "we want these servers during November/December and June/July but for the rest of the year we don't want them". Contracts have to be signed and so on and so forth.

    Question: So what are you going to do about it?

    Last year we spent considerable time, effort and money to sort out our web server situation and we moved to a much more flexible cloud and cluster setup. This has worked. It now makes sense that we continue those efforts and bring our file servers inline with the cloud ethos.

    We're currently in talks with a big CDN service, who already partner with big video game players like Steam, CCP and Wargaming, to get rid of our current dedicated file server setup and move our entire file serving efforts on to a CDN.

    If you don't know what a CDN is I won't bore you by going into detail about what it is (a simple Google search will surely enlighten you!), but I will bullet some key advantages it will have over our current setup:

    • Flexibility and scalability. There's practically no limit to the resources we can use and there's no time delay in making use of them, which means no bottlenecks. We contract for a set amount of usage and any overage due to one-off events, like a Steam sale, is charged at a standard and competitive rate.
    • Less administration and more secure. Maintaining 27 file servers (20 normal, 3 Premium, 4 static content) is a huge undertaking that requires a lot of server administration to keep up-to-date and secure. Moving to a CDN places this responsibility in the hands of a team of qualified individuals who are much better suited for the job, freeing us up to both not worry as much, and not work as much on this issue.
    • Increased performance and localisation. We currently have 14 download servers in the US and 6 download servers in the UK, but the Nexus has a global reach with many users from South America, Asia and Oceania. CDN networks have data centres distributed across the globe that should ensure you really will max out your connection when downloading from our servers, hopefully, irrespective of where you are in the world.

    Question: It sounds good, so why haven't you done this in the past?

    Partly because it wasn't necessary and partly because it costs more. Between 30%-70% more than our current dedicated file server setup depending on how much bandwidth we use. We've come to the realisation from our work on the cloud and cluster setup that this really has to be the future for us, and the added cost, although tough, is necessary to secure the future of the sites. We need to be able to move fast during these sorts of situations which is something we cannot do with a dedicated server setup.

    Question: When?

    As soon as possible. We're testing out the feasibility of the CDN for our setup as I'm typing this.