ALL FILES

View:

  • 23 January 2015

    NMM 0.60 Alpha release with profiling, ready for testing

    posted by Dark0ne Site News
    It's been a year in the making but we're finally ready to release what we think will be our final build for the next major step in NMM functionality, profiling. The download link for this build is at the bottom of this article, however, before you rush off, I highly recommend you read this article as it contains important information about this version of NMM.


    What is profiling?

    Profiling is the ability to create and switch between multiple different mod setups within NMM to facilitate different playthroughs of your game. One profile might be for your axe wielding Nord with mods specific to that character while another might be for your Argonian thief with active mods that you wouldn't need in your Nord playthrough. Or, you might just want to test a new setup without worrying about breaking your stable, current setup. That's fine, just make a new profile and do what you want; if it doesn't work all you have to do is switch to your Stable profile. With profiling, you can pick and choose what mods you have active, in what order, for the specific playthrough of the game you want.

    While this functionality in itself is powerful, it's the added things that the Nexus as a site can do with it that excites me. We want to build on this functionality (note the future tense here, we haven't done this yet...) so that people can share their mod profiles with others, including their entire mod list and load order. Having an issue with your mod build or load order? Publish it via NMM and make it public on the site and others can take a look and see where the issue might be. Have a favourite mod reviewer on YouTube but you're tired of asking what mods they have installed to get your game looking like theirs? Ask them to publish their NMM profile on the site and you can then use this functionality to setup your game just like theirs. Want to back up your profiles so if your computer explodes all your mods and load orders are saved on the site, ready to be used by you should the worst happen? It's all possible. It's quite an exciting prospect, which is why we're putting in the time with this major update to NMM.


    Why is it an Alpha? If it doesn't work for me can I go back easily?

    This version of NMM makes such major changes to the way in which your mods interact with the game folders and we're such a small team here (2 programmers and me testing...) we cannot properly test this version of NMM enough to release it even under the "beta" moniker which we hide behind when there are bugs in the versions we release. As such, we are releasing this version of NMM as an Alpha release. That means it is not an automatic update that everyone can get when they start up NMM and see there's a new version and the only way you can get this version of NMM is by downloading it from the link in this news article. It will remain in alpha until we've had enough feedback from people using the alpha to confidently release it as a beta.

    If you are one of those people who doesn't agree with NMM still being a beta and only wants to update to the latest version when it's confirmed as completely stable then I'd recommend not updating to this version. Additionally, If you're the type to moan about this, please, save yourself the time and don't bother (unless you want to donate a couple of hundred thousand pounds to the Nexus coffers so we can get some full-time bug testers using a myriad of different computer setups).

    If you do upgrade to this version please do it with the knowledge that it's likely to have some bugs, but also with the knowledge that we here on the NMM team are extremely thankful to you for taking the risk and offering to help us out. It's people like you who help us to make NMM better.

    To help you out, and to make sure you don't destroy your carefully tuned current NMM setup, when you install version 0.60 of NMM it will automatically create a backup of your current NMM setup in your NMM folder. If anything at all goes wrong with 0.60 for you and you get so fed up you no longer want to help us out all you need to do is reinstall version 0.53.x to the same folders and use the "Restore backup" functionality. You're then back to exactly where you were before you installed 0.60. No fuss. We've slapped up a quick wiki tutorial on how to do that, just in case.

    With the backup functionality just mentioned it's my hope we can convince plenty of you to help us to test this version. The more people we have testing and reporting bugs the sooner we can get this rolled out and move on to some of that other cool functionality I mentioned.


    Installing NMM 0.60 and the migration procedure

    As per usual, to install this update of NMM simply run the installer executable and ensure you point NMM to the same folders as your current NMM setup. If you install 0.60 and no mods are showing, you've directed NMM to the wrong folders (and yes, that's your fault and not NMM's).

    This version of NMM is a major update which completely changes the way in which NMM installs and interacts with files in your game folders. Because of this, 0.60 has a migration tool that will upgrade you from your current version to 0.60. The migration tool will automatically install any mods you have active in your current version of NMM and set it up as before. This can take a long time, depending on how many mods you have installed. However...

    Unfortunately, because of the way scripted installers work (the mods you download where they have different options you select in NMM while installing the mod), NMM has no way of knowing what options you have picked once you've installed a mod. So while the migration tool can install any mods that don't use scripted installers easily, during the migration process you will have to interact with any and all mods that have scripted installers again. I understand, for some, that's quite a big ask. I will say this though, once 0.60 is out of Alpha, it will become the only version of NMM that continues to be updated. We will not be maintaining a profile version of NMM and a non-profile version of NMM, so the sooner you get it out the way and upgrade to version 0.60 of NMM, the sooner you can get on and continue to enjoy the bug fixes and new functionality we're releasing each month.


    Reporting bugs and hot fixing

    Considering the major changes in this version of NMM it would be unrealistic to assume it's completely bug free. We think it's bug free right now as we can't find any bugs, but 3 people testing something on specific setups can't test every possible variation. Hence why we need you.

    As always, bug reports should be directed to our bug tracker on the forums. We've setup a specific category for this alpha version, away from the normal NMM bug tracker category, to make this easier for everyone. You can find it here.

    We will be fixing bugs as fast as possible as soon as we find out about them and can replicate them. Remember that the more information you can provide us about the bug, what is causing it and how we can replicate it, the faster we can fix it for you. Keep an eye out in the site news or on the Alpha file page for information about bug fixes.

    Please note: version 0.53, which we released recently, added a new "mod activation queue" system. Version 0.60.0a is not using this mod activation queue functionality as we are still considering version 0.53 as "in testing" rather than stable and we didn't want to complicate things by adding two relatively major changes into one big update.


    Downloading 0.60.0a

    We've decided to release this version of NMM via a Skyrim Nexus file page. In doing so, we can actively track the number of users who have downloaded this version of NMM, so we know how much help we're getting, and also quickly release hot fixes via the same mechanism.

    NMM still works on all games listed on the NMM download page. The Nexus does not have cross-game functionality (one file page for multiple games) and we chose Skyrim Nexus as it is the most familiar and most used Nexus site.

    To download this version simply head to the file page and download the file as normal.

    All that's left for me to say is a big thank you, once again, to all those who choose to help us out and run the risk with installing this new version. While the risk is mitigated with the back-up functionality, we still appreciate you taking the plunge early with this alpha.
  • 17 January 2015

    NMM 0.53 release and fixes

    posted by Dark0ne Site News
    We released NMM version 0.53.0 at the beginning of the week which includes a new mod activation queue window. The idea is to allow you to queue up multiple installations or uninstallations of your mods as well as not locking out the other functions of NMM while your mod actions are being processed.

    This update also enables you to select multiple mods and decide whether to install them or uninstall them, potentially saving a lot of clicking if you're doing a fresh install of hundreds of mods. Where before you would have to select each mod individually and either double-click it or press the activate button on the side, you can now select all the mods you want to install at once using standard Windows drag and drop, shift+click or ctrl+click controls.

    Version 0.53.0 and the hotfix version 0.53.1 showed some performance issues for a minority of users and also lacked a progress column that accurately showed how far along the mod installation or uninstallation was for each mod in the queue. We believe we have remedied both these issues in version 0.53.2 and ask that users update to this latest version and let us know if they're getting any more issues in this regard.

    Version 0.60, complete with mod profiling, is just around the corner and version 0.53 is a necessary stepping stone on the way to this release as it also has a "load backup" functionality. When/if you install version 0.60 it will automatically create a backup of your NMM directory (not its entire contents, for the hard-drive space conscious among you). If, for whatever reason, you need to revert back to version 0.53 from 0.60 this backup file can be used, by version 0.53, to quickly restore your NMM to exactly how it was. Such functionality is extremely important as version 0.60 presents a massive overhaul to the way NMM stores and processes mods. Something that is going to be a bit of a pain if you are someone who has hundreds of mods installed as all your scripted mods will need to be reinstalled and interacted with again. We didn't want people to be afraid about trying version 0.60 for fear that it might break their NMM setup, and thus, we needed to release version 0.53 for this purpose to provide a fail safe backup option for those users who are struggling with 0.60 or come across bugs that prevent NMM from working entirely.

    NMM Version 0.53.2 is up now and available for download.
  • 06 January 2015

    The Nexus State of the Union - 2015

    posted by Dark0ne Site News
    As far as I'm concerned I'm dubbing 2014 "The Year of Stability and Sleep" for the Nexus. The first year where we finally grappled with and beat the recurring downtime issues that have plagued the site ever since its inception back in 2001, and, once we finally stabilised our new cloud and cluster setup around about the middle of the year, myself and the staff have had practically 5 months of uninterrupted sleep. No more text messages or phone calls at 3am telling us the sites are down again. That, in itself, makes 2014 a success for me.

    Indeed, the sites have been so stable that it enabled me to take a bit of a hiatus from the Nexus over the final few months of 2014 to focus on some other businesses, to travel more and see more of my friends across the UK. While I've been away the programmers have been working on some long term projects for the Nexus and the moderation staff have continued to do an excellent job. Before I talk about that and what we've got in store for 2015, lets look at the year of 2014 in figures, because statistics tend to speak louder than words.

    New members: 1,788,372 (26% increase)
    New NMM users: 1,392,499 (50% increase)
    New files: 18,640 (20% increase)
    New file authors: 6,912 (18% increase)
    Total downloads: 268,197,960 (39% increase)
    New Image Share images: 126,775 (32% increase)

    The percentage increases are based on statistics from April 2005 up until December 2013.


    Stability

    In the recent Steam sale we saw traffic reach 8,000 concurrent users at any one time generating 100 page views a second. Both traffic and new registrations exceeded that of last December and during the sale we were gaining a new member to the network every 10 seconds. You wouldn't have known it from using the sites though, where load times remained normal and the sites were fine to browse and download from. The only down-time we've had since July has been caused by issues unrelated to traffic, or us, and have been beyond our control.

    I couldn't be happier. I freaking love sleep.


    Downloads

    Back at the start of September we changed our download serving mechanism away from the original system, which involved running and maintaining 24 individual, dedicated, high-capacity, physical servers located in the USA and Europe to a new CDN system, a distributed system of thousands of shared servers across the globe that we rent from a high profile company.

    This change has cut down maintenance time, increased redundancy, reduced reliability issues and increased our flexibility to allow us to go from serving 4GBit/second of downloads to 10GBit/second of downloads within seconds (which happened in the recent Steam sale...), something that we would never have been able to do before. If we wanted to increase our capacity before we had to buy more servers, set them up and transfer all the files over to them. This takes time. About 2 weeks to be exact, by which point the burst in traffic has past and you're now paying for additional servers that aren't being used. You see the dilemma? Now, if demand increases, we do nothing, the CDN simply allocates more bandwidth to us, logs the amount of bandwidth being used and bills us at the end of the month depending on how much we've used. It couldn't be simpler.

    For the vast majority this change brought improved download speeds (many users are now maxing out their 1MB/second normal member limits where before they were not), yet for a small minority this change was negative, either decreasing speeds or preventing downloads from working altogether. One of our programmers has been working on this issue full-time and the reports from the Download Diagnoser have been invaluable in helping him to better find and fix issues with the downloading system. We've helped out lots of people who were having issues already and we're continue to work on the system to improve it for as many people as possible. We haven't forgotten about it and we are still working on it.


    Nexus Mod Manager

    Feature updates for NMM were relatively mild in 2014. We've continued to squash as many bugs as possible but for most of the second half of 2014 programmer time has been spent on the much overdue profiling system.

    The profiling system completely changes the way NMM installs mods on your system and enables you to choose between different profiles for your games/mods quickly and easily. With one install of NMM you can have different profiles for different users in your house, or different profiles for different playthroughs of your game.

    It's taken a long time to get right because we didn't want to do any game/engine specific .DLL hacks. This profiling system has to work for every game that NMM supports, current and future, and not just for Bethesda games.

    The profiling build, version 0.60.0, is currently in internal testing and due out this month.

    Version 0.60 takes us one step closer to a full 1.0 release for NMM. On my roadmap, the only thing after profiling is a UI overhaul, smartening up, good documentation and a final round of bug squashing, which will likely take us in to 2016.


    Site and file statistics

    We completed a major update to the site back in November that now tracks, stores and presents statistics to mod authors in graph format so they can easily see how their files are doing on the site. We haven't launched this functionality yet as the script we're running to go through all the past 930 million downloads is taking quite a long time, as you can imagine, but once it's ready I'll put a news post up about it.

    For an idea of how it'll work/look you can check out our network statistics page which has been up for a while now. Rather than hide how the network is doing I've chosen to show, in plain sight, how active and busy it is. I don't have any shareholders to impress or reasons to hide the data, so take a look if you're interested. It's also great to direct Nexus haters to when they say, year after year, the Nexus is slowly dying. Sorry guys, but it isn't! ;)

    Mod authors will be able to choose whether to make their file statistics public for all to see or just keep it to themselves.


    New games

    The end of 2014 also marked the end of the first year of the Nexus supporting mods for all games. We now support mods for 147 games, up from 27 at the end of 2013, with 3,400 mods being provided for games we never would have supported had we not moved to our centralised system.


    Moderation

    In March of 2014 I started a thread in the private moderator forums titled "Consistency, transparency and strict guidelines on warnings, restrictions and bans". In it I laid out a set of formal guidelines for moderators, with their consultation, on specific, regularly seen infractions and the actions that all moderators should take in these circumstances.

    While it might sound crazy, up until that point there were no hard set guidelines (there was plenty of informal documentation) on moderating the Nexus that meant there were often inconsistencies in the way the Nexus was moderated depending on the member of the moderation team who was performing the action. Some moderators took a more rehabilitative approach, others a more hard-line approach, compounded by the fact I simply did not have the time to sit down, consult and draw up an extensive and exhaustive moderation document.

    With the introduction of the warning system back in 2013 the moderators were given more tools in tackling infractions of our TOS including, most importantly, the warning system. With this system in place, and with the moderators up to speed with using it, it was possible to formalise documentation to help consolidate and bring all the moderation team under a single doctrine on how to manage the most common infractions on the site.

    I'm very pleased with how this has panned out. Direct reports to me of moderator "abuse" dropped substantially in 2014. And I mean substantially. The warning system is being used for all but the worst of infractions on the site where before users were often simply banned outright.

    I'm aware that there are still some actions that might seem arbitrary and confuse some members as to why one member was warned and another one was outrighted banned. It's often down to the subtle differences. For example, in the case of uploading another mod authors work without permission, if the user seemingly isn't aware they're breaking the rules when doing this, and they still get everything else right (like proper accreditation) they'll receive a warning (despite the glaringly obvious terms they agree to when uploading their files...). If the user has uploaded the file and is knowingly aware they've broken our TOS in doing so (e.g. by saying "I know I'm not meant to do this but if someone doesn't like it then they can just take it down) they'll receive an outright ban. Because doing that is just plain stupid. And it still happens. Really.

    While there's still room for improvement, especially in the transparency department which I can't seem to get through to some moderators about for some reason, I'm pleased with the direction the moderation is going on the sites.


    Plans for 2015

    The major site update being worked on at the moment is a Facebook-style notification system that will completely replace the current notifications system. We want to make it as easy as possible for mod authors to be able to keep up-to-date with what's going on with their files, but it will also be usable by all users of the site, mod author or not, as the system will track a whole range of things on the site including new comments, new images, someone linking a video to your files, comments on your articles, replies to your comments, actions a member of your team performs on your files (someone you've given editing access to), actions moderators perform on your files or images, friendship requests, new uploads, versions or articles for files you're tracking as well as the return of author tracking that notifies you when your favourite authors upload new files, images or videos. Work continues on this functionality.

    I will also be commissioning some web design work on the sites to change certain visual aspects on the Nexus. I am not looking for a major overhaul of the sites like back in 2011, but many areas of the site could do with some smartening up, better UI functionality and sharper visuals. On top of that I would like to increase the usable content width on the sites from the current 960px to 1200px, which should allow more content to fit on to the user's visible page at any one time while increasing spacing. It'd be really nice if I could find a professional from within the community to do the work, as the people who know what the site needs the most are obviously the people who use it the most. When I'm ready I'll get a news post up about it, but if you're one such professional and you're interested in a side-job then by all means, let me know. Only professional web-designers though, please.

    Lastly, it is my intention to hire a full-time community manager for the Nexus. While typically this has been my role over the past 13 years I think anyone who's used this site for any length of time and has read my news posts and/or comments in response to them will know that I have public relation skills not too far off Christian Bale on the set of Terminator Salvation. It's simply not my forté. It'd be great to have someone with a happy-go-lucky attitude fronting the sites, representing the community among official game communities and interacting with mod authors from across the internet, not just on Nexus. Someone who's better than I am. Shouldn't be hard. The community manager will liaise between the users of the site, the mod authors, the site programmers and site moderation team to produce new features and ideas to be built for the site and NMM, in conjunction with me, and talk about issues that are directly affecting the community. On top of that they will also represent the Nexus abroad, among other communities, especially official game communities to discuss their needs and see if we can provide support for more games within the Nexus and help new mod authors get their mods on to the site. Once again, this job will go to someone within the community, who knows the community well. Preferably someone with a bit of experience in such things.

    While it sounds like I am shirking responsibilities I merely see it as a logical progression for the Nexus, one that involves a devolution of power and responsibilities away from me so as to reduce the reliance and bottleneck produced when one person is needed to manage all aspects of a business. I've always said it, I'm a jack of all trades, and most certainly master of none. Much like I began hiring on full-time programmers back in 2011 to take over those responsibilities and improve that aspect of the sites, it's very clear to me that this side of the sites needs to improve too, and I'm not good enough for it. I'll still be around, don't you worry, I'll just be around in the shadows, pulling on strings, doing the accounts, sorting out deals with ISPs and hosts, annoying the staff, confusing mod authors. You know, where I belong. What I'm good at.


    In conclusion

    We're still here. We're still working away. The Nexus is getting bigger. The Nexus pretty much works now. I'm happy with the way things are going. I'm now able to sleep.

    Happy new year.
  • 05 November 2014

    Slow download speeds? Please use our Download Speed Diagnoser

    posted by Dark0ne Site News
    Back in August we moved to a new CDN server setup for serving our downloads. For many, download speeds have stayed the same or improved with this move but for some, download speeds have gotten worse.

    We're continuing to work on the download setup within the CDN as we continue to learn all the little intricacies involved in such a complex setup but we're struggling to troubleshoot a lot of people's download speed issues, a lot of the time, because we're not getting the right sort of feedback. For example, simply saying "I'm getting bad download speeds" isn't useful at all for us!

    As such, we've coded a small piece of stand-alone software which we've dubbed the Nexus Download Diagnoser. It's a very simple piece of software that will run a download speed test against our CDN server and generate a report for you to post for us to see which will provide us with a lot of information. Hopefully useful information that will help us to understand what is happening and why your download speeds are slow. You do not need to install the software in order to run it and you can download it here. Remember, only download it from a nexusmods.com domain, never from anywhere else.

    In the interests of security and privacy we've kept things very simple for you. The Nexus Download Diagnoser will obviously need to be able to connect to the internet so it can run the download speed test (so update your firewall as necessary) but it will not send or report any information automatically to us. If you want to share the information, you'll have to willingly post it to us, and you can see all the information in the report so you know exactly what you're sending us. On top of that we've signed the software with our "Black Tree Gaming Ltd." key, just like NMM, so you know the software is legitimately from us and not from someone malicious and we're releasing the source code of the software with the executable just for your own peace of mind. If you really really don't trust us for whatever reason, you can check the source code yourself!

    Hopefully with enough reports we will be able to form a decent picture of what's going on and work towards sorting it out.

    For more information about the Nexus Download Diagnoser and where to post your reports please head to the specially setup forum thread here.
  • 13 October 2014

    Origin offers Dragon Age: Origins free for a limited time

    posted by Tchos Game News
    Possibly to increase interest in the upcoming release of Dragon Age: Inquisition, EA has released the original Dragon Age: Origins on the unfortunately similarly-named Origin, their content delivery system, for free.

    As with any of their "On the House" specials, it will only be free for a limited time of unknown duration.

    This package contains the base game, with no DLC except for the Stone Prisoner.  The other DLC remains full price.

    Get it here, while you can.

    Update: Alas, as of the 14th of October, the promotion has ended.
  • 28 August 2014

    August holidays, Fileserver to CDN switchover and WYSIWYG editor for descriptions

    posted by Dark0ne Site News
    Hello, and let me start by apologising for the lack of updates recently. August is typically our holiday month where most of us take at least a week or two off (or 4, in my case) to have our holidays, rest and recuperate away from the sites. I'll doubly apologise to those people who've sent in support tickets via the contact form. As I am the sole point of contact for support tickets if I'm not around they go unanswered, and a lot have gone unanswered this month (though I've tried to at least cover the Premium Members who are having issues with their payments). So, sorry about that. Service should resume as normal come September.

    I'll also do a shout out to a few of you who spotted me at i52 last weekend at the Ricoh Arena. Always slightly odd when randomers shout out "Dark0ne" across a hall, across some tables during board games or drinking with friends at one of the bars but it was great to meet some of you and share some modding stories. Never be afraid to come up and say hello. I'm not a celebrity, just a regular Joe.

    Having said that, it's not like we've done nothing this month.



    File servers and CDN switchover

    During the Steam Summer Sale, and for several weeks afterwards, we experienced issues with our download servers due to the sheer number of people who were coming to the site to get their fill of mods, a problem we didn't originally have due to the fact the sites would crash before the download servers would. We've been working since then on a switch-over to a CDN service used by the likes of Steam and Eve Online so that this will never be an issue again.

    Earlier on in the week we released a new version of NMM that makes use of this CDN service and also switched our manual download mechanism over to the CDN service as well. You no longer choose the file server you want to download from, the CDN should automatically select the nearest server they have to your location and serve your file from there. Even if we wanted to there's no mechanism to manually choose what CDN server you download from; our provider has thousands of servers across the globe.

    A few people have come on to the forums to say that their download speeds have decreased and a few people have come on to the forums to say their download speeds have increased. We want to give the CDN a chance to balance itself out -- it does have a self-learning process that adapts to the way in which we use the CDN (e.g. the demographics and usage during the time of the day) and it can only learn this information with prolonged use. It needs to learn how we use their resources so it can allocate them accordingly. The hope is within a couple of weeks it will have enough information to sort itself out for everybody.

    This update will require another forced update for NMM users which we'll enforce some time next week. This is because we will be decommissioning our complete set of current file servers as they will no longer be used (and frankly, we cannot afford to keep both the old file servers and CDN service running. We'd be paying double for little reason). What we will do, however, is keep the Premium servers. While it's our hope that the CDN will completely fulfill your download needs I've noticed a few Premium Members stating they're getting slower speeds on the CDN (while others are claiming they can now download at 13MB+ a second...) so, out of respect, these will be kept and we'll add the option for Premium Members to use these again at some point soon.

    I'd love to hear your feedback on this but please note if you're getting slower download speeds than before then please be patient and give it some time. If it's still just as bad in 2 weeks time then please come back and let us know. One of the programmers was getting terrible speeds to begin with and now it's maxing out his connection consistently.



    What You See Is What You Get (WYSIWYG) editor in uploads

    Something that's been requested a lot over the years is a decent way to manipulate the BBCode usable in file and image descriptions to make file and image pages look prettier. Lots of mod authors have learnt the code themselves while lots of others have struggled.

    We introduced a WYSIWYG to the description fields to help with this. The editor should provide tools to help manipulate your description and show you exactly what it looks like on the page.

    It's not been without its bugs, and we know there are still a couple left to squash. If you're really struggling with it/hate it then if you click the "BBCode" button then it'll switch to code-mode, which is basically resetting the form field to how it used to be. No WYSIWYG stuff.
  • 21 July 2014

    Downloading issues update

    posted by Dark0ne Site News
    Following on from an update on the downloading situation made at the start of the month I want to write a quick post to say we're still working on the CDN setup I touted in the original update.

    Many people are (wrongly) blaming the new download system we added to the site back in the middle of June, which worked fine up until the Steam sales made everything go a bit crazy. To enable us to fully discount this update as a factor for those of you who are getting slow downloads and downloading issues we've re-enabled the old file server selection download method which we'll likely keep up and running until tomorrow morning for our own piece of mind. This will enable us to monitor the situation and conclude whether it's having any factor on the downloading issues.

    Talking of the Steam Summer sale, which ended at the start of this month, we're still seeing unchanged and unprecedented registration traffic on the network since the end of the sale which is the driver for all the download issues occuring. For example, yesterday we hit 9,500 new registrations and today, a Monday (which are typically our slowest days), we've already had 6,200 new registrations with 7 hours of the day still left to go. We're not entirely sure what's driving this big influx of new users well past the end of the Steam sale, all we know is it's happening and we're working to sort out our file server infrastructure to make sure it's no longer an issue. For a more detailed explanation of what we're doing please refer to the news article linked at the start of this post.
  • 12 July 2014

    Virus scan reports now visible on file pages, more security incoming

    posted by Dark0ne Site News
    After recent events by a malicious user to upload viruses to the Nexus sites and gain access to Nexus accounts (for reasons completely unknown to us as your accounts are worth very little in tangible terms!) we've been stepping up our public-facing security options.

    Thanks to the great support of VirusTotal, who have given us very generous access to their virus API for free, we've been able to integrate their virus scanning functionality into the Nexus file pages. VirusTotal is an online tool that will scan files you send it using over 50 of the most used anti-virus programs and generates a report showing how many of the anti-virus programs have flagged the file as a virus.

    Any new files uploaded to the site will be sent off to VirusTotal to be scanned and will not be downloadable by users until the report has come back all clean. If more than 4 anti-virus programs used by VirusTotal flag the file as suspicious the file will be sent to a quarantine that will require one of the moderation team to verify the file is either safe, or not safe, before it can be downloaded. This process should only take 5 to 10 minutes, but during this time your file will not be downloadable. While the Nexus has always provided instant upload/download functionality I think a wait of 5 to 10 minutes for added security and peace of mind is a worthwhile sacrifice to make. I'm also aware that there are certain types of mods, especially those that make use of TexMod, that get flagged as false positives quite regularly. While this might be frustrating for you we will endeavour to get your file online as soon as possible.

    The VirusTotal report generated for each uploaded file is easy to access by clicking the new icon present on the file tab of file pages. While the file scan report is quite conclusive you should always have your own anti-virus and anti-malware software installed to compliment this service and it should not replace software already on your system.

    We are currently, slowly, scanning through the entire back catalogue of 250,000 uploaded files on the Nexus at a rate of 20 files a minute which is going to take a week or two. But yes, it is our hope that every file, new and old, will be scanned at some point soon.



    Two-Factor Auth

    Our two-factor authentication system is relatively close to being completed as well but has been put on the backburner for the next week or two while we evaluate the CDN situation. This system will work in the same way as Steam and Facebook; if you login from an unrecognised location you will be sent a unique authentication code via email to verify it's actually you. You will be able to turn this system off in your preferences but we'd obviously recommend having it on for maximum security.



    Staff changes

    After the compromise of one of our staff accounts we have removed the ability for staff to upload files to file pages they are not authors of. If you're wondering why they had that functionality to begin with it was a commonly used feature by the staff to help authors who were struggling to upload their files here for one reason or another. The author would upload the file to dropbox or similar, the staff would download the file and then upload it to their page for them. Staff can no longer do this, but it should ensure that any compromises in staff accounts again would have less implications.

    Many of the staff features are hidden behind a second password gateway that is unrelated to the staff member's username and password. For instance, you can't ban someone without being logged in to a staff account and knowing this secondary username and password. All the staff have been told to never save this information in their browser and to simply write it down on a notepad near their PC. This was already present before the compromise and probably helped to limit the compromise substantially.



    Recent outages

    We've had a couple of outages this week. Earlier on in the week we had a couple of hours of down-time because someone who is in the same Cloud as us had some how managed to take our allocated IP addresses. Without any IP addresses you can't access the sites. We managed to sort that one out and our hosts have told us it shouldn't happen again, but it was completely out of our control.

    Last night was a sleepless night for us as we had some extended down-time as well. Our hosts were attempting to install a lot of expensive new hardware under some scheduled maintenance. It didn't go to plan for them and took a lot longer than expected without even being finished. After that our internal network was extremely unstable and has yet to be resolved. We're working with our hosts to get this sorted on their end. However, you might notice things being quite slow, or slower than usual, today. That's because we're only running on 3 of our 5 database nodes. Given how good they've been to the Nexus over the years we won't be kicking up a fuss over a couple of incidents but it is (here's the silver lining) nice to be talking about down-time that isn't actually something to do with our setup for once.

    We'll get there.
  • 02 July 2014

    Steam sale and downloading issues prompts CDN expansion

    posted by Dark0ne Site News
    Over the past 10 days we've had widespread reports of downloading issues on the sites that has gone hand-in-hand with the annual Steam summer sale promotion that sees games getting massively discounted on Valve's gaming platform. These downloading issues were caused, simply put, by the fact every single one of our 20 download servers was filled to capacity with people trying to download.

    If our registration statistics are anything to go by this year's summer sale was the most successful one yet for Steam. Over the past ten days we've averaged 8,200 new registrations a day including a new Nexus record of 14,505 new members in a single day beating the previous registration record set on November 26th 2011 of 13,570 new members just a couple of weeks after Skyrim's launch. Typically the Nexus will average 3,500 - 4,500 new registrations a day when something special isn't going on.

    When you have a huge influx of new members in a short space of time this has quite a detrimental effect on the file servers. While you can typically only browse the site one page/tab at a time, which helps us maintain our resources on the web servers, you can have many downloads running at any one time. The inherent problem with having a huge influx of new people is that their downloading habits are different to "regular" users. As a new user you want to download a lot of mods all at once. You'll go through the top 100 and look up "best mod" lists on the internet and try and download as many as possible. As a "regular" user you've already done this, your mod list is pretty set, and you're now browsing the Nexus to see what's new, perhaps only downloading one or two new files a day to augment your current mod lists. So having a huge influx of 14,000 new users in a day is like adding an extra million regular users to the site over night for a short term period. The result was 20 file servers all serving 400 concurrent downloads each which meant during the Steam sale we were serving 8,000 concurrent file downloads at any given second and maxing out a 10Gbit line. That number would have likely been much higher if it weren't for the hard connection limits we've set on the servers. Hopefully you can appreciate that's a lot and the infrastructure you need to handle that has to be extremely powerful and resolute. While our file server infrastructure is powerful it's typically designed to handle around 6,000 concurrent downloads, and we average around 4,000-5,000 on a normal, non-Steam sale day.


    Question: Why has this only become an issue now?

    Aha, here's a silver lining (ahem). The reason this is the first time we've maxed our file servers is because this is the first time our web servers (the servers we use just to display the sites) have held under all this traffic. Secretly (ahem), we're patting ourselves on the back that the sites themselves were accessible for practically the entire Steam sale week, which means our new Cloud setup and centralised database cluster is finally working. We're obviously not happy about the file server setup so we're working to sort it out.


    Question: Why weren't you more prepared?

    I thought we were :)

    Back in January I posted that we had completely decommissioned our file server setup and we were moving from a 15 standard download server setup to a 20 standard download server setup, an increase in capacity of 33%. The inherent problem was, because our web servers always used to fail before the file servers did it meant we'd never thoroughly tested our file setup under extreme load conditions. Now that the web servers are up to scratch and holding under these conditions the file servers are taking on a lot more load. And so now we can react.


    Question: Why didn't you just buy more servers when the Steam sale started and it became apparent the load was too much?

    The file servers we need can't just be requisitioned overnight. They need to be ordered, delivered, plugged in and have all the firmware and updates applied before we can even get the entire file database copied on to the drives. That takes time, more time that the Steam sale was going to last.

    Picture the situation like a huge rock festival (lets take Glastonbury as it's only just finished) that comes to a very small town (population just under 9,000) in England once a year. 361 days of the year the local road infrastructure is completely fine, but 4 days a year, when the Glastonbury festival sets up in nearby fields, the roads are completely choked full of cars and the local residents can barely get out of their own town. Is it prudent for the local council to build an 8 lane highway to support a 3rd party event that may or may not happen from year to year that will only be used for 4 days of the year? I think not. In a similar vein, we'd be talking an extra $5,000 expense each month, minimum, to accommodate an event that happens once or twice a year.We can't just say to our server provider "we want these servers during November/December and June/July but for the rest of the year we don't want them". Contracts have to be signed and so on and so forth.


    Question: So what are you going to do about it?

    Last year we spent considerable time, effort and money to sort out our web server situation and we moved to a much more flexible cloud and cluster setup. This has worked. It now makes sense that we continue those efforts and bring our file servers inline with the cloud ethos.

    We're currently in talks with a big CDN service, who already partner with big video game players like Steam, CCP and Wargaming, to get rid of our current dedicated file server setup and move our entire file serving efforts on to a CDN.

    If you don't know what a CDN is I won't bore you by going into detail about what it is (a simple Google search will surely enlighten you!), but I will bullet some key advantages it will have over our current setup:

    • Flexibility and scalability. There's practically no limit to the resources we can use and there's no time delay in making use of them, which means no bottlenecks. We contract for a set amount of usage and any overage due to one-off events, like a Steam sale, is charged at a standard and competitive rate.
    • Less administration and more secure. Maintaining 27 file servers (20 normal, 3 Premium, 4 static content) is a huge undertaking that requires a lot of server administration to keep up-to-date and secure. Moving to a CDN places this responsibility in the hands of a team of qualified individuals who are much better suited for the job, freeing us up to both not worry as much, and not work as much on this issue.
    • Increased performance and localisation. We currently have 14 download servers in the US and 6 download servers in the UK, but the Nexus has a global reach with many users from South America, Asia and Oceania. CDN networks have data centres distributed across the globe that should ensure you really will max out your connection when downloading from our servers, hopefully, irrespective of where you are in the world.



    Question: It sounds good, so why haven't you done this in the past?

    Partly because it wasn't necessary and partly because it costs more. Between 30%-70% more than our current dedicated file server setup depending on how much bandwidth we use. We've come to the realisation from our work on the cloud and cluster setup that this really has to be the future for us, and the added cost, although tough, is necessary to secure the future of the sites. We need to be able to move fast during these sorts of situations which is something we cannot do with a dedicated server setup.


    Question: When?

    As soon as possible. We're testing out the feasibility of the CDN for our setup as I'm typing this.
  • 16 June 2014

    Site updates: stability, premium donations, endorsements, downloading, apostrophes and full-stops

    posted by Dark0ne Site News
    It's been a while since I last wrote about the updates we've made on the sites, of which there have been quite a few, just some have been quite subtle. For my part I've been extremely busy over the past few months (here, have a picture of my feet dangling over the edge of the grand canyon) in between trips to the US and moving home. Thankfully the Nexus to-do list is neverending so the other staff have had more than enough to keep themselves busy. As there's so much I'll just break it down into more manageable sections.


    Stability
    Have you noticed? The sites seem to be up a lot more than they used to and we haven't had to use "maintenance mode" in a long time (past when we accidentally overloaded the servers last week). That's because we finally think we've cracked our database cluster issues. Since the update we made last week we haven't had a single database blip (that wasn't caused by something we did) and needless to say we are freaking happy about this, especially since one of the guys has been working on this for over a year now.

    We're currently sorting out an issue when downloading via NMM and those annoying "retries" that can occur and then we'll be all set, at least we think we will.


    Premium donations
    About a month ago we silently released an update to the "Donations" system that now allows users to donate Premium Membership to other members on the site. At the time I wanted to write a big news post about it but I just didn't have the time, so now I've got to try and be concise within this news post.

    Where before you could only donate direct money to a mod author there's now a second option to "Buy Premium membership for this user" on the user's profile and when using the "Donate" button on file pages.

    You can turn off this option if you are uninterested in being donated Premium Membership from within your preferences. A user cannot have more than 5 years of Premium Membership bought for them, or more than a life-time membership (if you try and donate life-time membership to someone who already has it the site won't let you).

    We've also incentivised this system for the donor as well. If you donate Premium Membership to another user we'll apply ~25% of the Premium time you purchased to your account as well, free of charge. For example if you buy 1 month of Premium Membership for another member you'll receive 1 week of Premium Membership for yourself as well along with all the benefits that come with that (including being a Supporter for life). The time "stacks" as well, so if you buy 4 people 1 month of Premium Membership you'll have 1 month of Premium Membership yourself.

    The Premium Membership system is an unfortunate necessary evil. Without it I just couldn't afford to pay the server bills in this day and age of adblockers and diminishing advertising revenues. I hate asking people for money, or relying on it, so when people willingly share it with me I want to make sure they're looked after as much as possible.

    I get the occasional lifetime member ask me if there's a way they can donate more money to the site. This is the way you can do that while also giving back to your favourite mod authors or users on the site. And thank you.


    Changes to the way manual downloads are served
    We've changed the way we serve manual downloads to you via the pop-up. While before when you went to download a file manually you would be presented with a list of file servers to choose from we are now moving to a quicker, more automated system that will pick the least over-loaded server from a geographic location you've specified in your preferences. If you haven't specified your preferred location then the first time you go to download a file you'll be asked to pick one.

    There are several reasons for doing this:
    • Right now we have 23 file servers. As we continue to expand the number of file servers we provide it's going to become untenable (or very ugly) to show them all to you so you can make a selection.
    • Many people seem to grow attached to a certain file server and will only ever download from that server even when it's overloaded. This causes an unnecessary load on the network we'd rather avoid.
    • It allows us to code algorithms that manage our file server network more closely, ensuring that load is evenly balanced.
    • It removes a click from the downloading operation which in turn speeds up the downloading process.


    We're monitoring this system closely to ensure it's working properly. If you set the wrong location when the system first asks you, or if you want to change locations for any reason, you can do so from your preferences area.


    Endorsement changes
    Something the mod authors talk about regularly in the mod author forums is the low endorsement to download ratio on their file pages. Mods that have millions of unique downloads can have a comparatively low amount of endorsements. There's a multitude of reasons for this, which have been discussed to death, but during the last discussion that happened a couple of months back I said I would get some things added to the site that could help to combat a few of these issues.

    To begin with we've added added an endorsement reminder pop-up to the sites. This is something we had a few years back but it never really worked brilliantly. This time around the pop-up will provide you a list of files you've downloaded recently that you have yet to endorse. You can quickly endorse the listed files from within the reminder pop-up without any page reloads and if you've endorsed all your files already you won't see this pop-up. You can change how often this reminder is shown to you, if at all, from within your preferences (I think we're having a few issues with that right now, which means a few people might be getting spammed by the reminder, which we're in the process of fixing - should be fixed).

    There's a consensus among some mod authors (probably not a majority) that if you've downloaded a mod you should endorse it. I disagree with this philosophy. Because endorsements on the site are seen as a positive rating against a file it doesn't go that just because you've downloaded a file you think it's worth endorsing the file. Because this new reminder system checks if you have any files you have not endorsed, it's not fair to you if you're reminded about files that you have no intention of endorsing. As such we've added a new option when endorsing files that is labelled as "Abstain" and indicated as either a red cross or a thumbs up symbol with a red line through it. Abstaining from a file is just that; you are refraining from endorsing the file. It is not a negative mark against a file. The amount of people who have abstained from endorsing a file is not shown anywhere on the site. So what's the point? By abstaining you are telling the site that you are aware you can endorse the file but you don't want to. It's different from before because now the system knows that you aren't planning to endorse the file and, ergo, won't remind you to do so. You can change your mind later on by going to your download history and changing your abstain to an endorsement.

    Going hand-in-hand with the download changes listed above we've also added a sort of mini-endorsement reminder to the download pop-up window. If you have unendorsed mods you will be presented a list of 5 mods from your download history that you have yet to endorse which you can quickly do from within the pop-up without affecting your current download. We hope to update this system so that the list is organised in such a way that the unendorsed mods shown to the user will also belong to the author of the file you're downloading from first. For example, if you've downloaded 3 files from the author already that you've yet to endorse then you will be shown 5 files, 3 of which will be the ones that belong to the file author. Our early attempts at getting this put in, however, overloaded our servers so we're looking in to implementing this author preference system as soon as possible. If you have endorsements turned off on your file then the endorsement list will not be shown on your downloads.

    It's my hope that by presenting users with these convenient ways in which they can quickly endorse or abstain from endorsing files we'll see an increase in endorsement ratios for files. I'd be interested to hear from authors as to whether they've noticed any sort of noticeable bump in their endorsements over the next week.


    Using apostrophes and full-stops in file names
    We've changed a lot of things since the last time I was directly involved in programming a major recoding of the sites in 2007 but one thing that's never changed is the inability to use apostrophes and full-stops in file names. We've now, finally, updated the sites so you can use proper grammar in your file names and use full-stops to denote version numbers in your uploaded files.


    Smaller updates and bug fixes
    On top of these updates there's also been a number of smaller updates and fixes applied to the sites:

    • Banned members will now be notified of why they were banned when they attempt to login to the Nexus sites.
    • Fixed the filters in the Image Share not working.
    • Fixed an issue where user uploaded images were showing on file pages before the author had verified them.
    • Implemented our IP ban list on the sites when originally it only worked on the forums. If you are getting this error and you haven't been banned in the past then please get in contact via the contact form (obviously letting me know your IP address) so I can sort it out for you.
    • Fixed an issue where the "required files" system wasn't working properly.
    • The tracked files system no longer breaks the home page.
    • Authors who upload mods for games we haven't approved yet will now be emailed when we approve the game.



    Known issues
    We are aware of two ongoing issues with the sites right now. An issue with the tracking and notification system not properly informing users of certain updates to certain files, and an issue with comments sometimes going crazy and not being posted to the proper file page thread. Our work on fixing these is ongoing but is likely tied to our continual efforts to sort out the database cluster and ensure our code is efficiently reworked to synergise with this system.