It’s been a long day but 8 hours after taking the forums offline we’ve now put them back online with the necessary updates in place. There’s still quite a bit of work to do, tweaks and performance enhancements to make, but the majority of the functionality should now be in place and we’ve re-enabled mod publishing and commenting on the sites.
We’ve all got to get used to the changes that have been made by this latest version and we’re still trying to find our way around things and remember all the old settings we had in place before. While we use Invision Board for our forums we’ve made a hell of a lot of edits and tweaks to both the software and the servers over the years to be extremely optimised for the amount of traffic the sites receive. This latest update has wiped out all of those optimisations so we’ve got to try and get it back to how it all was before.
We’ve been testing out the new software over the past couple of weeks and while we can test for most things, we can’t stress test for the 100’s of requests the forums receive every second when things go live. Please bear with us, everything should be as it should be soon.
In the meantime if you’re one of our forum users feel free to do some exploring. I’m sure a lot has changed in the software. Don’t like how the posting forms now look? Me either.
Harvesting can be time consuming and can lead to over encumbrance. Let's look at some mods that fix those problems.
Thumbnail image for this video is 'The Love of Talos' courtesy of xsbullx.
Download links for reviewed mods:
Mage Backpack by Tumbajamba
FrankDema Travel BackPack
Download links for previewed mods:
Destroy the Thieves Guild - Final version
Download links for other mods mentioned:
Frostfall - Hypothermia Camping Survival
Something we rolled out on the sites a couple of weeks ago without any kind of announcement was a new file publishing system when adding file pages to the Nexus. The inherent problem is thus; if you want to (properly) release a mod on the Nexus you need to first start a file page, add a good long description, add and setup all your images, upload your files and setup any videos and mirror links. If you do this without hiding the file (so that others can see it) you run the risk of some hasty (and annoying) users asking you why you haven’t uploaded your files/images yet. If you do it while your file is hidden then your file is losing “valuable” exposure in the new today and recent files pages and columns.
So what we’ve done is added an extra step to the process. You no longer choose whether you want your new file page to be visible or hidden. All file pages start off hidden and you now need to “publish” your mod before it becomes visible on the site. Until you publish your mod your file page will have no comment thread, no upload date and you have all the time in the world to upload and setup everything exactly how you want it before your mod is visible to everyone. It also guarantees you complete control over when the file is released to the public, and as the upload date isn’t updated until you publish the file, you can have your file unpublished for as long as you like without suffering any drawbacks.
For an added example of why this might be handy, take if you will a new mod you’d like to upload to Skyrim Nexus. You think this mod is going to be the next big thing. Now the Hot File system on Skyrim Nexus has a 7 day turn around time, so it only shows the most endorsed files uploaded over the past 7 days. If your mod is 1GB+ in size it might take you a very long time to upload it to the site, anything from a few hours to days depending on your upload speed. So by the time your file is actually up on the site and ready to be endorsed you’ve lost a hell of a lot of time, and perhaps dented your chances at hitting the (sometimes) coveted hot files. With the publishing system in place, your mod only gets properly added to the site when you click that publish button, ensuring you’re absolutely ready to launch. Now, if you don’t care about the hot files or your mod’s exposure then this isn’t a big deal for you, but I know it is for some. So that’s what the publishing system is for, and it’s going to come in handy for the partial down-time I’m about to tell you about.
We’re hoping that on Monday (the 11th of February) we’re going to be taking the forums down for a long overdue software update. Now the forums are pretty integral to the operation of these sites, and lots of functionality is linked to it; the commenting system, the user login and registration system, messaging, chat, and so on and so forth. We’ve been running some tests over the past couple of weeks, doing some mock runs, and we believe we can keep the majority of the site’s functionality operational during this forum downtime. You’ll still be able to browse the sites, download mods, look at images and even read comment threads, but you won’t be able to browse the forums or post any new posts or comments either on the forums or in file or image comment threads. Similarly new registrations will be locked down during this period.
As the forums are going to be locked down during the upgrade we’re also going to lock down the publishing of new mods to the database, as publishing a mod during the forum upgrade would result in the mod comment topic being lost. As such, you will be able to add or edit new file pages, upload files, images and everything else you do before publishing a mod, but you won’t be able to actually publish the mod so it’s visible to the public until the forum upgrade is finished. This only affects new file additions to the site. You’ll be able to manage any current files already published on the site no problem. We’re doing this for the integrity of our data, and for the safety of maintaining the effort you’ve gone into getting your file up on the site.
We’ll be putting notifications up on the site to let you know when we’ve taken the forums down and I’ll try to keep you updated. This is one of these things where any number of problems can occur along the way. We’ve tried to account for everything (and there’s a hell of a lot to account for considering how integral the forums are to the running of the sites) but it’s logical to assume that at some point something unexpected is going to happen that we’ll have to react to. So while I’d like to tell you that this forum downtime will probably only last 6 hours, it could last 12 hours, a day, or hell, even longer. But I hope not.
I want to use this article to talk about the ads on the Nexus network, the introduction of a new ad format and the choices we’re going to offer you in regards to these ads. We’re making Supporter membership an ad-free experience for the one-off cost of £1 and I’ll also be delving into how we’re planning to incentivise the endorsement system a little to try and increase the download-to-endorsement ratio for mod authors. First let me explain our bandwidth throttling system, which is going to be used to incentivise aspects of the site explained later on in the article.
Last October when we upgraded the download system we improved our bandwidth throttling capabilities and upped the bandwidth limit on the site for normal members from 500kb/sec to 1MB/sec. The increase is obviously good, because you should be getting faster downloads, though it’s confused some people as the bandwidth limit is across all your files you’re downloading. So if you’re downloading 1 file, the maximum speed you can get is 1MB/sec. If you’re download 4 files the maximum speed you can get across all your files is 1MB/sec, or 250kb/sec per file if it averages out. So the more files you concurrently download, the slower speeds you’ll get across all of them. I wanted this more manageable bandwidth throttling in place so that we could provide incentives for members of this community who help the site, or who are particularly productive and active.
Something I touched on in my last blog post was the financial side of the Nexus; what I do in regards to money, why I do it and how the sites cope. We’re planning on completely overhauling the server infrastructure of the network this year to massively improve reliability and sustainability into the foreseeable future. The obvious obstacle in such an endeavour is the huge cost, and it’s my job to find ways to not only afford our current overheads but also to save up for this new infrastructure on top of all the current overheads.
As a network we have huge overheads to pay for; 18 super powerful servers, an average bandwidth consumption of 2.8Gbit/second (which is over 800TB of bandwidth a month), 4 dedicated, full-time programmers for the sites and NMM, software expenses, accounting and so on and so forth. As you can imagine, the costs are extremely high, but we do get by.
We get by for two reasons; ads, and Premium Members. They’re both ridiculously important to the survival of these sites, and if either source were to dry up then the Nexus would no longer exist. That’s not scaremongering, that’s just a fact. We can’t run these sites completely ad free like, for instance, Steam Workshop can because, unlike Valve, we can’t subsidise our mod distribution service with the ridiculous profits we’re taking from sales of games on Steam. Mods are our primary dedication, not a secondary service we provide as a means to bolster our primary sales business.
There’s two inherent problems with ads. The first is my lack of focus on direct ad sales, that I touched on in my last blog post, which means that we scrape the barrel in terms of ad revenue. Because we’re not directly selling our inventory to premium advertisers we’re not making mega-bucks off of them like the big gaming powerhouses with their huge ad sales teams. The second is the prevalence of ad blockers on the internet. Software and plugins like AdBlock and NoScript are extremely simple to install programs that eradicate almost all the adverts on the internet. And that’s great for you, because no one likes seeing ads. But it’s not great for webmasters out there who rely on advertising to afford the upkeep on the sites.
There’s various different reasons why people use adblockers. We all hate ads, and if you have a choice then you’d obviously choose to not see ads over seeing ads. Some people worry about security, some people just don’t care about the sites they’re visiting and just want an ad free experience, and others seem to understand that websites need to make money to afford the upkeep but think that their adblocking doesn’t affect revenue as they never look at/click on the ads when browsing anyway. And that’s where you’d be wrong. To clarify the point, I’ll quickly explain the three main advertising revenue models; CPA, CPC and CPM:
- The Cost Per Action model pays out money for a specific action a user might perform after clicking an ad. Say it’s an ad to sign up for a second hand car newsletter. If you click the ad and add your email address to the newsletter, there’ll be a payout for the displayer of the ad.
- The Cost Per Click model pays out money when a user actually clicks on an advert on a site. Pretty self explanatory.
- The Cost Per Mille models pays out money per one thousand (mille) impressions shown. You don’t need to interact with the advert to be paid, you don’t even need to click on it. If that ad gets shown to the user, you’ll get paid. And that’s what we use.
So as we use the CPM model of advertising, you can see that even if you never look at an ad and certainly never click on one, just by that ad being shown you’re helping to support the site through ads. The same cannot be said if you use an adblocker because the ad never gets shown.
In this day and age I have a lot of respect for those people who could install an adblocker but don’t, or for those people who have an adblocker installed but choose to disable it on the sites they like and want to support. Even if you can’t afford a Premium Membership, If you do this, you’re supporting the Nexus financially and helping to pay for the upkeep of the sites. Without you, we couldn’t afford to keep these sites running. And I respect that.
It’s at this point you hit this sort of moral dilemma. You’re a webmaster running a site that absolutely depends on the advertising revenue from the sites in order to keep them up and going. There are people out there either not using adblockers or who are using adblockers but have them disabled when browsing the site. These people get a lesser browsing experience with ads, but they’re helping to support the upkeep of these sites. And then there are people out there who are blocking the ads on this site, and getting a better browsing experience for it, and who contribute nothing to the upkeep of these sites. Is it fair? If you’re an advocate of adblockers you’d probably say yes, it’s fair. As a webmaster who relies on advertising revenue, I’d say no. It’s not fair.
The inherent problem is thus; if everyone visiting these sites blocked the ads then this site would no longer exist. Once again, that’s not scare or guilt mongering, that’s a fact. I don’t think it’s fair that people who choose to see the ads should get a worse experience browsing these sites than those people who choose not to see the ads but who don’t help to support the sites.
So what am I going to do? Two things.
First of all we have two different types of financial incentives on this site, that can be purchased through your account on the forums. Supporters and Premium Membership. Premium Membership provides all sorts of bells and whistles to your Nexus experience like uncapped downloads, multi-threaded downloads, Premium only file servers, full download history and so on. It ranges in cost from £2.49 a month (around $4) to £39.99 for a lifetime membership (around $64). If you buy a timed Premium Membership, let it expire and then don’t renew the payment we move you automatically into the Supporter membergroup. You have supported these sites financially, and you deserve to be recognised even after your payment has expired.
A Supporter membership on the network costs £1 (around $1.60). It’s a one-off payment that makes your account a Supporter account for life and it works on all Nexus sites, current and future. You get a couple of very minor bonuses like access to the Supporter-only Image Share section and an increase in your Private Message capacity from 100 to 500. We’re now going to make it so that Supporter memberships also provide an ad-free experience on the sites.
Why make Supporter membership ad-free? It’s very simple really; if everyone who used this site for more than a couple of downloads donated a one-off £1 payment to the upkeep of these sites then not only could we get rid of all the advertisements completely, but we would have around 15 programmers, a server network that would only go down if Skynet attacked, no download speed limits and an all round better experience. So it makes sense that, for the princely sum of £1, you have more than paid your dues to the Nexus and you should be rewarded with a completely ad free experience.
When I look at the reasons people give for why they use adblockers I often see “ad free membership is too expensive”. I don’t think that argument could be used here now.
That’s the first thing we’re going to do, and we’ll implement that within the next month. The second thing we’re going to do is drop the download speed limit on the Nexus to 750kb/sec, a 25% drop on the current 1MB/sec download speeds. What we’re then going to do is run some code that will tell us if you’re seeing our ads or not. If you’re seeing our ads, we’re going to provide you with a bonus download speed of 250kb/sec, to take you back up to the original 1MB/sec. You’re helping to financially support these sites, and I honestly believe you deserve a service above those people who do not.
I want to be careful here and ensure that we don’t cripple the crap out of the service we provide to people using adblockers. If you want to use an adblocker and block the ads on this site then that’s completely your prerogative, and I don’t want to do something silly like say “you can’t download unless you turn it off”. That’s dumb and it causes unnecessary friction between you and the site, and that is not my intention. My intention is to recognise those people who do not block the ads on this site and incentivise the support they’re showing by having the ads turned on. You use an adblocker out of choice, and now you can decide whether not seeing ads on this site is worth a slower download speed or not. I’m sure most of you will still prefer to not see ads for whatever reasons you might have to use the adblocker in the first place. That’s absolutely fine, that’s completely your choice, and for you, your use of these sites will basically be exactly the same apart from some slightly slower download speeds.
Once again, that’s something we’ll implement in to the sites over the coming month.
Before I move on from talking about adblockers, I just want to add a quick note. Some people even go the extra mile and ask me if clicking the ads every once in awhile would help us more. While I thank you very much for even considering the idea my answer is always the same; please only click on an ad if you’re actually interested in what the ad is about. It serves no purpose for you or us if you just click an ad for the sake of clicking it. It artificially inflates our numbers and doesn’t help the advertisers who want to see a return on their investment. So once again, thanks a lot for even thinking about it, but honestly, don’t bother unless you actually like what you see.
Lastly, on the ad front itself, I want to announce my intention of adding a new ad format to the sites on top of the current ad formats. Site skin ads. You’ve probably seen them around on other sites if you run at resolutions above 1152 pixel widths; they’re ads that make use of the blank spaces either side of the central content area. It is my intention to start selling this space to gaming-related only advertisers in the not too distant future. Now, before you grab your torch and pitchfork I’m going to put your fears to rest. I know site skins can be really annoying. I know, because they annoy me when I visit other sites. So this is going to be optional, and you’re going to be able to turn them off in your preferences. And why would you not want to turn them off? Well, aside from the fact they’re going to help to pay for all the new servers we’re buying, we’re going to incentivise it with the bandwidth throttling again. If you choose to leave the site skins on, we’ll give you a further 250kb/sec bump on your bandwidth, for a total of 1.25MB/sec speeds. Ooooo, aaaaaah. And we won’t penalise you if you turn them off. So if you use an adblocker and block all the ads, you’ll get download speeds of 750kb/sec irrespective of any settings. If you have ads on but block the site skins in your preferences you’ll get 1MB/sec. If you have everything on you’ll get 1.25MB/sec. It’s not exactly much, I know, but the thought of providing (even slightly) better service to those people who grin and bare the pain of ads at least makes me feel better, while helping to support the upkeep of the sites.
Right, I’m done talking about ads now. I’m already 2,200 words in and I’ve still got to talk about how we’re going to incentivise the endorsement system.
So the endorsement system. The endorsement system is there to provide mod authors with a feel good factor; a feeling that what they’ve uploaded to the sites is actually being used, and liked, by others. It tells them that they’ve done well. So what’s the problem? The problem is the very low download to endorsement ratios currently on the site.
If we take the most endorsed file on Skyrim Nexus as a case study, SkyUI, we see it has 32,501 endorsements and 1.01 million unique downloads. So of the 1.01 million individual people who have downloaded the mod, only 32,500 have endorsed it. As a percentage that means only 3.2% of people who downloaded SkyUI actually bothered to endorse it. Now you could try to argue that perhaps only 3.2% of people actually still use SkyUI, or actually like it, but I think you’d be very, very wrong. Using SkyUI as an example is slightly biased as they’re at the high end of the percentage scale, most other files have much lower download to endorsement ratios, and these sorts of percentages can dishearten some mod authors.
So what can we do to try and get more people using the endorsement system? Let me think, let me think...oh yes, our bandwidth throttling system! Here’s the plan:
We’re going to provide increased download speeds to those users of the site who make use of the endorsement system extensively. If you have endorsed 75% or more of the mods you’ve downloaded over the past 30 days then we’ll give you a speed increase. Say, an extra 250kb/sec. But wait, there’s an obvious problem with this, isn’t there? If you say to people “endorse all your mods and you can download faster” what are they going to do? They’re going to endorse all the mods irrespective of whether they think the mod was worth endorsing or not. I can see lots of mod authors liking THAT idea, but I don’t. I want the endorsement system to at least have some sort of meaning to it, rather than as a means to an end for faster download speeds. Case-in-point, Valve added a badge to Steam that you could get only if you rated mods on the Steam Workshop. And you needed this badge in order to become a “Pillar of the Community”, whatever that is. Turns out people care about this stuff, so what happened? People (some of whom didn’t even use the Workshop) had to go to the Workshop and find a mod to rate before they could get the badge. So they went to the front page, rated any mod they could find and left. What’s the freaking point in that? The result was a load of mods that were already on the front page for being popular now being even more popular because people were rating the mods without even playing them. Facepalm moment.
So what are we going to do about it?
We’re going to change the endorsement system. You will now either be able to endorse a mod, or abstain from endorsing it. What’s abstaining from endorsing? Basically it’s a conscious choice you’ve made that means “I don’t think this mod deserves an endorsement, but I’ve still used the endorsement system”. It is NOT the same as a thumbs down, or a down vote, or anything of the sort. The total number of people who have abstained from endorsing a mod will not be displayed anywhere on the sites, and it’s only for our records. Why? Because if you download 100 mods over the course of 30 days and you only think 50 deserve an endorsement by your own standards, then it’s unfair that you shouldn’t get a speed boost just because you haven’t hit a 75% endorsement rate. So if you endorse 50 mods and abstain from endorsing 25-50 mods, you’ll get the speed boost. Make sense? I hope so, I’ve just written 2,700 words non-stop and even I’m struggling to understand what I’m writing.
So now we’ve got all sorts of different download speeds you can achieve based on the choices you make on this site. And I like offering choice. So lets run-through the scenarios and the download speed limits that will be provided based on your choices:
- You block all ads, you don’t use the endorsement system, 750kb/sec
- You block all ads, you use the endorsement system, 1MB/sec
- You do not block ads, you turn off the site skin ads, you don’t use the endorsement system, 1MB/sec
- You do not block ads, you turn off the site skin ads, you use the endorsement system, 1.25MB/sec
- You do not block ads, you turn on site skin ads, you don’t use the endorsement system, 1.25MB/sec
- You do not block ads, you turn on site skin ads, you use the endorsement system, 1.5MB/sec
We’ll try and add some bars/meters/notifications to the UI of the site in your member area and when you’re downloading that tells you what limit you’re currently hitting and for what reasons. It should also help to inform the myriad of people who don’t read these news articles what’s happening, and what they can do to increase their download speeds.
I know that all this talk of incentivisation through bandwidth throttling has a few gaping holes in the plan. What if you’re on dial-up, or 1MB/sec download speeds are a dream to you? All this talk is pretty pointless huh? Yeah, it is. Sorry about that. What if you don’t care about download speeds? Well, yeah, pretty pointless too. But I wanted to look for ways that we could incentivise aspects of this site for you, and I’ll continue to look for ways to incentivise things in the future. This is the first step, I’m sure, of many.
Talking about money, ads, adblockers, penalties and incentivisation can often leave a bitter taste in your mouth, but it’s one of these articles that I think can be quite informative. It’s often very hard to look at arguments from angles other than your own, and I hope that you can see my side of the argument, even if you completely disagree with it.
Oh the irony of it. I wake up this morning ready to write out my plans for ads on the site and instead find Google have slapped a Malware warning on many of the pages of Skyrim Nexus, due to the ads. How tittingly annoying.
First of all, sorry about the mess. Second of all, it should be “fixed” soon (not that anything is broken). So what’s happened?
When Google sends its bots throughout the web to maximise their search content the bots also run all sorts of tests. One such test is a malware test to see if the site is trying to do anything naughty with your computer. The site could be naughty because it’s deliberately trying to be naughty, or because it’s been hacked by naughty people. Either way, it flags the sites and servers that are naughty and stores them in a big database. This morning on a routine Google bot visit, the bot was served an ad that came from one of those flagged sites; an adserver Google has flagged as serving malware. So naturally, rather than Google blocking your browser from just seeing that ad server, Google has blocked half of Skyrim Nexus, irrespective of whether you are using an ad blocker, anti-virus, anti-malware or any myriad of possible solutions that mitigate and dissipate any threat. Annoying, huh? Queue the face palm.
So why are we serving an ad from a server flagged as serving malware? Well, that one’s even more complex, but I’ll try and break it down for you as quickly as possible. When you’re not doing direct ad sales (e.g. calling up Bethesda’s ad agency and asking if they’d like to purchase inventory on the sites directly), you pass your advertising inventory to ad firms who sell your inventory for you. They’ll give you high quality ads most of the time, but if they can’t sell your impression they’ll pass it on to ad markets, where 100s of ad suppliers are congregated in a form of bidding technology, where the highest bidder for that impression gets to show their ad in that space. This is naturally all done by technology and maths, lots and lots of maths, in a split second. Ad firm A has decided your impression is worth $0.001. Ad firm B has decided it’s worth $0.0011. Ad firm B gets to display their ad to you. Unfortunately, if Ad firm B have been hacked, or have stupidly not vetted the ads they’re showing properly then Google might flag them as naughty malware suppliers. If a Google bot comes to your site when an ad is shown from that naughty ad supplier, BAM, you’re a malware supplier too. And that’s what has happened. Wondering why Google doesn’t just block any access to that supplier, rather than blocking every site that might have used an ad from the supplier? Me too.
I’m always working with my ad suppliers to increase the quality of the ads and remove any crappy suppliers. Today we were unlucky, and now we’ve got to wait for the Google bots to come back around, recheck the sites and confirm that they’re safe to use. They are safe to use. We haven’t been hacked or infected, and you can tell pretty easily. Go to Oblivion Nexus, or Morrowind Nexus, or any other Nexus site. Notice how Google is saying nothing about them? Skyrim Nexus was just unlucky. Showing the wrong ad at the wrong time.
As always you should be doing things to mitigate any damage that malicious code can do to your computer. An adblocker will not protect you from a hacked site, just hacked ads. You should have a good anti-virus program installed on your PC. And that copy of Norton anti-virus you got with your PC isn’t one of them. Any one of AVG, Avast, Comodo, Panda or Avira (which are all free) coupled with Ad-Aware and Malware Bytes will keep you super safe 99.99999% of the time. An adblocker will mitigate your chances of getting attacked by a malicious ad, but not completely prevent it and it will also severely dampen the Nexus’s ability to stay afloat financially. If you want to use one, that’s your choice, but you do hurt the Nexus by doing it.
Once again I apologise for this annoying situation, and we now wait on Google to come back around and sort this out.
I love most things about Skyrim, but here are three things I do not like, and luckily there are three mods that fix those things.
Thumbnail image for this video is 'Ignis II' courtesy of kaldaar.
Download links for reviewed mods:
No Sneak Attack Sound
Download links for previewed mods:
WIP - Immersive Bounty Hunting
Yes, I’m sorry, here’s another one of those 5 page, 2,500 word rambling nonsense blog posts I like to make from time to time to show you that I’m not dead and we’re still moving forward. If you have no interest in servers, money or talking about my narrow views on corporate greed, then I would suggest skipping this one!
We’re about one month into the new year now and I wanted to share with you one of our main priorities for this year, as it’s as important to us as it is to you.
Perhaps one of the Nexus’s biggest pitfalls since its inception has been the stability of the service. I don’t have any uptime statistics on hand to report on but I think it’s pretty safe to assume that we’re not hitting the 99.97% uptime that most big sites try to aspire to. Not only is it annoying because you guys can’t access the site, or particular services on the sites (like the downloads) at random points, but it’s annoying for the staff to be awake at 3am in the morning restarting services and troubleshooting database crashes, and it’s annoying for me to be running a service that isn’t 100% reliable. Moving forward I’d like the Nexus to be taken seriously by game developers, and it’s hard to be taken seriously when you can’t guarantee service.
It’s not as though other sites and companies out there don’t have reliability issues. I’ve been locked out of DotA 2 many times recently because the Steam servers have been down, for instance, and that’s from a multi-billion dollar company like Valve. But the problem with the Nexus is that it’s a regular occurrence.
I can attribute this problem to all sorts of systemic issues right through the Nexus, from the way I’ve set up the business to the way we’ve run the site and prioritised things. It’s not as simple as pointing the finger at the lack of server resources, or hardware failures, or the code, or the DDoS’ers, or being a victim of our own success or any one thing, it’s a multitude of things. But the highest priority of this year is to resolve this issue and make the Nexus as stable and redundant as it possibly can be.
So lets take a look at some of these systemic issues and then I’ll explain what we’ve been doing, and what we will be doing to make the situation better and ensure the Nexus is future-proof for the foreseeable future.
The largest factor of all with the stability issues has been the sheer popularity of the Nexus sites coupled with having an inappropriate server architecture to accommodate the demand on sites that are extremely database (and ergo, resource) intensive. I’ve blogged many times on the popularity of these sites and the difficulty in keeping them up with the load placed upon them, so I won’t bother to go into the numbers again. So you’re thinking, “OK, why not buy more servers then?”. The answer isn’t in needing more servers, the answer is in needing to restructure the architecture of the servers and network we currently have so that the combined resources of all the servers can be used to keep the sites going.
Right now we have a situation where we have lots of lower traffic sites (Far Cry, Neverwinter, The Witcher, Morrowind and so on), some high traffic sites (Forums, Oblivion, Fallout 3, New Vegas) and one super mega ridiculous traffic site (Skyrim). Typically speaking almost every site on the internet can fit onto a single powerful dedicated server. Depending on the size of the sites you can even fit hundreds or thousands of normal sites onto a single dedicated server. We have 6 servers dedicated to just serving the Nexus sites (not the file servers, we’ve got 12 of them!). The problem though, is that Skyrim Nexus, and the forums, are not normal sites and they’re at a point where they can no longer fit on one single dedicated server. Similarly we’ve upgraded the hell out of the servers so we can’t make them any more powerful than they are now.
We’ve reached this point where Skyrim Nexus has outgrown being able to run on a single super-powerful dedicated server, so how do we resolve this issue? The solution is in server clustering, which is a technology that lets you pool together the resources of multiple servers to act as one super mega server, much like SLI allows you to connect up and combine multiple video cards in your PC to dramatically increase your frames per second. Unfortunately server clustering isn’t as simple as connecting an SLI bridge connector to your video cards. It’s a lot more complex.
Server clustering is not only complex, it’s also expensive. We have 6 web servers at the moment. We can’t just flip a switch in the servers we currently have and turn on clustering. We’ve got to buy completely new servers, set them up for clustering and then transfer the network on to these clusters. That means running our current setup in parallel with the new one until everything is transferred which means paying for the original 6 servers plus the new servers we need to buy to form our clusters. That’s a lot of money.
And therein lies another systemic issue with the way things are setup. Money. The Nexus sites have remained completely independent; free of corporate interest and investment for its entire 11 years and it shall remain so for the very foreseeable future. The only investment these sites have had was the initial £10,000 I chucked in to the sites when I rebranded the sites as the Nexus back in 2007. I’m the sole owner and sole decision maker of the sites. There’s no outside interest, board of directors or investors pulling the strings behind the scenes. Similarly no game developers have any influence or sway over me. The buck stops at me.
If I wanted to I could make a business plan (I don’t have one, by the way) and go to Silicon Valley, pitch the idea to a load of private and angel investors, secure (potentially hundreds of) thousands of dollars in investment money and make a proper business out of it like many gaming sites and networks have done over the past few years. However, I then become answerable to shareholders and investors who are looking for a return on their investment as fast as possible. To be frank, F’ that.
Similarly it’s just me and 4 other programmers working on the Nexus. We have absolutely no one doing ad sales. I mean it, we have no ad reps at all. Others in the industry gawp at such an oddity. That’s why the ads you see on the site (if you don’t block them) are pretty crap, and in return we get pretty crap rates. While other networks have entire ad sales teams securing them crazy $1-$10 CPM rates on their ads, we don’t. We don’t get anywhere near that. So why don’t I hire some ad reps to better sell the inventory and use that money to pump it all back in to the sites? The reasoning is very similar to my private investment reasoning; when the focus of your business is on increasing your ad sales, and on ensuring a prompt ROI to your investors, you begin to lose sight of what your original goals were and instead focus on one very simple goal: making money. And money isn’t what I’m doing this for. Indeed, if money was my aim I’d be doing all these things I just mentioned, because the Nexus would be a cash cow. Case-in-point; I know sites that have 5-10 programmers working for them and 25-50 ad sales reps. Yes, that’s a 1:5 ratio of people working on content to people working on making money. To me, it’s crazy to have more people working on selling than actually improving and producing the content that sells. But that’s business for you, and I’m not a good businessman.
What this all breaks down to is limiting the stakeholders in the Nexus. Right now you guys, the people who use the site, the mod authors, the downloaders, the people on the forums, YOU are the Nexus site’s biggest stakeholders. If I don’t appease you then these sites cease to have a point. If I seek private investment, or start directly selling the Nexus site ads then my biggest stakeholders become the shareholders and the advertisers on these sites. My focus gets shifted from serving and pleasing you, the users, to serving and please people who have no interest in you. And the point of the site changes from being about modding to being about making money. That’s not what I want at all. There may come a time in the future when direct ad sales and private investment are exactly what the Nexus needs, but that time is not now.
You guys are really, really good when things go wrong on these sites. By and large the reaction is often tame and supportive rather than stressed and raging. I like to think it’s because you know we’re not some corporate mega-money machine that’s cutting costs by cutting corners, but just 5 gamers trying to provide the best service we can. I don’t want to change that, because being greeted with “Ah that sucks, I hope you can fix it soon! Good luck!” is better than being attacked with “WTF this is the worst pile of crap I’ve ever used and you should be ashamed” when something goes wrong.
I want to retain that focus on you guys being the primary stakeholders in the future of the Nexus, which means it takes a lot of monetary planning and saving to buy more servers and invest in expensive technology like server clustering while other sites can simply throw their private investment resources or ad sales money at the problem. That’s why it takes a long time. Avoiding private investment and direct ad sales is a conscious decision that isn’t without its pitfalls, but one that I think is worth it to retain the core values of what these sites were set up for in the first place; to provide mods authors with an easy platform to share their work with others that will stand the test of time.
So setting up server clustering is currently one of our biggest priorities, and we’ll be setting that all up in the very near future, but in the run up to all this we’ve spent (and are still spending) some considerable time right now focusing on the software side of things.
Over the Christmas period, while the “normal” members of the staff were enjoying a forced two week break, Axel was working on an error logging system for the Nexus.
One of the most annoying aspects of bug hunting and troubleshooting is when someone leaves a comment on one of these news articles, or on the forums, or on the tracker that something is broken. Typically it will go something along the lines of “Downloads are broken at the moment”. To which my response is “.......” , followed by much hair pulling. Downloads are broken? What downloads are broken? On what site? What files? Is it all files or just one file? Is it only happening on one Nexus site or all Nexus sites? Is it just small files or large files? What error are you getting? Is it happening 100% of the time or just some of the time? What browser are you using? Have you tried using another browser? Have you tried turning your PC on and off again? Have you tried logging out and in again? Did downloads ever work for you? Have you installed any new browser plugins, firewalls or anti-virus programs recently? What time did this happen? These are but some of the questions we need answered to actually troubleshoot the issue, especially if all the staff try downloading and it works fine.
What I wanted was a system that would aggregate and parse all the error logs the servers produce and present it to the staff in a system that can help us easily pinpoint not only errors and problem areas of the site, but also pinpoint specific times when the sites are worse than others to help us troubleshoot the problem. Typically the error logs that servers produce are all flat-file text documents. Line after line after line of errors with timestamps that can run up to gigabytes in size. It’s extremely hard to make use of these error logs without having a system to properly display the information, and there’s nothing worse than being told something isn’t working when it works for you and wondering if it’s affecting just one person, 1% of people 25% of people or even more. With the error logging system we can now see that “wow, yes, at 10am today we had 5 times more errors than we usually do”. It’s helping us to investigate things more and we’ve already applied numerous hot fixes to the sites over the past month that have patched up errors and slow areas of the site.
Similarly at this very moment we’re working on some more improvements to the downloading system for both manual downloads and downloads through NMM. Right now, if one of the file servers has hiccupped it can be a real pain in the ass trying to download something. These hiccups generally only last for minutes at a time, but during that minute it can make it hard to download any files, especially small files. With that in mind, we’re going to present the file server selection screen on all files now, irrespective of size. If a file server is down, you can quickly select another one to use. We’re also trying to implement a seamless redirect system incase you choose a file server that isn’t working for whatever reason. If the file server you choose isn’t working, the site will simply try another one until it finds a file server that is working. You won’t really notice a difference (except far fewer, or no errors at all!), although if you typically get fast speeds on only one or two servers you might get slower speeds as your download might be served from a different file server from the ones you normally pick, if they’re down.
This concept of seamlessly being moved to a server that works is very similar to our plans with the sites and servers in general. Right now, if there’s a hardware or network failure on one of the servers a Nexus site is on, that Nexus site becomes unavailable. Once our full clustering solution is done we’ll have a load balanced, redundant solution that means all sites are being served from all servers. If one server goes down, the other servers pick up the strain but the sites still work. It reduces the bottlenecks and also reduces our single point of failure problems. And finally, clustering restores the status quo by making “buy more servers” the viable solution to our strained server issues. If the network needs more power you just tack another server in to the cluster and you’ve boosted the resources available to the network. That’s not possible with the current system. So for me, this is quite exciting. For you, it’s more like “I don’t care, just make it work”.
I think I’ve gone on for long enough now. When I wrote my blog piece on Nexus development and expansion philsophy I was interested when I got a lot of emails and messages from people who were surprised I thought that hard about the process. I like to use these blog pieces to indeed show you that yes, I don’t just sit around all day watching stocks go up and down on my monitors and playing DotA 2, but I think pretty damn hard about this network. The choices I make aren’t just knee-jerk “oh, I guess we’ll just do that then” solutions but plans that have been made out and expanded upon over a long period of time in consultation with others. We’ve wanted to do server clustering for years now (indeed, I mentioned it in that first YouTube video I did), but we’ve only now been in a position to actually afford it. And that’s why I’m excited, even though the topic is pretty boring.
SkyUI is the most endorsed mod on all of the Nexus sites, and with the addition of the Mod Configuration Menu (MCM) it has become even more essential on most peoples Skyrim mod list. However there are people who still prefer the original games UI and for them the lack of the MCM is a major setback.
But have no fear, the SkyUI team thought of that and have provided a way for everyone to enjoy the advantages of MCM.
Skyrim Script Extender (SKSE)
Dragonborn just became available for pre-purchase on Steam, and will be available for playing February 5th. Grab it now for 20 $/€!
In case you don't know what Dragonborn is all about yet, here's a small description for you:
"With this official add-on for The Elder Scrolls V: Skyrim, journey off the coast of Morrowind, to the island of Solstheim. Encounter new towns, dungeons, and quests, as you traverse the ash wastes and glacial valleys of this new land. Become more powerful with shouts that bend the will of your enemies and even tame dragons. Your fate, and the fate of Solstheim, hangs in the balance as you face off against your deadliest adversary – the first Dragonborn."
Thanks to Camonna Tong for the find!
It’s been a month and a half since our last update to the Nexus Mod Manager, but, being the merciful slave master that I am, the Nexus programmers were given a two week break over the Christmas period so they could remind their friends and loved ones what they look like. That’s all over now however and it’s back to business as usual. I’m here to announce the fruits of our labours as we release NMM version 0.40.0.
This version of NMM includes our preliminary work on a feature that’s been begged for by some and outright rudely demanded by others since we first put NMM out in the public domain a year and a bit ago. That feature is mod categories, and the ability to organise your mods in to your own categories.
After updating to the latest version, NMM will ask you if you’d like to use the default Nexus site categories or whether you’d like to set up your own from the start. If you choose to let NMM specify your initial categories for you then your mods will be categorised based on the categories your mods are under on the Nexus site the mod is from. So if you have 10 armour mods, 10 weapon mods and 10 NPC mods the category manager will place your mods into those three categories automatically for you. If you choose not to let NMM set up your categories then all your mods will be in the “Unassigned” category and you can start from scratch. It’s probably important to note if you do choose to let NMM do your initial category setup you’re not forced to use those categories. You can rename them, delete them, add new ones and move mods around the categories with no limitations. The initial category setup is just a template for you to use, so it’s not one method or another.
The category manager button to the left of the mods tab allows you to do multiple different tasks to influence your categories including the ability to add more categories, collapse/expand all the categories, reset your categories to the Nexus defaults, reset all your mods to the “Unassigned” category, remove all your categories entirely and toggle hidden categories (categories with no files in them).
Hidden categories are probably worth expanding upon. If you select to use the Nexus categories NMM will add all the categories the Nexus site for that game uses, even if you don’t have mods in that category yet. These categories will be hidden unless you want to see them. However, if you download a mod from a category that hasn’t been used yet that category will then become visible, as it’s now being used. So NMM will automatically place your mods in to the correct Nexus category once the download is complete.
Something I desperately wanted in this version before I would allow it to be released is the ability for you to drag and drop your mods into categories, and also to multi-select mods by using the ctrl or shift keys like you would in windows to select multiple files. By using shift and ctrl you can select more than one mod to move at any one time, which should save you some time if you’re one of those horder-types who has millions of mods.
If you don’t care about categories or you prefer the old way of managing your mods then that’s no problem, we’ve added a button that will quickly let you switch between category view mode and old view mode.
When introducing this feature you might have noticed I mentioned this was our preliminary work on the category controls. The category view mode required a whole new control method to be made in NMM, which has not only enabled categories to work but has also enabled column sorting as well (you can now click the column headers to sort your mods ascending/descending by name, install date, version, author, etc.). However, column sorting does not work in the old view mode, and I’m aware that the old view mode will provide a quicker way of seeing things like ordering your mods by the last install date so you can quickly see which mods you last installed. What we haven’t had time to do is bring the new control method in to replace the old method, so we’re running both controls side-by-side at the moment, the old control method with no column sorting and the new control method with categories and column sorting. We’ll continue to allow you to switch between a categorised mod management option and a non-categorised mod management option, but the non-categorised option needs updating. We’ve got several other little tweaks and improvements we’d like to make which means this feature is probably about 80% done. However, as this version of NMM also contains quite a few bug fixes and a couple of other notable features we thought it would be better to get it out now, in this state, rather than later when it’s fully complete. So have no fear, the book on the new control method and category management is not yet closed.
Also worth noting in this release, we’ve added load order importing and exporting functionality to the plugin list and, more importantly (for me), NMM no longer looks like it’s crashed by becoming unresponsive when it’s doing intensive tasks. I’m sure you’ve had times in NMM when you’ve tried to check for latest versions or update other content and the program has become unresponsive and you’ve wondered if it’s crashed. Well, it’s very likely it hasn’t crashed but is instead very busy working away on what you’ve asked it to do. Unfortunately with no visual confirmation that it’s doing anything at all it can sometimes be confusing trying to work out if it’s crashed or still alive and breathing. We’ve added a new progress meter to these time intensive tasks that will show you what NMM is up to and that yes, in fact, it is working!
Here’s the complete change log:
New Feature: Categories for mods
New Feature: Loadorder Import/Export functionality.
New Feature: The logout button will now return the user to the login screen instead of just closing the program.
New Feature: Using the new mod manager control, you will be able to sort mods using the column headers.
New Feature: added progress bars when NMM is busy in intensive tasks.
Bugfix: Crash when clicking the "Get missing mod info" button and NMM was unable to find any match on the Nexus server.
Bugfix: Crash when the user enables too many plugins.
Bugfix: Crash loading non-existent or invalid plugins in the loadorder file.
Bugfix: Fomod/Omod IsMatching Version crash.
Bugfix: Overwrites Yes/No to Mod when there's no mod to overwrite.
Bugfix: NMM not saving Latest Version when it was already present in the info.xml file.
Bugfix: NMM will always use the info.xml file in the cache folder instead of the one in the original mod archive.
Bugfix: NMM renaming mods when searching for updates.
Bugfix: StripInvalidChars method crashing when the path string contained quotes.
Bugfix: NMM crashing when reading malformed version numbers in the info.xml file.
Bugfix: "Get missing mod info" button removing Install Date and Endorsement status.
Jump to page