Oh the irony of it. I wake up this morning ready to write out my plans for ads on the site and instead find Google have slapped a Malware warning on many of the pages of Skyrim Nexus, due to the ads. How tittingly annoying.
First of all, sorry about the mess. Second of all, it should be “fixed” soon (not that anything is broken). So what’s happened?
When Google sends its bots throughout the web to maximise their search content the bots also run all sorts of tests. One such test is a malware test to see if the site is trying to do anything naughty with your computer. The site could be naughty because it’s deliberately trying to be naughty, or because it’s been hacked by naughty people. Either way, it flags the sites and servers that are naughty and stores them in a big database. This morning on a routine Google bot visit, the bot was served an ad that came from one of those flagged sites; an adserver Google has flagged as serving malware. So naturally, rather than Google blocking your browser from just seeing that ad server, Google has blocked half of Skyrim Nexus, irrespective of whether you are using an ad blocker, anti-virus, anti-malware or any myriad of possible solutions that mitigate and dissipate any threat. Annoying, huh? Queue the face palm.
So why are we serving an ad from a server flagged as serving malware? Well, that one’s even more complex, but I’ll try and break it down for you as quickly as possible. When you’re not doing direct ad sales (e.g. calling up Bethesda’s ad agency and asking if they’d like to purchase inventory on the sites directly), you pass your advertising inventory to ad firms who sell your inventory for you. They’ll give you high quality ads most of the time, but if they can’t sell your impression they’ll pass it on to ad markets, where 100s of ad suppliers are congregated in a form of bidding technology, where the highest bidder for that impression gets to show their ad in that space. This is naturally all done by technology and maths, lots and lots of maths, in a split second. Ad firm A has decided your impression is worth $0.001. Ad firm B has decided it’s worth $0.0011. Ad firm B gets to display their ad to you. Unfortunately, if Ad firm B have been hacked, or have stupidly not vetted the ads they’re showing properly then Google might flag them as naughty malware suppliers. If a Google bot comes to your site when an ad is shown from that naughty ad supplier, BAM, you’re a malware supplier too. And that’s what has happened. Wondering why Google doesn’t just block any access to that supplier, rather than blocking every site that might have used an ad from the supplier? Me too.
I’m always working with my ad suppliers to increase the quality of the ads and remove any crappy suppliers. Today we were unlucky, and now we’ve got to wait for the Google bots to come back around, recheck the sites and confirm that they’re safe to use. They are safe to use. We haven’t been hacked or infected, and you can tell pretty easily. Go to Oblivion Nexus, or Morrowind Nexus, or any other Nexus site. Notice how Google is saying nothing about them? Skyrim Nexus was just unlucky. Showing the wrong ad at the wrong time.
As always you should be doing things to mitigate any damage that malicious code can do to your computer. An adblocker will not protect you from a hacked site, just hacked ads. You should have a good anti-virus program installed on your PC. And that copy of Norton anti-virus you got with your PC isn’t one of them. Any one of AVG, Avast, Comodo, Panda or Avira (which are all free) coupled with Ad-Aware and Malware Bytes will keep you super safe 99.99999% of the time. An adblocker will mitigate your chances of getting attacked by a malicious ad, but not completely prevent it and it will also severely dampen the Nexus’s ability to stay afloat financially. If you want to use one, that’s your choice, but you do hurt the Nexus by doing it.
Once again I apologise for this annoying situation, and we now wait on Google to come back around and sort this out.
I love most things about Skyrim, but here are three things I do not like, and luckily there are three mods that fix those things.
Thumbnail image for this video is 'Ignis II' courtesy of kaldaar.
Download links for reviewed mods:
No Sneak Attack Sound
Download links for previewed mods:
WIP - Immersive Bounty Hunting
Yes, I’m sorry, here’s another one of those 5 page, 2,500 word rambling nonsense blog posts I like to make from time to time to show you that I’m not dead and we’re still moving forward. If you have no interest in servers, money or talking about my narrow views on corporate greed, then I would suggest skipping this one!
We’re about one month into the new year now and I wanted to share with you one of our main priorities for this year, as it’s as important to us as it is to you.
Perhaps one of the Nexus’s biggest pitfalls since its inception has been the stability of the service. I don’t have any uptime statistics on hand to report on but I think it’s pretty safe to assume that we’re not hitting the 99.97% uptime that most big sites try to aspire to. Not only is it annoying because you guys can’t access the site, or particular services on the sites (like the downloads) at random points, but it’s annoying for the staff to be awake at 3am in the morning restarting services and troubleshooting database crashes, and it’s annoying for me to be running a service that isn’t 100% reliable. Moving forward I’d like the Nexus to be taken seriously by game developers, and it’s hard to be taken seriously when you can’t guarantee service.
It’s not as though other sites and companies out there don’t have reliability issues. I’ve been locked out of DotA 2 many times recently because the Steam servers have been down, for instance, and that’s from a multi-billion dollar company like Valve. But the problem with the Nexus is that it’s a regular occurrence.
I can attribute this problem to all sorts of systemic issues right through the Nexus, from the way I’ve set up the business to the way we’ve run the site and prioritised things. It’s not as simple as pointing the finger at the lack of server resources, or hardware failures, or the code, or the DDoS’ers, or being a victim of our own success or any one thing, it’s a multitude of things. But the highest priority of this year is to resolve this issue and make the Nexus as stable and redundant as it possibly can be.
So lets take a look at some of these systemic issues and then I’ll explain what we’ve been doing, and what we will be doing to make the situation better and ensure the Nexus is future-proof for the foreseeable future.
The largest factor of all with the stability issues has been the sheer popularity of the Nexus sites coupled with having an inappropriate server architecture to accommodate the demand on sites that are extremely database (and ergo, resource) intensive. I’ve blogged many times on the popularity of these sites and the difficulty in keeping them up with the load placed upon them, so I won’t bother to go into the numbers again. So you’re thinking, “OK, why not buy more servers then?”. The answer isn’t in needing more servers, the answer is in needing to restructure the architecture of the servers and network we currently have so that the combined resources of all the servers can be used to keep the sites going.
Right now we have a situation where we have lots of lower traffic sites (Far Cry, Neverwinter, The Witcher, Morrowind and so on), some high traffic sites (Forums, Oblivion, Fallout 3, New Vegas) and one super mega ridiculous traffic site (Skyrim). Typically speaking almost every site on the internet can fit onto a single powerful dedicated server. Depending on the size of the sites you can even fit hundreds or thousands of normal sites onto a single dedicated server. We have 6 servers dedicated to just serving the Nexus sites (not the file servers, we’ve got 12 of them!). The problem though, is that Skyrim Nexus, and the forums, are not normal sites and they’re at a point where they can no longer fit on one single dedicated server. Similarly we’ve upgraded the hell out of the servers so we can’t make them any more powerful than they are now.
We’ve reached this point where Skyrim Nexus has outgrown being able to run on a single super-powerful dedicated server, so how do we resolve this issue? The solution is in server clustering, which is a technology that lets you pool together the resources of multiple servers to act as one super mega server, much like SLI allows you to connect up and combine multiple video cards in your PC to dramatically increase your frames per second. Unfortunately server clustering isn’t as simple as connecting an SLI bridge connector to your video cards. It’s a lot more complex.
Server clustering is not only complex, it’s also expensive. We have 6 web servers at the moment. We can’t just flip a switch in the servers we currently have and turn on clustering. We’ve got to buy completely new servers, set them up for clustering and then transfer the network on to these clusters. That means running our current setup in parallel with the new one until everything is transferred which means paying for the original 6 servers plus the new servers we need to buy to form our clusters. That’s a lot of money.
And therein lies another systemic issue with the way things are setup. Money. The Nexus sites have remained completely independent; free of corporate interest and investment for its entire 11 years and it shall remain so for the very foreseeable future. The only investment these sites have had was the initial £10,000 I chucked in to the sites when I rebranded the sites as the Nexus back in 2007. I’m the sole owner and sole decision maker of the sites. There’s no outside interest, board of directors or investors pulling the strings behind the scenes. Similarly no game developers have any influence or sway over me. The buck stops at me.
If I wanted to I could make a business plan (I don’t have one, by the way) and go to Silicon Valley, pitch the idea to a load of private and angel investors, secure (potentially hundreds of) thousands of dollars in investment money and make a proper business out of it like many gaming sites and networks have done over the past few years. However, I then become answerable to shareholders and investors who are looking for a return on their investment as fast as possible. To be frank, F’ that.
Similarly it’s just me and 4 other programmers working on the Nexus. We have absolutely no one doing ad sales. I mean it, we have no ad reps at all. Others in the industry gawp at such an oddity. That’s why the ads you see on the site (if you don’t block them) are pretty crap, and in return we get pretty crap rates. While other networks have entire ad sales teams securing them crazy $1-$10 CPM rates on their ads, we don’t. We don’t get anywhere near that. So why don’t I hire some ad reps to better sell the inventory and use that money to pump it all back in to the sites? The reasoning is very similar to my private investment reasoning; when the focus of your business is on increasing your ad sales, and on ensuring a prompt ROI to your investors, you begin to lose sight of what your original goals were and instead focus on one very simple goal: making money. And money isn’t what I’m doing this for. Indeed, if money was my aim I’d be doing all these things I just mentioned, because the Nexus would be a cash cow. Case-in-point; I know sites that have 5-10 programmers working for them and 25-50 ad sales reps. Yes, that’s a 1:5 ratio of people working on content to people working on making money. To me, it’s crazy to have more people working on selling than actually improving and producing the content that sells. But that’s business for you, and I’m not a good businessman.
What this all breaks down to is limiting the stakeholders in the Nexus. Right now you guys, the people who use the site, the mod authors, the downloaders, the people on the forums, YOU are the Nexus site’s biggest stakeholders. If I don’t appease you then these sites cease to have a point. If I seek private investment, or start directly selling the Nexus site ads then my biggest stakeholders become the shareholders and the advertisers on these sites. My focus gets shifted from serving and pleasing you, the users, to serving and please people who have no interest in you. And the point of the site changes from being about modding to being about making money. That’s not what I want at all. There may come a time in the future when direct ad sales and private investment are exactly what the Nexus needs, but that time is not now.
You guys are really, really good when things go wrong on these sites. By and large the reaction is often tame and supportive rather than stressed and raging. I like to think it’s because you know we’re not some corporate mega-money machine that’s cutting costs by cutting corners, but just 5 gamers trying to provide the best service we can. I don’t want to change that, because being greeted with “Ah that sucks, I hope you can fix it soon! Good luck!” is better than being attacked with “WTF this is the worst pile of crap I’ve ever used and you should be ashamed” when something goes wrong.
I want to retain that focus on you guys being the primary stakeholders in the future of the Nexus, which means it takes a lot of monetary planning and saving to buy more servers and invest in expensive technology like server clustering while other sites can simply throw their private investment resources or ad sales money at the problem. That’s why it takes a long time. Avoiding private investment and direct ad sales is a conscious decision that isn’t without its pitfalls, but one that I think is worth it to retain the core values of what these sites were set up for in the first place; to provide mods authors with an easy platform to share their work with others that will stand the test of time.
So setting up server clustering is currently one of our biggest priorities, and we’ll be setting that all up in the very near future, but in the run up to all this we’ve spent (and are still spending) some considerable time right now focusing on the software side of things.
Over the Christmas period, while the “normal” members of the staff were enjoying a forced two week break, Axel was working on an error logging system for the Nexus.
One of the most annoying aspects of bug hunting and troubleshooting is when someone leaves a comment on one of these news articles, or on the forums, or on the tracker that something is broken. Typically it will go something along the lines of “Downloads are broken at the moment”. To which my response is “.......” , followed by much hair pulling. Downloads are broken? What downloads are broken? On what site? What files? Is it all files or just one file? Is it only happening on one Nexus site or all Nexus sites? Is it just small files or large files? What error are you getting? Is it happening 100% of the time or just some of the time? What browser are you using? Have you tried using another browser? Have you tried turning your PC on and off again? Have you tried logging out and in again? Did downloads ever work for you? Have you installed any new browser plugins, firewalls or anti-virus programs recently? What time did this happen? These are but some of the questions we need answered to actually troubleshoot the issue, especially if all the staff try downloading and it works fine.
What I wanted was a system that would aggregate and parse all the error logs the servers produce and present it to the staff in a system that can help us easily pinpoint not only errors and problem areas of the site, but also pinpoint specific times when the sites are worse than others to help us troubleshoot the problem. Typically the error logs that servers produce are all flat-file text documents. Line after line after line of errors with timestamps that can run up to gigabytes in size. It’s extremely hard to make use of these error logs without having a system to properly display the information, and there’s nothing worse than being told something isn’t working when it works for you and wondering if it’s affecting just one person, 1% of people 25% of people or even more. With the error logging system we can now see that “wow, yes, at 10am today we had 5 times more errors than we usually do”. It’s helping us to investigate things more and we’ve already applied numerous hot fixes to the sites over the past month that have patched up errors and slow areas of the site.
Similarly at this very moment we’re working on some more improvements to the downloading system for both manual downloads and downloads through NMM. Right now, if one of the file servers has hiccupped it can be a real pain in the ass trying to download something. These hiccups generally only last for minutes at a time, but during that minute it can make it hard to download any files, especially small files. With that in mind, we’re going to present the file server selection screen on all files now, irrespective of size. If a file server is down, you can quickly select another one to use. We’re also trying to implement a seamless redirect system incase you choose a file server that isn’t working for whatever reason. If the file server you choose isn’t working, the site will simply try another one until it finds a file server that is working. You won’t really notice a difference (except far fewer, or no errors at all!), although if you typically get fast speeds on only one or two servers you might get slower speeds as your download might be served from a different file server from the ones you normally pick, if they’re down.
This concept of seamlessly being moved to a server that works is very similar to our plans with the sites and servers in general. Right now, if there’s a hardware or network failure on one of the servers a Nexus site is on, that Nexus site becomes unavailable. Once our full clustering solution is done we’ll have a load balanced, redundant solution that means all sites are being served from all servers. If one server goes down, the other servers pick up the strain but the sites still work. It reduces the bottlenecks and also reduces our single point of failure problems. And finally, clustering restores the status quo by making “buy more servers” the viable solution to our strained server issues. If the network needs more power you just tack another server in to the cluster and you’ve boosted the resources available to the network. That’s not possible with the current system. So for me, this is quite exciting. For you, it’s more like “I don’t care, just make it work”.
I think I’ve gone on for long enough now. When I wrote my blog piece on Nexus development and expansion philsophy I was interested when I got a lot of emails and messages from people who were surprised I thought that hard about the process. I like to use these blog pieces to indeed show you that yes, I don’t just sit around all day watching stocks go up and down on my monitors and playing DotA 2, but I think pretty damn hard about this network. The choices I make aren’t just knee-jerk “oh, I guess we’ll just do that then” solutions but plans that have been made out and expanded upon over a long period of time in consultation with others. We’ve wanted to do server clustering for years now (indeed, I mentioned it in that first YouTube video I did), but we’ve only now been in a position to actually afford it. And that’s why I’m excited, even though the topic is pretty boring.
SkyUI is the most endorsed mod on all of the Nexus sites, and with the addition of the Mod Configuration Menu (MCM) it has become even more essential on most peoples Skyrim mod list. However there are people who still prefer the original games UI and for them the lack of the MCM is a major setback.
But have no fear, the SkyUI team thought of that and have provided a way for everyone to enjoy the advantages of MCM.
Skyrim Script Extender (SKSE)
Dragonborn just became available for pre-purchase on Steam, and will be available for playing February 5th. Grab it now for 20 $/€!
In case you don't know what Dragonborn is all about yet, here's a small description for you:
"With this official add-on for The Elder Scrolls V: Skyrim, journey off the coast of Morrowind, to the island of Solstheim. Encounter new towns, dungeons, and quests, as you traverse the ash wastes and glacial valleys of this new land. Become more powerful with shouts that bend the will of your enemies and even tame dragons. Your fate, and the fate of Solstheim, hangs in the balance as you face off against your deadliest adversary – the first Dragonborn."
Thanks to Camonna Tong for the find!
It’s been a month and a half since our last update to the Nexus Mod Manager, but, being the merciful slave master that I am, the Nexus programmers were given a two week break over the Christmas period so they could remind their friends and loved ones what they look like. That’s all over now however and it’s back to business as usual. I’m here to announce the fruits of our labours as we release NMM version 0.40.0.
This version of NMM includes our preliminary work on a feature that’s been begged for by some and outright rudely demanded by others since we first put NMM out in the public domain a year and a bit ago. That feature is mod categories, and the ability to organise your mods in to your own categories.
After updating to the latest version, NMM will ask you if you’d like to use the default Nexus site categories or whether you’d like to set up your own from the start. If you choose to let NMM specify your initial categories for you then your mods will be categorised based on the categories your mods are under on the Nexus site the mod is from. So if you have 10 armour mods, 10 weapon mods and 10 NPC mods the category manager will place your mods into those three categories automatically for you. If you choose not to let NMM set up your categories then all your mods will be in the “Unassigned” category and you can start from scratch. It’s probably important to note if you do choose to let NMM do your initial category setup you’re not forced to use those categories. You can rename them, delete them, add new ones and move mods around the categories with no limitations. The initial category setup is just a template for you to use, so it’s not one method or another.
The category manager button to the left of the mods tab allows you to do multiple different tasks to influence your categories including the ability to add more categories, collapse/expand all the categories, reset your categories to the Nexus defaults, reset all your mods to the “Unassigned” category, remove all your categories entirely and toggle hidden categories (categories with no files in them).
Hidden categories are probably worth expanding upon. If you select to use the Nexus categories NMM will add all the categories the Nexus site for that game uses, even if you don’t have mods in that category yet. These categories will be hidden unless you want to see them. However, if you download a mod from a category that hasn’t been used yet that category will then become visible, as it’s now being used. So NMM will automatically place your mods in to the correct Nexus category once the download is complete.
Something I desperately wanted in this version before I would allow it to be released is the ability for you to drag and drop your mods into categories, and also to multi-select mods by using the ctrl or shift keys like you would in windows to select multiple files. By using shift and ctrl you can select more than one mod to move at any one time, which should save you some time if you’re one of those horder-types who has millions of mods.
If you don’t care about categories or you prefer the old way of managing your mods then that’s no problem, we’ve added a button that will quickly let you switch between category view mode and old view mode.
When introducing this feature you might have noticed I mentioned this was our preliminary work on the category controls. The category view mode required a whole new control method to be made in NMM, which has not only enabled categories to work but has also enabled column sorting as well (you can now click the column headers to sort your mods ascending/descending by name, install date, version, author, etc.). However, column sorting does not work in the old view mode, and I’m aware that the old view mode will provide a quicker way of seeing things like ordering your mods by the last install date so you can quickly see which mods you last installed. What we haven’t had time to do is bring the new control method in to replace the old method, so we’re running both controls side-by-side at the moment, the old control method with no column sorting and the new control method with categories and column sorting. We’ll continue to allow you to switch between a categorised mod management option and a non-categorised mod management option, but the non-categorised option needs updating. We’ve got several other little tweaks and improvements we’d like to make which means this feature is probably about 80% done. However, as this version of NMM also contains quite a few bug fixes and a couple of other notable features we thought it would be better to get it out now, in this state, rather than later when it’s fully complete. So have no fear, the book on the new control method and category management is not yet closed.
Also worth noting in this release, we’ve added load order importing and exporting functionality to the plugin list and, more importantly (for me), NMM no longer looks like it’s crashed by becoming unresponsive when it’s doing intensive tasks. I’m sure you’ve had times in NMM when you’ve tried to check for latest versions or update other content and the program has become unresponsive and you’ve wondered if it’s crashed. Well, it’s very likely it hasn’t crashed but is instead very busy working away on what you’ve asked it to do. Unfortunately with no visual confirmation that it’s doing anything at all it can sometimes be confusing trying to work out if it’s crashed or still alive and breathing. We’ve added a new progress meter to these time intensive tasks that will show you what NMM is up to and that yes, in fact, it is working!
Here’s the complete change log:
New Feature: Categories for mods
New Feature: Loadorder Import/Export functionality.
New Feature: The logout button will now return the user to the login screen instead of just closing the program.
New Feature: Using the new mod manager control, you will be able to sort mods using the column headers.
New Feature: added progress bars when NMM is busy in intensive tasks.
Bugfix: Crash when clicking the "Get missing mod info" button and NMM was unable to find any match on the Nexus server.
Bugfix: Crash when the user enables too many plugins.
Bugfix: Crash loading non-existent or invalid plugins in the loadorder file.
Bugfix: Fomod/Omod IsMatching Version crash.
Bugfix: Overwrites Yes/No to Mod when there's no mod to overwrite.
Bugfix: NMM not saving Latest Version when it was already present in the info.xml file.
Bugfix: NMM will always use the info.xml file in the cache folder instead of the one in the original mod archive.
Bugfix: NMM renaming mods when searching for updates.
Bugfix: StripInvalidChars method crashing when the path string contained quotes.
Bugfix: NMM crashing when reading malformed version numbers in the info.xml file.
Bugfix: "Get missing mod info" button removing Install Date and Endorsement status.
Update: Not getting any confirmation after applying for the beta? Be sure to check your spam folder! It has been reported to end up there quite a lot.
Great news! The beta sign-up for The Elder Scrolls Online started today!
Be among the first to set foot in the massive world of Elder Scrolls Online! The Beta will begin soon, so be quick about signing up!
Now, remember everyone; this is not meant to be for you to just play the game before launch - this is a beta, and as such, you should not be playing the game just to have fun - you should be TESTING the game, and reporting your findings to ZeniMax Online Studios. If this is not your intent, don't sign up.
But enough of that, to celebrate this, Bethesda Softworks released a Cinematic!
I must say, this was a very well done Cinematic, and I really enjoyed it. Hope you do as well!
In this video I give my impressions of the Frostfall mod. Spoiler alert: I think it is brilliant.
Thumbnail image for this video is courtesy of TomAraya.
Frostfall - Hypothermia Camping Survival
Happy news for both PC players and Playstation 3 players!
PC: You will be exploring Solstheim on February 5th!
Playstation: Sometime in February, you will get to explore Solstheim, followed by building yourself a home, and after that you will get to either kill or team up with some vampires! All this in February! And that is not all; update 1.8 will arrive just before Dragonborn. You think Bethesda is ending it with just giving you (in order of release) update 1.8, Dragonborn, Hearthfire and Dawnguard? Nope. During the launch week, all DLCs are 50% off on PSN!
Why are the alliances fighting, what are their motivations, and why have they banded together as they have? Well, the loremaster at ZeniMax Online Studios, Lawrence Schick answers some of these questions, and more in this developer video.
It is only five (5:31) minutes long, so do not expect a detailed explanation to each alliance. The video is primarily to get sort of a perspective of what each alliance stand for, and why they think they need to enter Cyrodiil and take the throne.
Jump to page