Jump to page
As you might know we've been struggling with our forum server after the past few weeks and have been working towards getting them on the same Cloud setup that's worked so well for us with the main sites themselves.
However, we'd struggled to understand why, all of a sudden, the load times on the forums went from fine to absolutely awful in little to no time at all. There wasn't any sort of traffic spike and the servers weren't being constantly DDoS'd so there was no real reason for it. We knew it wasn't our database because the forums are running on the same database cluster that the sites are using; so the sites would have been slow as well.
We came to the conclusion that it must be related to how heavily we've integrated our forum software, Invision Board, with the sites themselves. All membership information, profiles, file comments and discussions and bug reports, image comments, video comments and Premium Membership information is handled and called through Invision Board and each time we add a new game to the site we were creating 17 new forum categories to support it, to the point where we had close to 3,000 forum categories on the forums. It seems Invision Board doesn't play nice with growth like that. At all. For the technically minded, Invision Board seems to store things like your read topics for every single forum inside variables before every single page load, so if there's 3,000 forums, that's a lot of server resources being used on each page load.
We've been experimenting with a number of different tweaks and changes to the forum servers over the weekend. Because it was the final weekend of the Steam Summer sale we once again saw record numbers of users on the sites (36,000 new members this weekend alone and 7,500 page views a minute). We were very pleased to see the sites held excellently during this time with page load times well within normal ranges and the traffic helped us to not only benchmark potential issues with the sites in light of a Fallout 4 release that's not too far away but also to analyse what would help relieve the load on the forum servers.
Today we've made some relatively large changes to the forum category structure. Most notably, almost all games on the network now only make use of a single forum for all discussion (with the exception of the big games like Skyrim, Fallout 3 and so on). We've done this not because we particularly want to but because the forum software itself cannot handle having 3,000 categories (and growing each time we add a new game to the network). All file, image, video and article comment topics now take up a single category each (as opposed to a category for each game on the network) and all the old topics and posts for files already uploaded have been moved to these single categories. While you can still access the "forum view" mode for file and image comments you will no longer be able to access the file and image comment forum category directly. Because all the files and image comment topics for all games on the network are now mingled together in a single forum category it seems pointless to open up the category for viewing; you'll never be able to find what you're looking for by browsing through that category anyway.
Making this small change has not only reduced our forum category count from 3,000 to 1,200 categories (and we haven't finished doing spring cleaning yet which will likely drop that figure to below 1,000) but it's drastically (DRASTICALLY) reduced the load times on the forum server. Essentially, the forums are now back to being quick and, in fact, better than ever.
We'll now manually add more forums (like forums for "mod requests") as and when they are needed. As it is, a lot of the games we support have no need for forums here and they're added automatically as a token gesture. They might be used, they might not. Point is, they're there, and we can always add more categories for popular games as and when they're necessary.
We've still got some spring cleaning to do and we're still going to go ahead with moving over to the cloud setup for the forum server. The cloud setup has showed its worth to us many times already, but case-in-point, this weekend the NMM virtual machine that handles logins and retrieving metadata for the Nexus Mod Manager became overloaded due to the sheer number of requests being made of it. As soon as we found out about the issue we were able to double the RAM and CPU cores available to the virtual machine, on the fly, within 5 minutes. Problem gone.
Aaaaaand, as I type this, I've just been informed of a DDoS taking place against our servers. It's being looked in to as we speak and I can't speak highly enough of our friends (well, actually my friends, two of whom I lived with in Reading after university and one of whom is a childhood friend from my early secondary school days) at Krystal.co.uk who handle a lot of our hosting requirements for the sites and are always there for the sites whether it's 3pm or 3am if we're having issues. If you're a Brit who needs hosting or has a crap hosting provider and is looking for reliable hosting with great service you really can't do better than these guys. And I wasn't paid for that little advertisement, they just deserve the exposure for the times they're there for the servers, fighting off the kiddies out there who want to ruin it for everyone else (they were also up at 3am on Friday morning helping us to bring back the forum server that decided it didn't want to wake up from a restart).
We're aware the forums have been largely inaccessible and now down since 11pm (GMT) on the 18th of June. In an unwelcome throwback to the 2013-2014 Nexus years the RAID controller on the forum server has seemingly packed in. Perhaps the forum server wanted to give us one last gift before we (had planned) to transition to the new cloud-based system over the weekend?
Thankfully all our database operations are stored on a 5-server database cluster unrelated to the forum server, so things like the member database, posts, topics, etc. are all fine, dandy and indeed all working from the NexusMods.com site as usual. The forums, including new registrations, are not however.
We're continuing to look in to it, but it's coming up on 3AM now and this might take a while. Just a heads up.
I just wanted to provide a quick update. While I haven't been able to reach a computer at all this weekend I'm aware that the servers have been slower than usual over the past few days.
The root cause of this issue is a database cluster node that got knocked out Saturday afternoon, meaning we've been running on 4 nodes instead of the usual 5. On top of that we're currently seeing higher than usual traffic, about 22% more than a usual Sunday, so not only are we running at 20% less power capacity but we're also serving 22% more users. I'm not entirely sure what the traffic bump is attributed to but it could perhaps just be an influx in people replaying Fallout 3 and New Vegas in response to the announcement of Fallout 4 last week.
We're overdue an update news post on the recent changes we've made to both the site and one of the largest updates to NMM we've ever done, due to my extremely limited time at the moment, but that will come as soon as possible.
Lastly, we're still aware of the slowness on the forum side and we're still working on transitioning the forums over to a new server system that should resolve this issue.
Fallout 4 has just been announced complete with a teaser trailer from Bethesda.
Details are obviously scarce right now, but we won't need to wait for much longer as the E3 reveal for Fallout 4, complete with gameplay (one would assume) is coming in under 2 weeks time at E3 on June 14th.
What I can say is, from watching the trailer, it looks like you'll roleplay a dog and be the trusty companion of a Vault Dweller...
As a long time mod tester and also as a lover of great screenshots I was really happy to see a new mod that gave us Free Camera mode and a Debug/Developer console. I am looking forwards to seeing some of the great screenshots that can now be made.
Debug Console Enabler
It has started! The game is a few days old, the official modding tool has yet to be released, but the modding community is already on the job.
Mods having bugs in them is nothing new. If you've used or released at least a few mods in your time you'll have come across certain "features" in the mod that probably aren't supposed to be and weren't intended to be there.
Up until now if you came across a bug in a mod you'd have to report it either in the file's comments or as a private message to the author of the mod themselves. Very few mod authors have set up their own bug reporting systems on their own hosting solutions for reporting bugs on their files.
Today we've released a first edition of a new bug reporting system on Nexus file pages. This system is opt-in, which means if you want to make use of it on your file pages you need to go to "Edit Attributes" on your files and then set "Allow users to add bug reports about your file" to "Yes" under "User Permissions". The bug reporting system will then be turned on for your file page. You don't need to turn on or use this new system if you don't want to and you can turn it off at any time.
The bug reporting system creates a new tab on your file page and from this tab users can post new bug reports to the system and mod authors can respond to and control the contents of the bug reporting system, which works in a similar way to the commenting system. You can specify the status of the bug (being looked at, known issue, duplicate issue, solved) and the priority of the bug report in relation to all the others (low, medium or high) as well as removing the bug if it isn't relevant.
Either you or the user can decide to set the bug report to "private" if you'd like to keep it just between the two of you, which might be handy if you need to share information about your setup that you don't want other people to be able to see.
Mod authors can also manage and move bug reports made in the file comments into the bug reporting system. This will help to keep your file comment system more tidy while providing obvious signs that you've recognised what the user has commented on is a bug with a link to the bug report in question.
This is our first go at the bug reporting system so we're happy to listen to constructive feedback on ways to improve it for you.
Changes to the file image uploading system
This has been a much requested feature over the past few months. We've now updated the file image uploading system to allow you to upload as many images as you want without having to reload the page every 5 images. We've even set it up so you can drag and drop one or more images straight from your hard-drive to the box on the website to save you some time.
We are still aware of and working on fixing the issue with images displaying as broken for the first 10 minutes or so after you upload new images to the site. It's to do with our caching system on our static content servers. All 8 of them.
Since the announcement of paid modding last week the internet has been awash with discussion on the topic from all angles and extremes. I myself have written over 10,000 words on the topic over the past month.
I have had to remind myself that I had the privilege of having a bit of forewarning about this happening; I had time to go through all the mental states, think of lots of the permutations and think of all the different arguments for and against the system, so when the announcement came, the shock for me was not knowing when it would happen or how they would do it, rather than not knowing it would happen at all.
I want to begin to draw a line under this ordeal. This does not mean I want to forget it ever happened or think that this won't crop up again in the not too distant future. It will. Of course it will. But life on the Nexus must go on, and in order for us to move forward we need to stop talking about the events of the past week so much and continue on where we left off before this all started while trying to pick up the broken pieces in our community.
Ironically (I know), before I sign off on the topic for now, I have taken part in two articles in the public media over the past week. It is my intention to do no more news articles or interviews in the media about this topic unless some new developments occur. I have already respectfully refused to do a few because of this. Of the two articles I took part in, one was a Q&A with Rock, Paper, Shotgun, the other was a close to 2 hour discussion with TotalBiscuit and Brumbek, author of SMIM . You can find that interview on Youtube now, and it should be on TotalBiscuit's SoundCloud by the end of the day. The former gives you some insight in to my take on things, and the latter is a very off-the-cuff conversation between the three of us. And a conversation is all it is. It is not a debate, it's not an attempt at an unbiased news article approaching the situation from every angle and it doesn't cover or intend to cover all the angles of this situation. But if you're interested in hearing a few opinions on a few different aspects of the topic, or if you want to hear my voice coming through an £8 Logitech desktop microphone I bought 10 years ago because I hate using headset microphones, then you might be interested.
I will say that, since finding out about this development a month ago, and since the actual launch of it last week, my opinions and views have been swayed massively during this time. I've read some excellent points from mod users, mod authors and game developers and I've read a lot of really, really dumb points too. But the point is, there are some excellent points and arguments to be made from all angles in this topic and, if you're anything like me, you want to try and see it from every angle and come to your own conclusions from as enlightened a perspective as possible. I might look back on what I have said in a day, a week, a month, a year or a decade and think "Heck, I got it wrong there!" or "Nah, I don't agree with that any more". For me, it's not about having principles and stubbornly sticking to them as much as it is approaching the situation with an open mind with the willingness to have my opinion changed by well thought out and reasonable critique. I encourage all of you to try and do the same, no matter what your view on the topic is.
On the topic of donations
The paid modding fiasco has put a spotlight on donations for mod authors. Of course, we've had a donate button on the sites since 2012 but many have said it isn't prominent enough and many didn't even realise we had one.
I like the idea of a donation system and I don't have a problem with a more prominent donation system. What I don't want is a site littered with adfly links, advertisements/demands for donations from mod authors and desperate begging for money. We need to find a system that is tasteful, not insulting to the senses, universal, yet still effective.
Many have brought up payment platforms like Patreon and Flattr. I would love to integrate these platforms, especially if they have a decent API where we can plug it seamlessly into the sites and make it look nice. Unfortunately that's not my choice to make. Legally, and for my piece of mind, this needs to be run by Bethesda to ensure they're OK with the idea. I am aware that at least one mod author is talking with representatives of Bethesda right now on this topic and we will wait to see what they have to say. If they give the go ahead for Patreon then we'll begin work on integrating it however we can. If they say they're not OK with it, we won't.
Change isn't going to come over night and I'm not going to rush out lots of big sweeping changes straight away. If you're a mod author with thoughts on this then you should go to the private mod author forums (a part of our main forums) to make your thoughts known with all the others discussing it. But in the mean-time our rules are the same as they always have been on donations. You can't charge money for your mods, specifically ask for or talk about donations in your descriptions/images/file comments and you can't solicit or advertise other products or services (like adfly links, or a few G2A links I saw crop up recently).
We're in the process of reviewing all the information being given to us on donations, at which point I'm sure some changes will be made, but in the mean-time it's business as usual.
The community manager position
I publicised a paid job opening on the sites a few months ago for a full-time community manager role. A few weeks back we took the listing down as we'd had over 300 applications, some of which were absolutely fantastic. I've read every single application and have created a "short" list of 30 names that I was in the process of whittling that down to a smaller number still when this fiasco hit, which has since taken all of my time.
I know, now more than ever, I really need this role to be filled, but the application process is taking a long time. I'm obviously taking it very seriously and analysing all the candidates for suitability. Because there are many, many really good fits who have applied already I will be sending out emails to people on the "short short" list in the not too distant future to get them to answer a few questions.
I wanted to update you all that the role has not been filled yet, I haven't sent out any emails, and the process is still on-going. And a big thank you to those of you who applied (and a big "what the heck, guys?" to those of you who sent in an application without telling me what your Nexus username is in your application...duh!!!).
Valve and Bethesda have announced that they have removed the payment feature from Steam Workshop effective immediately. Refunds have been sent out to anyone who paid for a mod using the system.
Bethesda's blog piece originally defended the system and was then later updated. However, I think it's an amazingly good read. You may not agree with everything written within it, but it is well written and shares key insights in to Bethesda's thinking when they went in to this endeavour. Frankly, if they had written and released that blog piece when they'd first announced the paid workshop functionality then it would have helped to alleviate some of my fears. It's a shame that it wasn't done. I particularly appreciated their comments on DRM, which I feel might have been slightly pointed at me:QUOTESome are concerned that this whole thing is leading to a world where mods are tied to one system, DRM'd and not allowed to be freely accessed. That is the exact opposite of what we stand for. Not only do we want more mods, easier to access, we're anti-DRM as far as we can be. Most people don't know, but our very own Skyrim DLC has zero DRM. We shipped Oblivion with no DRM because we didn't like how it affected the game.
Excellent words. Thank you, Matt (or whoever wrote it).
I know many people will disagree with me, but I can't help but feel sorry for Bethesda. I understand this was their own doing. But in between all the drama of the past few days it's very easy to lose sight of the fact that this is the developer who released an amazing SDK for Morrowind, when they didn't have to, an amazing SDK for Oblivion, when they didn't have to, an amazing SDK for Fallout 3 (that worked with New Vegas), when they didn't have to, and an amazing SDK for Skyrim, when they didn't have to. In a gaming industry that was and is running further and further away from modding. We shouldn't have had reason to doubt them, but unfortunately a lack of good communication with the community at large prior to releasing the tools has completely bewildered the entire community and contributed in a big part to the resounding amount of resentment towards the new system.
Earlier on today I wrote another 2,500 words for a Q&A interview with Rock, Paper, Shotgun. I'm interested to see if they'll still print it considering it's now mostly a moot point, but it raised some interesting questions and answers which are directly related to how I think (or thought) this could have been handled a bit better to avoid what has become this rather terrible PR nightmare for Valve and Bethesda. If they're not planning on releasing it then I'll have a think as to whether it would be worth posting up as a concluding blog piece on the topic. Though it may be better not to beat the dead horse further and draw a line under this extremely damaging incident in the hope that we can move on from it quickly. Would you even like to read more of my banal twitterings? God knows I've waffled on way too much these past few days as it is.
Jump to page