Oblivion

Downloading and file serving changes

  • Comment
Happily and thankfully a week after I wrote my 2,600 word End of Year review/essay we managed to stabilize our new server setup and run through some quick but efficient code updates that have enabled us to run the sites almost down-time free over the past week. No database desyncs, no maintenance windows, just a few times where we've taken the pages down for a few minutes to perform code updates. Paint me f'ing happy on that. And vindicated in my own eyes. No doubt we've still got work to do, but the important thing is that things are way better now.

In the same article I also mentioned that during January we would be decommissioning all 21 of our file servers (2 static content servers, 15 file servers, 4 Premium file servers) and replacing them with new servers. The old file servers were coming to the end of their life-line and running out of hard-drive space fast so we needed to upgrade and update the whole line up. We're now replacing those 21 file servers with 26 new file servers (4 static content servers, 20 file servers, 2 Premium file servers with probably more Premium servers to come). As you can imagine that's quite a job; we've got to set up each server with our custom server modules and configurations as well as transfer the 4TB or so of current files there are on to every single one of those servers (so we've got to transfer about 100TB of data over the next month). It's time consuming to be sure. We're currently setting up the first new servers which requires these new servers sync from the old ones, this is causing some issues with the network which is causing some downloads to fail (more than usual) or not start at all. Once we've got these first new servers setup we can sync the files between all the new servers, from the new servers themselves, thus reducing the load on the old setup in the mean-time.

I know there's been some issues with downloading reliably through NMM recently, especially when downloads don't start at all. In those instances it's important (for your own sake, not mine) that you know how to do the simplest of things: how to add mods to NMM without using the "download with manager" button. It's so simple I might as well just write it here:

  • Instead of clicking "Download with manager" click "download manually" on the file you want to download.
  • A window will pop up with a list of file servers to download from. Pick a file server. Typically the fastest server will be the one with the least users, not the one that's closest to you.
  • Your download should start within 1-30 seconds. Let the download finish.
  • When your download has finished, open NMM.
  • Open the folder where your file downloaded to.
  • Drag the file in to the "mods" window in NMM.
  • OR: in the "mods" window, click the button that looks like a jigsaw piece with a plus (+) symbol next to it and navigate to your file on your HDD.
  • NMM will add your mod just as if you'd used the "download with manager" button.


That's it. If you download from the Nexus and don't change the filename then NMM will recognise the file and the file will work exactly as if you had downloaded it through NMM itself. There seems to be some myth that if you download manually then NMM won't show you any new mod versions. That's wrong/a lie. NMM cannot show you mod versions for mods not downloaded via the Nexus (so if you download a mod from the Workshop and add it to NMM then NMM won't show you any new mod updates/versions for that file), hopefully for very obvious reasons, but it will show you updates for any Nexus mods whether you downloaded the file directly through NMM or manually via the site.

At times like these you will find it easier and quicker to download your files manually and then add them to NMM rather than using the "download with manager" button and have it failing on you a lot.

While we're working on the file servers we're directing a bit of attention to how NMM handles downloads, and especially downloads that fail. Right now NMM will try and find a working server that fits your download location preferences as well as possible. However if that server fails then it doesn't automatically try another server, it just fails. We're trying to add more fail safes in to the program so you're left with no failed connections.

Similarly we're having to implement a download queuing system based on feedback I've personally been getting from a few...interesting individuals. With the ever increasing interest in projects like Skyrim GEMS and S.T.E.P. there's a lot of users who are downloading large quantities of mods all at once. I'm getting support tickets from users wondering why they're struggling to download 200 mods all at the same time in NMM. Hopefully I don't need to tell you why downloading 200 files at the same time isn't good for the servers (or respectful to the other users of the site). Now this is partially our fault as we haven't hard-coded any limits in to NMM in this regard. NMM will let you download as many files as you want. They probably won't all work, but it'll let you do it. We've got hard-coded limits on our servers (which I believe are 16 connections for normal users and 64 for Premium, with an increase for Premium as they can use 4 threads, 4 multiplied by 16 is 64). With currently 15 file servers that's a theoretical maximum of 240 concurrent downloads, but that works on the assumption that NMM will pick servers that you aren't already maxing out, which it won't. But I digress, trying to download 200 mods all at once is really stupid and us not having hard-coded limits in the program is equally stupid. As such we are going to be introducing a download queuing system. Normal members will be able to download 5 files concurrently, Premium members will be able to download 10 files concurrently. You can add as many files as you want to your download list but only 5 (or 10) will download at any one time. Once a download finishes a new download in the queue will start automatically. Such functionality is commonplace on many sites and will enable us to provide a more stable and fair service to all users.

We hope to get this new NMM version out to coincide with the completion of our transition over to the new file servers. It's likely this version will be another "forced" update, in that you'll need to update to this version in order to be able to use NMM online, because it's updating and fixing things that are the cause of issues on the network. I'll have a word with the programmers to see if we can update the error you receive when trying to login to an unsupported version of NMM rather than the generic error that is currently used.

In the meantime I hope you're enjoying a much smoother site experience, even if the downloads are currently a little iffy. The important thing is: we're getting there. Like I said we would.

191 comments

Comments locked

A moderator has closed this comment topic for the time being
  1. Lokie7
    Lokie7
    • premium
    • 3 kudos
    Wow ! Lots of work going on by the Nexus Team. It has been interesting to watch the Nexus "Evolve" over the years. The (Your) pursuit of the Best Functioning and Diverse site for Mods is really showing and paying off.
    I would like to Thank the entire Nexus Team, Past and Present, for all of your' Hard Work, Time and Dedication that goes into providing a (this) site for Modders to Display/Offer their Hard Work and for users, like me, to access them and Enjoy. You have Created a Fantastic Community herein, IMO.
    I always enjoy coming to this site and usually wind up spending a few hours here just checking stuff out. It's always fun, interesting and informative. "Shopping" for Mods is fun in itself. My Thanks to the Modders, one and All.
    I have gotten anywhere from a few hours to Days/Months of extended and more Satisfying, Rich and Immersive gameplay due/from to this Site alone.
    For all of those reasons and a few more, I will continue to support this site as long as I can.
    Thanks Again, to you all !!
    Respects, Lokie7
  2. William2143
    William2143
    • supporter
    • 0 kudos
    NEED HELP PLS
    i download NMM i get to the choose language part i hit ok on English and i get an error " The system cannot find the path specified" so pls help
  3. Cattus99
    Cattus99
    • member
    • 0 kudos
    Hi.  I haven't been able to log on with NMM for a while.  About fours weeks ago I couldn't and then I was away and now this week I couldn't either.  I just downloaded the latest  NMM .47.3  but no go.  If the server or other things are down where can I find that out?  I've searched a bit for that info.  If things aren't down then what gives?  Thanks to all for any help you can give.
     
    .....Some time later.
     
    Ok after trying for a half hour then writing this post I got onto the network - I guess i just have to write about a problem here in the forum and it gets resolved?  haha..... Mr. Car Mechanic:  my car makes this awful noise when I turn the steering wheel.  Let me show you......oh....the noise is gone.....well then never mind.
  4. Thandal
    Thandal
    • Moderator
    • 183 kudos


    I just updated to the latest version of NMM, and now can't seem to get any thing to down load. I am a premium member, have logged in to both NMM and the Nexus web page but when I try to down load by clicking the "Download using NMM" button nothing happens. It seem that the web page is not talking to NMM at all. What is the fix in simple terms, as my knowledge has been depleated.


     
    Are you logged-in to the Nexus using NMM?  (Is the little "buddy" icon in the bottom-left corner of the window green, or red?)
  5. Foegel
    Foegel
    • member
    • 0 kudos
    Nice
  6. Olddog22
    Olddog22
    • premium
    • 0 kudos
    I just updated to the latest version of NMM, and now can't seem to get any thing to down load. I am a premium member, have logged in to both NMM and the Nexus web page but when I try to down load by clicking the "Download using NMM" button nothing happens. It seem that the web page is not talking to NMM at all. What is the fix in simple terms, as my knowledge has been depleated.
  7. jrtaylor001
    jrtaylor001
    • member
    • 0 kudos
    As i read through alot of the comments ,i see many people are really demanding for no apparent reason, the site is a tremendous help to people over all , and extends gameplay for many , many, many hours, they should not "bite the hand that feeds them ".
  8. Michele Magus
    Michele Magus
    • premium
    • 9 kudos
    I had the same issue here.
    As I had no plan to wait an hour while NMM kept hammering a dead server, I did a manual download and grouped the files by hand.
  9. Mebantiza
    Mebantiza
    • premium
    • 346 kudos
    Download with NMM has not been able to route download requests to the various mirrors effectively, now for sometime. Its (almost)always either;

    File does not exist
    or
    Server unavailable
    or
    Server busy.

    Somehow, 98% of the time, NMM manages to que the WORST possible server it can. Yet, when I manually click on say, Seattle, it just works 99% of the time, the first time. Probably not saying something you dont already know, but it makes me a little better at least mentioning it....
  10. petray
    petray
    • member
    • 1 kudos


    I think we're just arguing semantics with the use of the word "customer". I think a better word would be "stakeholder". You have a vested interest in the Nexus working properly and we have a vested interest in keeping you happy, but at the end of the day the only demands I will listen to are the demands of Premium Members who would like a refund or help getting their account to work. It's their money, so that's their prerogative.
     
    "Customers" suggests you have paid for a service, something you either haven't, or you have but on the clearly stated understanding that the payment is seen as a "donation with benefits" rather than payment for services rendered. Much like how you can make a voluntary contribution to an art exhibition or a charity event in exchange for access to special areas or special privileges. Using the word customers suggests I owe you something, something I strongly disagree with. The only people I owe anything are Premium Members, who I owe money should they not be satisfied, and my staff, whom I rely on. Anything else is a privilege, and not a right from which you gain the ability to make demands of me, for which I get most displeased.


    Well put - even if I don't actually agree with your definition but that is my priviledge!