Sooner or later the game session will become so laggy that fast forward becomes slow motion and your computer starts melting while you endure the half-hour long trip your armada takes from point A to point B across the galaxy. What I discovered was that if I lowered the amount of ships and squadrons on the map I was playing, the game would run more smoothly for a longer time. I tested the theory by simply not building any trade ports or refineries on a very large map with  ~500 nodes, and as I guessed the game ran smoothly throughout a large portion of the session (the AI still built their networks and slowed down the game a bit eventually - but not as much as it used to be).

I believe it has to do a lot with the amount of units moving around the map that the game has to keep track of. I don't have all the tech-wisdom to pinpoint as to exactly why but I'm guessing from what little I know; the game draws the meshes and the textures of each unit since you can see them visually represented in the game, and so they must be drawn, they have animations as well, the game calculates each vessel's pathway and what their values are (health, armor, speed, goods) and so on and after a while with a couple of hundred trade ships in your empire alone moving around the galaxy slows down the game so much that the game cease to be fun and ends up being a chore to play through. Now the trade ships are essential to have to get a large credit income but on a large map with a lot of planets, moons and asteroids to conquer, populate and tax you'll end up with a lot of credits anyway, and since you can sell the resources you constantly collect automatically for each metal & crystal node you have, you'll end up being rich in the end, not as fast though but still. On small to medium sized maps you'll be more wise to use trade ports (unless it's a zerg-rush map) but on large ones you should aim to conquer and expand and develop the planets to hold maximum population and the right tech and if lucky you'll also have the artifact to boost taxes and pop. With that you can fuel your empire and deploy your starbases all over the map.

So if you play on a large map, then I recommend that you go with the "NO trade ports and refineries" route to ensure that the game runs well for as long as possible. If you have to use them be strategic and build them on the right planets for them (those that have a boon that boosts trade for example). After a while you shouldn't need trade ports or refineries if you have secured enough planets. I've tried the approach on the maps I've made and it works, got one third of the map at the end of the session and staved off both the pirates and the AI, researched everything and even got rid of a few AI players with culture overthrow. Depends on difficulty but should work but most importantly the game was still purring like a kitten, almost, think it coughed up a hairball at the end when the AI had a few trade ships running around.

I've noticed that each trade port spawn 4 or 5 trade ships (have to double check it later), so imagine if you control 50 planets, all crammed with trade ports and refineries and the AI builds them as well, on top of that all players roams around with massive armadas that everyone ends up having and you have visual on all the nodes in the map, that's a lot to draw, calculate and keep in memory, so it's not so strange as to why the games lags so bad during mid to late game.

So if you experience a slow down in game speed and everything you do in the game begins to feel like an eternity to accomplish, demolish the trade ports and the refineries, dock your defensive squadrons that guard planets that are out of harms way, or take a break do something else and begin a new session without using trade ports/refineries or only utilize a few of them if absolutely necessary, and notice the difference. I still miss having my massive trade network and watch hundreds of cargo ships tirelessly flying all over my part of the map generating income to me but I don't miss the super slow sessions that took forever to win either.

Another tip is to save and reload, it helped my sessions a bit but only for a short while before the game began to lag again.

Quick checklist:

1) Large map = No trade ports & refineries in general, only use when absolutely needed and on the right planets with the right planetary bonuses.
2) Hangars and starbases with squadron modules located deep within your empire = Dock them, they're of no use being outside and ready to fight since you'll only get the occasional scouts running around deep within your borders unless you're playing against a player that rams right through.
3) Use more efficient armadas or only run around with capital ships with some utility ships/carriers.
4) Save and reload the save file see if it alleviates it a little.
5) If none above works, well... you've got eliminate your opponents faster and nuke their holdings into the void. ;)

I hope the info helps somewhat.