When i first started playing this game, i'd always go really slow when it came to colonizing planets and made sure they were up and fully upgraded etc before moving on to the next. I read that rapid expanding is much more efficient and a better way to win the game so lately i've been trying that.
Very much so; fast expansion not only provides you the best early-game economic base (lots of extractors) but also provides you plenty of room to grow in the mid-late game. I often don't need to touch logistic planetary upgrades (which can get ludicrously expensive) until well over an hour into the game, and those cost-savings can be funneled into more units to crush my enemies and expand my holdings further.
You still need to buy civilian population upgrades to avoid underdevelopment tax. Asteroids require one upgrade, full-sized planets require two.
However, it seems that when i do this, while it is generally effective in getting me planets, eventually i come to a planet where i colonize it then all of a sudden the enemy just jumps in with like 3 capital ships and a whole mob of frigates.
Your best defense against this is scouting, so you know where the enemy is and what kinds of assets they have. If the first you hear from an attacking enemy is them knocking at your door, it's probably already too late to set up a good defense. If you're outnumbered and don't have any aces up your sleeve, it's usually a good idea to abandon your front-line world and regroup. Unless the enemy has a Marza or Vulkoras (or a Jarrasul above level 6) it usually takes a few minutes to bombard a world, and this can give you lots of time to prepare for a counter-attack.
I also put up 2 millitary labs immediately and start building LRMs and a colony frigate to go in the opposite direction/take care of big militias
As TEC, I would build a second capital ship to lead this force, probably either a Marza or a Sova. This has a dual-benefit of providing you a tank to absorb militia attacks (thus avoiding frigate casualties) and also allowing you to gain levels on this capital ship. It's very nice to be able to have the "cavalry arrive" and a level 4 capital ship join the battle in your favour.
If you're taking casualties with this group, that alone might explain your problem. Ten LRM cost almsot as much as a capital ship, so if you lose that many over the course of several battles that's going to set you back dramatically.
Generally speaking i ususally get in about 5 or 6 planets, and start a trade chain easily. Its right about then though that The AI comes in with just a massive fleet and wipes out one or both of my edge planets and thats where it falls apart.
If the enemy is coming from both sides when you're that size, it sounds to me like you're "fast" expansion isn't quite fast enough. For 5-6 planets, you should typically be 15-20 minutes into the game. Presuming you're playing against the normal AI, your description of three capital ships and a "whole mob" of units seems to point more towards 30-40 minutes into the game, if not longer if they're hitting on both fronts.
I just have no clue what to do. Basically, i can't expand both the fleet AND my empire cause it seems as though im already using most of my resources just expanding one or the other. what should i be doing?
You want to keep your fleet low and focus on expansion and economy in the early minutes of the game, but you also want to scout voraciously. You want to know where the enemy is, what they have, and what planets are likely to be the "middle ground" where you meet. Keep scouting and reorganizing your priorities based on what you see.
When you get close to meeting the enemy (2-3 jumps away from each other) start focusing heavily on military. Produce large numbers of units so you can withstand his first offensive or (even better) push him back from the get-go.
If you want more in-depth advice, feel free to post a replay and I'll critique it for you.