I think a lot depends on the add-ons when it comes to speed. I don't have as many add-ons on my portable version.
I use the portable at work as well because it has IE and we are not allowed to install anything. However, the portable does leave traces on the PC so I am probably screwed no matter what.
ALso- my portabke is an earlier version. It was a 'custom made' portable I downloaded off of Lifehacker that got pulled a few days later. Apparently FF does not permit anyone to create a portable version package (add-ons already in place, theme, etc) for any kind of distribution. I have never updated it because I really like the way it was customized and it runs faster than my full install at home.
Now- If you have the same version at home as you do on your portable, and the home is is slower----I might think maybe your home PC needs some maintenance and updating.