So I take it from your reply to RudeDog that you ARE on DSL????
If thats the case I think you need to be sure you are testing UPLOAD speed and not download , they are 2 different tests! Even if you are getting that type of speed it is not guaranteed. ISPs oversubscribe their products all the time and at any given moment the bottom can drop out of your bandwidth!
I sometimes run a small server off my cable at home to test new maps and I get exactly what you describe ALL the time. I then go check my upload speed and sure enuf its < 100kbps. Must have dozens of people suddenly downloading their porn all at the same time..lol.
Heres a great test program to test your Upload speed:
http://bandwidthplace.com/speedtest/fut.html
Like Rude said, you are only gonna get about 6 players max on cable/dsl. Set the maxpings in your config to about 150 and your maxrate down to about 6500 and that will help a bit.