If it's laggy after 4 people, you probably have a bad upload speed. Goto http://www.dslreports.com and test your speed. With that upload number, you can determine what your maxrate is, which could make your server stable for about 6 people.
Goto dslreports, get your approximate upload speed from their tests there, and post it here, then i'll tell you how to set your maxrate.
2003-01-02 08:06:29 EST: 934 / 134
Your download speed : 934105 bps, or 934 kbps.
A 114 KB/sec transfer rate.
Your upload speed : 134680 bps, or 134 kbps.