You dont have to remove all your mods.
Also most all mods are server side only and you dont need to have them installed.
Always disable mods your not currently using.
You can simply rename the .pk3 file back to .zip file.
Remember also pk3 files are loaded in alphabetical order.
so if the name of the pk3 file is after "P" then it will overwrite.