From the UT FAQ:
http://www.unreality.org/cleaned/
The automatic installation of mutators does not start when I doubleclick them. What helps?
If this doesn't happen, the association of the UMOD extension to the UT installation program is broken. In this case, you have to do the association manually. Do the following:
1. Open the Windows Explorer or any Windows folder view. Choose [View]®[Options]®[File Types].
2. Look through the list for a file type "Unreal Module" with extension UMOD. If it exists, press [Edit…]. If it does not exist, press [New type …] and add a new file type with extension UMOD.
3. Under "Actions" look for an action called "Install …". If it already exists, [Edit…] it. If it does not exist, create it with [New…]
4. Under "Application used to perform action", put in:
[your correct path} …\UnrealTournament\System\setup.exe install "%1"
For example for me, this would be:
D.\Games\ UnrealTournament\System\setup.exe install "%1"