Chances are that your assembly has a pin in the registry which stops it from being uninstalled. You can find these here:
* HKEY_CURRENT_USER\Software\Microsoft\Installer\Assemblies\Global
* HKEY_LOCAL_MACHINE\Software\Classes\Installer\Assemblies\Global
Just remove the keys, and you can now uninstall them from the GAC