The set screw probably has Locktite on it. It will take extra pressure to get it to loosen. Maybe some heat (hair dryer maybe) will help to loosen it. Try to hold the blade holder firmly when you apply pressure to the set screw. Maybe remove the holder from the saw so you can get a better grip on it. I use a medium strength locktite.
Larry