Locals have reacted to plans to move asylum-seeking families out of a Norfolk hotel for men after hundreds filled the streets to protest.