Locals have said the town centre has declined in recent years, reporting a rise in anti-social behaviour and empty shops.