Residents close to the former brickworks are plagued by dust, noise and bad smells, a survey finds.