Four of the five Chalcots Estate towers in Camden, north London, were deemed unsafe after they were found to use cladding similar to that on the Grenfell Tower, widely blamed for the rapid spread of the massive blaze last week that is presumed to have killed 79 people.
Some 27 high-rise buildings in 15 local authorities have already failed urgent fire tests conducted after the Grenfell fire, the British government announced on Saturday, raising fears that thousands more may need to leave their homes.
Around 4,000 residents from all five Chalcots towers were initially evacuated, but one of the five was later deemed safe and residents allowed to return.
Other residents faced chaos, with temporary accommodation offered in a local leisure centre and hotels, but some refused to move.
Camden council leader Georgia Gould said 83 residents had refused to leave, adding that the situation "will become a matter for the fire service".
Outside one of the leisure centres, evacuees accused the authorities of sowing "panic".
"At 8.30pm (local time) yesterday they told us: ‘You have to leave.’ I don’t understand why," said Murtaza Taha, 27. "They made people panic. Inside [the centre], they are all afraid, they are all crying."
"They say they are going to find us a place to stay, they say for two to four weeks, but you never believe the council. If they say weeks, they mean months," he added.
Rosie Turner, 27, said she had initially refused to come because of concerns over her nine-week-old baby.
"There is nothing for him here, everyone is on top of each other," she said. "They should have done it in a proper and organised way, we could have arranged to stay with our family."
"Today I’m gonna go back to my flat, I don’t care."
Council leader Ms Gould acknowledged it was "a scary time" but vowed "to make sure that residents stay safe.
"The cost we can deal with later," she added.
The council has booked hotels across London and the works are expected to take up to four weeks.
In an update early on Saturday, the council said it had secured "hundreds of hotel beds for Chalcots’ residents".
Prime minister Theresa May, meanwhile, said on Saturday that the government would do "what is necessary" to ensure people would have somewhere to stay.
* Agence France-Presse