Evacuation of the 22-storey Clare House is underway, with families being moved out of the tower block in Bow that's been declared structurally unsafe.
A survey completed following government guidance for tall buildings found Clare House "cannot be kept safe" without major refurbishment or demolition, and residents have been asked to leave.
The families are to receive £9,500 as compensation and for disturbance, with all removal costs being covered.
Officials from Clarion Housing, which manages the former Tower Hamlets Council block, held a meeting with tenants and leaseholders on Friday.
However, a confrontation developed and the police were called.
“Next thing we knew they called in the police,” tenant Carlton Boulter told the East London Advertiser.
“Some people got angry because they had been offered flats which were withdrawn and offered B&Bs instead. Things got heated.”
Clarion admitted staff "felt threatened" and called police, who took no action.
The families then staged an impromptu meeting that evening on the forecourt of the block, with Tower Hamlets mayor John Biggs and Bethnal Green and Bow MP Rushanara Ali.
Structural defects were discovered at Clare House when a sprinkler system was being installed earlier this year.
Tower blocks like Clare House on the Monteith Estate near Victoria Park were constructed by concrete panels bolted together.
But the slabs were never reinforced as the government ordered after the 1968 Ronan Point disaster at Canning Town, when a kitchen gas explosion blew out several panels, killing three people.
Mayor Biggs said this week: "One assumed that all reinforcements had been done following Ronan Point.
"Some families have spent 50 years in Clare House and to be told they've got to get out in two days is ridiculous."
He was referring to initial conversations which saw Clarion ask residents to be out by Monday.
Clarion's chief executive Clare Miller said: “This is not a decision we have taken lightly. The focus right now is doing everything to support residents moving into temporary accommodation."
The Red Cross and Communities First will be at Clare House to support the evacuation while round-the-clock fire patrols are in the building.
Bethnal Green and Bow MP Rushanara Ali told the Advertiser: "We could have supported Clarion if we knew a bit earlier. It's all been haphazard."
Clarion is paying all removal costs in addition to £6,500 each once families move into permanent new homes, plus £3,000 disturbance payment.
The future of Clare House is yet to be decided.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules here