What went wrong?
G4S was awarded a 15-year contract in 2011 to operate Birmingham Prison.
Mr Clarke said there had been an "abject failure" of contract management and delivery.
In his correspondence, he pointed to a "dramatic deterioration" in conditions following a riot in 2016 and described a lack of order, with those perpetrating violence able to act with "near impunity".
Groups of staff had locked themselves in their own offices.
In his letter, Mr Clarke said: "It was often difficult to find officers, although we did find some asleep during prisoner lock-up periods."
He said "ineffective frontline management and leadership" were at the heart of the prison's problems.
He wrote: "The inertia that seems to have gripped both those monitoring the contract and delivering it on the ground has led to one of Britain's leading jails slipping into a state of crisis that is remarkable even by the low standards we have seen all too frequently in recent years."
Of the 16 privately run jails in the UK, G4S has contracts to operate five of them, including Birmingham.
Ministers believe the others the company runs - Altcourse, Oakwood, Parc and Rye Hill - are performing well.
But in 2016 the company was forced to transfer the management of Medway Secure Training Centre to the government, after BBC secret filming showed staff allegedly mistreating children held there.
Further concerns about the company emerged last year after another undercover investigation into the G4S-run Brook House immigration removal centre, near Gatwick Airport.
What happens next?
The justice secretary now has 28 days to develop an action plan for HMP Birmingham.
Prisons Minister Rory Stewart said: "What we have seen at Birmingham is unacceptable and it has become clear that drastic action is required to bring about the improvements we require."
A new governor and 30 extra staff are being brought in and the capacity of the jail will be cut to 900 prisoners.
Under the new regime at Birmingham, Paul Newton, who has spent 30 years in the Prison Service and is currently governor at Swaleside jail in Kent, will take charge.
Mr Stewart said while Birmingham faced its own "particular set of challenges", it must start to "live up to the standards seen elsewhere".
"We have good, privately-run prisons across the country," he said.
Jerry Petherick, managing director of G4S Custody & Detention Services, said: "HMP Birmingham is an inner-city remand prison which faces exceptional challenges, including increasingly high levels of prisoner violence towards staff and fellow prisoners.
"The wellbeing and safety of prisoners and prison staff is our key priority and we welcome the six-month step-in and the opportunity to work with the Ministry of Justice to urgently address the issues faced at the prison."
The prison will be returned to G4S when sufficient progress has been made.
Andrea Albutt, president of the Prison Governors Association, said: "We need to stop overcrowding, we need to define what is the purpose of prison and then fund it adequately."
Read more news of London on our site.