In the contest of the championships so far, Muir led through the first lap before slipping back as the pace dropped.
With 30m to go, the 24-year-old Scot had battled back into contention only to be overtaken by American Jenny Simpson and South Africa’s Caster Semenya in the final strides.
“I gave it everything I could,” said Muir, who missed out on bronze by seven hundredths of a second.
“I tied up in the last 15 yards. I knew it was close. It happened so late in the race. I couldn’t react, but I wouldn’t have been able to because I was so tired.”
With Ethiopian defending champion Genzebe Dibaba nowhere and the Netherlands’ Sifan Hassan, fastest woman in the world this year, tying up at the death, it was a frenzied end to a race as messy as it was thrilling.
Muir’s fellow Briton, Laura Weightman, crossed the line in sixth place.
Minutes earlier, Olympic champion Omar McLeod had given Jamaica its first gold medal of these championships as he powered to 110m hurdles gold.
But it was the 1500m that had the capacity crowd on its feet, and while Muir’s effort was brave, there will be debate about whether her tactics were the right ones for the occasion.
After a first lap of 65 seconds she slowed it down to 71secs on the next, with Hassan and Simpson coming past her as she ceded control.
At that stage Olympic 800m champion Semenya was way back, but the South African used her speed in the home straight as USA’s Simpson once again timed her own effort to perfection as Dibaba went backwards.
McLeod, 23, a sub-10 second runner over 100m flat, had held off Sergey Shubenkov by a tenth of a second in 13.04secs, with Hungary’s Balazs Baji in bronze.
Shubenkov is competing here as a neutral athlete, one of 18 Russians cleared by the International Association of Athletics Federations’ doping review board following the World Anti-Doping Agency report into their nation’s state-sponsored doping programme.
But there was no fairytale return to this stadium for American 2012 Olympic champion Aries Merritt, who had a kidney transplant operation two years ago.
Rojas beats Ibarguen as Hitchon misses out
Rojas’ 14.91m in the penultimate round stole back a lead that Ibarguen had herself taken back with her third-round 14.89m, the Colombian going close again in the final round only to come up three centimetres short of gold.
There were hopes within the British team that Sophie Hitchon might replicate or even improve upon the hammer bronze she won at the Olympics a year ago.
But with Poland’s double Olympic champion Anita Wlodarczyk, responsible for the 13 biggest throws of all time, taking the most predictable gold of the championships with her fifth-round 77.90m, Hitchon’s final round throw of 72.32m was her biggest of the night but enough only for seventh.
The 26-year-old from Blackburn struggled to hold back the tears afterwards, her personal disappointment also bad news for a team that have already seen medal hopes Katarina Johnson-Thompson, Holly Bleasdale and Andy Pozzi miss out.
“I felt like I was in better shape and if I had the rhythm I had in qualification, you don’t know what could have happened,” said Hitchon.
“Of course I am [going to beat myself up about the result], that’s part of my personality.”
Van Niekerk matched by Britain’s Talbot
“I think I’m in the best shape of my life, so I’m just trying to go with it,” said Talbot. “I’m very grateful to be in the position I am.”
Van Niekerk goes for 400m gold on Tuesday night, but the South African’s great rival Isaac Makwala, the fastest man in the world this year, now has only the one-lap event in his sights after being struck down by food poisoning before his own 200m heat.
Britain’s Nethaneel Mitchell-Blake went through as an automatic qualifier after winning the seventh heat in 20.08secs, his compatriot Zharnel Hughes joining him as fastest loser.
British team captain Eilidh Doyle and Meghan Beesley both moved into the 400m hurdles semi-finals, but Jack Green is out after coming home fourth in his semi-final in 49.93 secs.
Paula Radcliffe, marathon world record holder: “Regarding Muir’s tactics, it’s so hard when you’re in that situation and racing – you’re the only one who can make those decisions. It nearly paid off.”
Brendan Foster, Olympic 10,000m bronze medallist: “A race plan with a second lap of 71 seconds isn’t a brilliant race plan in my view. It can play into the strength of the sprinters. Muir’s running in the last 400m was strong, she went a bit fast on the final bend which took a bit out of her. She was unlucky.”
Darren Campbell, Olympic 200m silver medallist: “The reality is that Mo Farah has the only medal. He is retiring from the track. What are we pinning our hopes on? Some guys have made finals. That is good for next time. But we have got something wrong. If medals are not won, the funding is cut. All this stuff the young athletes have been given will go.”