Just puncture the lining and drop all your concrete, brick, blocks and slabs in then you can look at what you will actually need to fill the rest of the space. I'm in the process of removing my concrete drive and creating a soak away, so I have dug out the clay soil by hand and removed it but the broken up concrete has been laid in the bottom and covered in gravel. This has saved me buying in hardcore ( most people were saying get the grab lorry in to remove the rubble)
Have to agree with this, I think that this whole thing is being over thought.
Personally, I would knock that wall down (between house and pool area) as suggested and turn most of that back to lawn and generally renovate, but definitely keep the pool even if I stopped heating it. It looks like a really nice pool and to me it seems a shame to remove it.
alternatively though, as per Tony’s post, do all of the demolition, pull the slabs, and after puncturing the pool to have it drain I would bulldoze (mini digger) all of it into the hole and then just reevaluate. Worst case you get a giant hardcore and then topsoil delivery and do the same. This needn’t be a hugely expensive task and if you do fill it in I would leave the structure well alone below ground personally.