There's airflow behind the fridge/freezer, it will dissapate into the large ambient and the cooling power required is very low due to the insulation and size. 24 to 48hours for an avaerage fridge/freezer to get to operational level.
The air around the condenser is being recharged by convection, and the energy flow is different, you are getting rid of energy rather than pull it from the sourrounding. If you are cooling the surrounding earth with the pipes then there must be some thermal energy flow back to replace than energy otherwise the pipes will be at equilibrium with the soil and you'll be stuck. If not then the soil has infinite thermal capacity and we're looking at free power.
This highlights the problem which is you're not understanding how a heat pump works.
The condenser in a fridge is generating the heat it is not being "recharged" the fridge works the same way as a heat pump they all just heat pumps.
In the fridge analogy the ground is the inside of the fridge and the back of the fridge is inside the house.
So while the fridge is highly insulated and cold inside, heat still gets into the fridge and the temperature will go up and the compressor needs to work to bring the temperature back down again.
The fridge doesn't just get colder and colder till it reaches absolute zero, it sits at a 5'C or what ever the chosen inside temperature is. However the back of the fridge is much hotter than the surrounding air because while the heat energy is being removed from inside the fridge, some additional energy gets added by the compressor.
You need to stop thinking about it as hot and cold, you need to start thinking about it as energy.
You also need to consider that energy moves only in one direction which is from a higher to a lower energy state - so the heat moves into the cold.
In the ground it is the heat from the ground moving into the cold pipes the pipes to not make the ground cold, the ground warms the pipes up. if the energy wasn't there then it would not warm the pipes and the system would stop working, for there not to be energy in the ground it would need to be at absolute zero -273'C
If there is not enough energy in the ground then the system is less efficient and more energy needs to be added from the pump, but there is no situation where the system would just stop working because the ground was cold.
This gets very complicated but its not regulated by electronics its regulated by the pressure of whatever you are compressing and how that pressure changes round the system.
The working of a fridge or freezer is exactly the same, just one gets colder than the other, that's managed by changing the pressures in the system.
The more efficient system extracts more energy from the ground so if you are making the ground really cold and not needing to add as much energy from the pump you have a more efficient system.