Toger Blog

Varnish Cache and req.backend.healthy

An odd issue I ran into the other day: I had a Varnish 3 instance that had logic hinging on req.backend.healthy to show a special error page if all the backends were down. That logic inexpliciably triggered even though all my backends were up! After much head-scratching I identified the issue: one of my historical VCLs was still loaded that no longer had any healthy backends (due to repeated autoscaling up / down), and although the current definition of that director had healthy backends, the historical one did not. Varnish has a habit of not letting go of old VCLs even if you specify vcl.discard on them. So, req.backend.healthy will show the director as down if any prior definition of that director is down. Since the only way to definitively remove VCLs from memory is a restart (which flushes the memory cache), this makes req.backend.healthy fairly unreliable.

This is in v3 and may not apply to v4 anymore.