The Great VOIP VLAN Shuffle (Or: How My ISP Quietly Ghosted My Phone System)

I've been slowly moving my network away from "whatever VLAN felt right at the time" towards something a bit more considered - proper segmentation, one purpose per VLAN, that sort of thing. It's the kind of structural tidy-up that never quite justifies its own project, right up until something else forces you to touch the same bit of config anyway.

That something was the looming IP renumbering. My ISP (not my employer) are moving me off a routed /29 onto a single address, which meant every bit of 1:1 NAT on R5 (my long-suffering 887VA, quietly holding the FTTC circuit together since roughly the dawn of time) needed rewriting from scratch. With the router already open and the NAT table in bits on the workbench, so to speak, it made sense to finish the VLAN restructuring properly at the same time rather than bolt the PBX onto the DMZ again out of habit. So the PBX moved onto its own dedicated VOIP VLAN - 192.168.63.0/24, if you're the sort of person who likes specifics, with the PBX itself parked at 192.168.63.11 via a DHCP reservation on pfSense and the Unifi switch port retagged to match. I even added all the relevent information in to netbox.

In isolation this is a fairly dull change. Create VLAN, move port, update reservation, watch the PBX come back up on the new subnet, feel smug for approximately four minutes. The actual work was in the NAT rewrite that had to go with it, because you can't just 1:1 NAT a PBX any more when you've only got one public IP to your name. Web traffic (80/443) will get consolidated behind HAProxy, SMTP stayed as direct port forwards, and SIP got its own set of static PAT entries for 5060/5061 with a narrowed RTP range — 10000 to 10099 rather than the enormous range FreePBX defaults to, because Cisco IOS has no concept of "just forward this whole range" and I was not about to write ten thousand individual NAT lines by hand like some sort of masochist. A hundred ports is plenty for a home PBX unless you're planning on running a call centre, in which case, good luck to you.

All of which I got working, tested, and quietly proud of. Calls going out, fine. Calls coming in - also fine, as far as I could tell, right up until the point where my dad mentioned he'd tried ringing me three times over the weekend and got nothing. Not "rang out," not "went to voicemail." Nothing. Dead air, like the number didn't exist.

Cue the diagnostic spiral that anyone who's ever run their own phone system will recognise instantly: check the PBX logs (nothing incoming at all), check the pfSense firewall logs (nothing hitting the WAN interface either), check the NAT config on R5 about four times in case I'd fat-fingered a port during the rewrite (I hadn't, though by this point I genuinely wanted to have, because at least that would've been my fault and therefore fixable in the next five minutes). Everything I controlled looked correct. Which is always the least reassuring outcome, because it means the problem is somewhere you don't control.

Eventually - and I mean eventually, this had apparently been broken for the best part of a week by the time I noticed - I did a packet capture on the WAN side and started comparing notes with anyone else on the same voice platform I could find. Turns out my ISP had changed where they were sending inbound call signalling - instead of the dedicated address they'd previously targeted, they'd started pointing it at my main router address instead, and nobody had thought to mention it. My NAT rules were still built around the old target, so every inbound call attempt landed on an address that wasn't expecting SIP traffic at all, and simply vanished. No email. No account notice. No banner in the customer portal. Just a silent, one-sided change on their infrastructure that broke a working setup overnight. It's the network equivalent of someone deciding to redirect your post to a different letterbox and not telling you - you only find out when the letters stop turning up.

Getting anyone at the ISP to actually engage with the problem was its own adventure. The moment I mentioned I was running my own router rather than the one they'd supplied, the conversation had a habit of grinding to a halt - support scripts don't really account for "Cisco device, PBX behind it, and yes I do know what a NAT table is, and this setup hasn't changed in over a year" and there's only so many times you can explain that you're not asking them to configure anything, just tell you where they're currently sending the traffic, before the conversation loops back to "have you tried the router we gave you." I hadn't touched their router in years. It's in a drawer somewhere, being smug about its retirement.

They still haven't actually fixed it on their end - 'We have had to perform IP regrades over the past few weeks.' - Is the message I finally got from their support. What did fix it was giving up on waiting and adding a NAT rule of my own on pfSense, redirecting the traffic hitting that address straight through to the PBX regardless of what my ISP thinks they're doing with it. Which works - calls land, phone rings, everyone's happy - but it's not really a fix, it's a workaround for someone else's unfixed problem, sat there being magically functional for reasons that have nothing to do with them and everything to do with me routing around the damage.

The bit that actually annoys me, with the benefit of hindsight, is that none of my monitoring caught it either, and - small confession - I still haven't fixed that part. I've got Prometheus and blackbox_exporter watching more or less everything else in this house, and not one of them checks whether inbound SIP signalling is actually reaching the PBX from a source I'd expect. It's very much on the list. It has been on the list since this happened. It will probably still be on the list next time something silently breaks, because that appears to be how these things go.

It's not just the voice side, either. I've been wanting to get my mail server onto IPv6 for a while now - if only to stop worrying about IPv4 reputation and blocklists for five minutes - but my ISP doesn't do reverse DNS delegation for the address space, and won't set up PTR records on request either. For outbound mail that's not exactly optional; without a working PTR record half the internet will quietly bin your messages on arrival. So the IPv6 migration sits there, blocked, for reasons that have nothing to do with my own configuration and everything to do with the provider simply not supporting a fairly basic bit of address management. It's one more entry on the growing list of reasons IPv4 is looking less optional and my current ISP looking less permanent.

So: VLANs properly structured, NAT rewritten ready for the renumbering, and a workaround duct-taped over a problem my ISP still hasn't acknowledged fixing. I've been with them well over fifteen years now, largely out of loyalty and because their actual customer service - when you can get someone on the phone who understands what a NAT table is - has genuinely been good. But sitting here with a pfSense rule quietly papering over a fault nobody upstream has bothered to close out, and an IPv6 migration that's going nowhere because of missing reverse DNS delegation, I'm starting to eye up alternatives on both fronts: a different voice provider, and possibly a different ISP altogether, ideally one that still does static IPs, supports IPv6 properly, and doesn't require me to explain what a Cisco router is every time something goes wrong. Fifteen years of loyalty is a lot to walk away from, but it turns out there's a limit.

About the author

Tim Wilkes is a UK-based security architect with over 15 years of experience in electronics, Linux, and Unix systems administration. Since 2021, he's been designing secure systems for a telecom company while indulging his passions for programming, automation, and 3D printing. Tim shares his projects, tinkering adventures, and tech insights here - partly as a personal log, and partly in the hopes that others will find them useful.

Want to connect or follow along?

LinkedIn: [phpsytems]
Twitter / X: [@timmehwimmy]
Mastodon: [@timmehwimmy@infosec.exchange]


If you've found a post helpful, consider supporting the blog - it's a part-time passion that your support helps keep alive.

⚠️ Disclaimer

This post may contain affiliate links. If you choose to purchase through them, I may earn a small commission at no extra cost to you. I only recommend items and services I’ve personally read or used and found valuable.

As an Amazon Associate I earn from qualifying purchases.