Why can't I reach my subdomains on one machine, but can on another?

[kosymodo]

@the-evaluator as you've so been been so helpful, here's today's update...

In desperation, I rebooted my router, hoping it might give a quick fix. Not sure if that was what worked, but I can now access the router via unifi.ui.com No idea what the issue there was...something temporary though

Ubiquiti support answered my ticket and said:

Rest assured, we will help you find a solution to the issue you are facing. When I visit at bitwarden.DOMAIN.co.uk it works for me fine. However, pihole.DOMAIN.co.uk doesn't works for me as well. Looking at the support file, I don't see any port forward rule for the host server with IP 192.168.1.100 with port 8080. Can you please re-verify your settings & configuration for your server? Re-check your port forward setting for the server 192.168.1.100, I don't see any port forward setting for this IP address. (The issue with the pihole address was that he'd literally used pihole.domain.co.uk instead of substituting domain with the actual domain name, d'oh!)

Additionally, the issue discribed appears to be related to the hairpin NAT problem. I see that you have a drop firewall rule configured under LAN_IN to disable communication between the LAN/VLAN network. This could prevent your traffic from being masquerade correctly on the LAN/VLAN network. I'd recommend to delete this firewall rule and then check and see if everything works fine.

[3410:433789] -A UBIOS_LAN_IN_USER -m set --match-set UBIOS_644836185d7a4a7ccfb96efd src -m set --match-set UBIOS_644836185d7a4a7ccfb96efd dst -m comment --comment 00000001095216680482 -j DROP

If the issue still persists or need further assistance, please share the updated copy of the support file, with exact IP & port forward details of the server used to guide you in the right direction. Also, run nslookup from your local client to check on what IP address those domain are being resolved, share the output results.

I tried pausing the drop firewall rule, but it made no difference... This was expected, as I can't see what the relevance of that firewall rule is to my issue, and everything had been running perfectly for months, even with that rule enabled

I then tried the nslookup queries he'd asked for...

nslookup from my MBP (not working) shows:
Server: 2606:4700:4700::1111
Address: 2606:4700:4700::1111#53
Non-authoritative answer:
Name: bitwarden.DOMAIN.co.uk
Address: 188.74.119.159

nslookup from my work laptop (working) shows:
Server: pi.hole
Address: 192.168.1.99
Non-authoritative answer:
Name: bitwarden.DOMAIN.co.uk
Addresses: 2a10:bcc0:83e8:1::2b
188.74.119.159

Noticing the difference between the IPv6 server on my MBP and the IPv4 server on my laptop, I thought I'd stumbled across a fix...

If I disable IPv6 on the MBP, then run nslookup again, I get:

Server: 192.168.1.99
Address: 192.168.1.99#53
Non-authoritative answer:
Name: bitwarden.DOMAIN.co.uk
Address: 188.74.119.159

Bitwarden then works. Once I re-enable IPv6 it stops working. Therefore, I thought to myself that it must be an IPv6 issue. However...

On my iPhone, which I have set to manual DNS servers and only specified IPv4 addresses (8.8.8.8 and 9.9.9.9), which means it can't be using IPv6, I still can't access Bitwarden until I enable my VPN

Does any of this make any sense to you, and does it provide any insight into what might be the root of the problem?

 

[the-evaluator]

On my iPhone, which I have set to manual DNS servers and only specified IPv4 addresses (8.8.8.8 and 9.9.9.9), which means it can't be using IPv6, I still can't access Bitwarden until I enable my VPN

Does the iPhone itself have an IPv6 address? If it has then it is likely still using IPv6 irrespective if you only specifying IPv4 DNS servers as a DNS server will return AAAA records irrespective if it itself has IPv6 connectivity.

How long have you had an IPv6 internet connection? MacOS prefers IPv6 over IPv4 so if there isn't a firewall rule in the UDM-P to allow that traffic over IPv6 and there's a AAAA, then I can see why MacOS is having problems.

 

[kosymodo]

Does the iPhone itself have an IPv6 address? If it has then it is likely still using IPv6 irrespective if you only specifying IPv4 DNS servers as a DNS server will return AAAA records irrespective if it itself has IPv6 connectivity.

Good call - although I’d followed instructions on how to ‘disable’ IPv6 on my iPhone (https://discussions.apple.com/thread/254787535?answerId=258925543022&sortBy=rank#258925543022 - though it’s a workaround, rather than actually disabling anything), it still has an IPv6 address

How long have you had an IPv6 internet connection? MacOS prefers IPv6 over IPv4 so if there isn't a firewall rule in the UDM-P to allow that traffic over IPv6 and there's a AAAA, then I can see why MacOS is having problems.

Well, I’ve had IPv6 enabled on my UDM-P ever since I set it up, so many months ago. However, whether IPv6 was actually working since I changed ISP to Lit Fibre is a contentious question!

So, if this all boils down to an IPv6 misconfiguration/missing firewall rule, rather than a cert issue or a DNS issue, I’m relatively happy, as the fix will be somewhat easier! However, I’ll need to have a think now about how to add such a rule into the UDM-P. I’ll also need to play around some more tomorrow to actually nail down this hypothesis that IPv6 is the root of the issue...

Many many thanks for your help this far, it’s been a baffling one!

 

[the-evaluator]

Well, I’ve had IPv6 enabled on my UDM-P ever since I set it up, so many months ago. However, whether IPv6 was actually working since I changed ISP to Lit Fibre is a contentious question!

I think that's key. When did you change to Lit, and which ISP were you with before? If you were with an ISP that doesn't give IPv6 addresses or the config of you UDMP didn't match the IPv6 requirements of that ISP then I would expect you'd have no IPv6 connectivity at all. Having said that, do Lit is PD for IPv6?

So, if this all boils down to an IPv6 misconfiguration/missing firewall rule, rather than a cert issue or a DNS issue, I’m relatively happy, as the fix will be somewhat easier! However, I’ll need to have a think now about how to add such a rule into the UDM-P. I’ll also need to play around some more tomorrow to actually nail down this hypothesis that IPv6 is the root of the issue...

Yeah, that's what it feels like. Adding the rule should be easy enough, just match the IPv4 rule. Have a look at the 'WAN_IN' rules (I think) and see what you find.

Many many thanks for your help this far, it’s been a baffling one!

No problem, it's been good to get the grey matter working

 

[kosymodo]

I think that's key. When did you change to Lit, and which ISP were you with before? If you were with an ISP that doesn't give IPv6 addresses or the config of you UDMP didn't match the IPv6 requirements of that ISP then I would expect you'd have no IPv6 connectivity at all. Having said that, do Lit is PD for IPv6?

I switched in April (2024), from Giganet. I've had IPv6 enabled since my days with Aquiss (switched from Aquiss to Giganet in April 2023). Aquiss made it easy to run IPv6, Giganet less so, Lit give no info whatsoever. Therefore, my settings have been bodged from advice on here, like this https://forums.overclockers.co.uk/threads/lit-fibre.18955934/post-37427071 (the bit relating to DHCPv6 Prefix Delegation)

Yeah, that's what it feels like. Adding the rule should be easy enough, just match the IPv4 rule. Have a look at the 'WAN_IN' rules (I think) and see what you find.

OK, will have a play!

MacOS prefers IPv6 over IPv4 so if there isn't a firewall rule in the UDM-P to allow that traffic over IPv6 and there's a AAAA, then I can see why MacOS is having problems.

OK, so this explains why even though both Windows laptop and MBP are getting IPv6 addresses, only the MBP is having problems. Howeever, it doesn't explain why the new MBP is having problems where the old MBP wasn't. Perhaps this is just a very unlucky coincidence with Lit making a pigs ear of their IPv6 routing, as per the same post I referred to earlier https://forums.overclockers.co.uk/threads/lit-fibre.18955934/post-37427071 (the bit relating to broken IPv6 routing)

 

[kosymodo]

Adding the rule should be easy enough, just match the IPv4 rule. Have a look at the 'WAN_IN' rules (I think) and see what you find.

Hmmm, not sure I see what I'm supposed to do here. The relevant IPv4 (i.e. Internet In) rules were automatically added when the port forwarding was set up. In the screengrab below, the only rules which I've added are the ICMP rules. Therefore, I'm unsure how I can replicate the rules as Internet v6 In rules?

 

[the-evaluator]

I've not needed to setup access to internal stuff from the internet over v6 before but as a test I just allowed 22/tcp traffic from my externally hosted UniFi controller to a Pi.

These instructions will allow access that matches your port forwarding though personally I wouldn't leave the source as 'any' unless that was absolutely necessary,

Go to firewall rules and select 'Internet v6' and hit create entry.

Type - Internet V6 In
Name - Whatever you like
Protocol - Whatever you need
Source Address Group - Any
Source Port Group - Any
Destination Address Group - If you haven't already set them up, click New and add, give it a name and appropriate IPv6 address
Destination Port Group - Add a new one that matches just the port you're wanting access to

Save and give the UDMP a minute or two to provision. Test.

If you want, I can verify connectivity over IPv6 from here if it still isn't working for you.

Now, with that all said why do you have it set this way? Doing stuff with port forwarding isn't overly secure. Personally I'd look to get your DNS sorted so that bitwarden.domain.co.uk (for example) resolves to an internal IP address and you could then bin off the port forwarding. If you gave it just an A record then your clients would have no choice but to use IPv4 to reach it.

 

[kosymodo]

Now, with that all said why do you have it set this way? Doing stuff with port forwarding isn't overly secure. Personally I'd look to get your DNS sorted so that bitwarden.domain.co.uk (for example) resolves to an internal IP address and you could then bin off the port forwarding. If you gave it just an A record then your clients would have no choice but to use IPv4 to reach it.

OK, I'll answer your post in reverse, as I think the answer to this part will determine what needs to be done to the initial part.

The reason for the port forwarding is nothing to do with Bitwarden directly. The port forwarding is what I set up when configuring Nginx Proxy Manager to act as reverse proxy for all my services hosted on subdomains. Maybe I didn't explain this very well earlier in the thread. So, my setup is something like this:

192.168.1.200
Pi running Portainer
Multiple containers including Nginx Proxy Manager

192.168.1.99
Pi running Pihole

192.168.1.100
Pi running Portainer
Single Bitwarden container

192.168.1.70
Synology NAS
Multiple containers running in Container Manager

192.168.1.71
Pi running Home Assistant

So, by exposing just the .200 Pi using ports 80/443 I'm able to expose all the other services. The reason why I have port 80 mapped to 8765 and port 443 mapped to 8907 is, I think while desperately trying to remember back, to resolve conflicts between other services running on those ports. Does that sound about right?

Therefore, in terms of port forwarding, I don't necessarily think that the bit about getting my DNS sorted is correct? I have to have those port forwards/firewall rules in place for this setup, or at least that's my understanding?

Am I making any sense so far? Hopefully so! In which case what I now need to understand is what I need to do in terms of IPv6, and whether any action is required to set up any Internet v6 In rules?

In terms of the A and AAAA records, all subdomains are set up exactly the same, with the A record being my IPv4 WAN address and the AAAA record being my IPv6 WAN address. I suppose that the key thing here is having the IPv6 WAN address correct, which I'm taking from the UniFi Network dashboard

 

[kosymodo]

Still not working

I tried creating the Internet v6 rules for both ports 80 and 443 as follows:

Type - Internet v6 In
Name - Nginx80 (or Nginx443)
Action - Accept
Protocol - TCP/UDP (mirroring the IPv4 rules)
Before Predefined - ticked
Match Opposite - unticked
Source Address Group - Any
Source Port Group - Any
MAC Address - blank
Destination Address Group - created a new group with the IPv6 address of my .200 Pi entered
Port Group - 8765 (or 8907)
Advanced - set to Auto

Tried on my MBP and on DuckDuckGo I get the error 'An SSL error has occured and a secure connection to the server cannot be made', whilst on Chrome I get the error 'NET::ERR_CERT_AUTHORITY_INVALID' with the offending cert being the unifi.local cert (which makes sense, as the cert should be the cert sitting in NPM, which incidentally doesn't expire until 12/1/25)

 

[the-evaluator]

The reason for the port forwarding is nothing to do with Bitwarden directly. The port forwarding is what I set up when configuring Nginx Proxy Manager to act as reverse proxy for all my services hosted on subdomains. Maybe I didn't explain this very well earlier in the thread. So, my setup is something like this:

I get that you're using Nginx (my knowledge of which is extremely rusty) but that doesn't necessitate port forwarding and externally accessible services. Are you specifically wanting them available externally?

In terms of the A and AAAA records, all subdomains are set up exactly the same, with the A record being my IPv4 WAN address and the AAAA record being my IPv6 WAN address. I suppose that the key thing here is having the IPv6 WAN address correct, which I'm taking from the UniFi Network dashboard

One of the key differences between IPv4 & IPv6 is NAT. The lack of it (mostly) with IPv6 specifically. If you're accessing your Nginx host externally over IPv6 then your AAAA record needs to point to the IPv6 address of the Nginx box, not at the WAN interface of your router. There's no port forwarding with IPv6 so if your AAAA record points to the WAN interface if your router than that's all you'll be able to reach.

 

[kosymodo]

I get that you're using Nginx (my knowledge of which is extremely rusty) but that doesn't necessitate port forwarding and externally accessible services. Are you specifically wanting them available externally?

Well yeah, particularly because if I didn't, I wouldn't be able to access Bitwarden from anywhere outside my local network. There are other services I'd also miss out on, so for the sake of argument let's say I'll provide external access to all of them.

One of the key differences between IPv4 & IPv6 is NAT. The lack of it (mostly) with IPv6 specifically. If you're accessing your Nginx host externally over IPv6 then your AAAA record needs to point to the IPv6 address of the Nginx box, not at the WAN interface of your router. There's no port forwarding with IPv6 so if your AAAA record points to the WAN interface if your router than that's all you'll be able to reach.

Ahhh, now this I believe is what you call hitting the nail on the head! So, if I'm understanding correctly, by using the WAN interface of my router, there's little wonder I'm getting the unifi.local cert error! It's trying to find the Bitwarden SSL cert in that location, finds the unifi.local cert, and errors. Is that right?

 

[john_smith]

I'd expose nothing externally and VPN home when you need bitwarden.

 

[kosymodo]

I'd expose nothing externally and VPN home when you need bitwarden.

But I use Bitwarden for every single website I have a password for. If I had to VPN home each time I wanted toi enter a password into a website, I'd be constantly enabling a VPN...

Am I missing something obvious?

 

[john_smith]

Can bit warden run a cache locally?

 

[kosymodo]

Can bit warden run a cache locally?

No idea - it's not Bitwarden 'full fat' I'm hosting myself, it's actually Vaultwarden (the rewritten server implementation). I'll do some research!

 

[john_smith]

Sorry to send you off on a wild goose chase.

 

[the-evaluator]

Well yeah, particularly because if I didn't, I wouldn't be able to access Bitwarden from anywhere outside my local network. There are other services I'd also miss out on, so for the sake of argument let's say I'll provide external access to all of them.

Ok, makes sense. Still not something I'd want to do, but that's your decision.

Ahhh, now this I believe is what you call hitting the nail on the head! So, if I'm understanding correctly, by using the WAN interface of my router, there's little wonder I'm getting the unifi.local cert error! It's trying to find the Bitwarden SSL cert in that location, finds the unifi.local cert, and errors. Is that right?

Yep, exactly. If you want to keep using IPv6 then it should just be a case of making sure that the Internet v6 In rule you created points to the IPv6 address of the target system and updating the AAAA record in the same way.

 

[kosymodo]

Ok, makes sense. Still not something I'd want to do, but that's your decision.

Yep, exactly. If you want to keep using IPv6 then it should just be a case of making sure that the Internet v6 In rule you created points to the IPv6 address of the target system and updating the AAAA record in the same way.

Glad to report that all's now working fine

Thanks yet again for all your help with this, it really is appreciated! I have no idea how things had been working OK until now with the wrong address on the AAAA records, and the lack of firewall rule, so will just place it in my 'mysterious' box! I've definitely learnt from this experience, and you never know who it might help in the future if they're searching for a similar issue...

 

[the-evaluator]

Glad to report that all's now working fine

Thanks yet again for all your help with this, it really is appreciated! I have no idea how things had been working OK until now with the wrong address on the AAAA records, and the lack of firewall rule, so will just place it in my 'mysterious' box! I've definitely learnt from this experience, and you never know who it might help in the future if they're searching for a similar issue...

Not a problem, happy to help. I wonder if it was previously working because the Mac was preferring IPv4 over v6, I can't think why it'd do that though. I like your plan of forgetting about it and moving on.

IPv6 has a few key differences to IPv4 which unless you're aware can easily catch you out.

 

[Rainmaker]

Glad to report that all's now working fine

Thanks yet again for all your help with this, it really is appreciated! I have no idea how things had been working OK until now with the wrong address on the AAAA records, and the lack of firewall rule, so will just place it in my 'mysterious' box! I've definitely learnt from this experience, and you never know who it might help in the future if they're searching for a similar issue...

Saved my fingers, I was just cracking knuckles to write in here. Good on you chaps. This came up for me recently, too. You have to remember, OP, that the reverse proxy is only for your masqueraded IPv4 WAN IP. The IPv6 stuff is completely separate and doesn't need forwards etc. You need to think of them separately. The single (?) public IPv4 address on WAN from your ISP forwards traffic to your reverse proxy (i.e. Nginx), which only listens on 443/tcp and terminates the TLS with its cert and relays the traffic to the appropriate backend service and port. All well and good. With IPv6 though, whether SLAAC or DHCPv6, you are needing a firewall traffic rule (rather than a forward) to allow traffic to the actual server port in question (eg Vaultwarden 8000/tcp, nzbget 6789/tcp).

Those machines now suddenly also need a TLS cert of their own, as you can't rely on a single reverse proxy with a single wildcard cert for your domain. Any IPv6 connections are coming in direct, with the server itself sitting facing the Internet instead of your firewall and reverse proxy. It's two completely separate streams of traffic, and causes headaches for stuff like subdomains. You might have vaultwarden.domain.com:443 for IPv4, because of the reverse proxy listening helpfully on 443 for you. However, you'll still be needing vaultwarden.domain.com:8000 for IPv6 unless you, for example, run every server on separate IPs and have them all listen on 443, maybe by using LXC on Proxmox or something. It's a bit of a pain running dual stack, but it's nice once you have it all figured out.