Why can't I reach my subdomains on one machine, but can on another?

kosymodo

kosymodo

kosymodo

Associate
Joined
18 Jun 2020
Posts
403
Location
Warminster
Hey folks - this is a bit of a weird one (to me anyway). I'd appreciate some advice!

I have three laptops, the first being my work Windows laptop, the second being a personal Windows laptop, and the third being a new personal MacBook Pro. What I'm finding is that I can access my subdomains on both Windows laptops, but not on my MBP. For example, I have Bitwarden self-hosted at bitwarden.DOMAIN.co.uk. I can access Bitwarden on both Windows laptops, but if I try to access on my MBP it throws up errors. The same applies to other subdomains, such as pihole.DOMAIN.co.uk.

Something which might help diagnose the issue is that when trying to access pihole.DOMAIN.co.uk, the error message I get is "the certificate for this site is invalid", and when I click to accept the risk and visit the site, it takes me to my router's homepage, but with the correct address in the address bar, albeit suffixed with additional info (i.e. the browser tab is called UniFi OS, and the address bar shows https://pihole.domain.co.uk/login?redirect=/admin/login.php)

What I find odd about the above example is that my reverse DNS is handled by Nginx Proxy Manager, in which the Pihole proxy host is configured as http://pihole.DOMAIN.co.uk, rather than https://pihole.DOMAIN.co.uk

However, the same happens when I try to access subdomains which do have SSL certificates, such as https://synology.domain.co.uk

Any ideas what's going on? More than likely something I'm completely overlooking, but it would be good to know what this is!
 
the-evaluator said:
Does the MBP resolve bitwarden.domain.co.uk correctly? When you say access, are you referring only to HTTP/HTTPS access? What about SSH or something else?
Click to expand...
Thanks @the-evaluator for your reply...

No it doesn't, this is the issue

Access to me = stick it in the address bar of a browser and expect to reach the site (e.g. my Bitwarden vault or my Pihole dashboard)

In terms of SSH, I can SSH into the hosts (Pi's running on 192.168.1.100 and 192.168.1.200), is that what you mean?
 
the-evaluator said:
Sounds like a DNS issue to me. Is the Macbook using DoH?
Click to expand...
I haven't set up anything specific on the MBP, the DNS is my Pihole which runs Unbound

the-evaluator said:
What browser(s) have you tried?
Click to expand...
OK, so just testing a bit more...it doesn't work in DuckDuckGo browser, it does work in Safari, it doesn't work in Chrome. The Bitwarden app doesn't work (I get an error 'Failed to fetch')

the-evaluator said:
Can you SSH into something from the MBP using the FQDN or can you only do it by IP address?
Click to expand...
The only things I ever SSH into are my Pi's and my router (UDM-Pro), none of which I'd ever use an FQDN to access. What else could I try to SSH into? I'm happy to try anything :)
 
the-evaluator said:
I'm trying to work out what's broken, but I'm fairly sure it's DoH causing it
Click to expand...
Seems like it could actually be a cert error...

A networking friend suggested running openssl s_client -connect synology.domain.co.uk:443 on my MBP and the result was:
depth=0 CN = unifi.local
verify error:num=18:self signed certificate
 
Just throwing this out there, but would mDNS be causing the DNS issue? I'm frantically Googling, and came across mDNS as the cause of something vaguely similar

mDNS is enabled on my UDM-Pro. I have a LAN network (VLAN ID 1) and an IoT network (VLAN ID 10). Do I need mDNS to allow devices on LAN to communicate with devices on IoT? Advice on a forum post (albeit 7yrs old) I found says this "If you don't know why you'd need it, then you don't". I don't really understand why I'd need it, though I appreciate that having 2 VLANs isn't what Joe Public does, but pretty common for those frequenting this forum! I'm therefore stuck between a rock and a hard place in knowing whether my set-up requires it
 
the-evaluator said:
Interesting. Please do the same thing on one of the other laptops.

I can see why hitting your WAN IP (it'll be hairpinning) on port 443 will return the SSL certificate of your UDMP so next step is to work out why the traffic is ending up there.
Click to expand...
Same result on 2 other laptops - but both can access via browsers
 
the-evaluator said:
Err. Then I'm stumped.
Click to expand...
As am I - everything was working fine with my previous MBP. I have set this new one up as a fresh install, and am experiencing issues. Perhaps my next move will be to restore the backup from my previous MBP and see if that overwrites some default settings somewhere
 
@the-evaluator In between our last posts and now, I've been away, had Covid, now back with the bit between my teeth!

I had a bit of a eureka moment last night, where I made a realisation...the problem vanishes as soon as I turn on a VPN.

What I was finding is that the issue was spreading to all my devices, so whereby my iPhone had previously been unaffected, it was showing the same signs of not being able to log into Bitwarden. I then realised that my personal Windows laptop was running a VPN, so thought I'd try turning on the VPN on my iPhone, and voila!

So, all devices work as expected when VPN is on. No devices work as expected when VPN is off. (My work laptop is kinda 'odd' in so much as it runs a split-tunnel VPN for work purposes)

I have tried manually altering the DNS settings on my iPhone to 8.8.8.8 and 9.9.9.9, but no luck. Therefore perhaps not a DNS issue?
 
the-evaluator said:
Have you changed router or firmware at some point?
Click to expand...
Haven't changed my router (UDM-Pro) since 18mths ago. As for firmware, I'd have to check the logs, but I can't remember a recent update.

This does look suspiciously like something related to my router. Just tried accessing it via unifi.ui.com and can't. Also tried on my phone, via the UniFi app, on 4G and can't. I can access it internally though, so at least I have that!
 
@the-evaluator as you've so been been so helpful, here's today's update...

In desperation, I rebooted my router, hoping it might give a quick fix. Not sure if that was what worked, but I can now access the router via unifi.ui.com No idea what the issue there was...something temporary though

Ubiquiti support answered my ticket and said:

Rest assured, we will help you find a solution to the issue you are facing. When I visit at bitwarden.DOMAIN.co.uk it works for me fine. However, pihole.DOMAIN.co.uk doesn't works for me as well. Looking at the support file, I don't see any port forward rule for the host server with IP 192.168.1.100 with port 8080. Can you please re-verify your settings & configuration for your server? Re-check your port forward setting for the server 192.168.1.100, I don't see any port forward setting for this IP address. (The issue with the pihole address was that he'd literally used pihole.domain.co.uk instead of substituting domain with the actual domain name, d'oh!)

Additionally, the issue discribed appears to be related to the hairpin NAT problem. I see that you have a drop firewall rule configured under LAN_IN to disable communication between the LAN/VLAN network. This could prevent your traffic from being masquerade correctly on the LAN/VLAN network. I'd recommend to delete this firewall rule and then check and see if everything works fine.

[3410:433789] -A UBIOS_LAN_IN_USER -m set --match-set UBIOS_644836185d7a4a7ccfb96efd src -m set --match-set UBIOS_644836185d7a4a7ccfb96efd dst -m comment --comment 00000001095216680482 -j DROP

If the issue still persists or need further assistance, please share the updated copy of the support file, with exact IP & port forward details of the server used to guide you in the right direction. Also, run nslookup from your local client to check on what IP address those domain are being resolved, share the output results.

I tried pausing the drop firewall rule, but it made no difference... This was expected, as I can't see what the relevance of that firewall rule is to my issue, and everything had been running perfectly for months, even with that rule enabled

I then tried the nslookup queries he'd asked for...

nslookup from my MBP (not working) shows:
Server: 2606:4700:4700::1111
Address: 2606:4700:4700::1111#53
Non-authoritative answer:
Name: bitwarden.DOMAIN.co.uk
Address: 188.74.119.159

nslookup from my work laptop (working) shows:
Server: pi.hole
Address: 192.168.1.99
Non-authoritative answer:
Name: bitwarden.DOMAIN.co.uk
Addresses: 2a10:bcc0:83e8:1::2b
188.74.119.159

Noticing the difference between the IPv6 server on my MBP and the IPv4 server on my laptop, I thought I'd stumbled across a fix...

If I disable IPv6 on the MBP, then run nslookup again, I get:

Server: 192.168.1.99
Address: 192.168.1.99#53
Non-authoritative answer:
Name: bitwarden.DOMAIN.co.uk
Address: 188.74.119.159

Bitwarden then works. Once I re-enable IPv6 it stops working. Therefore, I thought to myself that it must be an IPv6 issue. However...

On my iPhone, which I have set to manual DNS servers and only specified IPv4 addresses (8.8.8.8 and 9.9.9.9), which means it can't be using IPv6, I still can't access Bitwarden until I enable my VPN

Does any of this make any sense to you, and does it provide any insight into what might be the root of the problem?
 
the-evaluator said:
Does the iPhone itself have an IPv6 address? If it has then it is likely still using IPv6 irrespective if you only specifying IPv4 DNS servers as a DNS server will return AAAA records irrespective if it itself has IPv6 connectivity.
Click to expand...
Good call - although I’d followed instructions on how to ‘disable’ IPv6 on my iPhone (https://discussions.apple.com/thread/254787535?answerId=258925543022&sortBy=rank#258925543022 - though it’s a workaround, rather than actually disabling anything), it still has an IPv6 address

the-evaluator said:
How long have you had an IPv6 internet connection? MacOS prefers IPv6 over IPv4 so if there isn't a firewall rule in the UDM-P to allow that traffic over IPv6 and there's a AAAA, then I can see why MacOS is having problems.
Click to expand...
Well, I’ve had IPv6 enabled on my UDM-P ever since I set it up, so many months ago. However, whether IPv6 was actually working since I changed ISP to Lit Fibre is a contentious question!

So, if this all boils down to an IPv6 misconfiguration/missing firewall rule, rather than a cert issue or a DNS issue, I’m relatively happy, as the fix will be somewhat easier! However, I’ll need to have a think now about how to add such a rule into the UDM-P. I’ll also need to play around some more tomorrow to actually nail down this hypothesis that IPv6 is the root of the issue...

Many many thanks for your help this far, it’s been a baffling one!
 
the-evaluator said:
I think that's key. When did you change to Lit, and which ISP were you with before? If you were with an ISP that doesn't give IPv6 addresses or the config of you UDMP didn't match the IPv6 requirements of that ISP then I would expect you'd have no IPv6 connectivity at all. Having said that, do Lit is PD for IPv6?
Click to expand...
I switched in April (2024), from Giganet. I've had IPv6 enabled since my days with Aquiss (switched from Aquiss to Giganet in April 2023). Aquiss made it easy to run IPv6, Giganet less so, Lit give no info whatsoever. Therefore, my settings have been bodged from advice on here, like this https://forums.overclockers.co.uk/threads/lit-fibre.18955934/post-37427071 (the bit relating to DHCPv6 Prefix Delegation)

the-evaluator said:
Yeah, that's what it feels like. Adding the rule should be easy enough, just match the IPv4 rule. Have a look at the 'WAN_IN' rules (I think) and see what you find.
Click to expand...
OK, will have a play!

the-evaluator said:
MacOS prefers IPv6 over IPv4 so if there isn't a firewall rule in the UDM-P to allow that traffic over IPv6 and there's a AAAA, then I can see why MacOS is having problems.
Click to expand...
OK, so this explains why even though both Windows laptop and MBP are getting IPv6 addresses, only the MBP is having problems. Howeever, it doesn't explain why the new MBP is having problems where the old MBP wasn't. Perhaps this is just a very unlucky coincidence with Lit making a pigs ear of their IPv6 routing, as per the same post I referred to earlier https://forums.overclockers.co.uk/threads/lit-fibre.18955934/post-37427071 (the bit relating to broken IPv6 routing)
 
the-evaluator said:
Adding the rule should be easy enough, just match the IPv4 rule. Have a look at the 'WAN_IN' rules (I think) and see what you find.
Click to expand...
Hmmm, not sure I see what I'm supposed to do here. The relevant IPv4 (i.e. Internet In) rules were automatically added when the port forwarding was set up. In the screengrab below, the only rules which I've added are the ICMP rules. Therefore, I'm unsure how I can replicate the rules as Internet v6 In rules?

 
the-evaluator said:
Now, with that all said why do you have it set this way? Doing stuff with port forwarding isn't overly secure. Personally I'd look to get your DNS sorted so that bitwarden.domain.co.uk (for example) resolves to an internal IP address and you could then bin off the port forwarding. If you gave it just an A record then your clients would have no choice but to use IPv4 to reach it.
Click to expand...
OK, I'll answer your post in reverse, as I think the answer to this part will determine what needs to be done to the initial part.

The reason for the port forwarding is nothing to do with Bitwarden directly. The port forwarding is what I set up when configuring Nginx Proxy Manager to act as reverse proxy for all my services hosted on subdomains. Maybe I didn't explain this very well earlier in the thread. So, my setup is something like this:

192.168.1.200
Pi running Portainer
Multiple containers including Nginx Proxy Manager

192.168.1.99
Pi running Pihole

192.168.1.100
Pi running Portainer
Single Bitwarden container

192.168.1.70
Synology NAS
Multiple containers running in Container Manager

192.168.1.71
Pi running Home Assistant

So, by exposing just the .200 Pi using ports 80/443 I'm able to expose all the other services. The reason why I have port 80 mapped to 8765 and port 443 mapped to 8907 is, I think while desperately trying to remember back, to resolve conflicts between other services running on those ports. Does that sound about right?

Therefore, in terms of port forwarding, I don't necessarily think that the bit about getting my DNS sorted is correct? I have to have those port forwards/firewall rules in place for this setup, or at least that's my understanding?

Am I making any sense so far? Hopefully so! In which case what I now need to understand is what I need to do in terms of IPv6, and whether any action is required to set up any Internet v6 In rules?

In terms of the A and AAAA records, all subdomains are set up exactly the same, with the A record being my IPv4 WAN address and the AAAA record being my IPv6 WAN address. I suppose that the key thing here is having the IPv6 WAN address correct, which I'm taking from the UniFi Network dashboard
 
Still not working :(

I tried creating the Internet v6 rules for both ports 80 and 443 as follows:

Type - Internet v6 In
Name - Nginx80 (or Nginx443)
Action - Accept
Protocol - TCP/UDP (mirroring the IPv4 rules)
Before Predefined - ticked
Match Opposite - unticked
Source Address Group - Any
Source Port Group - Any
MAC Address - blank
Destination Address Group - created a new group with the IPv6 address of my .200 Pi entered
Port Group - 8765 (or 8907)
Advanced - set to Auto

Tried on my MBP and on DuckDuckGo I get the error 'An SSL error has occured and a secure connection to the server cannot be made', whilst on Chrome I get the error 'NET::ERR_CERT_AUTHORITY_INVALID' with the offending cert being the unifi.local cert (which makes sense, as the cert should be the cert sitting in NPM, which incidentally doesn't expire until 12/1/25)
 
the-evaluator said:
I get that you're using Nginx (my knowledge of which is extremely rusty) but that doesn't necessitate port forwarding and externally accessible services. Are you specifically wanting them available externally?
Click to expand...
Well yeah, particularly because if I didn't, I wouldn't be able to access Bitwarden from anywhere outside my local network. There are other services I'd also miss out on, so for the sake of argument let's say I'll provide external access to all of them.

the-evaluator said:
One of the key differences between IPv4 & IPv6 is NAT. The lack of it (mostly) with IPv6 specifically. If you're accessing your Nginx host externally over IPv6 then your AAAA record needs to point to the IPv6 address of the Nginx box, not at the WAN interface of your router. There's no port forwarding with IPv6 so if your AAAA record points to the WAN interface if your router than that's all you'll be able to reach.
Click to expand...
Ahhh, now this I believe is what you call hitting the nail on the head! So, if I'm understanding correctly, by using the WAN interface of my router, there's little wonder I'm getting the unifi.local cert error! It's trying to find the Bitwarden SSL cert in that location, finds the unifi.local cert, and errors. Is that right?
 
john_smith said:
I'd expose nothing externally and VPN home when you need bitwarden. :-)
Click to expand...
But I use Bitwarden for every single website I have a password for. If I had to VPN home each time I wanted toi enter a password into a website, I'd be constantly enabling a VPN...

Am I missing something obvious?
 
You must log in or register to reply here.
Back
Top Bottom