More info required please. What's the subnet of your LAN, and what's the subnet for your Docker instance running AGH? Is DHCP being handled by the router still, or by AGH? If your LAN subnet is 192.168.1.0/24, your Docker machine (NAS) itself has 192.168.1.5, and your Docker AGH subnet is 172.16.0.0/24 then you should be using 192.168.1.5 for the DNS address. Make sure there are no firewalls in the way, including allowing access on the NAS too; for ports 53, 80, 443, 784, 853 - UDP and TCP.
Make sure AGH is set to listen on 0.0.0.0 (not a particular IP) and ensure the access controls aren't in use, or if they are that all relevant subnets are allowed - for example 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/24.
There are no device limits, or indeed limits of any kind (unless you configure them yourself). How are you adding the DNS to the Asus router? Are you listing it as the router's upstream DNS, or are you setting it as the DNS to be handed out by DHCP?