If you're gonna use it as a 'real' server, then you don't need the GUI. It only eats resources which you could use for other purposes. Install SSHd, configure sshd.conf to disallow root logins (login as a regular user and then use sudo to become root), and if needed you could configure a firewall (like IPTABLES) to only allow SSH access from a batch of IP addresses, if security is a concern (or use certificates).