A server install - what do you do to tune performance?

FirebarUK · 27 Mar 2007 at 12:03

This is purely out of interest. For those Linux / Unix sys-admins, what are the first things you do after installing a new server to tune performance or slim down the machine?

For me in most cases its just;

- set runlevel 3 for no X (or just use something like Blackbox or IceWM)
- disable unrequired services

How much further do some of you go? Tuning the scheduler perhaps? recompile a fresh kernel without all the rubbish you don't need? set some fancy kernel parameters?

Other than monitoring performance to see if it can be tweaked anywhere, I get the feeling I should be doing more tinkering

AndrewP · 27 Mar 2007 at 12:24

Usually just chkconfig unnecessary things off, keep an eye on top and the temperature sensors.

Most of our servers have plenty of performance overhead e.g. an Athlon 4000 and 4GB RAM box running a couple of internal AMP apps and email for 10 people

. Probably easier and more cost-efective to buy better hardware than spend ages optimising.

Una · 27 Mar 2007 at 12:25

On top of what you said I do,

- Disable loadable module support.
- Recompile the kernel to include either Grsecurity or SELinux and strip all the crap out I don't need.

Thats about all I can think of off the top of my head.

FirebarUK · 27 Mar 2007 at 12:36

How do you find dealing with SELinux? and more importantly - administering it. How do you find grsecurity too?
I'm looking into that and AppArmor for a project. I presume you use gentoo or debian for servers?

Probably easier and more cost-efective to buy better hardware than spend ages optimising.

Overprovision, certainly what we do tbh. Need to look into Virtualisation at some point

Una · 27 Mar 2007 at 12:43

FirebarUK said:
How do you find dealing with SELinux? and more importantly - administering it. How do you find grsecurity too?
I'm looking into that and AppArmor for a project. I presume you use gentoo or debian for servers?

Overprovision, certainly what we do tbh. Need to look into Virtualisation at some point

Hmm I got about 10 years of *nix usage behind me and im really a programmer not an admin so its not really a challenge to admin. Obviously its just hacking together configs - no pretty gui's

. SELinux has some really nice features and if your going to be running box's where you don't fully trust all your users its pretty handy. I have used quite a few linux distros for servers mainly debian based ones though. I use BSD on a few mind (OpenBSD).

For virtualisation I can recommend Xen if your going to be hosting multiple vps otherwise just go with something like vmware workstation.

FirebarUK · 27 Mar 2007 at 12:56

Cool

thanks for the info.

matja · 27 Mar 2007 at 13:06

Things I do :

Use/compile a kernel with Pentium II or higher native instructions for speedy sysenter/sysexit acceleration of syscalls (generic i686 and below kernels don't)
for IDE/SATA drives, make sure DMA is enabled via hdparm (hdparm -d1c1u1m16)
for SCSI and hardware RAID devices, tweak the read-ahead with "blockdev --setra"
If the machine is on a network with other gigabit devices, setup NIC(s) to use jumbo frames with MTU of 9000.
If using ext3, create filesystems with 4096 byte block size, full data journaling (not default, suprisingly)
If the machine is running a 32bit kernel and will never have more than 512MB of RAM (rare for Linux servers now), configure direct mapping of page tables (implies a 880MB limit), rather than slower, indirect mapping (for the larger 4GB/64GB limits).
Enable kernel CFQ I/O scheduler (won't improve performance, but will make the system more responsive under heavy I/O load)
Use a SMP kernel only on a SMP machine, otherwise you'll have unnecessary locking overheads (unless you use a recent 2.6.18+ kernel with runtime binding of locking functions ("SMP-alternatives"))
Careful selection of low-overhead devices (Realtek NICs must DIEEEEEE)
... lots more I can't remember right now

Una said:
On top of what you said I do,
- Disable loadable module support.

That was a good idea when LKM rootkits were going arround, but now theres rootkits that don't depend on module support, it's not really worth the hassle imo. Especially when the 2.6 kernel series does some funky entrypoint mapping during module loading to make modules as fast as compiling them into the kernel.

Una · 27 Mar 2007 at 13:28

matja said:
That was a good idea when LKM rootkits were going arround, but now theres rootkits that don't depend on module support, it's not really worth the hassle imo. Especially when the 2.6 kernel series does some funky entrypoint mapping during module loading to make modules as fast as compiling them into the kernel.

Well yeah of course there are rootkits which don't require module support but its better to narrow the scope as much as possible. I just don't see the point in having loadable modules enabled when you have a custom compiled kernel with all your stuff in. Sure if you need to upgrade some hardware, but then your going to have to down the box anyhow so doing a recompile/upgrade is no big deal.. Not to mention by that time your prolly going to be doing a kernel upgrade anyhow.

matja · 27 Mar 2007 at 13:39

Having run several servers of wildly contrasting hardware in production environments with monolithic (in the retrospecively, misguided, interests of security) and heavily modular kernels, I'll have to disagree with you on that point - unless you have no concern over availability or uptime.

I've had countless occasions where I've needed to reload modules with different options when changing hardware configuration while the machine is on (network, SCSI, RAID), which would otherwise require a reboot and to specify the options via the boot loader. Even if your hardware doesn't change, I wouldn't recommend compiling-in your drivers, for the flexibility that you lose whilst doing so.

From a security standpoint, if someone is able to get root privileges in order to load an LKM rootkit in the first place then you have bigger fish to fry, imo.

celliott · 27 Mar 2007 at 13:41

Disable X, update kernel, secure /tmp drives with noexec, a few other important config files like sysctl.conf and then proceed to optimizing whatever services are installed.

Garp · 27 Mar 2007 at 15:41

FirebarUK said:
This is purely out of interest. For those Linux / Unix sys-admins, what are the first things you do after installing a new server to tune performance or slim down the machine?

For me in most cases its just;

- set runlevel 3 for no X (or just use something like Blackbox or IceWM)
- disable unrequired services

How much further do some of you go? Tuning the scheduler perhaps? recompile a fresh kernel without all the rubbish you don't need? set some fancy kernel parameters?

Other than monitoring performance to see if it can be tweaked anywhere, I get the feeling I should be doing more tinkering

For absolute starters, I never ever install a GUI on a *nix server. There is absolutely no need for it. Everything can, and should IMO, be done through the command line. GUI is for desktops not servers.

For ease of upgrading and lack of hassles its rare for me to arse about with custom kernel compilation. Once the server has booted there isn't a huge amount of performance boosts to be gained by recompiling your own IME with the more modern distros; particularly with any 64bit system as the kernel will take advantage of newer processor features.

In an ideal world there should be no need to disabled uneeded services, because they shouldn't be installed in the first place. Taking a distro like Ubuntu Server for example, you only get little more than the bare minimum necessary to boot and get you to a command prompt. If its not on it cant be utilised to compromise your box, and its one less thing to worry about patching. (speaking of which, one of the first things I do is install openssh-server and then disable root logins for ssh)

FirebarUK · 27 Mar 2007 at 16:17

There is absolutely no need for it. Everything can, and should IMO, be done through the command line. GUI is for desktops not servers.

I don't think I'd go so far as to say point-blank that there is absolutely no need for it. I know a lot of people I work with are Windows Admins and simply want a GUI. E.g; They administer their servers whilst sat in front of them and quite like using neat on a RedHat box to manage network settings etc.
Sure, if your a hardcore Nix user then you will most likely prefer using the commandline and won't need a GUI. Yet a GUI can be awfully handy to some.
I don't think running a GUI on a server should automatically make someone an admin-lolzor-noob or something.

Garp · 27 Mar 2007 at 17:03

FirebarUK said:
I don't think I'd go so far as to say point-blank that there is absolutely no need for it. I know a lot of people I work with are Windows Admins and simply want a GUI. E.g; They administer their servers whilst sat in front of them and quite like using neat on a RedHat box to manage network settings etc.
Sure, if your a hardcore Nix user then you will most likely prefer using the commandline and won't need a GUI. Yet a GUI can be awfully handy to some.
I don't think running a GUI on a server should automatically make someone an admin-lolzor-noob or something.

I would still disagree I'm afraid.. from hard experience working here, if customers are not aware of Linux to a sufficient degree to be able to do everything through the shell, then they shouldn't be using it; particularly on any kind of public facing server, or any server with access to the internet. Too many GUIs and portals are designed with usability in mind and not security. If you want a GUI, use Windows. It really doesn't take long to learn to admin a box from the shell, with the added bonus that you can do that remotely (for security allowing a public-private key pair only for login.), without having to put up with horribly laggy VNC.
I'm not trying to be snobby.. its just that a GUI is:
a) A waste of disk space (at least Fluxbox and WindowMaker are reasonable size, but RH and most other server distros use Gnome or KDE which is just bloatware)
b) A terrific waste of processing power.
c) Another point to be compromised, requiring regular patching and checking, though again gnome and KDE are worse in this regards with the large number of dependant libraries and the like.
d) Encourage lazyness and shabby practices, and potential insecurity, through allowing any Tom Dick and Harry to easily change things without understanding the security implications, or without paying attention to them.

FirebarUK · 27 Mar 2007 at 17:15

I certainly see your points and will agree to disagree.

Running a Window Manager (understandable imo) / Desktop Manager (heaven forbid) certainly does create a new attack vector, but its risk factor may be so small (e.g; the system is an internal asterisk server, not facing the net) as to be ignored.

I dont think iceWM, *box etc are a terrific waste of resources (for modern machines, even heavily utilised ones - especially if they are used to good effect) and certainly aren't bloaty, admittedly as you say.

I also don't think its fair to exclude people from using Linux just because they can't make a cup of tea through the command-line. Linux certainly has its uses, especially the ability to get a FREE server.

The couple of servers I looked after didn't run any X stuff, but a server running OES Linux that I used, did.