It's a code library that intercepts instructions to the standard DirectX library and replaces the result. Or in another way, it injects code into the game to enhance the graphics in some way. Usually this is applied to lighting calculations, anti aliasing and effects.
ENB mods will generally slow you down. However, if you have GPU power to spare it can be a none issue.
There are some instances where such mods have been used to work around bugs within video drivers, and even game engines, to produce better performance.
In practical terms, you drop a .dll file into the games' executable folder and away you go. Often some form of config ini or application can be used to tweak settings.
Security measures of online games can consider this code injection a hack attempt. Just something to be aware of.