There exists a vulnerability involving environment variables and setuid/setgid programs under SunOS 4.0 and higher.
A dynamically-linked program that is invoked by a setuid/setgid program has access to the caller's LD_* environmental variables if the setuid/setgid program sets the real and effective UIDs to be equal and the real and effective GIDs to be equal before the dynamically-linked program is executed. A vulnerability exists if the UIDs and GIDs are not equal to those of the user that invoked the setuid/setgid program.
In particular, SunOS /usr/lib/sendmail, /usr/bin/login, /usr/bin/su, and /usr/5bin/su are vulnerable to this problem.
In-house and third-party software can also be impacted by this vulnerability. For example, the current versions of rnews, sudo, smount, and npasswd are known to be vulnerable under SunOS.
This or similar vulnerabilities have been found in other unix operating systems.
It seems Sun's solution is to call the dynamicly linked programs without both the real and effective uid and gid being the same. This is rather subobtimal as third party programs are left vulnerable. A better solutio is to mark a process as having changed it's uid or gid within the kernel. The dynamic linker can then query this information and use the LD_* variables depending on the results.
$ mkdir /tmp/mylib
$ cp libevil.so /tmp/mylib
$ export LD_LIBRARY_PATH=/tmp/mylib