What horrors of custom/closed-source systems have you battled with?
I have recently (late last year) turned my passion/hobby for computers, technology, and open source software/open standards into my a career. My primary role is sysadmin for a debian based collection of specialized servers. The whole network is primarily Windows centric, but all the Windows VMs run on Linux phisical hosts, and all my systems run Debian. The group I'm responsible for are all orchestrated via a custom application. There are 2 orchestration systems, no backups, no source for the custom software or even packages, and there hasn't been updates since 2009 perhaps.
While I have an adequate knowledge/skill base to administer these systems, the custom software is a huge challenge. My mentor is a programmer with plenty of linux experience, but he has no way to delve into this closed and custom system to get a better idea of how it works. So the fear of dependencies breaking the entire system keeps us from changing anything unless it's to fix an issue.
This is an incredibly stable and well designed system, but before we got here it had remained unmaintained, and the original developers have moved on. Meanwhile I can't help but think that if the company had maintained commitment to this whole collection, they could be world leaders in the area these systems operate. Failing that, if the developers (or company) had left the code for internal reference, or better yet released it as open source, I would not spend my time caressing the land-mine, so to speak.
What hurdles have you encountered in your environment?