Senior Linux System Administrator

Duties & Responsibilities

• Develop and document standard operating procedures for systems provisioning, installation, upgrading, migration; host and service monitoring; configuration management; problem troubleshooting; and other systems administration aspects.
• Manage and configure services, software, and hardware configurations.
• Take direction from the team lead and work independently within a team-environment.
• Mentor junior team members.
• Learn new technologies and skills.
• Develop and maintain custom tools in Python, Perl, Ruby, and Bash to automate managing the UNIX infrastructure.
• Use Puppet to maintain thousands of physical and virtual Linux CentOS systems and associated applications.
• Develop policies and procedures for standard systems administration tasks (packaging, OS installs, patch management).
• Build open-source software from source code.
• Document procedures, systems, services, and software. Provide ad-hoc training where appropriate.
• Provide UNIX support to NCBI's internal developers and researchers.
• Provide technical solutions to a wide range of problems and user requests through an established trouble ticketing system.
• Troubleshoot, research, design, and implement solutions to complex systems problems while accounting for technical limits, overall cost, and security policies.
• Assist with planning and implementation of major OS upgrade, deployments, and architectural changes.

Requirements

• Willingness to learn new technologies and skills.
• Significant experience in Linux Systems Administration (7+ years).
• Comfortable with Unix command line tools.
• Highly versed in Linux administration and best practices.
• Fluent in programming/scripting skills for automation in at least two of Bash, Ruby, Python, and Perl.
• General understanding of Linux internals (system calls, file systems, processes, etc).
• General understanding of TCP/IP and related protocols (HTTP, FTP, NFS, etc…).
• Comfortable working with users of varying levels of IT knowledge.
• Experience with building open-source software from source code.
• Able to work with minimum supervision on assigned tasks, and work with others both within and outside of the team.
• Strong communication skills, especially with regards to documentation.

Bonus Points

• Experience in a complex, dynamic, production, public-facing Linux environment.
• Experience with OpenShift, Kubernetes, GKE, or similar technologies.
• Knowledge of cloud technology such as AWS and GCP.
• Experience with automated configuration management, such as Puppet (preferred), Ansible, Chef, bcfg2, CFEngine, etc.
• Experience with version control (Git is preferred).
• Experience supporting large web application environments and development.
• Experience with enterprise network architectures.
• Experience with monitoring, alerting, and observability tools.