Skip to content

Business Outcomes

Your Only Sysadmin Left: What to Do With Infrastructure Nobody Understands

Executives and engineering managers who lost their only sysadmin can learn how to secure access, get the environment audited and choose who runs it next.

By Stack Harbor Engineering · Published · 10 min read

When your only system administrator (sysadmin) leaves, secure access first, get a written audit of the environment second, and choose who runs it third. The four realistic choices are a replacement hire, a contractor, a managed operations provider that runs the environment in place, or a migration to something simpler. In most cases, new people can still operate or migrate an undocumented environment, as long as the company can recover administrative access.

A sysadmin is the person who builds and runs servers, cloud accounts, backups and monitoring. Key-person risk is the exposure a company carries when one person holds knowledge or access that nobody else has. A runbook is a written procedure for a known failure or a routine task. This article is for an executive or engineering manager who now owns a production environment and did not build it.

What should you do in the first week after your only sysadmin leaves?

The first week after a sysadmin leaves is about control and safety, and none of it requires understanding how the environment works. Three decisions come first, in this order.

  1. Confirm that the company controls every account. List the domain registrar, the Domain Name System (DNS) provider, the hosting or cloud account, server logins, backups and monitoring. Every account should belong to a company identity, with at least two current people able to sign in. Rotate the credentials the departed person held as ordinary offboarding, whatever the terms of the departure.
  2. Pause non-essential changes. Hold upgrades, clean-ups and cost cuts until someone has a written picture of what depends on what. An urgent security fix is the exception, and it should be applied by someone who can also roll it back.
  3. Name who answers production alerts. Decide who is reachable if the site or application fails tonight, and what that person is allowed to do.

If the sysadmin is still working out a notice period, spend that time on handover: a recorded walkthrough, a list of every account and scheduled job, and the known fragile parts. For the third decision, the knowledge base explains how to set up on-call for a team that has never had one.

Why should you get a written audit before anyone changes the environment?

A written audit comes before any change because an undocumented environment hides dependencies that only show themselves when something is moved or removed. An audit here means a written read of what is running, where it runs, what depends on what, and where the risk sits. It is a document, and producing it should change nothing in production.

A useful audit of an inherited environment answers these questions:

  • What servers, services, databases and scheduled jobs exist, and what does each one do?
  • Who and what has access, and which of those accounts are no longer needed?
  • Do backups exist, where are they stored, and has a restore ever been tested?
  • How far behind are operating system and software patches?
  • What is monitored, who receives the alerts, and which alerts are noise?
  • Which risks come first, and is the recommendation to document, stabilize or migrate?

Small undocumented details matter. In one example scenario, a ten-year-old web application moved to a managed platform, and an undocumented override in a server file caused a hiccup during cutover, the moment traffic switches to the new environment.

For technical depth, the knowledge base covers detecting configuration drift, meaning changes made to production that nobody declared, and a takeover audit of a cPanel server. cPanel is a common hosting control panel.

Who can take over your servers after a sysadmin leaves? Four options compared

Four kinds of successor can run an inherited environment, and they differ on coverage, continuity and the risk each one leaves behind.

OptionWhat it gives youChoose it whenAvoid it whenMain risk
Replacement hireProduct context and daytime ownershipThe role was a full-time job of steady workYou expect one person to cover every nightThe same key-person dependency, rebuilt
Contractor or freelancerFast, bounded expertiseThe job is an audit, documentation or one defined repairYou need ongoing coverage or continuityKnowledge leaves again when the contract ends
Managed operations provider, in placeA team, a written scope and a defined coverage windowThe gap is coverage, continuity and routine, and the architecture is soundThe footprint is tiny, or the real problem is application defectsA scope that stops short of what you assumed
Migration to something simplerA documented environment on a supported platformThe current setup is fragile, unsupported or far more complex than the workload needsNobody has audited the environment yetSurprises on cutover day

The four options combine, for example a written audit followed by interim coverage while you recruit.

When should you hire a replacement sysadmin?

Hire a replacement sysadmin when the departed person was doing a full-time job of steady infrastructure work, or when the infrastructure is the product you sell. A colleague learns the application in a way an outside party rarely does.

Two conditions make the hire work the second time. Documentation is part of the job description. After-hours coverage is decided separately, because one person cannot carry production alerts every night and through every vacation.

Recruiting takes time, and interim coverage from a contractor or a provider can bridge the vacancy. A contractor also fits a bounded job such as the audit itself, but a single freelancer is another single person, so the deliverables should be documents the company owns. If a competent successor is already hired, a one-off audit and documentation project may be all the outside help you need. The article on whether to hire a DevOps engineer or outsource operations compares the two paths on coverage, continuity and cost. DevOps is short for development and operations.

When should a managed operations provider run the environment in place?

A managed operations provider fits when the gap is coverage, continuity and operational routine, and the work does not fill a full-time role. Managed operations means a provider runs the infrastructure layer under a written scope: monitoring, patching, backups and incident response. The contract that states the provider’s commitments is a service level agreement (SLA).

Running the environment in place means the provider operates it where it is today. Your hosting or cloud account stays yours, and no migration is needed to start. Ask any provider to confirm these terms:

  • The engagement starts with an audit, and nothing changes in production before the audit is delivered.
  • The provider works through access that you grant and can revoke, never through the departed person’s old credentials.
  • The scope names the coverage window: business hours, extended hours or 24/7.
  • Runbooks and an architecture diagram are deliverables that you keep if the contract ends.
  • The scope says where it stops. Defects in your application code normally stay with your developers.

On Amazon Web Services (AWS), a provider can operate an existing AWS account through a dedicated access role, with root credentials kept by the company. The article on who can manage AWS infrastructure for your company describes that handover, and the knowledge base lists the signs that an AWS environment has outgrown its team.

A managed operations provider is the wrong choice in three situations:

  • The whole footprint is one server and a storage bucket, so a contract is overhead.
  • Two or more engineers still own operations and do it well, so the fix is documentation.
  • Most incidents come from application defects, which need engineering time and not a new operator.

When is migrating to something simpler the right decision?

Migrating is the right decision when the audit shows that the current setup is the problem. Typical reasons are software that no longer receives security updates, a single server with no recovery path, or a design far more complex than the workload needs.

Migration is a poor first move. Moving an environment nobody understands carries the unknowns into cutover day, when traffic switches to the new environment. The audit comes first, and it sometimes concludes that no move is needed.

In one example scenario, a legal-technology software company of about 35 people depended on a lone sysadmin and one developer for its production knowledge, and nothing was written down. The audit found the architecture “fine for the workload” and proposed no rebuild. The company stayed on AWS, and ten runbooks were written in the first four weeks of that engagement.

What drives the cost of each option?

The four options have different cost drivers, so compare them over the same period and for the same hours of coverage.

  • Replacement hire. Salary and benefits, recruiting effort, the ramp-up period in an undocumented environment, tooling, and the cost of the vacancy while the role is empty.
  • Contractor. Contract work is usually billed by time. Discovery in an undocumented environment is hard to estimate, so ask for a fixed price for the audit and a separate quote for any repair.
  • Managed operations provider. The number and class of environments, architecture complexity, the coverage window, the response targets in the SLA, the number of monitored assets, and any stabilization work the audit identifies.
  • Migration. The number of sites or workloads, data size, dependency complexity, tolerance for downtime at cutover, data-residency needs, and the length of the support window after launch.

The cost of waiting can be measured too. List the tasks only the departed person could perform, then estimate what one day of downtime costs in lost orders, contract penalties and staff time. The article on how managed cloud hosting is priced explains the common pricing structures.

How do you reduce key-person risk in production infrastructure?

Key-person risk in production infrastructure falls when knowledge and access are shared and written down, whoever runs the environment. Five controls do most of the work:

  1. Every account belongs to the company, and at least two people or parties hold working access.
  2. A current architecture diagram shows what runs where and what depends on what.
  3. Runbooks exist for the common failures and the routine tasks, and they are updated after each incident.
  4. Restores are tested on a schedule, and the result is recorded.
  5. Documentation is a named deliverable in the job description or the provider contract.

The knowledge base shows how to write a runbook entry for a recurring alert.

What should you decide in the first 30 days? A checklist

Answer these seven questions in writing during the first 30 days after the departure.

  1. Does the company control every account, and have the departed person’s credentials been rotated?
  2. Who answers a production alert tonight, and what may that person do?
  3. Who will write the audit, and by what date?
  4. Has a backup been restored successfully since the departure?
  5. Was the role a full-time job of steady work, or a few hours a week plus emergencies?
  6. When are your customers active, and which hours need coverage?
  7. Whoever runs the environment next, which documents will the company own at the end?

Steady full-time work points to a replacement hire with documentation in the job description. Thin coverage and a sound architecture point to a provider operating the environment in place. A fragile or unsupported platform points to a planned migration after the audit.

Where Stack Harbor fits

Stack Harbor is a Canadian managed cloud operations provider (MSP) for production environments, headquartered in Montréal. It serves small and mid-sized businesses and agencies of roughly 5 to 200 people that run production systems without a 24/7 operations team of their own. Its clients include software-as-a-service (SaaS) companies, agencies and publishers.

Stack Harbor’s solutions page states that “Most engagements are takeovers”, and the first phase is an assessment of what is running and where the risk sits. Stack Harbor writes that assessment as part of the initial consult, and its site says: “You get the assessment whether you engage us or not.”

Ongoing operations are sold as Managed Cloud Ops, which runs in four phases: audit, stabilize, operate and improve. Stack Harbor operates environments on AWS, Azure, Google Cloud, OVH, DigitalOcean, dedicated servers and cPanel/WHM. It also operates its own Canadian infrastructure in two data centres in Montréal. It does not require a migration. What it can do depends on the access you grant, network reachability and the condition of the environment.

For an existing AWS account, the AWS operations offer works through a dedicated role in AWS Identity and Access Management (IAM), and the root account stays with the client. When the audit points to a move, Migration Services starts with discovery, a small paid scoping engagement that produces a written migration plan and a fixed-fee quote. Asked whether it can migrate hosting after the person who set it up has gone, the migration page answers: “That is the most common starting point.”

Coverage under the Management, Monitoring & Support service is scoped as business hours, extended hours or 24/7. A trained on-call rotation works to runbooks designed by senior engineers, and senior engineers own Severity 1 incidents, meaning production is down.

Every engagement ships with a written runbook and an architecture diagram, which are deliverables that you keep. Stack Harbor does not publish prices. It delivers a fixed scope and quote after the assessment, as the page on how Stack Harbor scopes and prices explains. That page also lists the cases where a retainer is the wrong purchase.

FAQ

Questions buyers ask about this

Who can take over our servers after our only sysadmin leaves?
Four kinds of successor can run the servers: a replacement hire, a contractor, a managed operations provider that operates the environment where it is, or a migration team that moves it to a simpler platform. The right choice depends on how much steady work the environment generates, which hours need coverage and what a written audit finds. The options also combine, for example an audit followed by interim coverage while you recruit.
The person who set up our hosting is gone and nothing is documented. Where do we start?
Start with control, then knowledge. Confirm that the company owns the domain registrar, DNS, hosting or cloud account, backups and billing, and rotate the credentials the departed person held. Pause non-essential changes and name who answers production alerts in the meantime. Then commission a written audit of the environment before anyone repairs, upgrades or moves anything.
Can a provider still operate or migrate an environment that nobody documented?
Yes, in most cases, as long as the company can recover administrative access. A competent provider audits the live environment, maps its dependencies and writes down what it finds before changing anything. What the provider can do depends on the access you can grant, whether it can reach the systems over the network, and the condition of the environment.
Should we replace our only DevOps engineer or move to a managed provider?
Replace the engineer when the role was a full-time job of steady infrastructure work, or when infrastructure is the product you sell. Choose a managed provider when the real gap is coverage at night and during vacations, continuity, and routine such as patching and backups. A provider can also cover the vacancy during recruiting, so the company decides with the audit in hand.
Can a provider operate an AWS account that was set up by someone who is no longer around?
Yes. A provider can operate an existing AWS account without owning it. The safe pattern is a dedicated access role that the company can revoke, root credentials that stay with the company, and a log of the provider's actions that the company can review. The first piece of work is normally a review of access, cost and backups left behind by previous administrators.
How do we reduce key-person risk in our production infrastructure?
Put every account in the company's name and make sure at least two people or parties hold working access. Require a current architecture diagram, runbooks for common failures and routine tasks, and restore tests on a schedule. Write documentation into the job description or the provider contract as a named deliverable, so the knowledge stays when a person or a supplier leaves.

About Stack Harbor

Stack Harbor is a Canadian-owned managed services provider headquartered in Montréal, Québec. It is an MSP for production cloud environments, not office IT. It offers three services: Cloud Infrastructure, an Application Platform, and Management, Monitoring & Support. Its own infrastructure runs in two data centres in Montréal, and its operations team also manages environments that clients run on AWS, Azure, Google Cloud and other providers. Stack Harbor works with SaaS companies, agencies, publishers and WordPress and WooCommerce operators that run production systems without a 24/7 operations team of their own, in English and French.

About Stack Harbor →

Share

Tell us what you are running. We will tell you what needs to change.

Book a consult with Stack Harbor. We review your current environment, the risks in it and your support needs, then give you a fixed scope and price after the scoping call.