Case studyInfrastructure & operations
Moving production workloads back onto owned hardware
Production workloads that ran at an external provider were moved back onto owned infrastructure, cutting the recurring cost without users noticing a change.
- Role
- IT Network & System Administrator
- Period
- November 2023 — April 2024
- Status
- Finished
- Used
Context and problem
The company production services ran at a cloud provider, and the bill came around every month without stopping. Meanwhile our own servers were sitting there, not fully used. The question was never whether those workloads could be moved. It was how to move them without users noticing that anything had changed.
What made it hard
- The systems being moved are used every day. A small mistake during the move is felt by users immediately.
- Multi-gigabyte databases had to stay intact and consistent after changing location.
- After the move, all day-to-day maintenance belongs to the internal team, with no provider to call.
- Backups were still a manual routine that depended on one person remembering, and that became a bigger risk once everything ran on our own hardware.
Constraints that could not move
- No service downtime could be visible to users during the migration.
- Existing server capacity had to be enough, since the point of the work was to cut cost, not to buy equipment.
- The legacy desktop application and its databases could not be modified to make the migration easier.
- The change window was limited, so every step had to be testable and reversible.
My role and responsibilities
- Migrated 5 to 6 production workloads from the cloud provider onto virtual machines in our own infrastructure, one at a time.
- Administered production Linux and Windows Server environments alongside the databases serving ERP users.
- Kept the corporate network path, routing stability, and remote connectivity working through the existing routers.
- Wrote scheduled jobs that back up databases and system state to self-hosted storage, removing the manual backup routine.
How I went about it
- 01
Ordering the migration by risk
The least used services moved first as a rehearsal of the working method. The busiest ones moved last, once the procedure had been proven on lighter load.
- 02
Preparing the destination before moving anything
Destination virtual machines were built and tested first, including storage capacity and network paths, so the migration would not turn into repair work halfway through.
- 03
Moving data and applications separately
Databases were moved and checked for integrity first, then the application was pointed at the new data source. That way one failed step does not damage the others.
- 04
Automating the backup routine
Database dumps and system-state capture run on a schedule, with results stored on self-hosted storage so they remain reachable without depending on a single server.
Moving workloads back onto your own hardware often sounds like a step backwards. Here the reason was simple: a monthly bill that never stopped, while machines we already owned were not fully used. What made the work hard was not the move itself, but that people were using those systems every single day.
Why move back instead of scaling down
The first option considered was reducing cloud capacity until the bill dropped. That was rejected because the workload does not shrink, and reduced capacity becomes its own problem as usage grows. Moving everything out has a clearer boundary: the work happens once, then the recurring cost stops.
Order of migration is the risk control
There was no single big night. The least used services moved first, not because they were easiest, but because if the method was wrong the damage would be small and could be fixed without users noticing anything. Once the procedure held up, the busiest workloads followed. For the same reason, data and applications moved in two separate steps: the database was moved and checked first, then the application was pointed at it.
Backups that no longer depend on memory
Backups used to be manual, which meant depending on whoever remembered to run them. Once everything lived on our own hardware that risk grew, because no provider held a second copy. The routine was replaced by scheduled jobs that capture database dumps and system state to self-hosted storage.
Results in short
5 to 6 production workloads moved from the cloud provider onto our own virtual machines, and the migration ran with zero service downtime. The recurring monthly cost paid to the cloud provider went down, although the saving was never measured, so no figure can be quoted. Database and system-state backups now run on a schedule instead of as a manual routine.
Outcomes
Moving 5 to 6 production workloads from the cloud provider onto our own virtual machines completed with zero service downtime.
No downtime window was announced or taken during the migration.
verified
The recurring operating cost paid to the cloud provider went down after the migration.
The size of the saving was never measured, so no figure can be quoted.
Database and system-state backups now run on a schedule, replacing the manual routine.
verified
With every system in one place, troubleshooting became more direct because the network and servers are under our own control.
Technical detail
This part only matters if you want to judge the technical side. Everything above is enough to understand the outcome.
Open the implementation detail3 decisions
Production workloads were moved from AWS onto Proxmox VE in our own infrastructure. The production environment covers Linux and Windows Server alongside multi-gigabyte Microsoft SQL Server databases serving high-concurrency ERP users. Routing and VPN connectivity run on MikroTik, and backups are scheduled Bash jobs on cron writing to self-hosted Nextcloud storage.
- Moving workloads one at a time rather than all of them in a single maintenance window.
- This stretches the work out and requires each step to be tested separately. What it buys is the ability to reverse one step without taking the whole service down with it.
- Writing backups as scheduled jobs to self-hosted storage instead of a managed backup service.
- No subscription cost appears and the data stays under our control, but monitoring whether the backups succeeded becomes an internal responsibility, and there is no vendor to call when nothing shows up.
- Keeping the existing routers and network topology during the migration.
- Avoiding two large changes at once makes the cause of any disruption easier to pin down. The cost is that the network keeps its design debt after the migration is done.
The deciding factor was not the migration tooling but the order of work. What I would do differently is deliberately measure cost before and after, because a saving with no number behind it cannot convince anyone later.
What I learned
- Migrating systems that are in daily use is more about sequence than about tooling. Starting with the least used ones makes the first mistake cheap.
- An unmeasured saving is still real, but it cannot persuade people. Measurement should be set up before the work starts.
- A backup that depends on one person remembering is not finished, even when the data is being written.