You probably know
more than you can
say out loud.

What gets you hired

They're points on a shared-responsibility spectrum. With IaaS like EC2 you get raw VMs and manage everything from the OS up, maximum control, maximum ops. PaaS like RDS or App Service runs the OS, patching, and scaling for you, so you just ship code. SaaS like Gmail is a finished product you only consume. The trade-off is control versus operational burden: lower for flexibility, higher to move fast and stop maintaining undifferentiated infrastructure.

What gets you hired

A region is a geographic location like eu-west-1, chosen for user latency, data-residency law, and which services are offered there. Inside it are Availability Zones, isolated datacenters with their own power, cooling, and network, far enough apart to fail independently but close enough for low-latency replication. The AZ is your unit of isolated failure: run multi-AZ so a datacenter outage does not take you down, and add multi-region only for DR or global latency, because it brings real complexity and cost.

What gets you hired

Vertical scaling is making one machine more powerful, going from say 4 vCPUs and 16 gigs of memory to 16 vCPUs and 64. Horizontal scaling is adding more machines behind a load balancer so they share the work. Vertical is simple, but it has a hard ceiling at the largest instance type the provider sells, and a resize usually means a stop and start, so I am taking a couple of minutes of downtime. I treat it as a quick fix. Horizontal is what the cloud is really built for. I can run an autoscaling group with a minimum of 3 instances and a maximum of 20, scaling out when average CPU crosses about 60 percent and back in when traffic drops. The catch is that scaling out only works smoothly if the tier is stateless, so any of those 20 instances can serve any request and sessions live in a shared cache rather than on the box. So for a web tier I scale out. For a single relational database I usually scale up first, because sharding is a big commitment. Then I add read replicas, which works well when the workload is read heavy, something like 10 reads for every write, and only shard when the write path itself is the ceiling.

What gets you hired

High availability really means the system keeps serving despite component failures, and the core mechanism is eliminating single points of failure. So I walk the architecture and find every SPOF, a lone load balancer, a single database, everything in one availability zone, and remove it. The tool is redundancy: run multiple instances of each tier, N+1 or more, and spread them across failure domains like availability zones so one zone going down does not take the service with it. Redundancy alone is not enough though, you need automatic failover: health checks detect a failed instance and traffic shifts to the healthy ones, or a standby is promoted, fast and without waiting for a human. And the subtle part that catches people is shared fate, redundant copies that secretly share a dependency, the same zone, the same config rollout, the same primary database, so they fail together. Real HA means the copies are genuinely independent. And you measure all this in nines of availability, which tells you how much redundancy and how fast a failover you actually need.

What gets you hired

A stateless service keeps no client or session state on the instance itself. The state lives in an external store, a database, a cache like Redis, or a session store. That 1 property has big consequences. Because no instance holds anything special, any request can be served by any instance, so load balancing is trivial and I can go from 3 pods to 30 during a traffic spike without worrying about where a user's state lives. It also makes instances disposable. One can crash or be replaced and nothing is lost, so recovery is just bringing up a fresh instance, which for us was under 30 seconds, and autoscaling works cleanly because scaling down does not strand any data. A stateful service is the opposite. It holds data locally, so I either need sticky sessions to pin a user to the same instance, or replication to copy state between instances. Losing an instance can mean losing data, and both scaling and failover get much harder because instances are not interchangeable. So the design rule I follow is to push state out of the compute tier into purpose-built stores, keep the request-handling services stateless, and confine the genuinely stateful parts, the databases and the queues, to systems that are actually designed to manage state with replication and failover.

What gets you hired

Retries are necessary for transient failures but dangerous if done naively. First, retry only the right things: transient errors like timeouts or 503s, and only operations that are safe to repeat, idempotent ones, because retrying a non-idempotent write can double-charge or duplicate data. Second, use exponential backoff, wait progressively longer between attempts, say one second, then two, then four, instead of immediately hammering, so the dependency gets room to recover. Third, and this is the part people forget, add jitter: randomize the backoff so all the clients do not retry in perfect lockstep, because synchronized retries arrive in waves and re-overload the service. Fourth, cap the retries and the total time, and pair retries with a circuit breaker so that once a dependency is clearly down, you stop retrying entirely and fail fast rather than piling on. The failure mode all of this prevents is a retry storm, or thundering herd: a dependency has a brief blip, every caller retries immediately and repeatedly, that retry traffic multiplies the load on an already-struggling service and turns a momentary glitch into a sustained outage. So retries with backoff, jitter, caps, and a breaker, not retry-until-it-works.

What gets you hired

An operation is idempotent if doing it multiple times has the same result as doing it once. A read is naturally idempotent, and setting a field to a specific value is, but add 10 dollars to a balance is not, because applying it twice moves the balance by 20. This matters enormously in distributed systems because duplicates are not an edge case, they are guaranteed. A client hits its 30 second timeout and retries even though the first request actually succeeded, a queue like SQS delivers at least once, a load balancer re-sends. On a busy service even a 0.1 percent duplicate rate is thousands of duplicate requests a day, and if the operation is not idempotent those become double charges, duplicate orders, or corrupted counters. The standard tool is an idempotency key. The client attaches a unique key, often a UUID, the server records which keys it has processed, typically with a 24 hour retention, and if the same key arrives again it returns the original stored result instead of doing the work twice, so a retried payment charges once. Where I can, I also design operations to be naturally idempotent, for example set order status to shipped rather than increment a counter, and where I cannot, I dedupe by key. Idempotency is what makes retries and at least once delivery safe, which is why it underpins reliable systems.

What gets you hired

Graceful degradation means that when something fails, the system drops to reduced functionality instead of collapsing. If a non-critical dependency is down, I keep the core experience working, serving cached or default data, hiding or stubbing the affected feature, rather than failing the whole request. In practice I put a short timeout on that call, around 200 milliseconds, and a circuit breaker that trips once the error rate crosses about 50 percent, so one sick dependency cannot eat all my threads. The key decision is whether to fail open or fail closed. Fail open means I let the operation proceed despite the failure, favoring availability. That is right when the dependency is not essential. If the recommendations service is down, I serve the page without personalized recommendations instead of returning a 500. Fail closed means I deny the operation on failure, favoring safety. That is right when proceeding would be unsafe. If the authorization service is unreachable, I must deny access rather than let everyone through. So I decide per dependency, based on the cost of being wrong. Anything touching authentication, authorization, or payments fails closed, because failing open there is a breach or a financial loss. Cosmetic or enhancement features fail open and degrade so the core stays available. The mistake to avoid is a blanket policy, because the right choice depends entirely on what that dependency guards.

What gets you hired

First I'd scope it, every request or one endpoint/region, and since when, so I know the blast radius. Then the highest-yield question: what changed in the last hour, a deploy, a config flip, a traffic surge, a dependency. I'd form a hypothesis and confirm it cheaply, error rate by version and recent deploy events, before drowning in logs. If a deploy lines up, I'd roll back first to stop the bleeding and confirm 500s drop, then root-cause from the release diff and a trace of the failing path. Finally I'd add whatever was missing, a symptom-based alert or a regression test, so it can't page someone silently again.

What gets you hired

RPO is how much data loss is acceptable, so it sets how often I back up or replicate. RTO is how long I can be down, so it sets how fast I have to recover. Both numbers come from the business and then drive the design, not the other way round. An RPO of 5 minutes means continuous replication or frequent snapshots, not a nightly dump that can lose 24 hours of orders. An RTO of 15 minutes means a warm or hot standby, because restoring a 2 terabyte database from cold backup will take hours. So I map the pattern to the agreed numbers. Backup and restore is cheapest but recovery runs in hours, pilot light gets you to tens of minutes, warm standby to minutes, and active active is near zero at the highest cost, and I pick the cheapest option that genuinely meets the RPO and RTO. I also protect the backups themselves, copied cross region and cross account, encrypted, and immutable with a lock of say 35 days, so one regional outage or one deletion event cannot take primary and backups together. Then I run real restore drills, at least quarterly, and time them against the RTO, because an untested backup is a hope, not a recovery plan.

What gets you hired

A circuit breaker protects against cascading failure. When a dependency is slow, every call ties up caller resources, threads and connections, waiting on doomed requests. If your timeout is 30 seconds and you have a pool of 200 threads, a dependency that stops responding drains that pool in seconds, and then the caller itself starts failing and the problem propagates upstream. One slow dependency takes down the whole chain. The breaker counts failures, and once they cross a threshold, say 50 percent of the last 20 calls, it opens, and subsequent calls fail fast in about a millisecond instead of hanging. After a cool off, typically around 30 seconds, it goes half open and lets a single trial request through. If that succeeds it closes and normal traffic resumes, if not it stays open. Two things make this powerful. Failing fast frees the caller resources so it stays healthy, and not hammering a struggling dependency gives it room to recover, which is what actually breaks the cascade. The complementary pattern is the bulkhead. I isolate resources per dependency, so each downstream gets its own thread or connection pool, maybe 20 threads each rather than one shared pool of 200. Then one slow dependency can only exhaust its own compartment. The name comes from ship compartments, a breach floods one compartment, not the whole hull. Together, breakers and bulkheads stop a single failing dependency from sinking the entire service.

What gets you hired

These are three overload-protection mechanisms with the same underlying philosophy: at capacity, it is better to reject or slow some traffic cleanly than to let everything collapse. Rate limiting caps how many requests a given client or tenant can make in a time window, usually with a token-bucket algorithm, which enforces fairness and prevents one noisy caller from consuming all the capacity. Load shedding is what you do under genuine overload: deliberately reject the excess requests, returning a clear 429 or 503, so that the requests you do accept are served well, instead of trying to handle everything and degrading until the whole service times out for everyone. Backpressure works in the other direction, it signals upstream to slow down, through bounded queues, blocking, or flow-control protocols, so the producer rate is matched to what the consumer can actually handle, rather than letting unbounded work pile up and exhaust memory. The unifying idea is serve some, not none: a service at its limit should gracefully shed or throttle the overflow to protect its core throughput, because a controlled partial rejection is vastly better than an uncontrolled total meltdown where latency explodes and everything fails. So I design these in deliberately rather than hoping load never exceeds capacity.

What gets you hired

Caching helps reliability in two ways. It reduces load. On a read heavy service with a 10 to 1 read to write ratio, a cache at a 95 percent hit rate means the database sees roughly one twentieth of the reads, which protects it from overload and improves both latency and effective capacity. It also improves availability. With stale while revalidate, the cache keeps serving slightly old data even when the backing store is slow or briefly down, so a 30 second database blip never becomes a user facing outage. But caching is not free, and the failure modes are the interesting part. The classic one is a stampede, or thundering herd. A hot key expires and every request misses at once and slams the backend together, which can take down the very database the cache was protecting. I mitigate that with request coalescing or a lock so only one request recomputes the value, and with jittered TTLs, so a 5 minute TTL becomes 5 minutes plus or minus 30 seconds and the keys do not all expire on the same tick. There is also staleness, so I need a deliberate invalidation strategy. And the cache becomes a dependency itself. If it dies or comes back cold after a restart, 100 percent of traffic hits the database at full force. So I plan for cache redundancy and for warming or gracefully degrading with a cold cache. Caching is a powerful reliability tool that brings its own reliability concerns to design for.

What gets you hired

Putting a queue between services decouples them in time, and that buys real resilience. The producer hands work off and moves on, so it does not block on or fail because of a slow or unavailable consumer, and the consumer drains at its own pace. The queue also smooths load. If a marketing email drives a 10x spike for 5 minutes, the queue absorbs it and the consumer keeps processing at a steady 500 messages a second instead of being overwhelmed, which is hugely valuable for spiky workloads. Queues also give reliable processing primitives. Failed messages are retried automatically, and after a few attempts, I usually set 3 to 5 with backoff, a dead letter queue captures them so a poison message is isolated for investigation instead of being lost or blocking everything behind it. The honest trade offs. You move from synchronous to eventual consistency, so the work lands seconds or minutes later, and the user experience and any read after write expectation has to accommodate that. You add operational complexity, another system to run and monitor, with queue depth and consumer lag as key signals, and I will alert if depth stays above a few thousand for more than 5 minutes. And because most queues are at least once, duplicates happen, so consumers must be idempotent, and I think about ordering if the workload needs it. A queue is a strong resilience pattern, but I add it deliberately, knowing it trades immediate consistency and simplicity for that resilience.

What gets you hired

Limiting blast radius is about containing failure so 1 problem hits a fraction of the system, not all of it, and it goes beyond plain redundancy. The strongest lever is partitioning: divide users or tenants into independent cells or shards, each a self-contained stack. With 10 cells, a cell failure is a 10 percent outage rather than a 100 percent one, and you can update 1 cell at a time. Then isolate across failure domains: spread across at least 3 availability zones and, where it matters, multiple regions, and use bulkheads internally so a misbehaving dependency is confined to its own compartment, say a connection pool capped at 20 for that one caller. The thing senior engineers really watch for is shared fate, the hidden global dependencies and global actions that can fail everything at once: 1 config push applied everywhere simultaneously, 1 shared database or cache, 1 control plane. Those defeat all your redundancy if they go bad, so I try to eliminate or shard them, and where a global change is unavoidable I stage it. That is the last piece: roll changes out progressively, canary at 1 percent, then waves, then region by region, with automated rollback on error-rate regression, because deploys and config changes are the biggest single cause of correlated, everything-at-once outages. Redundancy stops a component failure being fatal. Cells, isolation, no shared fate and staged rollouts stop any single failure or change becoming global.

What gets you hired

Chaos engineering is the practice of proactively injecting failure into a system to learn how it actually behaves under stress, killing instances, adding latency, making a dependency error, so you discover weaknesses in a controlled experiment instead of during a 3am incident. The crucial thing is that it is disciplined, not random breakage. Each experiment is hypothesis-driven: I define the normal steady state with a metric, then state a hypothesis like the system should stay within its error budget even if this dependency slows by 500ms, and run the experiment to confirm or refute it, so I learn something specific. I bound the blast radius carefully: start in staging or on a small slice of production traffic, have a kill switch to abort immediately if it goes wrong, and only widen the scope as confidence grows, never an uncontrolled free-for-all. And I close the loop, the point is not to break things, it is to find the weaknesses, fix them, and then automate the experiments so the resilience does not silently regress as the system changes. Done this way, chaos engineering turns assumptions about resilience into verified facts, which is exactly what you want before a real failure tests them for you. The classic origin is Netflix Chaos Monkey randomly terminating instances to force engineers to build for instance failure.

What gets you hired

First I make the target concrete, because nines are abstract: 99.9% is roughly 8.8 hours of downtime a year, 99.99% is about 53 minutes, and 99.999%, five nines, is about 5 minutes a year, which is so tight that essentially no human can be in the recovery loop, it demands fully automated detection and failover. Then I price it: each additional nine is roughly an order of magnitude more engineering and operational cost, multi-region active-active, sub-minute automated failover, eliminating every single point of failure, and a lot more rigor and on-call maturity. I also raise that availability compounds across dependencies: if a request flows through several services that are each 99.9%, the end-to-end availability is the product, lower than any single one, so you cannot promise more than your dependencies deliver unless you add redundancy around them. With all that on the table, I push back on the round number and anchor the SLO to actual business impact, what does an hour of downtime actually cost, what do users genuinely need, because over-targeting wastes large amounts of money and slows down delivery for reliability nobody needs, while under-targeting hurts the business. Very often the honest answer is that 99.9% is plenty and five nines is not worth the cost, and my job is to have that conversation with real numbers rather than just nodding and trying to build it.

What gets you hired

The shared responsibility model splits security into the provider's job and mine. The provider handles security of the cloud, the physical data centres, the hardware, and the virtualization layer. I handle security in the cloud, my data, who can access it, my network rules, and patching anything I run. The important part is that the line shifts by service. On a raw virtual machine I own the operating system, so I am the one applying that kernel patch within my 30 day window. On a managed serverless function the provider owns the runtime and I own only my code and the permissions attached to it, which might be one role with 3 specific actions on 1 bucket. This matters because the breaches are almost never the provider's hardware failing. Gartner has said for years that through 2025 around 99 percent of cloud security failures would be the customer's fault, and what I see matches that, a storage bucket left public, or an identity with a wildcard policy that grants every action on every resource when it needed 2. So when I join a team I assume the hard part of the job is on my side of the line, and I go looking for the public bucket and the over-permissive role first.

What gets you hired

CIA is confidentiality, integrity, and availability, and I like it because it gives me a checklist I can walk any design through. Confidentiality is making sure only authorized parties can read the data. I cover that with encryption, AES-256 at rest and TLS 1.2 or higher in transit, plus access control and least privilege, so a role gets the 3 actions it needs and nothing more. Integrity is making sure data is not tampered with and that changes are traceable. I cover that with hashing, signatures, and audit logs, typically kept for at least 90 days and often a year for compliance. Availability is making sure the system is there when people need it. I cover that with redundancy across 3 availability zones, tested backups, and DDoS protection, sized against whatever target we have promised, say 99.9 percent, which is about 43 minutes of downtime a month. The real point is that the three pull against each other and you need all of them. I have seen a system with beautiful encryption and no restore test, and when a bad deploy corrupted the data, a 4 hour outage turned into 2 days. Confidentiality was strong and availability failed anyway. So I walk the design through all three and go looking for the leg nobody funded.

What gets you hired

Authentication is about identity, proving who you are, with something like a password, a token, or a certificate, often strengthened with MFA. Authorization is about permission, deciding what that identity is allowed to do, through roles, policies, or scopes. The order is fixed: you authenticate first, then authorize, because you cannot decide what someone may do until you know who they are. They also fail differently, which is a useful tell: a 401 means authentication failed, I do not know who you are; a 403 means authentication succeeded but authorization did not, I know who you are and you are not allowed. Keeping them separate matters because a lot of access-control bugs come from checking identity but forgetting to check permission.

What gets you hired

They protect against different attacks. Encryption in transit, which in practice means TLS 1.2 or 1.3, protects data while it moves across the network, so an attacker tapping the wire or sitting in the middle sees ciphertext. Encryption at rest protects stored data, on disk, in a database, in object storage, usually AES-256, so someone who steals the media or grabs a storage snapshot also gets ciphertext. You need both because they cover different attack points and neither substitutes for the other. TLS does nothing for a backup that walks out of the building, and at rest encryption does nothing for traffic sniffed on the wire. The part that actually takes thought is key management. I use a managed KMS, turn on automatic rotation, typically a new key version every 365 days, and rely on envelope encryption, where a data key encrypts the data and a master key encrypts that data key. That is what lets me rotate in 1 operation instead of re-encrypting terabytes. One caveat I would be honest about: at rest encryption does nothing against someone with valid application access, because the app decrypts transparently. That is what authorization is for.

What gets you hired

The application should never hold a long lived access key. Those get committed to git, baked into images, or printed into logs, and because they never expire, 1 leak from 3 years ago is still a live liability, and rotating it means chasing down every place it was copied. Instead I attach a role to the workload: an instance profile for a VM, IRSA or workload identity for a Kubernetes pod, a managed identity on Azure. The platform then injects short lived credentials, typically valid for about 1 hour and auto rotated well before they expire, and the SDK picks them up from the metadata endpoint with no code change. There is nothing static to steal, and anything that does leak dies on its own within the hour. I scope that role to least privilege, only the specific actions on the specific resources it needs, so a compromised service can read its 1 bucket and nothing else. And for things outside the cloud, like a CI/CD pipeline, I use OIDC federation, so the pipeline trades its identity token for a role with a token that lasts about 15 minutes instead of storing a permanent key in the CI settings. The goal is simple: no static long lived credentials anywhere.

What gets you hired

Symmetric encryption uses 1 shared key for both encryption and decryption, AES-256 being the standard. It is fast, gigabytes per second on modern hardware, so it is ideal for bulk data, but it has a key distribution problem, because both sides have to already share that secret safely. Asymmetric uses a keypair, a public key and a private key, RSA-2048 or an elliptic curve like P-256. The public key can be handed to anyone, so distribution is solved, but it is on the order of 1000 times slower, so I would never encrypt a 5 gigabyte file with it. The elegant part is that you combine them. You use asymmetric to authenticate the server and agree on a symmetric session key, then switch to fast symmetric encryption for the actual traffic. That is exactly how TLS works, the handshake is asymmetric and costs 1 round trip in TLS 1.3, and everything after it is symmetric. Asymmetric also gives you digital signatures, signing with the private key and verifying with the public key, which proves authenticity and integrity. So it is not that 1 of them is better. It is fast but distribution hard versus slow but distribution solved, and in practice they are used together.

What gets you hired

Shifting left means moving security out of a late gate and into the everyday workflow, catching issues at commit and build where a fix costs minutes rather than the days it costs once it is live. In the pipeline I layer complementary checks, because each one catches different things, and I keep the whole set under about 10 minutes so people do not route around it. On the code, SAST for the source, SCA to flag vulnerable third party dependencies, and secret scanning so credentials never get committed, ideally with a pre commit hook as well. On the artifact, container image scanning for OS and package CVEs, and IaC scanning like tfsec or checkov for misconfigured Terraform or Kubernetes, plus image signing and an SBOM so I know the provenance of everything that ships. After deploy to a test environment, DAST exercises the running app for issues static tools cannot see. The judgment part is gating. I fail the build on critical and high findings and let mediums through as warnings with a 30 day fix window, and I spend real time tuning thresholds and triaging false positives. On one team the scanner opened over 200 findings on day one and most were noise. If the pipeline cries wolf, developers will find a way around it and you have made things less secure, not more.

What gets you hired

A few of the big ones. Injection, like SQL injection, where untrusted input becomes part of a query. The defense is parameterized queries and prepared statements, never string concatenating SQL. Broken access control, which sits at number 1 in the current OWASP Top 10, where a user reaches data or actions they should not, and the defense is enforcing authorization server side on every request with default deny, not hiding the button in the UI. Changing an id in a URL from 41 to 42 should return a 403, not somebody else's invoice. Cross site scripting, where attacker script runs in another user's browser, defended with output encoding and a Content Security Policy. Security misconfiguration and vulnerable components, so default credentials, public buckets, unpatched libraries, defended with hardened defaults, patching, and dependency scanning. And SSRF, where you trick the server into calling internal endpoints, defended with egress restrictions and protecting the instance metadata endpoint, which is why IMDSv2 exists. The theme is that input validation alone is not enough. Each class has its own specific mitigation, and you need all of them.

What gets you hired

I work it in layers. Start with a minimal or distroless base image, because every extra package is attack surface and another CVE to patch. Swapping a full OS base for distroless has taken images I have worked on from around 900MB and 150 plus known CVEs down to about 80MB and a handful. Then harden the runtime, a non root USER, dropped capabilities, and a read only root filesystem, and absolutely no secrets baked into image layers, because they persist in the history even if a later layer deletes them. Scan images in CI for known vulnerabilities, pin base images by digest rather than a floating tag so builds are reproducible, and rebuild on a schedule, weekly is a reasonable default, to absorb upstream patches. Then prove provenance, which is the supply chain part, sign images with something like cosign and publish an SBOM, so I can verify an image really came from my pipeline and know exactly what is inside it. Finally enforce it at the cluster boundary with an admission policy that rejects unsigned images, untrusted registries, or any container asking to run as root. So scanning is one layer. The real protection is minimal base, runtime hardening, signed provenance, and admission enforcement working together.

What gets you hired

Defense in depth means layering independent controls so no single failure is catastrophic. Network controls, identity and access controls, application hardening, and data protections like encryption, stacked so an attacker who gets through 1 layer still meets the next. The reason is simple, any single control can fail or be bypassed, so you never want the whole system resting on the perimeter. Zero trust is the modern evolution of that idea, centered on identity. The old model trusted anything inside the network, which is how one phished laptop turned into a full breach. Zero trust drops that assumption and treats every request as untrusted until proven otherwise, authenticating and authorizing it on its own merits regardless of where it comes from. The 3 principles are verify explicitly, enforce least privilege, and assume breach. So these are not competing ideas. Zero trust is defense in depth applied on the assumption that the attacker may already be inside, which is why internal service to service calls get mTLS with certificates that rotate on the order of every 24 hours, and why each service gets a policy scoped to the specific calls it makes, rather than implicit trust because it happens to be in the same VPC.

What gets you hired

First, rotate, immediately. The instant that key hit a public repo I treat it as compromised, because bots scrape new GitHub commits and exposed keys get used in under 60 seconds in the worst cases, so I disable and delete it right away. The common mistake is to delete the commit, force-push, and think it is handled. It is not. The key was scraped and cached within seconds, so removing it from history does not un-leak it, only revoking the credential does. With that done I assess blast radius in CloudTrail, and I look at the whole window from the commit timestamp forward, not just the last hour: was the key used, are there unexpected API calls, new resources like 20 large GPU instances spun up for crypto mining in a region we never touch, or IAM changes. If it was used, I run it as a real incident, revoke derived sessions, roll affected resources, and hunt for persistence like new users or backdoor access keys. Then prevention: secret scanning and pre-commit hooks so this is blocked at the source, and the deeper fix, move to short-lived role-based credentials, tokens that expire in an hour, so there is no long-lived key to leak in the first place. I would also ask why a static key existed at all.

What gets you hired

At scale the unit of isolation is the account, so I use a lot of them, often one per team per environment, which for 10 teams across dev, staging and prod is already 30 plus accounts. That sounds like a lot until you remember an account is a hard blast-radius boundary: a mistake or compromise in one cannot reach another. Over the top I put guardrails with Organizations SCPs, org-wide denies that no account-level IAM can override, like blocking anyone from disabling CloudTrail, restricting to the 2 or 3 regions we actually approve, or denying risky actions outright. I centralize logging into a dedicated, locked-down account so CloudTrail and Config data cannot be tampered with by a compromised workload account, and I use single sign-on with short-lived assumed roles, typically 1 hour sessions, instead of long-lived IAM users scattered everywhere. Every new account comes from a landing zone that applies the secure baseline automatically, so provisioning takes minutes rather than a week of manual setup, and the root user of each account is MFA-protected and never used for daily work. Finally I make it observable: GuardDuty for threat detection, Security Hub and Config for posture and drift, aggregated org-wide so I can see and respond centrally. The theme is structure and guardrails, not fixing things one resource at a time.

What gets you hired

I treat it as a structured exercise, not a brainstorm, because ad hoc thinking misses whole categories. First I map the system, components, data flows, and above all the trust boundaries, the points where data crosses from less trusted to more trusted, internet to app, app to database, calls to third parties. In a typical service that is 4 or 5 boundaries, which is a tractable number to work through in a 90 minute session with the team. Then I identify the assets actually worth protecting, PII, credentials, money, availability, so effort lands where impact is highest. Then I enumerate threats systematically rather than guessing, walking all 6 STRIDE categories at each boundary, spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege, which forces breadth. I prioritize by likelihood times impact, because you cannot mitigate everything, and I focus on realistic high risk attacks over exotic ones. For the top few, usually the top 5, I design specific controls, and for the rest I explicitly accept or transfer the risk rather than quietly ignore it. And I treat the model as living, revisiting it whenever the architecture changes, because one new integration can open a new boundary. The goal is to reason about risk before we build, so security is designed in rather than bolted on.

What gets you hired

A VPC is my own isolated private network in the cloud. I pick an address block, typically something like 10.0.0.0/16, which gives me about 65,000 addresses, and every resource I create lives inside that boundary, separate from other customers. A subnet is a slice of that range, say a /24 with 256 addresses, and subnets are how I control where things sit. In a normal layout I would carve out 6 subnets, a public and a private one in each of 3 availability zones. The key distinction is public versus private. A public subnet has a route to an internet gateway, so that is where internet-facing things like load balancers and web servers go. A private subnet has no direct route from the internet, so databases and backend services go there. The database is reachable from the web tier but never directly exposed. On top of the layout, firewall rules enforce the isolation. My database security group only allows port 5432 from the web tier's security group, nothing else, so even if something lands in the wrong subnet it still cannot reach the data. Layout plus rules together is what makes the isolation real rather than just tidy.

What gets you hired

TCP is connection-oriented. It does a three-way handshake, which costs a full round trip before any data moves, and then guarantees reliable, ordered delivery with acknowledgements, retransmission, and congestion control. You are paying some latency and about 20 bytes of header per segment for correctness. UDP is connectionless, no handshake and no guarantees, with an 8 byte header, so it is lighter and lower latency, but the application has to tolerate loss and reordering itself. I choose TCP when every byte matters and the order matters, so HTTP, SSH, database connections. I choose UDP when timeliness beats completeness, so live voice and video, gaming, and DNS, where a single query and response fit in one small packet on port 53 and retrying is cheaper than setting up a connection. In a voice call, losing about 1 percent of packets is barely audible, but waiting 200 milliseconds for a retransmission is a jarring gap, so dropping is the right call. QUIC is the interesting middle ground. It runs over UDP to get that lightness, then rebuilds reliability and congestion control in user space, and it folds the transport and TLS handshakes together so a connection can be up in 1 round trip.

What gets you hired

First the browser turns the hostname into an IP address: it checks its own cache, then asks a DNS resolver, which walks root, TLD, and authoritative servers if needed. With the IP, it opens a TCP connection on port 443 using the three-way handshake. For https it then runs a TLS handshake, validating the certificate against a trusted CA and agreeing on a session key. Now it sends the HTTP request, GET with headers, which often hits a CDN or load balancer before the origin. The server returns a response, and the browser parses the HTML and pulls in CSS, JS, and images to render the page. I can go deeper at any layer, ARP and routing under TCP, or caching at the DNS and CDN level.

What gets you hired

When I look up a name, my machine asks a recursive resolver. If nothing is cached, the resolver walks the hierarchy: a root server points it to the TLD server for .com, which points to the domain authoritative nameserver, which returns the record. An A record is the leaf, it maps the name to an IPv4 address (AAAA for IPv6). A CNAME is an alias, it points one name to another name that then has to be resolved, which is handy for aliasing many subdomains to one canonical host. Every record carries a TTL, so resolvers cache the answer for that long. That TTL is the practical gotcha: set it low before a migration so changes propagate fast, because a high TTL means the old IP lingers in caches well after you have changed it.

What gets you hired

IPv4 is a 32-bit address space, about 4.3 billion addresses written as four dotted decimals, and we ran out of them. NAT was the workaround: many private hosts share a single public IPv4 for outbound traffic. IPv6 is 128-bit, written as hex groups, with an effectively unlimited supply, so it removes the scarcity that NAT was patching. The practical consequence is that with IPv6 every device can have its own globally unique address and you do not need NAT for conservation, so isolation has to come deliberately from firewalls rather than as a NAT side effect. The two are not interoperable, so in practice we run dual-stack during the transition, serving both until IPv6 is universal.

What gets you hired

The subnet is private because its route table has no route to an Internet Gateway, so nothing inbound or outbound flows directly. But the instance only needs to initiate outbound, not be reachable. So I add a NAT gateway in a public subnet and point the private route table’s default route at it. Traffic flows instance to NAT to IGW out, and replies come back the same path, while the instance stays unreachable from the internet because you cannot initiate inbound through a NAT. If it is just pulling from S3 or ECR, I would use a VPC endpoint instead to skip the NAT cost and keep it off the public internet.

What gets you hired

A security group sits on the instance or ENI and is stateful, so if I allow inbound TCP 443 the reply is allowed back out automatically and I never write a return rule. A network ACL sits on the subnet and is stateless, so I have to explicitly allow the return traffic too, which in practice means opening the ephemeral port range, 1024 to 65535, outbound. Forget that and connections just hang in one direction, which is a miserable bug to chase. Security groups are allow only and deny everything by default, so there is no deny rule to write. NACLs do both allow and deny, and they are evaluated in numbered order, lowest number first, so a deny at rule 90 wins over an allow at rule 100. That ordering is exactly what makes them useful for an explicit block. Day to day I rely on security groups for what each resource accepts, and I only reach for a NACL when I need a subnet wide rule, like denying a single bad CIDR that is scanning us.

What gets you hired

DNS is the lookup layer that turns a name like app.example.com into the address a client connects to, using records served by authoritative name servers. Route 53 is AWS's managed authoritative DNS: I create a hosted zone for the domain, define the records, and AWS runs the name servers. Beyond plain records, it offers routing policies, latency, weighted, failover, geolocation, plus health checks, so it can steer traffic or fail over to a backup endpoint when the primary is unhealthy. The thing to respect is TTL: records cache for their TTL, so changes are not instant. Before a cutover I lower the TTL in advance so clients pick up the new record fast.

What gets you hired

In Azure I would use a network security group, which is a list of inbound and outbound rules that allow or deny traffic by source, destination, port, and protocol. I can attach it to the subnet to cover everything in it, or to the VM's NIC for just that machine, or both, and then both rule sets apply. Each rule has a priority number: lower numbers are evaluated first and the first match wins, so a high-priority deny beats a lower-priority allow. Azure ships default rules that allow intra-VNet traffic and deny inbound from the internet, so I layer my own higher-priority rules on top, for example allowing 443 from a specific source range and leaving everything else denied.

What gets you hired

CIDR notation says how many leading bits are the network prefix. A /24 means 24 network bits and 8 host bits, so 2^8 = 256 total addresses, for example 10.0.0.0 through 10.0.0.255. Usable is fewer: the network and broadcast addresses are reserved, leaving 254, and cloud providers reserve extra per subnet, AWS reserves 5, so you actually get 251. The key intuition is that a smaller number after the slash is a bigger block: a /16 is 65,536 addresses, a /28 is only 16. When I design, I size subnets to the workload with headroom, keep the ranges non-overlapping, and avoid one giant subnet because that hurts blast-radius isolation.

What gets you hired

Private IPs come from the RFC 1918 ranges, 10.x, 172.16-31.x, and 192.168.x. They are reusable inside any private network and are never routed on the public internet, which is why every home and VPC can use 10.0.0.0/8. Public IPs are globally unique and routable, so they are scarce and allocated. NAT bridges them: when a private host makes an outbound connection, the NAT device rewrites the source to a public IP and tracks the mapping so replies come back. The important property is direction. A NAT gateway gives private instances outbound internet access, for OS updates say, without making them reachable from the internet. To accept inbound traffic you need something public on purpose, a public IP, a load balancer, or a port forward.

What gets you hired

The handshake does two jobs: prove identity and set up encryption. Client and server first agree on a TLS version and cipher. The server then presents its certificate, and the client validates it against a trusted CA and checks the hostname and expiry, that is what stops a man in the middle from impersonating the site. Then they do a key exchange, typically ephemeral ECDHE, so both sides derive the same symmetric session key without ever sending it in the clear. From there the actual data uses that symmetric key, which is far faster than asymmetric crypto. So security comes from two things together: the certificate proves who you are talking to, and the negotiated session key keeps the traffic confidential and tamper-evident. TLS 1.3 does this in one round trip.

What gets you hired

The difference is which side it represents. A forward proxy sits in front of clients and makes requests outward on their behalf, so the destination only ever sees the proxy's IP, not the 500 laptops behind it. That is a corporate egress proxy, a content filter, an anonymizer. A reverse proxy sits in front of servers and accepts inbound requests on their behalf, so the client believes the proxy is the server. It is the thing that terminates TLS on port 443 and forwards to backends on plain 8080, load balances, caches, compresses, and routes by host or path while hiding the backend entirely. In cloud work almost everything I touch is a reverse proxy: an application load balancer, an Nginx ingress, an API gateway, a CDN holding static assets at the edge with something like a 60 second TTL. In one setup I worked on, 3 services sat behind a single ingress and clients only ever knew 1 hostname. So the short version is that a forward proxy protects and controls clients, and a reverse proxy protects and scales servers.

What gets you hired

A stateful firewall tracks the state of connections, so when it permits an outbound or inbound flow, the matching return traffic is allowed automatically, you only write the rule for the direction that initiates. A stateless firewall has no memory and evaluates each packet independently, so you have to allow both directions explicitly. In AWS this maps directly: security groups are stateful, so if I allow inbound on 443 the response leaves without a separate rule. NACLs are stateless, so I have to allow the return traffic as well, and since responses come back on ephemeral ports I need to open 1024-65535 outbound, which is the classic mistake that makes NACLs seem broken. So they are not better or worse, they operate at different layers with different rules, and the stateless one needs explicit return-path rules.

What gets you hired

For every packet the host looks up the destination IP in its routing table, which maps destination prefixes to a next hop. If the destination is on the same subnet, it delivers directly, using ARP to find the MAC address. If it is somewhere else, it hands the packet to a router. The catch-all is the default route, 0.0.0.0/0, which points at the default gateway and matches anything without a more specific entry, normally the way out to the internet. When several routes could match, longest-prefix match decides: the most specific prefix wins, so a /24 entry takes precedence over the /0 default. This is exactly the mental model I use when a cloud instance has no internet: I check whether the subnet route table has a 0.0.0.0/0 route to an internet or NAT gateway, because a missing default route is the usual cause.

What gets you hired

First I’d narrow the failure: DNS, a TCP connect on 1433, or auth, because each points to a different layer. Azure SQL has its own server firewall that denies by default, so I’d confirm the VM’s outbound IP or a VNet rule is allowed. Then the network path: the NSG on the VM’s subnet/NIC must allow outbound 1433 and there must be a route. For production I’d actually put SQL behind a Private Endpoint so it resolves to a private IP and never touches the public internet, which removes the public-firewall guesswork. Once the TCP path is open, any remaining failure is auth, credentials or Azure AD, not networking.

What gets you hired

It comes down to what the load balancer can see. A layer 4 load balancer works at the transport layer, routing by IP and port without inspecting the payload, so it is fast, cheap, and works for any protocol. A layer 7 load balancer understands the application protocol, usually HTTP, so it can route by hostname, URL path, header, or cookie, terminate TLS, do sticky sessions, and plug into a WAF. I reach for L7 for normal web apps where I want to send /api to one target group and /static to another, or centralize TLS. I reach for L4 when I need extreme throughput and low latency, when the protocol is not HTTP, or when I need to preserve the original client IP and pass TLS through untouched. So it is a deliberate trade-off, not newer-is-better.

What gets you hired

Latency is the delay for a packet to travel, usually measured as round trip time, and it is bounded by distance and hops, London to Sydney sits around 250ms no matter what you pay. Bandwidth is the capacity of the link, say 1 gigabit per second. Throughput is what you actually achieve, which is often well below bandwidth because of latency, packet loss, and protocol overhead. So a high bandwidth link feels slow when the workload is latency bound. If a page makes 50 small sequential requests and each pays a 100ms round trip, that is 5 seconds of waiting, and widening the pipe changes nothing. The fix is cutting round trips, batching, caching, or moving the content closer with a CDN. Loss matters too. TCP backs off on loss, and the congestion window caps throughput on long fat links, which is why a transfer across the world can crawl at 20 megabits on a gigabit line while showing under 1 percent loss. So before anyone buys more bandwidth I measure which dimension is the bottleneck, because for most slow apps it is round trips, not capacity.

What gets you hired

I work the layers in order. First DNS: dig the hostname to confirm it resolves to the IP I expect, since a stale record is a frequent cause. Then reachability: ping for basic latency, and traceroute or mtr to see where on the path packets stall. Then the port: curl -v or nc -vz to the specific port, where connection refused means nothing is listening and a timeout usually means a firewall or security group is silently dropping packets. That refused-versus-timeout distinction localizes fast. Then I separate the layers: if routing is fine but connect times out, I check security groups and NACLs; if it connects but returns errors, the problem is the app, not the network. I also check both ends, that the service is listening on 0.0.0.0 not just localhost, the load balancer targets are healthy, and the rules permit the port the whole way through.

What gets you hired

The load balancer runs periodic health checks against each target, an HTTP path or a TCP connect, and flips a target to unhealthy after a configured number of consecutive failures (and back after consecutive successes). I tune the interval, timeout, and thresholds to balance fast detection against flapping, because too aggressive a check will eject a healthy node during a brief GC pause. I also point the check at a real readiness endpoint that verifies dependencies, not a trivial handler that returns 200 before the app can actually serve. For deploys and scale-in, the key is connection draining, the deregistration delay: when a target is removed the LB stops sending new requests but lets in-flight ones finish before it disappears. I pair that with graceful shutdown in the app, stop accepting new work, drain, then exit, so a rollout never cuts an active request.

What gets you hired

HTTP/1.1 handles one request at a time per connection, so browsers worked around it by opening many parallel TCP connections, with keep-alive to reuse them. HTTP/2 multiplexes many streams over a single connection and compresses headers, which is a real efficiency gain, but because it still rides one TCP connection, a single lost packet stalls every stream, that is TCP-level head-of-line blocking. HTTP/3 fixes that by running over QUIC, which is built on UDP with independent streams, so a lost packet only delays its own stream, not all of them. QUIC also folds the TLS handshake into the connection setup for a faster start, often one round trip, and supports connection migration so a session can survive switching networks, like wifi to cellular. So each step targets a specific bottleneck: parallelism, then per-connection efficiency, then transport-level head-of-line blocking.

What gets you hired

A VPN builds an encrypted tunnel between two endpoints and encapsulates your traffic inside it, so it can cross an untrusted network like the internet privately and you appear to be sitting on the remote network. IPsec is the established site to site standard, and it is what most cloud VPN gateways speak. WireGuard is the newer option, roughly 4,000 lines of code against tens of thousands for IPsec, simpler and faster, and it is gaining ground quickly. The routing choice is full versus split tunnel. Full tunnel pushes 100 percent of client traffic through the VPN, so everything is protected and subject to corporate inspection, but you pay the extra hop, often 20 to 40ms added, and you concentrate every user's video calls and software updates through one gateway. Split tunnel sends only traffic bound for the corporate or cloud ranges, say 10.0.0.0/8, through the tunnel and lets the rest go straight out. That is faster and far lighter on the gateway, but the direct traffic skips corporate security controls entirely. So it is a deliberate trade off, full tunnel when I need maximum control and inspection, split tunnel for performance when only specific networks need the private path.

What gets you hired

I start from requirements: bandwidth, latency sensitivity, how consistent the performance must be, security and compliance, and how soon it must be live. With that, the options sort themselves. A site-to-site IPsec VPN is encrypted, cheap, and live in a few hours, but it rides the public internet, so a single tunnel caps around 1.25 gigabits per second and latency wanders, fine for modest or bursty traffic and as a quick start. A dedicated circuit, Direct Connect on AWS or ExpressRoute on Azure, gives consistent low latency, links from 1 up to 100 gigabits, and cheaper egress, around 2 cents a gigabyte instead of 9, at the cost of real money and typically 4 to 12 weeks to provision, which suits steady high-volume or latency-sensitive workloads. A common pattern is the dedicated circuit as primary with a VPN as encrypted failover, and BGP flipping over in well under a minute. At scale I avoid a mesh of peerings, which are non-transitive and grow as n squared, so 10 VPCs would need 45 links. I use a hub, Transit Gateway or Azure Virtual WAN, for centralized routing instead. Across all of it I make sure CIDRs do not overlap, routes propagate correctly, and traffic is encrypted where compliance demands it, even on a private circuit.

What gets you hired

A CDN runs edge points of presence around the world, often 300 or more, that cache content near users, but the interesting part is how a user is sent to the right one. There are two mechanisms, usually combined. DNS-based steering: the CDN authoritative nameserver returns a different edge IP depending on where the resolver is, so a user in Amsterdam and one in Sydney resolve the same hostname to different PoPs, and a short TTL, often 20 to 60 seconds, lets the CDN re-steer quickly. Anycast: the CDN announces the same IP address from many locations over BGP, and internet routing carries each packet to the topologically nearest announcement, with no per-user DNS decision needed. That is typically what takes a first-byte time from 200 milliseconds down to 20. Anycast is also why CDNs are resilient. If a PoP fails, its BGP route is withdrawn and traffic reroutes to the next nearest within seconds, and spreading 1 IP across hundreds of sites means a DDoS is absorbed across the whole fleet instead of hitting one target. So nearest-edge is not magic caching, it is DNS geo-resolution and anycast routing doing the steering.

What gets you hired

The signature gives it away: small packets succeed, large ones hang. That is almost always an MTU problem, not the application. MTU is the largest packet a link will carry, typically 1500 bytes, and a tunnel like a VPN adds header overhead that lowers the effective MTU. Normally Path MTU Discovery handles this: the sender marks large packets do-not-fragment, and any router that cannot forward one returns an ICMP "fragmentation needed" so the sender shrinks. The failure mode is when a firewall blocks those ICMP messages, the sender never gets told, so large packets are silently dropped while small ones sail through, a PMTUD black hole. That is exactly why a TLS handshake can stall (the certificate is large) or a big upload hangs. The fixes are to allow ICMP type 3 code 4 through, reduce the MTU on the tunnel interface, or clamp the TCP MSS so segments always fit the path. MSS clamping is the usual go-to in practice.

What gets you hired

Normal TLS authenticates only the server. It presents a certificate and the client stays anonymous at the transport layer. Mutual TLS makes it two-way, so both sides present and validate certificates and each service cryptographically proves its identity to the other. That is the foundation of zero-trust service-to-service security: a service is trusted because of its verified identity, not because it happens to sit inside the 10.0.0.0/8 network. A service mesh operationalizes this by running a sidecar proxy next to every service, so across a fleet of 200 pods you get mutual authentication and encryption on every hop without changing a single line of application code. The mesh runs its own CA and issues short-lived per-service certificates, often with a 24 hour lifetime and automatic rotation well before expiry, which means a stolen certificate is useless almost immediately. On top of that identity it layers authorization policy, which service may call which, so the payments service can call the ledger and nothing else, plus traffic management and consistent observability. The honest cost is a sidecar per pod, roughly 50 to 100 megabytes of memory and a millisecond or so of added latency per hop. For most estates that is a fair price. So mTLS is the identity primitive, and the mesh is what makes it practical at scale.

What gets you hired

Elasticity is capacity that grows and shrinks automatically with demand, adding instances when traffic spikes and removing them when it drops. On the sites I have worked with, peak traffic is often 4 or 5 times the overnight baseline, so that difference is real money. Pay-as-you-go is the billing model that goes with it. Instead of buying servers up front for peak load, a big capital expense that sits idle most of the day, I rent capacity by the hour or the second, which is an operating expense. The two only pay off together. Elasticity is what makes pay-as-you-go actually save anything, because if I scale from 10 instances down to 2 during the quiet 12 hours, I stop paying for peak capacity around the clock. The honest caveat is that the cloud is not automatically cheaper. A forgotten instance still bills, maybe 70 dollars a month, and an oversized database running at 8 percent CPU bills at full price all year. The savings come from right-sizing and from actually turning things off. Putting non-production environments on a nightly shutdown alone took one team's dev account from roughly 4,000 dollars a month to about 1,600.

What gets you hired

I think of a cloud bill in four categories. Compute is usually the biggest, often 50 to 60 percent of the total: instances, containers, and serverless functions, billed as size times time, so vCPU and memory hours, or per invocation. Storage is next: data at rest charged per GB-month, roughly 2 cents a GB for standard object storage, plus performance tiers for IOPS and throughput. That spans object storage, block volumes, and databases, and the easy-to-forget parts, snapshots and backups, quietly accumulate. Data transfer is the one people most often overlook. Egress to the internet runs about 9 cents a GB, and traffic crossing regions or even availability zones is metered too, usually a cent or two per GB each way, while ingress is free. So a chatty cross-AZ architecture, or serving large files without a CDN, can run up a surprising bill. And managed services, databases, queues, load balancers, each bill on their own per-request, per-GB, or per-hour model, so the convenience of managed infrastructure carries an ongoing cost. Having this map matters because optimization starts with knowing which category dominates. I optimize compute very differently from egress or storage, so my first move is always to open the cost breakdown and find the top 3 line items rather than assume it is all servers.

What gets you hired

Egress is sneaky because of an asymmetry. Getting data into the cloud is almost always free, but data leaving is billed, both internet egress to your users, around 9 cents a GB, and traffic crossing regions or even availability zones, which is typically a cent or two per GB in each direction. Because ingress is free, people stop thinking about transfer costs, and then egress shows up as a surprising line on the invoice. It comes from a few places: serving large files, images, video, downloads, straight to users without a CDN, so every single download egresses from the origin; cross-region replication; chatty microservices talking across AZs, which is billed cross-AZ traffic; and bulk exports out of the cloud. The fixes are architectural. A CDN is the big one for user-facing content. It caches at the edge, and with a healthy cache hit ratio of 90 to 95 percent only a small fraction of requests ever reach the origin, which is both cheaper per GB and faster. Beyond that, keep chatty services in the same AZ and region so their traffic is local, avoid needless cross-region hops, compress payloads, gzip alone often cuts JSON by 70 percent, and cache aggressively so I am not re-sending the same bytes. So I treat egress as a first-class design consideration, where data lives and how it flows, rather than something to discover on the bill.

What gets you hired

Right-sizing is matching the resources you provision, instance type, CPU, memory, capacity, to what the workload actually uses, instead of guessing on the high side. The reason it is the biggest source of waste is human: teams provision for peak load or pad generously for safety, deploy it, and then never revisit it, so a huge amount of capacity sits at low utilization, getting billed every hour for headroom that is never touched. It is not exotic services that blow the budget, it is fleets of oversized instances running at 10 percent CPU. The fix is to be data-driven: look at the actual CPU, memory, and throughput metrics over a representative period, and then downsize or switch to a more appropriate instance family so the resource fits the real demand, keeping enough headroom for genuine spikes but not for imagination. And critically it is continuous, not a one-off, because workloads grow and shrink, so right-sizing is a recurring practice, and I pair it with cleaning up zombies, idle instances nobody owns, unattached volumes, old snapshots, which are pure waste. So the message is: most cloud savings come from cutting over-provisioned, underutilized capacity, found through utilization data, not from clever tricks.

What gets you hired

These three models trade flexibility against price. On-demand is full price with no commitment, you pay per hour or second and can start and stop freely, which is maximum flexibility but the highest unit cost, so it suits unpredictable or short-lived workloads. Committed or reserved pricing means you commit to a certain usage level for one to three years in exchange for a substantial discount, often 30 to 70 percent, which is ideal for steady, predictable baseline load that you know will run continuously. Spot or preemptible instances are spare capacity sold at a deep discount, up to around 90 percent off, with the catch that the provider can reclaim them on short notice, so they are perfect for fault-tolerant, interruptible, or batch workloads that can tolerate being killed and restarted, and a bad fit for a stateful database. The real answer is not to pick one but to mix them to the shape of your usage: cover the always-on baseline with commitments to lock in the discount, absorb the variable demand above that baseline with on-demand for flexibility, and run anything fault-tolerant or batch on spot for the deep savings. So I look at the workload profile, what is steady, what is bursty, what is interruptible, and assign each part to the cheapest model that fits its risk and predictability.

What gets you hired

Tagging is the foundation of cost accountability. Without tags the cloud bill is one opaque number. You can see you spent 80,000 dollars last month, but not which team, service, or environment drove it, so you cannot target optimization or hold anyone responsible. Consistent tags, typically 4 or 5 required ones like team, environment, project, and cost-center, let you slice the bill along those dimensions, and the invoice turns into attributable data: this team spent 12,000 dollars, dev is somehow 30 percent of prod, this one project costs 9,000 a month. That allocation unlocks the two practices that actually change behavior. Showback, where each team sees its own spend and starts caring about it, and chargeback, where the cost lands on the team's budget so it becomes a real constraint they own. The catch is that this only works if tagging is consistent and complete. I have seen estates where 40 percent of spend lands in an unallocated bucket, and a half tagged estate gives you a half useful breakdown. So I enforce it: a documented required tag set, automation that applies tags at provision time, tags inherited from the Terraform provider default_tags, and policy as code that flags or rejects untagged resources, with a target of 95 percent or better allocation coverage. Tagging is not about tidiness. It is the prerequisite for seeing where the money goes, which is why every FinOps effort starts here.

What gets you hired

Both are about matching what you pay for to what you actually need, instead of paying for peak capacity continuously. Autoscaling adds instances when demand rises and, just as importantly for cost, removes them when demand falls, so you are not running and paying for peak-sized capacity twenty-four hours a day when you only need it at busy times. The key is to scale on a signal that reflects real load, CPU, request rate, or queue depth, with enough headroom to absorb spikes and the lag of spinning up new capacity. The easiest concrete win, though, is scheduling non-production environments: dev, staging, and test environments almost never need to run overnight or on weekends, yet they often run 24/7 by default, so simply shutting them down outside working hours cuts their cost by roughly two-thirds, and it is low-risk because nobody is using them then. And for workloads that are very spiky or mostly idle, serverless or scale-to-zero is a cost lever in its own right, because you pay only when code actually runs rather than for an always-on server sitting idle. So autoscaling for production demand, scheduling for non-prod, and scale-to-zero for spiky workloads, all the same principle: only pay for capacity when it is doing useful work.

What gets you hired

Storage grows forever if you let it, so I manage it by access pattern rather than by deleting things in a panic. The core idea is tiering. I match the storage class to how often the data is really read. Hot for frequently accessed data, an infrequent access tier for data touched maybe once a month, and an archive or cold tier for data that is rarely read but must be kept for compliance, each step roughly 3 to 5 times cheaper per GB. Then I automate it with lifecycle policies, rules that transition objects as they age and expire them at the end of retention. On a logs bucket that usually looks like standard for 30 days, infrequent access at 30, archive at 90, delete at 365, so cost management runs itself instead of depending on someone remembering. Alongside that I hunt orphaned storage, which is pure waste. Unattached volumes left behind when instances were deleted, snapshots kept years past their retention, incomplete multipart uploads, forgotten datasets. On one account that cleanup alone took the storage bill from about 4,000 dollars a month to 1,600. The trade off to respect is that cold tiers are cheap to store but charge for retrieval and add latency, minutes to hours for archive, so I tier on genuine access data and check that retrieval charges would not erase the savings if the data gets read more than expected. So, tier by access frequency, automate transitions and expiry, and continuously clean up orphans.

What gets you hired

The mistake is comparing only the hourly sticker price. Yes, a managed service, say a managed database, costs more per hour than the same engine running on a raw instance, but that price is buying operations: the provider handles patching, backups, high availability, failover, scaling, and a lot of the monitoring. When you self-host, that work does not disappear, it moves to your engineers, who now own setup, OS and engine patching, version upgrades, configuring and testing failover, capacity management, and getting paged at 3am when it breaks. That engineering time is real money and often the larger cost, especially because skilled engineer hours are expensive and finite. On top of labor there is operational risk: a self-managed system is one botched upgrade or one missed backup away from an outage or data loss whose cost can dwarf years of the hourly savings. So the right lens is total cost of ownership, infrastructure plus engineering labor plus the expected cost of operational risk, not just the bill line. When I do that comparison, managed services usually win, because the per-hour premium is less than the loaded cost of doing it well yourself, and the honest exception is at large scale or with a specialized requirement, where self-hosting can genuinely pay off and you have or can justify the dedicated expertise to run it reliably.

What gets you hired

The whole point is to be proactive, because if you only look at the invoice you find out weeks late, and one forgotten p3 GPU instance at roughly 3 dollars an hour is about 2,200 dollars a month burned quietly. So first I set budgets per team, project or account, with alerts at 50, 80 and 100 percent of the expected spend, plus a forecast alert, so someone gets notified mid-month while there is still time to act. On top of that I turn on cost anomaly detection, which flags a sudden spike against the normal pattern, say a day that jumps 40 percent above the trailing 14 day baseline. That is what catches the classic disasters, a developer who left a big GPU box running, a misconfigured loop spinning up resources, an export accidentally egressing 5 terabytes at 9 cents a gigabyte, within hours instead of on the monthly bill. I also keep spend continuously visible with a dashboard and a 30 minute cost review each month with the teams that own the spend, so cost is a monitored metric, not a once-a-month surprise. And for the genuinely catastrophic cases I add guardrails, service quotas and limits, and policy as code that blocks launching the largest instance types without approval, so the worst runaways are prevented outright rather than merely alerted on. Budgets and alerts, anomaly detection, continuous visibility, and preventive guardrails, all aimed at catching a cost problem long before the invoice arrives.

What gets you hired

I start with visibility, tagging and cost tooling to attribute spend, because usually the top 5 line items are 70 or 80 percent of the bill and blind cuts are how you cause an outage. First I remove pure waste, idle and oversized instances, unattached volumes, dev environments running 168 hours a week when they are needed for 45, which is risk free and often 15 to 20 percent straight off. Then I right-size from real utilization, if a box sits at 8 percent CPU for 30 days it does not need that shape, and I cover the steady-state baseline with Savings Plans, roughly 1 year commitments for 30 to 40 percent off, leaving burst on demand and spot only for fault-tolerant work. But the biggest wins are architectural. Lifecycle-tiering cold objects to infrequent access and archive after 30 and 90 days, a CDN and caching to cut egress and compute, autoscaling down off-peak, serverless for spiky loads. Egress at 9 cents a gigabyte and idle capacity are the silent killers. On the last estate I worked, that took the bill from about 40,000 dollars a month to 26,000 without touching redundancy. Throughout, I tie every cut to SLOs. If the availability target needs 3 availability zones, I do not collapse to 2 to save money.

What gets you hired

FinOps is the practice of bringing financial accountability to cloud spend, and the key is that it is cross functional, engineering, finance, and product together, not finance unilaterally controlling the bill. The model has three phases. Inform, which is visibility and cost allocation through tagging and dashboards, and my bar there is that at least 95 percent of spend is attributable to an owning team, because unallocated spend is spend nobody will fix. Optimize, the levers like right sizing, commitment discounts, and eliminating waste. And Operate, making it continuous governance and culture rather than a one time cleanup. It has to involve engineers because engineers drive the spend through their architecture and resource choices, so cost is really decided at design time, and a finance review 30 days after the invoice cannot undo those decisions. So the way you make engineers care is visibility and ownership, show each team the cost of what they run, attributed to them, give them a budget with an alert at 80 percent of it, and cost becomes a metric they watch beside latency and reliability. That beats policing, because a central team chasing overspend after the fact does not scale across dozens of teams and does not change the decisions that caused it, whereas engineers who see the cost of their own choices start optimizing naturally. And I frame the goal as value for money, not minimizing the bill, because cost is one dimension of good engineering, not the only one.

What gets you hired

Total cloud spend on its own is almost meaningless, because it has no context. If the bill went from 100,000 dollars a month to 130,000, that could be waste, or it could simply mean the business grew and you are serving far more customers. You cannot tell which from the total, and that is exactly why reduce the bill is a bad goal. Unit economics fixes it by adding a denominator, cost per unit of business value, per customer, per transaction, per active user, per request. That ratio actually means something, because it relates what you spend to what the business gets. With it you can separate healthy growth from waste. If spend rises 30 percent while customers double and cost per customer drops from 4 dollars to 2.60, that is efficient scaling and it is good news. A cost per customer that is flat or climbing while you grow is the real warning sign, it says the architecture is not scaling efficiently and there is genuine inefficiency to find. This reframing matters because it points optimization at efficiency and scalability rather than blindly slashing the total, which can starve the business by cutting capacity that supports growth. It is also the language leadership and finance speak, they care about gross margin and cost to serve, not raw infrastructure dollars. So I measure and optimize unit cost, treating cloud spend as an investment that should scale sub linearly with the value it produces, rather than a bill to minimize.

What gets you hired

Permissions are set for three identities, owner, group, and other, each with read (4), write (2), execute (1) that you sum. So 644 is owner read-write (6), group read (4), other read (4): a typical data file. 755 is owner read-write-execute (7), group and other read-execute (5): what you want for a script or a directory, everyone can run or enter it but only the owner can modify it. One gotcha: on a directory, the execute bit means you can traverse into it, not that it runs.

What gets you hired

I find the PID with ps aux | grep or pgrep, or watch it live in top. Then I send SIGTERM first with plain kill, which asks the process to shut down cleanly, flush buffers, close sockets, and exit on its own terms. Only if it ignores that do I escalate to kill -9, SIGKILL, which the kernel enforces immediately with no chance to clean up, so it can leave corrupt files or stale locks. So -9 is the last resort, not the first move.

What gets you hired

Environment variables are inherited key-value settings, you export VAR=value and child processes see it, which is how config like AWS_REGION reaches a program. PATH is a colon-separated, ordered list of directories the shell searches to resolve a command; "command not found" usually means it is not on PATH or not executable. Pipes (|) feed one command stdout into the next command stdin, which is how you compose small tools. Redirection sends streams to files: > overwrites, >> appends, < feeds input, and 2> handles stderr separately, with 2>&1 merging stderr into stdout, which matters when you want to capture errors too.

What gets you hired

First df -h to see which mount is actually full, because it is often not /, it is /var sitting at 100 percent from logs. Then du to localize it, du -ahx /var, sorted with sort -rh and piped to head -20, ranks the 20 heaviest directories and files and usually one of them is obviously the problem. If df says the disk is 100 percent full but du only accounts for, say, 20 gigabytes of it, that gap is the classic trap. A process is holding a deleted file open, so the space is not returned until that handle closes. lsof piped through grep deleted finds it, and restarting that process, often a logger, frees the space instantly. I would also run df -i for inode exhaustion, because millions of tiny cache or session files can hit 100 percent of inodes while df -h still shows free space, and the error message looks identical. Then I fix the cause rather than just buying time: logrotate with a size cap and 7 days of retention, clean the package and build caches, and an alert at 80 percent so it warns me before it silently refills.

What gets you hired

I start with systemctl status, which gives me the failed state, the exit code, and the last 10 log lines. The exit code alone often tells the story, a 203 means the ExecStart binary was not found or is not executable, while a plain 1 usually means the app itself started and then died. Then journalctl -u for the service, with -n 100 and --no-pager, gives me the full error, and that normally says exactly why it stopped. Next I read the unit itself with systemctl cat, checking the ExecStart path, the User it runs as, the WorkingDirectory, and any After or Requires dependencies, because the cause is very often a wrong absolute path, a missing environment variable, or a permissions problem, like the service user not being able to read a file or bind to a port below 1024. If it is still unclear, I run the ExecStart command by hand as that service user, which strips systemd out of the picture and shows me the raw error. And after any edit to the unit file I run daemon-reload before restarting, because otherwise systemd is still running the old definition, and that is the step people forget most.

What gets you hired

Locally, ss -tlnp shows what is listening, the port, and the owning process, and crucially what address it bound to, 127.0.0.1 is localhost-only and a frequent cause of "works locally, not remotely". To test a remote service I use curl for HTTP or nc -vz host port for a raw TCP check. Then I layer the diagnosis: DNS resolves (dig)? Connection refused means nothing is listening; a hang usually means a firewall or security group dropping packets; if it connects but errors, the problem is the app, not the network. Ping alone is not enough since it only tests ICMP, which is often blocked.

What gets you hired

First I read load average against core count, load 8 means different things on 2 cores versus 16. Then I split the cause in top: high user CPU points to a busy app, but high I/O wait (%wa) means disk or network is the bottleneck and the CPU is idle waiting, and heavy swap (free -h) makes everything slow for a memory reason. I find the offender with htop for CPU/memory or iotop/iostat for I/O. Before acting I decide if it is legitimate load that needs scaling or a runaway that needs a fix; renice to de-prioritize, or fix the process, rather than blind-killing. Then I confirm recovery and add a limit or alert so it cannot silently recur.

What gets you hired

I run a quick vitals sweep before diving in. Load and CPU with uptime/top, including the 1/5/15-minute trend. Memory with free -h, and dmesg or journalctl -k for OOM kills. Disk with df -h and df -i, a full disk or exhausted inodes breaks things silently. Then logs, journalctl -e and the relevant /var/log, plus dmesg for kernel and hardware messages. Finally the service itself, systemctl status, is it listening on its port (ss -tlnp), and can it reach its dependencies. That order tells me which layer is actually failing instead of assuming it is the app.

What gets you hired

I add it with crontab -e, five time fields plus the command, like "0 2 * * * /opt/app/backup.sh" for 2am daily. The thing that bites people: cron runs with a minimal environment, it does not source your profile, so PATH is bare and your env vars are gone. So I use absolute paths everywhere, set required env in the crontab or the script, and never assume a working directory. I redirect output explicitly, ">> /var/log/job.log 2>&1", because cron silently mails output you will never read. To debug a no-run, I first confirm cron actually triggered it in the cron/syslog, then read the job log for the real error, which is usually a missing binary on PATH or a relative path.

What gets you hired

There are four levers. On-Demand is pay-as-you-go with no commitment but the highest unit price, right for spiky or short-lived workloads. Reserved Instances and Savings Plans give up to about 72% off for a 1 or 3 year commitment; I lean toward Compute Savings Plans because they commit to a dollar-per-hour spend rather than specific instances and even cover Fargate and Lambda, so they survive instance-family changes. Spot is spare capacity at up to 90% off, but AWS can reclaim it on a 2-minute notice, so I only put fault-tolerant, stateless, or batch work there. The real skill is the mix: cover the steady baseline with Savings Plans, absorb bursts with On-Demand, and run interruptible workloads, CI runners, batch processing, stateless web tiers behind an ASG, on Spot. That layered approach typically cuts compute cost dramatically without risking the workloads that cannot tolerate interruption.

What gets you hired

I match the class to the access pattern. S3 Standard for hot, frequently accessed data. Standard-IA for data I rarely touch but must get instantly, cheaper storage but with a per-GB retrieval cost and a 30-day minimum, so it backfires for data you actually read often. If access patterns are unknown or shifting, Intelligent-Tiering auto-moves objects between tiers for a small monitoring fee, which removes the guesswork. For archives there is a Glacier family: Instant Retrieval for millisecond access at archive prices, Flexible Retrieval for minutes-to-hours, and Deep Archive, the cheapest, where restore can take around 12 hours. The pattern is that price falls as retrieval gets slower and as you commit to leaving data put. Then I automate it with lifecycle policies: new objects in Standard, transition to IA after 30 days, to Glacier after 90, and expire after a year. The gotchas I watch are minimum storage durations and retrieval fees, because misjudging access frequency can make a cheaper class more expensive overall.

What gets you hired

Every request starts from an implicit deny, so nothing is allowed unless something grants it. AWS then evaluates all applicable policies with a key rule: an explicit deny always wins. If any policy, identity-based, resource-based, an SCP, or a permission boundary, explicitly denies the action, it is denied no matter what else allows it. If there is no explicit deny, AWS looks for an explicit allow across identity-based policies (on the user or role) and resource-based policies (on the resource, like an S3 bucket policy); if it finds one, allow, otherwise the default deny holds. On top of that, Organizations SCPs and permission boundaries act as ceilings, not grants: an action has to be allowed by the principal policy and also be within the SCP and boundary. So the mental model is default deny, explicit deny overrides everything, then an allow within the ceiling. That framing is what keeps me from both locking people out and handing out more than intended.

What gets you hired

An Auto Scaling Group keeps a fleet between a minimum and maximum around a desired count, launching instances from a launch template and spreading them across AZs for resilience. For scaling I default to target tracking: I pick a metric and a target, like keep average CPU at 50% or requests-per-target at some number, and AWS adds or removes capacity to hold that target, which is far less fiddly than hand-tuned step thresholds. Step and scheduled scaling are there for predictable spikes or known traffic patterns. The ASG also does health-based self-healing: it replaces instances failing EC2 or ELB health checks, so it improves availability, not just throughput. The tuning that matters is warmup and cooldown, so a new instance is only counted once it is actually serving, avoiding flapping, and choosing a scaling metric that reflects real load. For a request-driven web tier I prefer requests-per-target or latency over raw CPU.

What gets you hired

I match the primitive to the pattern. SQS is a queue, it decouples a producer from a consumer that pulls work at its own pace, which is great for load leveling and work queues. It is at least once delivery, messages can sit up to 14 days, and there is a FIFO flavour when I need strict ordering and dedup, at roughly 3,000 messages a second with batching. SNS is pub sub fan out, one message pushed to many subscribers at once, which I use when several consumers each need their own copy. The classic combo is SNS into 3 or 4 SQS queues so each consumer gets a durable copy it can retry. EventBridge is an event bus that routes events by content based rules to many targets, with schema support, scheduling, and native AWS and SaaS integrations, so it is my choice for event driven architectures where producers and consumers should not know about each other. Kinesis is for streaming, ordered and replayable and high throughput, 1MB per second per shard on write, with a retention window I can set to 7 days or more so multiple consumers read the same stream and I can reprocess history. Think clickstreams, real time analytics, log pipelines. So the question I ask is, queue of work, broadcast, event routing, or an ordered replayable stream.

What gets you hired

A VPC endpoint lets resources in your VPC reach AWS services over the AWS private network instead of going out through a NAT gateway to the public internet. There are two kinds. A gateway endpoint is for S3 and DynamoDB only; it works by adding an entry to your route table and it is free, so it is the easy default for private S3 or DynamoDB access. An interface endpoint uses PrivateLink: it puts an elastic network interface with a private IP into your subnet and works for most other AWS services and even your own services published behind a Network Load Balancer; it is billed per hour and per GB. The reasons to use them are security and cost: traffic to the service never leaves the AWS network, and you avoid NAT gateway data-processing charges for that traffic, which adds up for something like heavy S3 access. I also attach endpoint policies to restrict exactly which buckets or actions are reachable through the endpoint.

What gets you hired

CloudFront is the AWS CDN: it caches content at edge locations near users, so most requests are served at the edge instead of traveling to the origin, which cuts latency and offloads the origin. I point it at origins, an S3 bucket for static assets, an ALB for the dynamic app, or a custom origin, and configure cache behaviors per path: cache /static aggressively with long TTLs, and pass /api through with minimal caching and the right cache key. Security matters: for an S3 origin I use Origin Access Control so the bucket only accepts requests coming through CloudFront and is not publicly reachable, and I add WAF for filtering and signed URLs or cookies for private content. The operational gotcha is invalidation: invalidations propagate slowly and can cost money, so instead of invalidating on every deploy I use versioned asset names like app.abc123.js, which lets the browser and edge cache forever and just fetch the new filename on release, while old ones age out. That combination, edge caching, locked-down origin, and fingerprinted assets, gives fast and safe delivery without cache-staleness pain.

What gets you hired

I reach for DynamoDB when the access patterns are well understood and high volume and I need single digit millisecond reads, typically p99 under 10ms, on key value or document shaped data, and I do not need ad hoc queries, joins, or complex transactions, which is where a relational database is still the right tool. The mental shift is that you model the table around your access patterns up front rather than around normalized entities. In practice that often means a single table design with composite partition and sort keys, plus a few Global Secondary Indexes, usually 2 or 3, for the alternate ways you need to query. The performance lever is the partition key, because DynamoDB shards by its hash and each physical partition caps out around 3,000 read units and 1,000 write units per second. A high cardinality key spreads load evenly. A skewed one, say partitioning by a status value or by a single large tenant, concentrates traffic on one partition and you get throttled while your table level capacity looks half idle. That is the classic hot partition problem. Adaptive capacity softens it, but the real fix is composing a key that distributes, for example suffixing with a shard number 1 through 10. For capacity I use on demand when traffic is unpredictable and provisioned with auto scaling when it is steady, and I default to eventually consistent reads, which cost half as much, opting into strong consistency only where I truly need it.

What gets you hired

I start with visibility, because optimizing blind just breaks things. Cost Explorer and the Cost and Usage Report show where the money actually goes, and I enforce tagging so I can attribute spend to teams and services and target the biggest line items first. Then waste: Compute Optimizer to rightsize over-provisioned compute, and a sweep for idle resources, unattached EBS volumes, stale snapshots, unused elastic IPs, idle load balancers, dev environments left running overnight. Next the structural levers: cover steady baseline usage with Savings Plans or RIs, move batch and interruptible workloads to Spot, and look at Graviton for cheaper price-performance. On storage, apply S3 lifecycle policies and appropriate classes. Then the one people miss: data transfer. Cross-AZ chatter and internet egress can quietly dominate a bill, and VPC endpoints and CloudFront cut a lot of it, plus keeping chatty components in the same AZ. Finally I make it durable with budgets, anomaly detection, and a regular cost review, so the savings stick instead of creeping back. The order matters: measure and attribute, kill waste, commit, then tackle storage and transfer.

What gets you hired

Cold starts are the famous one, a new environment adds startup latency, worse with big packages or VPC attachment, and I handle it with provisioned concurrency on latency-sensitive paths and lean packages. But at scale the bigger issues are elsewhere. Concurrency: there is an account limit, 1,000 by default, and bursts past it get throttled with 429s, so I use reserved concurrency to guarantee headroom for critical functions and to cap noisy ones. Downstream pressure is the classic trap: Lambda scales out so aggressively that it can open thousands of database connections and take RDS down, so I put RDS Proxy in front to pool connections, or buffer through SQS so a consumer drains at a controlled rate. Idempotency is non-negotiable because async and event-source invocations are at-least-once, so every handler needs a dedup key and safe retries, with DLQs or on-failure destinations so nothing is silently lost. And I respect the limits, the 15-minute max timeout, payload and /tmp sizes, and I orchestrate multi-step flows with Step Functions rather than chaining functions and reinventing retries and state. Serverless removes server management, but these operational edges are what you actually design around.

What gets you hired

I localize the layer before touching anything. First I split the 5xx in CloudWatch: ELB 5xx means the load balancer could not get a healthy response, no healthy targets or upstream timeouts, while Target 5xx means the app itself returned the error. That one split tells me whether I am chasing infrastructure or application. Then target health: are ECS tasks failing health checks and flapping in and out of the target group, and is the service hitting CPU or memory limits or being throttled? Then latency per hop, ALB target response time, ECS task metrics, and RDS, using X-Ray traces to pinpoint which segment is slow, the app, a downstream dependency, or the database. RDS is a prime suspect for this signature: I check CPU and freeable memory, but especially connection count and slow queries, because an exhausted connection pool or a missing index shows up exactly as intermittent slowness and 5xx under load. Throughout, I correlate with change, a recent deploy, a traffic spike, a scaling event, since intermittent problems that began at a known moment narrow the cause quickly. Only once I know the layer do I act, which might be fixing a query, adding RDS Proxy, tuning health checks, or scaling, rather than blindly restarting.

What gets you hired

First I would challenge the requirement, because active-active is the priciest, most complex option, so is this for latency, regional resilience, or data residency? Often active-passive DR is the better answer, so I confirm the driver. Assuming active-active is justified: routing is straightforward with Route 53 latency-based or geolocation routing plus health checks, which sends each user to the nearest healthy region and fails over automatically, with Global Accelerator as an option for non-HTTP. Stateless compute is the easy part, run the whole stack in each region behind its own ALB with auto scaling, and replicate artifacts and config. The hard part, always, is data. I decide per data class. DynamoDB Global Tables give genuine multi-region multi-master, but resolution is last-writer-wins and reads are eventually consistent, so it fits data that tolerates that. Aurora Global Database gives low-latency cross-region read replicas and fast promotion, but there is a single write region, so it is closer to active-passive for writes. S3 Cross-Region Replication handles objects. The thing a principal has to own is the consistency trade-off: for true multi-master I decide conflict handling deliberately, last-writer-wins, or partition writes so each record has a home region, or application-level merge for things like profiles, keep every operation idempotent, and I say the CAP choice out loud, during a partition, strongly-consistent data favors consistency over availability and I accept that. And I rehearse regional failover, because a DR path you have never tested does not exist.

What gets you hired

Kubernetes is a container orchestrator, and the core idea is desired-state reconciliation. Doing this by hand, running containers across a fleet of machines means manually handling placement, restarts, scaling, networking, and deploys, and that stops being practical fast. With Kubernetes I declare the desired state, this image, 3 replicas, this config, and controllers loop continuously, comparing actual to desired and closing the gap within seconds. Everything good falls out of that one idea. Self-healing, so a crashed pod is restarted and a node that goes unready has its pods rescheduled elsewhere. Horizontal scaling, where I can tell the autoscaler to hold pods around 70 percent CPU and it moves the replica count for me. Rolling updates, where by default it brings up 25 percent extra pods and takes at most 25 percent down at a time, so the app stays serving while the new version rolls in, and rollbacks when it does not. Plus service discovery and load balancing across whatever replicas exist right now. The honest caveat is that all of this is real operational complexity, a control plane to run, YAML to own, networking to understand. I would reach for it when the number of services or the scale genuinely justifies it, not for 1 small app that a managed container service would run happily.

What gets you hired

A cluster splits into the control plane and the worker nodes. On the control plane, the API server is the front door, listening on 6443, and everything talks through it, kubectl, the controllers, and every kubelet. It validates changes and persists them. etcd is the backing store, a consistent key-value database holding all cluster state, and it uses Raft, so production runs 3 or 5 members to keep quorum. That is also why etcd backups are non-negotiable, because losing etcd is losing the cluster. The scheduler decides which node each pending pod lands on based on resource requests and constraints. The controller-manager runs the reconciliation loops that keep actual state matching desired. On every worker node, the kubelet runs the pods assigned to it and reports status back, kube-proxy programs the networking that makes Services work, and a container runtime like containerd actually launches the containers. Knowing this split is what makes debugging tractable. Pods stuck Pending points at the scheduler and node capacity. A node going NotReady after roughly 40 seconds of missed heartbeats, with its pods evicted about 5 minutes later, points at that node's kubelet or network. Pods failing on exactly 1 node points at that node's runtime.

What gets you hired

A pod is the smallest thing Kubernetes schedules. It is 1 or more containers that share a network namespace, so they share an IP and can talk to each other over localhost, and they can share volumes. That is how sidecar patterns work. Pods are ephemeral by design. I almost never create them directly, because a bare pod has no safety net. If its node dies, nothing brings it back, there is no scaling, and there is no controlled way to update it. A Deployment fixes that by putting a controller in front. It manages a ReplicaSet that holds the desired replica count, so a pod that dies is recreated in seconds, and scaling from 3 replicas to 10 is one number changing in one field. It also gives me declarative rollouts. I update the image tag and the Deployment replaces pods gradually, by default surging up to 25 percent extra and keeping at least 75 percent of capacity serving, so the app stays available through the change. And it keeps revision history, 10 revisions by default, so if the new version starts throwing errors I can roll back with a single command and be on the old image in under a minute. So the Deployment is the controller that turns a fragile pod into a resilient, updatable workload.

What gets you hired

Pods are ephemeral and their IPs change every time they restart or reschedule, so you cannot point clients at a pod IP. A Service solves that: it provides a stable virtual IP and a DNS name that fronts a set of pods, selected by labels, and load-balances across the ones that are ready. The types differ by reach. ClusterIP, the default, is only reachable inside the cluster, which is right for internal service-to-service traffic. NodePort opens a fixed port on every node so the service is reachable from outside in a basic way, but it is rarely used directly in production. LoadBalancer provisions an actual cloud load balancer pointing at the service, which is the usual way to expose something externally on a managed cluster. And above all of these, for HTTP, an Ingress plus an ingress controller gives L7 routing, host and path based, TLS termination, so many services can share one entry point instead of one load balancer each.

What gets you hired

I match the controller to the shape of the workload. A Deployment is for stateless, interchangeable replicas, web servers, APIs, background workers, where every pod is identical and order does not matter. I can run 3 replicas or 30 and the rollout can replace them in any order, which covers most application workloads. A StatefulSet is for workloads that need stable identity and stable storage, so databases, message brokers, clustered systems. Each pod gets a stable ordinal name, kafka-0, kafka-1, kafka-2, a stable network identity, ordered startup and shutdown, and its own persistent volume that follows it across restarts. A Deployment simply cannot give you that. A DaemonSet runs exactly 1 pod per node, which is what node level agents need, log shippers, metrics agents, CNI or storage plugins. On a 40 node cluster that means 40 pods, and when node 41 joins, the DaemonSet schedules onto it automatically. For run to completion work rather than long running services I use a Job, or a CronJob for something scheduled, like a backup that runs at 2am. The classic mistake is running a stateful database as a plain Deployment and then wondering why the pod came back with a different name and an empty volume.

What gets you hired

I keep configuration out of the image so the same artifact runs everywhere. Non-sensitive config goes in a ConfigMap, injected as environment variables or mounted as files. Sensitive values, database passwords, API keys, tokens, go in Secrets, consumed the same way and kept out of git. The catch that trips people up is that a Kubernetes Secret is only base64-encoded, which is encoding, not encryption. By default, anyone who can read the object through the API or get at etcd can decode it. So to actually protect secrets I enable encryption at rest for etcd, lock down access with RBAC so only the right service accounts can read them, and I prefer mounting secrets as files over environment variables because env vars leak more easily into logs and child processes. For anything serious I integrate an external secrets manager, Vault or a cloud secrets service, through the Secrets Store CSI driver or the external-secrets operator, so the source of truth is a real vault with rotation and audit, not a base64 blob in etcd.

What gets you hired

They look similar but they do 3 different things. A liveness probe asks whether the container is still healthy, and if it fails, the kubelet restarts the container, so it is how you recover from a hang or a deadlock. A readiness probe asks whether the pod can serve traffic right now, and if it fails, the pod is pulled from the Service endpoints but not restarted. That is exactly what you want during warmup or when a dependency blips, you stop sending it traffic without killing it. A startup probe is for slow starting applications, and it gates the other 2, so liveness and readiness do not even run until startup succeeds. The defaults matter here, a periodSeconds of 10 and a failureThreshold of 3 mean a container is restarted about 30 seconds after it goes bad, and a JVM that needs 2 minutes to boot would be killed 4 times before it ever came up. A startup probe with a failureThreshold of 30 gives it a 5 minute budget and prevents exactly that. The big danger is a misconfigured liveness probe. If it is too aggressive, or worse, if it checks a downstream dependency, 1 slow database can restart loop every healthy pod you have at once. So liveness should test the app itself, and dependency or warmup conditions belong in readiness.

What gets you hired

A request is what the pod is guaranteed, and crucially it is what the scheduler uses to place it. If a pod requests 500 millicores and 1 gigabyte of memory, a node needs that much free to accept it at all. A limit is the hard ceiling on what the container may actually use at runtime. The behavior on overage is asymmetric, and that is the part people miss. If a container exceeds its CPU limit it gets throttled, slowed down but kept alive, because CPU is compressible. If it exceeds its memory limit it gets OOMKilled, exit code 137, because memory cannot be reclaimed on the fly. So a memory limit set 20 percent too low does not show up as slowness, it shows up as containers dying and restarting under load. Requests and limits together also set the QoS class, Guaranteed when they are equal, Burstable when the request is lower than the limit, BestEffort when neither is set, and that order decides who gets evicted first when a node runs out of memory. BestEffort goes first. So I set requests from observed real usage, roughly the p95 plus a little headroom. Too low and pods get starved or evicted, too high and I reserve 4 gigabytes on every node that nothing ever touches, which is money burned every hour.

What gets you hired

The scheduler works in 2 phases. First it filters, dropping any node that cannot run the pod, not enough free CPU or memory against the pod's requests, node selectors that do not match, taints that are not tolerated. Then it scores the survivors and places the pod on the best one. To influence that, there are two complementary mechanisms. Affinity attracts. nodeSelector is a plain label match, node affinity is the richer form with required and preferred rules, where a preferred rule carries a weight from 1 to 100, and pod affinity and anti affinity place a pod relative to other pods, so I can co locate two services that chat constantly or keep replicas apart. Taints and tolerations are the inverse, a taint repels every pod unless the pod carries a matching toleration, which is how you reserve nodes, say a pool of 4 GPU nodes or the control plane, for specific workloads. For resilience I use topology spread constraints or anti affinity with a maxSkew of 1 across zones, so 6 replicas land 2 per zone across 3 availability zones and losing one zone never takes the whole service down. And most pod stuck in Pending cases trace right back to this filter phase, no node satisfied the requests, the selectors, or the tolerations.

What gets you hired

When I change a Deployment it does not replace everything at once. It creates a new ReplicaSet and shifts pods over gradually, governed by two settings, maxSurge, how many pods above the desired count it may add, and maxUnavailable, how many it may take down at a time. Both default to 25 percent, so on 8 replicas it brings up 2 extra and takes at most 2 down. The thing that makes it zero downtime is the readiness probe. A new pod only receives traffic once it reports ready, and old pods are kept until the new ones are, so if the new version is broken and never goes ready the rollout stalls and 100 percent of traffic keeps hitting the healthy old pods. Rollback is fast because each rollout keeps the previous ReplicaSets in revision history, 10 by default, so kubectl rollout undo just scales the old one back up, usually inside a minute. For riskier changes I go past a plain rolling update to a canary, send 5 percent of traffic to the new version and watch error rate and latency for 10 minutes before going further, or blue green, run both and switch at once, typically with Argo Rollouts or a service mesh for metric driven promotion and automatic rollback. The mistakes I avoid are a missing readiness probe, which serves traffic to unready pods, and a maxUnavailable set so high it drops capacity mid deploy.

What gets you hired

The model starts with a flat pod network: every pod gets its own IP and can reach any other pod directly without NAT, and a CNI plugin like Calico or Cilium actually implements that networking. On top of that, a Service provides a stable virtual IP for a set of pods, and kube-proxy programs the data path, traditionally with iptables or IPVS rules, or eBPF in newer CNIs, to route the service IP to a healthy backing pod and load-balance. Service discovery is by name: CoreDNS resolves a Service DNS name to its ClusterIP, so applications connect to a name, not an ephemeral pod IP. For external HTTP traffic, an Ingress controller sits at the edge and does L7 routing by host and path plus TLS termination, so many services share one entry point. And for security, NetworkPolicies segment traffic, the important default being that without any policy all pods can talk to each other, so I add default-deny policies and then explicitly allow what is needed. When connectivity breaks, I work down this stack: DNS resolving, the Service has endpoints, the pods are ready, and no NetworkPolicy is blocking.

What gets you hired

There are 3, and the key is that they work at different layers. The Horizontal Pod Autoscaler scales out, adding or removing pod replicas against CPU, memory, or custom and external metrics, and a target like 70 percent CPU is a common starting point. It is the usual answer to load. The Vertical Pod Autoscaler scales up, adjusting a pod's requests and limits toward real usage, so a pod asking for 1 CPU but using 200 millicores gets right sized. You generally do not point VPA and HPA at the same metric on the same workload, because they fight each other. The Cluster Autoscaler scales the cluster itself, adding nodes when pods cannot schedule and removing nodes that sit underused for about 10 minutes, with Karpenter as the popular modern alternative that provisions right sized nodes in well under a minute. The part interviewers care about is how they compose. HPA can decide it needs 20 replicas, but if no node has room those extra pods sit in Pending until the node autoscaler brings capacity, so pod scaling without node scaling just changes where you queue. A complete story is usually HPA for load, a node autoscaler for capacity, and VPA or carefully set requests for right sizing.

What gets you hired

I always start the same way, kubectl describe pod and read the last 10 or so events, because the events section almost always states the reason directly. Then I read the status. Pending means it could not be scheduled, so I look for insufficient CPU or memory against the requests, and I have seen a pod ask for 4 CPU when every node only had 2 allocatable, a nodeSelector or affinity nothing satisfies, a taint with no toleration, or a PersistentVolumeClaim that is not bound. ImagePullBackOff means it could not pull the image, so I check the image name and tag for typos, whether the registry is reachable, and whether the imagePullSecrets are present and correct, which is the usual cause on a private registry. CrashLoopBackOff means the container starts and keeps exiting, and the restart backoff doubles up to a 5 minute cap, so I stop waiting and go to the logs, including kubectl logs --previous because the current container may only live for 2 seconds. I check the exit code, and exit 137 tells me straight away it was OOMKilled, usually a memory limit set at 128Mi for a JVM that needs 512Mi. Other common causes are a missing config or secret, a failed migration, or a missing dependency. Each status narrows the cause, so I diagnose from describe and logs rather than blindly restarting.

What gets you hired

The abstraction separates the request from the implementation. A PersistentVolume is actual storage in the cluster, and a PersistentVolumeClaim is a pod request for a size and access mode, say 20Gi ReadWriteOnce, and the pod mounts the claim and does not care about the underlying disk. In practice you rarely pre-create PVs by hand. A StorageClass defines a provisioner and parameters, for example gp3 EBS on AWS, so when the PVC is created the volume is provisioned dynamically and bound automatically, usually within about 30 seconds. Access modes trip people up. ReadWriteOnce means one node mounts it read-write, and ReadOnlyMany and ReadWriteMany allow many. Almost all cloud block storage is ReadWriteOnce, so if 3 pods on different nodes need to share a volume you need a shared file system like EFS, not a plain block volume. The reclaim policy, Retain or Delete, decides whether the underlying volume and its data survive when the claim is deleted, and for a database I want Retain so that deleting one claim does not destroy the data. And StatefulSets use volumeClaimTemplates, so a 3 replica set gets 3 stable PVCs, one per replica, that follow each pod across restarts and rescheduling. That is exactly what stateful workloads need.

What gets you hired

I treat it as layers, because no single control is enough. Identity and access: RBAC with least privilege, Roles and ClusterRoles bound to specific users or groups, a dedicated ServiceAccount per workload with only the permissions it needs, and never default cluster-admin; I also turn off automounting the default service account token where a pod does not call the API. Workload hardening: enforce the restricted Pod Security standard, run as non-root, read-only root filesystem, drop Linux capabilities, no privileged containers, no host network or host PID, so a compromised container is boxed in. Network: NetworkPolicies, and because the default is that all pods can talk to each other, I start from default-deny and explicitly allow required flows, and for service-to-service auth I use a mesh with mTLS. Supply chain and secrets: scan images for vulnerabilities and sign them, enforce trusted registries and these policies with an admission controller like Gatekeeper or Kyverno, enable encryption at rest for etcd, and pull real secrets from an external manager rather than relying on base64 Secrets. The theme is assume breach, least privilege at every layer, and limit blast radius if one container is compromised.

What gets you hired

My default is actually fewer, bigger clusters, because inside one cluster, namespaces, RBAC, resource quotas, and NetworkPolicies already give strong logical isolation, and every extra cluster multiplies upgrades, monitoring, and config surface. With a release every 4 months upstream, running 20 clusters means you are almost always mid upgrade somewhere. So I want a real driver before going multi cluster. The legitimate ones are blast radius and upgrade isolation, so a bad change does not take everything down and I can roll clusters one at a time; regional proximity and disaster recovery, clusters near users and in separate regions; hard isolation for compliance or untrusted tenants where a namespace is not enough; and raw scale, since a single cluster eventually hits control plane and etcd limits, and the official guidance tops out around 5,000 nodes. Once I am running a fleet, the key is consistency. Everything goes through GitOps with Argo CD or Flux, so each cluster's config comes from git and drift is detected and reconciled automatically rather than someone kubectl applying at midnight. I add a management layer for policy and a single view. Then I solve the cross cluster concerns, service discovery and networking through a multi cluster mesh, centralized observability and identity, and policy enforced uniformly with Kyverno or Gatekeeper. So, resist multi cluster until a real driver justifies it, then run it as a GitOps managed fleet so the extra clusters never become snowflakes.

What gets you hired

A container packages an application with its dependencies and runs it as an isolated process on the host, and the crucial part is that it shares the host kernel instead of booting its own operating system. A virtual machine is different in kind. It runs a full guest OS with its own kernel on a hypervisor, so it is emulating a whole machine. That difference explains the numbers. A container starts in well under a second, often around 100 milliseconds, because there is no OS to boot, and an image might be 50 to 200 megabytes. A VM takes 30 seconds or more to boot and carries a multi-gigabyte disk image and its own kernel and services in memory. So on one host I might comfortably pack 50 containers where I would fit a handful of VMs. The trade-off is isolation. Because each VM has its own kernel, the boundary is stronger, and that matters for untrusted code or strict multi-tenancy, where a single shared kernel is 1 exploit away from being the whole blast radius. Containers trade some of that boundary for density and startup speed. In practice they compose rather than compete. Containers very often run inside VMs, which is exactly what a managed Kubernetes node is, hardware-level isolation from the VM plus container packaging and speed on top.

What gets you hired

An image is an immutable, read-only template, built from a Dockerfile, holding the application, its dependencies, and a filesystem, stored as a stack of layers. A container is a running, or stopped, instance of that image. Docker adds one thin writable layer on top of those read-only layers and runs the process. The relationship is one-to-many, like a class and its objects. If I start 10 containers from the same 200 megabyte image, that image is on disk once, and all 10 share those layers, differing only in their own writable layer and runtime state. That is a big part of why containers are cheap to run, 10 containers cost me roughly 200 megabytes of image, not 2 gigabytes. The consequence people get bitten by is that the writable layer is ephemeral. Anything the container writes to its own filesystem, a database file, an uploaded image, is gone the moment the container is removed, and containers get removed all the time. So anything that has to survive belongs in a volume or in external storage, never in the container filesystem. The clean mental model is that the image is what you ship and version, and the container is a disposable process you can throw away and recreate in seconds.

What gets you hired

A Dockerfile is a text recipe of instructions that Docker runs top to bottom to build an image reproducibly. It starts with FROM, which sets the base image, and I would pick a slim one, since a node:20-slim base is maybe 80 megabytes against roughly 1 gigabyte for the full image. RUN executes commands at build time, installing packages or compiling, and every RUN adds a layer. COPY brings files from the build context into the image. ADD does the same but also handles URLs and tar extraction, which is exactly why I default to COPY unless I need those. WORKDIR sets the working directory, ENV sets environment variables, and EXPOSE documents which port the app listens on, say 8080, but it publishes nothing, that only happens at run time with a flag like -p 8080:8080. Then CMD or ENTRYPOINT define the process that runs when the container starts. The thing I always keep in mind is that most instructions create a cached layer, so order drives build speed and image size. If I copy my package manifest and install dependencies before copying the rest of the source, a code change reuses the dependency layer and the rebuild drops from 2 minutes to about 10 seconds.

What gets you hired

Docker caches the result of each instruction as a layer, and on a rebuild it reuses a layer if that instruction and its inputs have not changed. The key behavior is that as soon as one layer changes, every layer after it is invalidated and rebuilt, because layers stack on each other. So instruction order is a real performance lever. The classic optimization is to copy the dependency manifest and install dependencies before copying the application source. Dependencies change rarely, source changes constantly. On a Node service I worked on, copying package.json and package-lock.json and running npm ci first, then copying the rest of the code, meant a normal code edit only rebuilt from the source-copy layer onward and reused the cached dependency layer, which was the slow part. That took our image build from about 4 minutes down to under 30 seconds on a code-only change. If instead I copy everything and then install, all 900-odd packages get reinstalled on every single commit. I also add a .dockerignore to keep node_modules, .git, and local artifacts out of the build context, which keeps COPY layers from being invalidated unnecessarily and shrinks the context sent to the daemon. On that same project it cut the context from roughly 300 MB to about 2 MB, so even the upload before the build got faster.

What gets you hired

They serve different roles. ENTRYPOINT defines the executable that always runs and is not replaced by the arguments you pass to docker run. CMD provides the default arguments, or a default command, and is replaced by anything you pass on the run command line. The common pattern combines them: ENTRYPOINT is the program, CMD is its default flags, so the container always runs my binary but a user can override the arguments easily, and if I use only CMD, the whole command is overridable. Just as important is exec versus shell form. Exec form, the JSON-array syntax, runs the binary directly as PID 1, so it receives signals like SIGTERM and can shut down gracefully. Shell form runs the command through /bin/sh -c, which becomes PID 1 and can swallow the signal, so your app may not get SIGTERM and gets hard-killed after the timeout. So I default to exec form and use ENTRYPOINT plus CMD together for clean, overridable, signal-aware containers.

What gets you hired

By default a container attaches to a bridge network and gets its own internal IP, usually somewhere in 172.17.0.0/16. It can make outbound connections, but it is not reachable from the host until I publish a port with docker run -p 8080:3000, which maps host port 8080 to container port 3000. EXPOSE by itself only documents the port, it does not publish it. The part people miss is container-to-container communication. On the default bridge, containers can reach each other by IP but not by name. On a user-defined bridge network, Docker runs an embedded DNS server at 127.0.0.11 inside each container, so they can resolve each other by container name. That is why the right way to wire up a multi-container app is to put them on a shared user-defined network, or just use Docker Compose, which creates one automatically. There are also other modes. Host networking means the container shares the host network stack directly with no isolation and no port mapping, so a service listening on 3000 is just on the host's 3000. None disables networking entirely. Overlay networks connect containers across multiple hosts in a swarm. In my experience 9 times out of 10 a local connectivity problem is either a missing -p or relying on the default bridge instead of a user-defined network for name resolution.

What gets you hired

The starting point is that the container filesystem is ephemeral. Whatever is written to the writable layer disappears when the container is removed, so any data that must survive needs external storage. I learned that the hard way once when a docker compose down wiped about 3 weeks of local Postgres data. There are two main options. A named volume is storage managed by Docker, living under /var/lib/docker/volumes on the host, outside the container lifecycle. It is portable, not tied to a host path, and is the recommended way to persist real application data like a database, because I can stop, remove, and recreate the container 100 times without losing it. A bind mount maps a specific directory on the host into the container. It is perfect for local development, since I can edit source on the host and see it live inside the container in under a second with no rebuild, but it is coupled to the host path and filesystem layout, so it is less portable. There is also tmpfs, which keeps data in memory only and never touches disk, useful for sensitive scratch data. My rule of thumb is named volumes for persistent application and production data, and bind mounts for development workflows where I want my local files live in the container.

What gets you hired

A multi-stage build uses more than one FROM in a single Dockerfile. An early stage is the builder, it has the compilers, dev dependencies, and full source, and it produces the artifact, a compiled binary or bundled assets. Then the final stage starts from a slim or distroless base and copies only that artifact out of the builder, leaving everything else behind. The final image is where it pays off. Without multi-stage, a Node build image can easily sit at 1.1GB carrying the whole toolchain, package manager, and source, all dead weight and attack surface. With multi-stage I have taken the same app down to roughly 80MB on distroless, because the runtime image holds just the app and its runtime, with no compiler and no shell for an intruder to pivot from. The knock-on benefits are real. A 12 times smaller image pulls and starts faster, so a scale-up that took 40 seconds to pull now takes a few, it costs less to store across every tag you keep, and it has far fewer packages, so a scan that used to report 200 plus CVEs drops to a handful you can actually patch. I also name the stages so CI can target the builder stage to run tests before the final image is assembled.

What gets you hired

I attack it on several fronts because bloat usually has several causes. First the base image, which is often the biggest single piece: I move from a full OS base to a slim, alpine, or distroless image. Second, multi-stage build, so compilers, dev dependencies, and source stay in the builder stage and only the final artifact lands in a slim runtime image, that alone often cuts the most. Third, layer hygiene: I combine related RUN commands and clean up within the same layer, removing package-manager caches and temp files in the same RUN that created them, because deleting files in a later layer does not reclaim the space already baked into an earlier one. Fourth, keeping junk out: a .dockerignore so .git, node_modules, tests, and local build output never enter the context or image, and installing only production dependencies, not dev ones. I would also inspect the layers with docker history or a tool like dive to see exactly which layer is fat and target it. Usually the combination of a slim base plus multi-stage takes a 1.5GB image down to a fraction.

What gets you hired

Docker Compose lets me define a whole multi-container application declaratively in one YAML file, the services and their images or builds, the networks, volumes, and the dependencies between them, then bring it all up or down with a single command. It is excellent for local development and testing. On my last project one compose file stood up 5 services, the app, Postgres, Redis, a worker, and RabbitMQ, on a shared network so they resolve each other by service name, and a new joiner went from a fresh laptop to a running stack in about 10 minutes instead of a day of setup. Where it stops is production at scale. Compose is single-host by default, so everything lives or dies with that 1 machine, and it has no real scheduling across nodes, no cross-node self-healing, no rolling updates or rollbacks, and no autoscaling. For production I move to Kubernetes or a managed container service, which give me exactly those things, placing containers across nodes, restarting and rescheduling on failure, doing controlled rollouts, and scaling on demand. The clean way to put it is that Compose describes and runs a stack on one machine, mostly for development, and Kubernetes runs it resiliently across many machines in production.

What gets you hired

First I keep in mind that a container lives only as long as its main process: when that process exits or crashes, the container stops, there is nothing else holding it open. So I look at what that process did. docker logs shows its stdout and stderr, which usually contains the error or stack trace, and docker ps -a with docker inspect gives me the exit code, which is often diagnostic on its own, for example 137 means it was SIGKILLed, typically an out-of-memory kill. If it dies too fast to inspect, I override the entrypoint to drop into a shell, docker run -it --entrypoint sh on the image, and run things by hand to see what fails, or exec in if it stays up for a moment. Then I check the usual suspects: a command that simply runs and finishes instead of staying in the foreground, a missing environment variable or config file, a wrong path or permission, a startup crash, or the main process being backgrounded so PID 1 exits and takes the container with it. The principle is to diagnose from logs, exit code, and an interactive shell rather than blindly restarting.

What gets you hired

A container is not magic and not a VM, it is a normal Linux process with kernel features applied to it. Three primitives do the work. Namespaces give isolation, and there are 6 or 7 of them in common use, a PID namespace so the process sees its own process tree with itself as PID 1, a network namespace for its own interfaces and IP, plus mount, UTS for hostname, IPC, and user namespaces. cgroups, control groups, give resource control, they limit and account for CPU, memory, and I/O so one container cannot exhaust the host or starve its neighbours, and that is what enforces a 512Mi memory limit and triggers the OOM kill you see as exit code 137. The third piece is the union or overlay filesystem, image layers are stacked read-only and the container gets a thin writable layer on top with copy-on-write, which is how layering and the ephemeral writable layer actually work. Put it together and a container is just a process the kernel has isolated with namespaces and constrained with cgroups on a layered filesystem. Because it shares the host kernel instead of booting its own, it starts in tens of milliseconds where a VM takes 30 seconds or more, and that same fact means a Linux container fundamentally needs a Linux kernel underneath.

What gets you hired

I harden at both build time and run time. At build time, first, do not run as root, which is the default, meaning UID 0 inside the container. I create a dedicated non-root user, say UID 10001, and switch to it with USER, so a container compromise does not immediately hand over root. Second, keep secrets out of the image entirely. A secret put in an ARG, an ENV, or a copied file stays in the image history even if a later layer deletes it, and anyone who pulls the image can recover it with 1 docker history command, so I use BuildKit build secrets with --mount=type=secret for build-time credentials and inject runtime secrets through the platform, never baked in. Third, a minimal base, slim or distroless, which can take an image from around 1 GB down to under 100 MB and cuts the attack surface with it, pinned by digest for reproducibility, plus image scanning in CI to catch known CVEs. At run time I add the OS-level controls: a read-only root filesystem so the container cannot be modified, dropping Linux capabilities with --cap-drop ALL and adding back only the 1 or 2 that are genuinely needed, setting no-new-privileges to block privilege escalation, and never using --privileged unless there is a hard requirement. The mindset is least privilege end to end, and assume the container could be compromised, so limit what that buys an attacker.

What gets you hired

The key distinction is mutable tags versus immutable digests. A tag, including :latest, is just a movable pointer: it can be repointed to a different image whenever someone pushes, so pulling the same tag at two times can give you two different images, which is exactly how you get works-in-staging-broken-in-prod or non-reproducible deploys. An image digest, the sha256 content hash, is immutable and identifies the exact bytes, so pinning by digest guarantees you run precisely what you tested. The practice that follows is build once, promote the same artifact: I build the image a single time, tag it with something immutable and meaningful like a git SHA or version, and then promote that exact image, ideally by digest, through dev, staging, and prod, rather than rebuilding in each environment, because a rebuild can pull different base layers or dependencies and produce a subtly different image. :latest is fine as a human convenience pointer but should never be what production pins to. I also sign images so each environment can verify provenance before running. So: immutable tags or digests for what actually runs, promote one artifact across environments, and treat :latest as a label, not a contract.

What gets you hired

Helm is the package manager for Kubernetes, and it does 3 jobs. First, packaging. A real app is not 1 manifest, it is a Deployment, a Service, a ConfigMap, an Ingress, an HPA, a ServiceAccount, easily 8 or 10 YAML files, and Helm bundles them into a single versioned package called a chart, so I install the whole app as one unit. Second, templating. The chart is parameterized with values, so instead of copying the whole folder and hand-editing it per environment, I keep 1 chart and supply different values files, 2 replicas in dev and 6 in prod, different hostnames, different resource limits. Third, the release lifecycle. Helm installs a chart as a named release and keeps revision history, so an upgrade becomes revision 5 and a bad deploy is one helm rollback away, back on the previous revision in well under a minute. Plain kubectl apply gives me none of that history. And because charts are published to repositories or OCI registries, standing up something like an ingress controller or a Postgres becomes 1 command instead of authoring dozens of manifests by hand. So it is packaging, templating, and lifecycle together, which is a lot more than just applying YAML.

What gets you hired

A chart is the package. It is the templates, the default values.yaml, and the metadata in Chart.yaml, and on its own it is a blueprint that is running nothing. A release is one specific installation of a chart into a cluster, identified by a name, and it carries a revision history. That distinction is the one people blur, and it matters. From 1 chart I can create 3 releases, say a Postgres chart installed as billing-db, auth-db, and reporting-db, each with its own values and its own lifecycle, completely independent of each other. Every upgrade creates a new revision of that release, so after 4 upgrades I am on revision 5, and that history is exactly what lets me run helm rollback and be back on revision 4 in seconds. A repository is where charts are stored and distributed. That is the traditional Helm chart repo, an index over packaged charts, or increasingly an OCI registry, the same kind of registry that holds container images. So the flow is straightforward. I add a repository, pull version 1.2.3 of a chart from it, install that chart as a named release, and from then on I manage that release through its revisions rather than touching the chart itself.

What gets you hired

A chart ships a values.yaml that holds the default configuration, things like image tag, replica count, and resource limits. On our charts that default block was small, maybe 30 lines, but it drove everything. The files in the templates directory are Go templates that reference those values as {{ .Values.something }}, and at install or upgrade time Helm renders the templates against the values to produce the final Kubernetes manifests it applies. The power is in overriding without touching the chart. I can pass a custom values file with -f my-values.yaml, or set individual keys inline with --set replicaCount=6, which I mostly reserve for one-off or CI cases like pinning an image tag from the build. Precedence is layered and predictable, and later sources override earlier ones: the chart defaults first, then each -f file in the order given, then --set on top. So if values.yaml says 2 replicas, values-prod.yaml says 6, and I pass --set replicaCount=10, I get 10. That layering is exactly how one chart served all 3 of our environments, a base values.yaml plus a values-prod.yaml override, rather than maintaining separate copies of the manifests. When I am unsure what will actually land, I run helm template or helm upgrade with --dry-run and read the rendered output before it touches the cluster.

What gets you hired

A release moves through versioned revisions. helm install creates a named release at revision 1: Helm renders the chart with my values and applies the resulting manifests. helm upgrade applies a change, a new chart version, new values, or both, and creates a new revision while keeping the prior revisions in history, which I can see with helm history. Because of that history, helm rollback is a first-class operation: I can revert the release to a previous revision in one command, which makes a bad deploy quick to undo. helm uninstall removes the release and its resources. An important Helm 3 detail is where this state lives: Helm 3 stores each release, the rendered manifests and metadata for every revision, as a Secret in the release namespace. There is no Tiller anymore, which was the privileged in-cluster component in Helm 2 that was removed for security. So the release Secret is the source of truth that makes upgrade and rollback work.

What gets you hired

Helm templates are Go templates extended with the Sprig function library, which gives me roughly 200 functions on top of simple substitution: conditionals, loops with range, and helpers for strings, defaults, and encoding. Inside a template I have several built-in objects. .Values is the merged configuration. .Release gives me the release name and namespace, which is great for naming resources uniquely per release. .Chart exposes the metadata like name and version. .Capabilities tells me about the cluster and available API versions, so a chart can adapt, for example picking the right autoscaling API on a 1.23 cluster versus a newer one. For reuse, _helpers.tpl holds named templates defined with define and pulled in with include, which is how a chart avoids repeating the same 6 line label block across every manifest. Two practical details. Whitespace control with the dash syntax, {{- and -}}, keeps the rendered YAML valid, because a single stray space of indentation breaks the parse. And helper functions like default and required make templates robust, where required fails the render with a clear message if a needed value is missing. Before anything reaches a cluster I run helm template or helm install --dry-run and read the rendered output, which catches most mistakes in about 10 seconds.

What gets you hired

A chart is a directory with a defined layout, and there are really 4 things at the top level. Chart.yaml holds the metadata: the chart name, the chart version, the appVersion, and the dependencies. The distinction between version and appVersion matters. version is the version of the chart itself, so a pure templating fix might take it from 1.4.2 to 1.4.3, while appVersion is the version of the application it deploys, and the 2 move independently. values.yaml is the default configuration that the templates consume and that users override. The templates directory contains the Go-templated Kubernetes manifests, plus a couple of special files: _helpers.tpl for reusable named templates, and NOTES.txt, which is itself templated and printed to the user after install, handy for usage hints. The charts directory holds dependency subcharts, either vendored or pulled in by helm dependency update. Then there are supporting files: values.schema.json to validate supplied values against a JSON schema, and .helmignore to keep junk out of the packaged chart. Knowing this layout is what lets me open an unfamiliar 40 file chart and find what I need in a couple of minutes, and author a clean one myself.

What gets you hired

I keep one chart and vary only the values. There is a base values.yaml with the shared defaults, and then a small values file per environment, values-dev.yaml, values-staging.yaml, values-prod.yaml, containing only the differences. In practice those are tiny, maybe 15 lines each, covering things like replica counts, so dev runs 1 replica while prod runs 6, plus resource sizes and hostnames. At deploy time I layer them: helm upgrade with -f values.yaml -f values-prod.yaml. Later files override earlier ones, so the environment file wins over the base. For the truly dynamic or one-off values, the image tag from CI for instance, I use --set, which overrides the files and is convenient in a pipeline. The thing I deliberately avoid is forking the chart per environment, because 3 copies inevitably drift and every fix has to be applied in 3 places, and one of them always gets missed. With this layout the chart is the single source of truth for structure, and the per-environment values stay small, reviewable in a 2 minute pull request, and the only thing that differs.

What gets you hired

Dependencies are declared in Chart.yaml under dependencies, each with a name, a version constraint such as 12.1.x, and a repository, and helm dependency update resolves and vendors them into the charts directory. The common pattern is an umbrella, or parent, chart that aggregates several subcharts so a whole stack, the app plus its database and cache, installs and upgrades as 1 release instead of 3. The part people get wrong is how configuration flows. To configure a subchart from the parent, you nest its values under a key matching the subchart name in the parent values.yaml, so the parent owns the whole stack config. For values that many subcharts need, like a shared image registry or the environment name, you put them under .Values.global, which Helm exposes to every subchart, so 1 line sets the registry for all of them. And there are controls. A condition or tags field enables or disables a subchart, so the same umbrella can turn the bundled Postgres on in dev and off in prod where a managed database is used, and alias lets you include the same chart twice under different names, for example 2 Redis instances, one for cache and one for queues. Dependencies plus globals plus conditions are what make an umbrella chart clean rather than a mess.

What gets you hired

On a helm upgrade, Helm first renders the chart with the new values or chart version to get the desired manifests for the new revision. Then, importantly, Helm 3 does a three-way strategic merge to figure out what to change. It considers three states: the old manifest, which is what Helm last applied and reads from the release Secret, the new manifest it just rendered, and the live state of the objects in the cluster. Merging all three means it computes a correct patch even when something was changed out of band, it can tell the difference between a field you changed in the chart and a field something else modified live, which the older two-way approach in Helm 2 got wrong. It then applies the computed changes and records a new revision in the release Secret, keeping the prior ones for rollback. I usually pair this with --atomic, so a failed upgrade automatically rolls back instead of leaving a half-applied release, and --wait, so the upgrade only succeeds once the new resources are actually ready.

What gets you hired

Hooks let me run a resource at a specific point in the release lifecycle instead of as part of the normal manifest set. I annotate a resource with helm.sh/hook and a value like pre-install, post-install, pre-upgrade, post-upgrade, or pre-delete. The canonical use is a database migration. I define a Kubernetes Job that runs the migration and annotate it as a pre-upgrade and pre-install hook, so Helm runs that Job and waits for it to complete before rolling out the new application version, which guarantees the schema is ready first. Helm waits up to its default 5 minute timeout, so if a migration realistically takes 10 minutes I raise the timeout rather than let the release fail. For control, helm.sh/hook-weight orders multiple hooks and lower runs first, so a weight of -5 goes ahead of 0, and helm.sh/hook-delete-policy decides whether the Job is cleaned up after success, otherwise you look up in 6 months and find 300 completed migration Jobs in the namespace. The important caveat is that hooks live outside normal release reconciliation. They are not part of the tracked manifest set, so they are not rolled back automatically and Helm does not manage their drift. A failing hook fails the release, so I make migrations idempotent and handle cleanup deliberately.

What gets you hired

I separate templating problems from runtime problems. First I render and inspect, helm template, or helm install and upgrade with --dry-run --debug, shows me exactly the manifests Helm would apply, which catches templating mistakes, missing values, or malformed YAML before they touch the cluster. Then for a live release I use helm status for its state, helm get manifest and helm get values to see what was actually applied and the final merged values, and 9 times out of 10 the bug is a value I did not expect winning precedence, and helm history to see the revisions. Once I know the rendered resources I drop to kubectl describe, logs, and events on them, because the real failure, a bad image, a crashing pod, a missing secret, is at the Kubernetes level and Helm is just surfacing it. The specific Helm failure mode to know is a release stuck in pending-upgrade or pending-install, which happens when an operation is interrupted, for example CI is cancelled partway through, and I recover it with helm rollback to the last good revision, so if revision 8 is stuck I roll back to 7. Going forward I run upgrades with --atomic and an explicit --timeout, since the default is 5 minutes, so a failed upgrade auto rolls back instead of leaving the release wedged.

What gets you hired

They solve overlapping problems differently. Helm is templating plus packaging plus lifecycle. It parameterizes manifests through a templating language, packages an application as a versioned chart you can publish and share, and manages installations as releases, so a bad upgrade is 1 helm rollback away and, by default, Helm keeps the last 10 revisions to roll back to. That makes it the natural choice for distributing or consuming reusable, configurable applications, an off-the-shelf ingress controller or database, where first-class release and rollback matter. Kustomize is template-free. You keep plain Kubernetes manifests as a base and layer environment-specific overlays and patches on top, with no templating DSL, and it has been built straight into kubectl since 1.14 with the -k flag. That shines when you own your manifests and want simple, transparent, reviewable variation across 3 or 4 environments without learning a template language or fighting whitespace. So I choose Helm when packaging and sharing reusable apps with lifecycle management matters, and Kustomize when I manage my own manifests and value simplicity and transparency. And they are not mutually exclusive. You can render a Helm chart and apply Kustomize patches over the output, and GitOps tools like Argo CD support both natively, so I pick based on the use case rather than treating it as a rivalry.

What gets you hired

The shift is from imperative to declarative. Rather than a pipeline running helm install or helm upgrade against the cluster, in GitOps I declare the desired state in git, which chart, which version, and the values, and a controller continuously reconciles the cluster to match, Argo CD polling roughly every 3 minutes by default. Argo CD and Flux both consume Helm charts this way: I point an Application at a chart reference and a values source in git, the tool renders the chart, effectively helm template, and applies the result, and I get drift detection and auto-sync, so if someone edits the cluster by hand it is corrected back to git within a couple of minutes. For reproducibility I pin the chart version and the image tags in git, never a floating latest, so every deploy is auditable and rollback is just reverting 1 commit. I promote across environments by changing values in git through pull requests, which gives review and a clear history. The one thing I am careful about is secrets. I never commit plaintext secrets in values files. Instead I use SOPS-encrypted values, sealed secrets, or an external secrets operator that pulls from a real secrets manager, so git only ever holds encrypted or referenced secrets. So Helm provides the packaging and templating, and GitOps provides the declarative, audited delivery around it.

What gets you hired

A few things, starting with the big Helm 3 change: Tiller is gone. Helm 2 had Tiller, a privileged in-cluster component that was a real security liability; Helm 3 removed it, so Helm now acts with the user own kubeconfig and respects Kubernetes RBAC. That means I control who can deploy what and where through normal RBAC and namespace scoping, rather than a shared privileged service. For trust in the charts themselves, I use provenance: helm package --sign produces a signed .prov file and helm install --verify checks it, and I vet third-party charts instead of installing arbitrary ones blindly, since a chart can create anything. For distribution and reproducibility, I host internal charts in an OCI registry, pin both chart versions and image versions, ideally by digest, and scan the rendered manifests and referenced images in CI. And for hygiene at scale: keep secrets out of values entirely using SOPS or an external secrets operator, enforce a values.schema.json so misconfigured values fail fast, and back it with admission policy like OPA Gatekeeper or Kyverno so whatever a chart renders still has to meet org standards, non-root, resource limits, approved registries. The theme is treat charts as supply-chain artifacts: signed, pinned, scanned, RBAC-scoped, and policy-checked.

What gets you hired

GitOps is an operating model where git is the single source of truth for the desired state of your infrastructure and applications, and a controller running in the cluster continuously reconciles the actual state to match what is in git. Argo CD, for example, re-syncs on a 3 minute interval by default, so it is not a one-shot deploy. That is the real distinction from ordinary deploy-from-git: reconciliation is continuous, and the cluster is always being driven toward the declared state, not just at deploy time. The model gives you several things at once. A complete audit trail, because every change is a reviewed commit with an author, a timestamp, and history. Trivial rollback, because reverting is just reverting a commit and letting the controller reconcile, which on my last team turned a bad release into a 2 minute recovery instead of a scramble for the right helm command. Consistency, because the same declarative definitions produce the same environments. And drift correction, because if someone changes the cluster by hand, the controller notices within that sync window and pulls it back to what git says. It fits best with Kubernetes and declarative infrastructure, where I want every change versioned, reviewed, and applied automatically rather than someone running kubectl straight against production.

What gets you hired

In the push model, a CI pipeline outside the cluster holds cluster credentials and runs kubectl or helm to push manifests in. In the pull model, an agent running inside the cluster watches the git repository and pulls changes in, applying them itself. GitOps prefers pull for two reasons. First, security. With pull, no external system holds cluster credentials, so a leaked CI token cannot be used to deploy to or tamper with the cluster. When we moved 12 services off push pipelines, we deleted the long-lived kubeconfig secret from CI entirely, and the credentials stayed inside the trust boundary. Second, correctness. Because the in-cluster agent reconciles continuously, on a default 3 minute loop with Argo CD, it does not just apply once at deploy time. It keeps watching and corrects drift if someone changes the cluster out of band, which a push pipeline never sees, because it fires once and is done. I have watched a hand-edited replica count get reverted automatically inside a couple of minutes. So pull gives you a smaller attack surface and a self-healing, always-converging system, which is exactly what the GitOps model is built around.

What gets you hired

There are 4, often called the OpenGitOps principles. Declarative: the whole desired state of the system is expressed declaratively, so you describe what you want, not a sequence of imperative steps. Versioned and immutable: that desired state lives in git, so it is versioned, auditable, and immutable, and every change is a commit you can review and trace back to an author. Pulled automatically: software agents automatically pull the declared state from git, rather than a human running an apply. And continuously reconciled: those agents keep observing the actual state and work to make it match the declared state, correcting drift whenever the two diverge. Argo CD does that on a 3 minute sync loop by default, and Flux is similar, so drift typically gets healed within minutes rather than sitting there until someone notices. That fourth principle is what really separates GitOps from just storing manifests in git. The system is always converging on what git says, not only at the moment of a deploy. And the first two are what buy you the operational wins people care about, a full audit trail across every commit and a rollback that is one git revert away.

What gets you hired

Both Argo CD and Flux are CNCF graduated GitOps controllers that implement the pull-based, reconcile-from-git model for Kubernetes, and either is a fine choice. The difference is in flavor. Argo CD is application-centric and known for its strong web UI. I define Application resources that point at a repo path and a target cluster, and the UI shows sync status, the diff between git and live, and resource health, which teams love for visibility. It also has patterns like app-of-apps and ApplicationSets, which is how one Argo instance can drive 200 applications across several clusters without hand-writing each one. Flux is more of a composable GitOps toolkit, a set of controllers, source, kustomize, helm, and image-automation, that I assemble. It is more CRD-driven and modular, it has a lighter footprint, and its image automation is particularly strong, so it can watch a registry and open a commit bumping a tag on its own. So in practice I reach for Argo CD when a team wants the dashboard and clear visualization, and Flux when I want a lean, GitOps-native toolkit that composes well and automates image updates. Both reconcile continuously, Argo defaults to polling git about every 3 minutes on top of webhooks, and both support Kustomize and Helm.