Not “secret knowledge.” More like scars, pattern recognition, and things you only really believe after seeing them fail in production.
Hard-Won Engineering Truths
The bug is rarely where the stack trace points
It is often in an assumption, contract, deployment, config, clock, encoding, cache, or data shape.
Most production outages are boring
They come from bad defaults, missing rollback paths, expired certificates, untested migrations, overloaded queues, DNS, or someone changing “just a config.” The CrowdStrike outage on July 19, 2024 is now the textbook example — a content configuration update for Windows sensors caused a logic error that crashed millions of machines worldwide. Not a cyberattack. A config update. About 99% of affected Windows sensors were back online by July 29.
A rollback plan that has never been tested is a wish
Backups are irrelevant unless restores are tested
Many teams discover during an incident that they have backups but no usable recovery process.
“Temporary” code outlives company strategy
Migration scripts, one-off cron jobs, Excel exports, and shell scripts become critical infrastructure.
The database is usually the real application
Code comes and goes; schemas, dirty data, indexes, constraints, and migration history stay.
You can add microservices and still have a monolith
A distributed monolith is worse: same coupling, more network, more failure modes.
Caching is easy until correctness matters
Then you meet stale reads, dogpiles, invalidation bugs, TTL surprises, and “why did customer A see customer B’s data?”
Exactly-once delivery is mostly a business illusion
In practice, you design for retries, idempotency, deduplication, ordering limits, and reconciliation.
Every distributed system eventually teaches you humility
Timeouts, retries, partial failures, split brain, clock drift, and backpressure are not edge cases.
The network is not reliable, fast, secure, or homogeneous
Old engineers internalize the “fallacies of distributed computing.”
Time is a hostile API
Time zones, daylight saving time, leap years, leap seconds, local calendars, and “end of month” logic break serious systems.
Unicode is not solved just because your editor shows emoji
Normalization, collation, byte length, grapheme clusters, and mixed encodings still bite systems.
Floating point and money do not mix
Someone eventually learns this through a reconciliation nightmare.
Logs are part of the product
Good structured logs, correlation IDs, audit trails, and traceability save hours during incidents.
Monitoring tells you something is wrong; observability helps you ask why
A dashboard nobody owns becomes decorative wallpaper
High test coverage does not mean high confidence
The wrong tests can preserve the wrong behavior forever.
The best test is often a tiny production rollout
Feature flags, canaries, progressive delivery, and fast rollback beat heroic “big bang” releases.
Code review is less about catching bugs and more about shared ownership
It spreads context, design judgment, and operational awareness.
Most performance problems are not language problems
They are data-structure, query, network, or allocation problems.
Indexes are magic until they are not
A single missing, unused, or bloated index can decide whether a feature works at scale.
Queues do not remove load; they move it
They also hide failure until the backlog becomes tomorrow’s outage.
Rate limits are product decisions disguised as engineering decisions
Security cannot be bolted on at the end
Authentication, authorization, secrets, logging, dependency hygiene, and abuse cases must shape the design from day one.
Things Older Engineers Have Watched Come Around Again
- Mainframes became client-server, then web apps, then SPAs, then server-rendered apps, then edge rendering. The pendulum keeps swinging.
- RPC keeps reincarnating. CORBA, DCOM, Java RMI, SOAP, REST, GraphQL, gRPC, tRPC: different syntax, same desire to pretend the network is local.
- “This will replace SQL” has been said many times. Object databases, XML databases, NoSQL, document stores, graph databases, and lakehouses all found uses; SQL stayed.
- XML was once the future of everything. Then JSON became the simpler default. Now typed schemas, protobuf, Avro, OpenAPI, and contracts reintroduced structure.
- SOA and microservices are cousins. Many “new” microservice lessons were already learned painfully in enterprise SOA.
- Serverless is powerful, but not magic. It trades server management for cold starts, platform coupling, observability complexity, limits, and pricing surprises.
- Kubernetes is becoming the new application server. The CNCF annual survey frames Kubernetes as evolving from a container orchestrator into an operating layer for production systems and AI infrastructure.
- Platform engineering is old internal tooling with a better contract. The good version reduces cognitive load; the bad version becomes a ticket-driven internal bureaucracy.
- Low-code/no-code comes in waves. Access, Lotus Notes, Dreamweaver, Visual Basic, SharePoint, Airtable, Retool, AI app builders: empowerment and shadow IT arrive together.
- Every generation rediscovers “boring technology.” The mature move is often PostgreSQL, Redis, Linux, queues, object storage, boring HTTP, and simple deployment.
Practices That Separate Senior Engineers from Merely Senior Titles
- Design for deletion. Code that can be removed cleanly was probably designed well.
- Prefer explicit contracts over tribal knowledge. APIs, schemas, ownership, SLOs, and runbooks prevent archaeology.
- Use expand-and-contract migrations. Add new schema, dual-write or backfill safely, switch reads, then remove old fields later.
- Make operations boring. Boring deploys, boring alerts, boring rollbacks, boring dependencies. Excitement belongs in product value, not release night.
- Budget for maintenance as first-class work. Dependency upgrades, certificate rotation, schema cleanup, and observability are not “nice to have.”
- Own your dependencies. “It came from npm/PyPI/Maven/Docker Hub” is not a risk assessment.
- Use feature flags, but retire them. A stale feature flag is a hidden branch in production.
- Document decisions, not just systems. Architecture Decision Records are valuable because they preserve why a trade-off was made.
- Prefer small, reversible changes. This is one of the most reliable ways to move quickly without gambling.
- Measure delivery health, not busyness. DORA now describes five software-delivery performance metrics — the original four (lead time, deployment frequency, change fail rate, recovery time) plus reliability measured through SLOs — and ties them to delivery outcomes and team well-being.
- Incident reviews should be blameless but not toothless. The point is to fix systems, incentives, and detection — not to produce a ritual document.
- The best engineers ask, “How will this fail?” before asking, “How will this work?”
- The fastest code is code you do not run. The safest dependency is the one you do not add. The easiest system to operate is the one you do not build.
- Good abstractions reduce cognitive load; bad abstractions hide reality.
- Slowing a team down for one day so it does not lose three months later — that is often the senior engineer’s real job.
Recent Items on the Senior Engineer Radar
AI coding tools: useful, but verification is now a core skill
The 2025 Stack Overflow Developer Survey found that 46% of developers actively distrust AI tool accuracy versus only 33% who trust it (a mere 3% “highly trust”). Experienced developers are among the most cautious — 20% highly distrust AI tool output. AI changes code production faster than it changes software delivery, and the gap between the two is where trouble lives.
AI complacency and shadow IT
Thoughtworks’ Technology Radar (Vol 33, November 2025) flags “AI-accelerated shadow IT” and “coding throughput as a measure of productivity” as caution items. When non-engineers can spin up AI-generated apps in minutes, the governance question is not hypothetical — it is already happening.
Prompt injection is the new SQL injection-shaped lesson
OWASP’s 2025 LLM Top 10 treats prompt injection (LLM01:2025) as the central LLM application risk — user-controlled input that can alter model behavior in unintended ways. If you have been in software long enough, the pattern feels familiar. Different input vector, same class of vulnerability.
Software supply chain security is no longer theoretical
The 2024 xz Utils backdoor (CVE-2024-3094) showed that maintainer trust, release artifacts, build systems, and social engineering are part of the attack surface. Malicious code was embedded in xz versions 5.6.0 and 5.6.1. SLSA, provenance, SBOMs, and reproducible builds are becoming normal vocabulary. OpenSSF’s SLSA framework specifically targets preventing tampering and improving artifact integrity across the supply chain.
Regulation is becoming part of software engineering
The EU Cyber Resilience Act entered into force on December 10, 2024. Reporting obligations begin September 11, 2026, and main obligations apply from December 11, 2027. If you ship software with a European user base, this affects your build pipeline, your vulnerability disclosure process, and your documentation. It is no longer optional to track your dependencies.
Post-quantum cryptography is no longer sci-fi planning
NIST finalized its first three post-quantum encryption standards in August 2024: FIPS 203 (ML-KEM), FIPS 204 (ML-DSA), and FIPS 205 (SL-DSA). NIST encouraged administrators to begin transitioning. “Harvest now, decrypt later” is not a theoretical threat — it is an architectural decision you are making right now by choosing what crypto you deploy.
Memory safety is back as a board-level security topic
CISA’s Secure by Design initiative now includes “The Case for Memory Safe Roadmaps” and guidance on memory safety in critical open source projects. The original NSA advisory on memory-safe languages (2022) started the conversation; the ongoing push from CISA and partners makes it operational. If your stack is C/C++, this is your migration planning signal.
Typed languages are having another moment
GitHub’s Octoverse 2025 reports that AI, agents, and typed languages are driving the biggest shifts in software development, with TypeScript reaching the top spot. When AI generates more code, types become the contract layer that makes that code trustworthy.
Opinions Worth Holding
- Simple beats clever almost every time.
- Boring architecture plus excellent operations beats fashionable architecture plus chaos.
- The hardest part of software is not typing code; it is understanding the problem, coordinating humans, and preserving correctness while everything changes.
- Every system is legacy the moment it reaches production.
- The best engineers are not the ones who know every new tool. They are the ones who know which problems are genuinely new and which ones are old problems wearing new branding.