Tag Archives: devops

From 4 Months to 2 Years: Scaling Systems and Stepping into Seniority at Bazaarvoice

A follow-on from the smash hit that was “My First 4 Months at Bazaarvoice.”

While my first post focused on onboarding and adapting to a global organization, this post shifts focus to developing at scale, navigating major organizational shifts, and my journey to becoming a Senior Engineer.

What’s Been Going On? (The High-Level View)

To say the past two years have been busy would be an understatement. We’ve been operating at a massive scale, balancing long-term infrastructure modernization with the daily realities of keeping the lights on across long lived systems.

Here is a snapshot of what the Platform Engineering team and I have tackled:

  • Infrastructure Modernization: Migrated over 100 Kubernetes deployments from a long standing, self-managed KOPS cluster over to multiple AutoMode EKS clusters.
  • Cloud & Cost Optimization: Improved our overall AWS posture by enforcing consistent tagging standards, reducing waste, and redesigning infrastructure for our systems.
  • CI/CD Overhauls: Helped multiple development teams migrate from Jenkins setups to modern GitHub Actions pipelines.
  • Multi-Cloud Networking: Connected our AWS and GCP footprints via a private cross-cloud interconnect.
  • Developer Empowerment: Developed an Internal Developer Platform (IDP) to consolidate and accelerate our application development lifecycles.

Navigating the Team Merger: Cultivating a Shared Brain

Halfway through my tenure, our Platform Engineering organization underwent a major shift. Originally, we operated as two small, distinct teams: Cloud Engineering (my team, focused on our AWS footprint, cloud governance, and least-privilege access across 50+ accounts) and Developer Experience (DevEx).

Unifying into a single, cohesive team was a huge win for our operational health. Not only did it give us a far more sustainable on-call rotation, but it also allowed us to share deep system knowledge and enhance system resilience. 

When we merged into a single Platform team, we inherited 60+ new applications and services, including the long standing KOPS Kubernetes cluster and scattered CI/CD tooling.

Bridging this massive knowledge gap wasn’t going to happen overnight. To tackle it, we designed a deliberate knowledge-transfer strategy:

  • Targeted KT Sessions: Deep dives into large, complex systems led by the original maintainers.
  • Split On-Call Rotations: Pairing a DevEx member with a Cloud Engineer (Primary/Secondary) so engineers could safely shadow unfamiliar issues.
  • Leveraging Subject Matter Experts (SMEs): Recognizing that no single engineer can learn every bespoke system in a few months, we leaned on individual strengths. Everyone naturally upskills over time by collaborating with the resident experts.

Game Days: Turning Chaos into a Sandbox

While product development teams have hackathons to build new features, our platform team has benefited immensely from Game Days. These are safe, controlled sandbox environments where the team can work through real-world failure scenarios without risking production.

They have proven incredibly useful for two things: handing over newly developed systems, and reverse-engineering complex, long-standing architectures. 

Case Study: Cross-Cloud Interconnect

My teammate and I were responsible for building our Cross-Cloud Interconnect, a private connection linking our GCP and AWS VPCs. While we knew the architecture inside and out, the rest of the team were focused on other projects and lacked context.

To fix this, I led a series of Game Days. In a sandbox environment, I intentionally broke the infrastructure so other engineers could get hands-on experience troubleshooting errors like BGP session failures, overloaded capacity, and connection scaling. We also practiced modifying the Infrastructure as Code (IaC) that defines the system.

This served as the perfect final handover. It allowed us to stress-test our runbooks & documentation, fill in any missing gaps, and ensure the entire team felt confident supporting the system.

The outcome of this was that we had 100% up time across the Black Friday/Cyber Monday weekend where Cross Cloud Interconnect handled 2 Billion+ requests.

We used this same chaos engineering model to build confidence in our long standing systems rebuilding them from scratch in a sandbox to map out hidden dependencies and understand how they behaved under load.

Embedding with Development Teams

In Platform Engineering, it’s easy to become siloed in your own ecosystem and lose sight of what product teams are actually building. Over the past year, we made a conscious effort to break down those walls and embed directly with our internal “customers.”

Instead of just handing over infrastructure from afar, we partnered closely with development teams on two major initiatives:

  • Simplifying Architecture (Lambda to ECS Fargate): We collaborated with a development team to migrate a complex, multi-Lambda and Step Function setup into a streamlined AWS ECS Fargate deployment. Rather than treating this as a one-off fix, we used it as a blueprint for the future. We built reusable Terraform modules and shared GitHub Actions workflows, creating a repeatable “paved path” that any team at Bazaarvoice can now leverage to adopt Fargate instantly.
  • Standardizing the Core (The KOPS to EKS Migration): Migrating applications over to our new EKS clusters wasn’t just a technical shift; it was a chance to see the daily challenges of dev teams working with heterogeneous applications and infrastructure. Working side-by-side with various teams gave us insights into their unique tech stacks. We used these insights to drive organizational consistency, aligning disparate development workflows as we moved them into the new EKS ecosystem.

By shifting from an “isolated platform” mindset to a collaborative, high-touch model, we didn’t just ship improved  infrastructure, we built stronger relationships across the entire engineering organization.

Developing in the Era of AI

In the two years I’ve been here, AI tooling has shifted from a novelty to an absolute necessity. Tools like GitHub Copilot have evolved from simple unit-test generators into foundational workflow drivers. If you aren’t actively leveraging AI in your development workflow today, you are unintentionally slowing yourself down. 

This was majorly highlighted at Platform Con 2026 where a majority of talks and workshops were AI focused. Meaning it isn’t just a shift at our org, but across Platform Engineering as a whole. 

Recognizing this shift, I gave an internal talk sharing tips and tricks for maximizing Copilot’s utility covering prompt engineering, model selection, and the critical importance of providing rich repository context.

In the world of LLMs, things move fast. With the emergence of MCP (Model Context Protocol) servers, agentic workflows and repository-level configuration files, the landscape is shifting again, and keeping our engineering practices aligned with these advancements is a continuous focus.

Agentic Case Study: Automating IAM Audits

Historically, managing IAM permission changes meant tedious, manual reviews from the Platform team. By offloading the first pass to an AI agent, we now automatically scan every IAM PR for least-privilege violations and compliance policy drift. This cuts down manual toil for our team while giving developers instant feedback on security concerns.

The Path to Senior Engineer

Looking back at my first four months, my focus was inward: learning the communication loops (like moving “work out of DMs” into public Slack channels), understanding team work-ownership, and treating internal developers as stakeholders.

Moving into a Senior Engineer role over these past two years required shifting that focus outward. Seniority at Bazaarvoice hasn’t just been about handling larger technical footprints like EKS migrations or cloud networking; it’s been about delivering impact at an organizational level rather than a team level.

Whether it was leading Game Days to up skill my peers, building reusable project scaffolds that abstract away complexity for hundreds of developers, the goal has been the same: make the right way the easy way for every engineer at Bazaarvoice.

It’s been a fast paced, incredibly rewarding two years and I’m excited for what’s next. Want to get in on the action? Head over to our careers page

As a software engineer, how do I change my career to DevOps?

At Bazaarvoice, we’re big fans of cloud. Real big. We’re also fans of DevOps. There’s been a lot of discussion over the past several years about “What is DevOps?” Usually, this term is used to describe Systems Engineers and Site Reliability Engineers (sometimes called Infrastructure Engineers, Systems Engineers, Operations Engineers or, in the most unfortunate case, Systems Administrators, which is an entirely different job!). This is not what DevOps means, but in the context of career development, it carries the connotation of a “modern” Systems or Site Reliability Engineer.

There’s a lot of great literature about what a DevOps engineer is. I encourage you to read this interview of Google’s VP of Engineering, as well as Hixson and Beyer’s excellent paper on Systems Engineering and its distinction among Software, Systems and Site Reliability engineers. Although DevOps engineering goes beyond these technical descriptions, I’ll save that exegesis for another time. (Write me if you want to hear it, though!)

Many companies claim to hire or employ DevOps engineers. Few actually do. Bazaarvoice does. Google does, too, although they’re very hipster about it (they called it Site Reliability Engineering before the term DevOps landed on the scene, so they don’t call it DevOps because they had it before it was cool, or something). I don’t know about other companies because I haven’t worked at them (well, I haven’t worked at Google either, but they are pretty vocal about their engineering philosophies, so I’ll take them at their word). But there’s a lot of industry buzzwordium with little substance. This isn’t a jab at other companies (but really, Bazaarvoice is way cooler), it’s just a side-effect of assigning job titles based on pop culture. If you’re really a DevOps engineer, then you already know all of this, and you probably filter out a lot of this nonsense on a daily basis.

But we’re here to answer a specific question: If I’m already a software engineer, how do I become a DevOps engineer?

So, you’re a developer and you want to get in on the ops action. Boy, are you in for a surprise! This isn’t about installing Arch Linux and learning to write Perl. There’s a place for that kind of thing (a very small, dark place in a very distant corner of the universe), but that isn’t inherently what DevOps means.

Let’s consider a few of the properties and responsibilities of DevOps engineering.

A DevOps engineer:

  • Writes code / software. In fact, he is a proper software engineer.
  • Builds tools.
  • Does the painful things, as often and frequently as possible.
  • Participates in the on-call rotation (yes, for 2 a.m. production outages).
  • Infrastructure design.
  • Scaling systems (any system or subsystem — networking, applications, load balancers).
  • Maintenance. Like rebooting that frail vhost with a memory leak that no one’s bothered to fix or take ownership of.
  • Monitoring.
  • Virtualization.
  • Agile/kanban/whatever development methodology. It’s not so much that agile is “right.” It’s just the most efficient way to complete a work queue (taking into account interruptions and blockers). A good DevOps engineer has strong opinions about this!
  • Software release cycles and management. In fact, you might even see “development methodology” and software release cycles as the same thing.
  • Automation. Automation. Automation.
  • Designing a branch/release strategy for the provided SCM (git, Mercurial, svn, etc). Which you do have.
  • Metrics / reporting. Goes hand-in-hand with monitoring, although they are different.
  • Optimization / tuning. Of applications, tools, services, hardware…anything.
  • Load and performance testing and benchmarking, including performance testing of highly complex systems. And you know the difference between load testing and performance testing.
  • Cloud. Okay, you don’t really have to have cloud experience, but it can fundamentally change the way you think about complex systems. No one in a colo facility devised the notion of “immutable infrastructure.”
  • Configuration management. Or not. You have an opinion about it. (You’ve surely heard of Puppet, Chef, Ansible, etc. Yes?)
  • Security. At every layer.
  • Load balancing / proxying / replicating. (Of services, systems, components and processes.)
  • Command-line fu. A DevOps engineer is familiar with tools at his disposal for debugging, diagnosing and fixing issues on one or many servers, quickly. You know how pipes work, and you can count how many records contained some phrase in a log file with ease, for example.
  • Package management.
  • CI/CIT/CD — continuous integration, continuous integration testing, and continuous deployment. This is the closest thing to the real meaning of “DevOps” that a Systems Engineer will do.
  • Databases. All of them. SQL, NoSQL, whatever. Distributed ones, too!
  • Solid systems expertise. We’re talking about the networking stack, how hard disks work, how filesystems work, how system memory works, how CPU’s work, and how all these things come together. This is the traditional “operations” expertise you’ve heard about.

Phew! That’s a lot. Turns out, almost all of these skills are directly applicable to software engineering. The only difference is the breadth of domain, but a good software engineer will grow his breadth of domain expertise into operations naturally anyway! A DevOps engineer just starts his growth from a different side of the engineering career map.

Let’s stop and think for a moment about some things DevOps engineer is not. These details are critically important!

A DevOps engineer is not:

  • Easier than being a software engineer. (Ding! It is being a software engineer.)
  • Never writing code. I write tons of code.
  • Installing Linux and never touching your favorite OS again.
  • Working the third shift. (At least, it shouldn’t be; if it is, quit your job and come work with me.)
  • Inherently more “fun” than being a software engineer, although you may prefer it, if you’re into that kind of thing.
  • Greenfield. You’ll deal with old stuff in addition to new stuff. But as a good engineer, you care about business value and pragmatism.
  • An unsuccessful software engineer. Really: if you can’t write code, don’t expect to be a good DevOps engineer until you can.

A career shift

Here are a few things you should do to begin positioning yourself as a DevOps engineer.

  • Realize that you’re already an engineer, so becoming a DevOps engineer means you are just moving yourself to a different domain to grow and learn from a different direction.
  • Interview at a company that’s hiring DevOps. If you get hired, you’ll learn the operations side of things fast. Real fast. Or get fired. (Hint: you should disclose your experience honestly!) If you don’t get hired, you’ll learn what is still missing from your resume / experience that’s preventing you from becoming a full-time DevOps engineer. Incidentally, we’re hiring. 🙂
  • Tell your boss you want to become a DevOps engineer at your company. Your boss should help you to this end. If he/she does not, quit. Then come work at Bazaarvoice with me and a bunch of really awesome, super talented engineers working on some really awesome and challenging problems.
  • Obtain practical experience by using your skills as a software engineer to build tools rather than applications. Look at any of the open source projects Netflix has written for examples / ideas.
  • Learn OpenStack or some equivalent infrastructure-level project. (OpenStack has tremendous breadth, which is why I recommend it.) You can do this on your own time and budget. It’s not important whether OpenStack sucks compared to Rackspace Cloud. What’s important is that you understand all of the various components and why they are important. Have a wad of cash lying around? Learn Amazon Web Services or Google Compute Engine instead.
  • Bonus: learn about Apache Mesos and Kubernetes, and why they’re useful / important. Using them is less important than understanding them.
  • Participate in anything your team does involving operations — deployment, scale, etc. (See list above: “What DevOps is.”) If your team doesn’t do any of that (i.e., they send artifacts over to Operations and the Operations team does deployment), go over to the Operations team and sit in on a few deployments. You may be surprised!

Do I need to have deep operations experience to become a good devops engineer?

I’ve asked myself the same question. I come from a development background myself and only had less than a year of experience dealing with operations (part time) before becoming a DevOps engineer. (Granted, I had a referral vouching for me, which weighed in my favor.) Despite my less-than-stellar CS/algorithm skills (based on my complete lack of formal computer science education), I’ve had enough experience writing software that I could apply these concepts to systems in a similar fashion. That is, in the same way a software engineer needs to understand at what point his application code should be abstracted in support of future changes (i.e., refactored into smaller components), a DevOps engineer needs to understand at what point a component of his infrastructure should be abstracted in support of future changes (i.e., rebuilding all of his servers and rearchitecting infrastructure that’s already in production, despite the potential risk, in order to solve a problem, like allowing the architecture to scale to business needs). At its core, a good engineer is just as good whether he’s writing software or deploying it. Understanding complex systems and their interactions is a critical skill. But all of these are important for software engineers, whether you’re writing application code or not!

I hope this post helps you in your endeavor to become a DevOps engineer, or at least understand what it means to be a DevOps engineer at Bazaarvoice (as I said before, it may mean something totally different at other companies!). You should get your feet wet with some of the things we do. If it gets you tingly and excited, then come work with me at Bazaarvoice: http://www.bazaarvoice.com/careers/research-and-development/.

This article was originally posted as an answer on Quora. Due to surprising popularity, I’ve updated the article and posted it here.