Chapter 1: The Reality of Production Issues

As a software engineer in a tech firm that promotes an impressive 360-degree feedback system, I find the process both enlightening and daunting. Every six months, employees are encouraged to gather anonymous reviews from colleagues they've collaborated with. Personally, I appreciate the frankness of the feedback, as it fosters an environment of honesty.

The initial review was particularly challenging for me. I struggled to provide constructive feedback to my peers, and when I received my own review, I was taken aback by the candidness of their comments. They pointed out my weaknesses, which I later recognized as strikingly accurate. Initially, I thought their remarks were harsh. However, the culture here encourages transparency, which relieved me from the pressure of overly sugar-coating my responses.

During one of these reviews, I received a pivotal piece of advice: I tend to be overly cautious with my code changes out of fear of causing issues in production, despite the high standard of my work.

While I acknowledge this observation, I want to provide some context. We operate on a CI/CD model, meaning that any modifications I make to the main branch of our Git repository are immediately deployed to production after passing various unit tests and End-to-End Integration tests. This is a daunting prospect for two significant reasons:

1. Our codebase is vast, and it's impossible to know every corner of it to prevent regression errors.
2. If a commit fails during deployment due to a test case, it can block others from deploying their changes, necessitating a revert and communication with the entire development team.

This second point is a substantial drawback, but there’s a silver lining: our company is in the process of transitioning from a Monolithic to a Microservices Architecture, which I find incredibly exciting.

But I digress. Recently, I received this feedback just as I was preparing to merge changes to the Master branch. I have a personal rule against merging code on Fridays to avoid burdening the on-call team and to prevent worrying about potential bugs over the weekend. However, this particular Friday evening was an exception since I needed to submit my changes for User Acceptance Testing (UAT) by Monday.

The following morning, I woke up at 4:45 AM to check my company's Slack channel before catching an 8-hour flight. To my dismay, I noticed that the on-call team was paged.

Oh no. The timing couldn’t have been worse. With my flight approaching, I quickly informed a teammate that my changes might be the cause. I texted him after passing through security, asking him to review my last commit against the logs. Thankfully, no one else had deployed changes since my last commit, allowing him to swiftly revert and resolve the issue.

This was a nerve-wracking experience, and the anxiety of merging into the main branch is all too real. It’s time I reconcile with this fear.

Having now tarnished my self-imposed "I haven’t broken production yet" reputation—an image that was more in my mind than reality—I'm sitting on the plane pondering why I struggle with this fear. Here’s my analysis:

Cause: I’m uncertain about the protocol that follows discovering a production issue.

Action: I need to take a moment to understand what steps to take.

Cause: Our company has a procedure for documenting outages and discussing them. I often avoid this due to my reluctance to engage.

Action: I need to stop procrastinating and accept responsibility if an issue arises.

Cause: My tendency to rigorously test and retest before merging.

Action: I should improve my testing efficiency and enhance my coding practices.

And that’s a wrap. This entry is just a glimpse into my Tech Journal. After this experience, I hope to soon conquer my irrational fear of pushing changes to production, as it’s time for that to change!

Have you ever faced a production issue? What was your most challenging experience?

I strive to create content that encourages thought, learning, or motivation. If you enjoy my work, consider following my journey!

Chapter 2: Insights from the Field

In "Sharing SECRETS! You won't believe how detailed they are! Riviera staff teach me about production," the speaker delves into the intricacies of production processes and the valuable lessons learned from industry veterans.

In "I produced music for 1,000+ hours. This is what I learned…," the creator shares key insights gained from extensive experience, drawing parallels to challenges faced in tech and production.