DEV Community: Jesal Gadhia

How to break the cycle of firefighting and build engineering excellence

Jesal Gadhia — Thu, 02 Jan 2025 04:58:19 +0000

Exercise better foresight by identifying patterns, pinpointing what makes them persist, and understanding what could prevent it in the future.

Take a moment to count the urgent issues you handled this week. Production incidents. Customer escalations. Team conflicts. Deadline pressures. For many engineering leaders, these “firefighting” moments consume the majority of their workday, leaving little time for strategic thinking or meaningful improvements.

This constant cycle of reacting to problems is exhausting and expensive. Every hour spent handling preventable issues is an hour not spent on innovation, team development, or strategic planning. The cost compounds across your team, affecting morale, productivity, and your ability to deliver value.

But there’s a better way. By embracing systems thinking, you can transform your team from reactive to proactive, creating processes that run themselves and free you to focus on what matters most: innovation and growth.

Understanding the systems thinking iceberg

One powerful model to visualize systems thinking is the iceberg model. Just like an iceberg, where 10% is visible above water while the remaining 90% lies beneath the surface, this model reminds us that there’s much more to a situation than meets the eye. The visible problems are often symptoms of deeper, hidden issues.

Levels of thinking in the iceberg model

1. The event level

At the tip of the iceberg are the events — the immediate issues that catch our attention. These are surface-level incidents that often demand urgent responses. Examples include:

A critical system outage causes downtime.
A major customer complains due to a missed project deadline.
An unexpected defect is discovered in a recently released product.

While addressing these events is necessary, focusing solely on them keeps us in a cycle of reaction without addressing the root causes. It’s like repeatedly putting out fires without investigating what started them.

2. The pattern level

Just below the surface, patterns emerge when we observe similar events happening over time. Recognizing these patterns helps us anticipate issues. Examples of patterns might be:

Frequent system outages: noticing that system downtimes occur consistently during peak traffic hours.
Recurring customer complaints: observing that customer dissatisfaction spikes whenever a specific feature is updated.
Consistent project delays: realizing that projects often overrun their deadlines by similar timeframes.

By identifying patterns, we begin to see that events are not isolated incidents but symptoms of deeper issues that need to be addressed.

3. The structure level

Deeper still are the structures influencing the patterns we observe. Structures include processes, organizational hierarchies, tools, and policies that shape behavior. Examples of structural issues might be:

Inefficient resource allocation: teams are understaffed during critical periods, leading to burnout and mistakes.
Outdated processes: reliance on manual testing rather than automated testing increases the chance of defects slipping through.
Communication silos: departments not sharing information effectively, causing misalignment and delays.

These structures create the conditions that lead to the patterns and events we experience. By examining and modifying structures, we can influence the patterns and prevent undesirable events.

4. The mental model level

At the base of the iceberg lie the mental models — the beliefs, values, and assumptions that drive behavior within the system. Examples include:

A speed over quality mindset: believing that getting products to market quickly is more important than ensuring they are free of flaws.
Resistance to change: assuming that established processes don’t need updating because “they’ve always worked before.”
Underestimating the importance of communication: not valuing cross-department collaboration, leading to isolated teams.

These underlying mental models perpetuate the structures and patterns leading to recurring events. Changing mental models can be challenging but is essential for sustainable transformation.

Strategies for implementing systems thinking

With a clear understanding of the underlying issues, the next step is to implement strategies that foster systems thinking and build self-sustaining processes. Here are actionable steps you can take to apply these strategies in your organization.

Engage your team in analysis

Involving your team in analyzing the different levels of the iceberg model is crucial. Suppose your team is experiencing frequent project delays. Organize a retrospective meeting involving project managers, engineers, and other stakeholders to examine each level:

Events: document specific instances of delays and their immediate impacts.
Patterns: collect data on project timelines to identify common bottlenecks.
Structures: examine whether current project management tools and processes are effective.
Mental models: encourage team members to share their beliefs about deadlines, workload management, and quality standards.

By bringing together diverse perspectives, you can uncover hidden issues and foster a sense of ownership over the solutions. This collaborative approach ensures that any changes made are well-informed and more likely to be embraced by the team.

Identify leverage points

Focus on areas where small, strategic changes can have a significant impact. Leverage points often exist within processes, resource allocation, or communication channels. For example:

Implement a tiered support system: if customer complaints often arise due to slow response times from support, introduce first-line support for quick resolutions and escalate complex issues to specialized teams.
Streamline approval processes: if project delays are the product of lengthy approval procedures, simplify or automate approvals to speed up workflow.
Reallocate resources: if a team is consistently overloaded, redistribute tasks or hire additional staff to alleviate pressure and improve efficiency.

By targeting these leverage points, you can disrupt negative patterns and foster positive outcomes with relatively minimal effort.

Challenge mental models

Addressing and reshaping underlying beliefs is crucial for sustainable change. If there’s a “speed over quality” mindset leading to product defects:

Initiate a cultural shift: launch training sessions emphasizing the importance of quality and its long-term benefits. Share case studies where poor quality led to significant issues.
Promote open communication: if resistance to change stems from fear of the unknown, encourage transparency by involving team members in decision-making processes and explaining the benefits of new initiatives.
Encourage cross-functional collaboration: if departments work in silos, organize joint projects or team-building activities to break down barriers and foster a collaborative culture.

Changing mental models may require consistent effort, but it lays the foundation for lasting improvement.

Develop an action plan

An action plan turns insights gained from discussions into concrete steps you can follow to improve the situation. For optimizing cross-department collaboration:

Set clear objectives: aim to reduce project delays by 25% in the next quarter.
Define actions: implement regular inter-departmental meetings, use collaborative tools like shared project boards, and establish clear communication protocols.
Assign responsibilities: designate team leads to oversee collaboration efforts and ensure accountability.
Set timelines: roll out changes within the next month and schedule regular reviews to assess progress.

An effective action plan should be specific, measurable, achievable, relevant, and time-bound (SMART) to ensure accountability and track progress.

Monitor progress and adjust

Continuous monitoring allows you to measure the effectiveness of changes and remain flexible. After implementing new processes:

Track metrics: monitor KPIs such as customer satisfaction scores, defect rates, or project completion times. When metrics aren’t trending in the desired direction, analyze root causes and adjust your approach — perhaps by reallocating resources, modifying processes, or revising timelines.
Gather feedback: regularly solicit input from team members about what’s working and what’s not. Use this feedback to make targeted improvements, such as streamlining overcomplicated processes or providing additional support where teams are struggling.
Celebrate successes: recognize improvements and milestones to maintain momentum and motivate the team.

By staying attentive to results and being willing to adapt, you can ensure that the changes have a lasting positive impact.

Components of self-sustaining systems

Ensuring that all essential components are addressed will help build systems that are not only effective but also self-sustaining. Here are key elements to focus on:

1. Roles and responsibilities

Establishing clear organizational structure and team interactions forms the foundation of effective systems. While no longer current practice at Spotify, Henrik Kniberg’s influential 2012 model of scaling agile organizations demonstrates this through several key concepts.

The model organizes autonomous, cross-functional squads that operate like mini-startups, each owning a specific part of the product lifecycle. It scales collaboration through tribes — collections of squads working in related areas — to maintain coordination without sacrificing autonomy.

The model also builds expertise through chapters, where specialists across different squads share knowledge and practices. This structure exemplifies how clear roles and responsibilities can create self-sustaining systems that balance autonomy with collaboration.

2. Knowledge and skills

Developing a learning culture enables organizations to adapt and innovate continuously. Microsoft’s transformation under Satya Nadella, chronicled in his 2017 book “Hit Refresh,” showcases how even large enterprises can fundamentally reshape their culture:

The company focused on embedding a growth mindset where learning from failures is valued over knowing all the answers. This started with leaders modeling vulnerability by openly sharing their own failures and lessons learned. They implemented “failure retrospectives,” where teams analyzed setbacks not as problems to avoid but as valuable learning opportunities. Recognition systems were revised to reward innovative attempts and learning processes, not just successful outcomes.

They worked on breaking down silos by promoting cross-functional skill development and collaborative learning. This included rotating team members across departments, creating mixed-skill project teams, and establishing regular knowledge-sharing sessions where different departments teach each other about their work. They also implemented shared objectives that required multiple departments to collaborate, moving away from department-specific goals that encouraged isolation.

Microsoft also prioritized psychological safety which encouraged questioning, learning, and challenging the status quo. Leaders achieved this by actively soliciting diverse viewpoints in meetings, responding positively to challenges of their own ideas, and establishing “no-blame” policies for raising concerns. They created formal channels for anonymous feedback and suggestions, while also training managers to respond constructively to new ideas and concerns. Regular town halls and open forums were established where employees at all levels could question existing practices and propose alternatives without fear of negative consequences.

3. Processes and rituals

Establishing consistent operational practices ensures reliability and enables innovation. Drawing from their extensive experience operating large-scale systems, Google’s Site Reliability Engineering teams have pioneered practices that demonstrate this.

Their teams implement error budgets — a concept that gives teams a specific allowance for acceptable system failures. They define clear service level objectives (SLOs) that ensure technical decisions directly connect to customer impact.

Their teams also follow a structured post-incident analysis process that goes beyond just fixing immediate problems. Each incident triggers a formal review with a standardized template covering what happened, why it happened, its impact, and the broader systematic improvements needed. This systematic approach has helped Google build institutional memory and continuously improve their operational practices

4. Tools and equipment

Building and maintaining the right tooling infrastructure enables teams to work effectively. Netflix’s engineering organization, renowned for its open-source contributions and innovative practices, demonstrates this through several groundbreaking approaches.

Netflix pioneered chaos engineering through tools like Chaos Monkey, which randomly terminates instances in production to ensure systems can handle unexpected failures. This tool simulates real-world failures by deliberately causing outages during business hours when engineers are available to respond. This forces teams to build resilient systems that can handle failures gracefully. Netflix has built deployment pipelines that automate the process of moving code from development to production. These pipelines include automated testing, security scanning, canary deployments (rolling out changes to a small subset of users first), and one-click rollbacks if issues are detected.

For custom testing tools, Netflix developed and open-sourced several frameworks that others can learn from. The failure injection testing (FIT) framework allows teams to inject failures at the application level rather than the infrastructure level. For organizations looking to develop their own testing tools, Netflix’s engineering blog provides examples of their approach to testing distributed systems..

5. Observability and metrics

Creating comprehensive system visibility enables data-driven operations. AWS’s Well-Architected Framework, demonstrates effective observability through several key practices that have transformed how organizations monitor and respond to system behavior.

AWS implements integrated metrics, logging, and tracing that work together to provide complete system visibility. For example, when a customer experiences a slow checkout process, teams can trace the entire transaction path across different services, examine detailed logs of each component’s behavior, and correlate this with system-wide performance metrics. This three-pronged approach helps quickly identify whether the issue stems from database performance, network latency, or application code.

AWS aligns technical metrics directly with business outcomes through what they call “business-aware monitoring.” Rather than just tracking technical metrics like CPU usage or memory consumption, they monitor business-relevant metrics like order completion rate, revenue per hour, or customer engagement scores. When technical issues occur, teams can immediately see the business impact — for example, how a slight increase in API latency directly correlates to a drop in successful checkouts or customer engagement. This approach helps teams prioritize their responses based on actual business impact rather than just technical severity.

6. Performance and expectations

Implementing a performance management system fosters a culture of accountability and continuous improvement. Netflix’s widely studied high-performance culture, recently reaffirmed in their 2024 culture memo, demonstrates this.

Netflix hires experienced professionals who thrive in environments of freedom and responsibility, paying top-of-market compensation to attract and retain exceptional talent. To prevent burnout, they implement unlimited vacation policies and encourage actual usage, maintain flexible working arrangements, and explicitly discourage long hours. Instead of measuring time spent, they focus on impact delivered. Teams are kept intentionally lean, with each member bringing significant expertise and value, rather than distributing work across larger groups.

The company’s “keeper test” is a fundamental part of their performance management approach. Managers regularly ask themselves: “If this person were interviewing elsewhere, would I fight hard to keep them?” This isn’t about constant pressure, but rather about maintaining honest dialogue about performance and fit. If the answer is “no,” they provide generous severance packages rather than maintaining ongoing employment.

Netflix’s approach focuses on providing context rather than top-down control. In practice, this means leaders spend significant time sharing detailed information about company strategy, market conditions, and business challenges. For example, instead of dictating specific technical solutions, leaders might share comprehensive data about customer behavior, cost constraints, and strategic priorities, then trust their teams to make informed decisions.

Final thoughts

Picture this: Your phone buzzes with another escalation. But this time is different. Instead of jumping into firefighter mode, you see an opportunity for fireproofing.

Take that production incident, customer escalation, or team bottleneck and ask:

What patterns made this inevitable?
Which structures allow these patterns to persist?
What system could prevent this entirely?

Then build it. While others scramble from crisis to crisis, you’ll be methodically eliminating the conditions that create them.

This article was originally published on LeadDev.com on Dec 24th, 2024.

6 questions to ask yourself before applying to that job

Jesal Gadhia — Wed, 31 Jul 2024 15:00:00 +0000

The current job market is overflowing with competition, put yourself in better stead for meeting the job spec by asking yourself these 6 questions before applying.

Rejection stings. As engineers, we're wired to solve problems and achieve goals. In today’s job market, it can be tricky to get past the first stage, let alone a second or third, so a "no" can feel like a personal setback. But what if we could reframe potential rejection as a learning opportunity before it even happens?

Enter the premortem, a tool used in project management to anticipate potential failures before they occur. Applying the concept of premortems to your job search and anticipating what might cause your application to be rejected, can help you navigate the process more effectively. Here are six questions to ask yourself before applying to better align with the job spec.

1. Does my expertise match the demands of the company’s size?

Knowing the type of environment you thrive in and applying that to your search is an important step. Different company settings require different skill sets and experiences, so a mismatch in company size or stage can lead to difficulties in adapting to the pace.

For example, if you've spent the last five years at Google working on large-scale infrastructure projects, moving to a 10-person start-up might require you to wear multiple hats and work on tasks outside your current expertise. Conversely, someone used to the fast-paced, multifaceted nature of a start-up might struggle with the bureaucratic processes in a large corporation.

Action:

Create two versions of your resume: one tailored for smaller companies and another for larger organizations.
List three to five specific projects or experiences that demonstrate your adaptability to different company sizes.
Draft a paragraph for your cover letter explicitly addressing your fit for the company's size.

2. Can I effectively integrate into this team structure and working style?

Look into the team and wider org structure of the company you’re applying to and consider whether your skill set lends itself to the internal dynamics. Reflect on your collaboration style; do you prefer working as part of a specialized team or within a cross-functional one?

For example, if you're a machine learning specialist within a dedicated research team, transitioning to a product-focused team where you need to collaborate closely with UX designers, product managers, and software engineers might be challenging. You may need to adapt your communication style by simplifying technical jargon, learning to translate complex concepts for non-technical team members, and developing skills in areas like product management or user experience to contribute effectively.

Conversely, if you're accustomed to working in a cross-functional product team and plan to join a specialized research group, you might struggle to dive deep into a specific area of expertise and adapt to a more focused, research-oriented workflow.

Action:

Research the company's team structure (e.g., cross-functional, specialized) using their website or job description.
Prepare two to three specific examples of how you've successfully adapted to different team structures in the past.
List questions to ask during the interview about team dynamics and collaboration methods.

3. Does my area of expertise directly address the challenges and goals of this particular role?

Having the right skills doesn't guarantee you're the right fit for the team's needs. It's crucial to ensure that your expertise aligns with the specific requirements of the role you're applying for. This goes beyond just matching keywords on a job description; it involves understanding the nuances of the role, the team's current challenges, and the company's strategic goals.

Consider the depth and breadth of your expertise in relation to the role's demands. Some positions may require deep specialization in a particular area, while others might value a broader skill set. Additionally, the team may be looking for someone to fill a specific gap or bring a particular perspective that complements existing team members' skills.

It's also important to consider the trajectory of your career and how it aligns with the role's growth potential. A position that doesn't allow you to utilize your core strengths or develop in your desired direction might not be the best fit, even if you have the required skills on paper.

For example, you might be applying for a full-stack role, but if the team is looking for more back-end expertise and your strength lies primarily in front-end development, this could be a misalignment. Despite your full-stack development experience, your limited depth of experience on the server side could be a dealbreaker for the hiring team if they're specifically looking to bolster their back-end capabilities.

Alternatively, if you're a generalist full-stack developer applying to a role that requires deep specialization in a particular area like database optimization or scalability, your breadth of knowledge might not compensate for the lack of in-depth expertise they're seeking.

Action:

Create a table mapping your skills to the job requirements, identifying any gaps.
Choose three key skills from the job description and write a brief statement for each, explaining how you've applied them in past roles.
Identify any areas where you may lack experience and prepare talking points about how you will mitigate the skill gap.

4. Have I researched the demands of this industry?

If you're transitioning to a new industry altogether, you may have some other facets to consider before hitting "send" on your CV. Moving fields will involve adapting to different regulations, stakeholder expectations, and industry-specific knowledge. This can present a steep learning curve, so you have to be sure you're well-equipped to walk into this role.

To capitalize on transferable skills, identify core competencies from your current role that are valuable across industries. These might include project management, data analysis, problem-solving, or leadership skills. Then, frame these skills in the context of the new industry.

For instance, if you've managed tech projects in the software industry and are transitioning to healthcare, you could highlight:

Project management: Emphasize how your experience managing complex software projects can translate to overseeing the implementation of healthcare IT systems.
Stakeholder communication: Showcase how your ability to communicate technical concepts to non-technical stakeholders can be applied when explaining medical technologies to healthcare professionals.
Regulatory compliance: If you've dealt with data privacy in tech, highlight how this experience can be valuable in understanding and implementing HIPAA compliance in healthcare.
Problem-solving: Illustrate how your approach to troubleshooting software issues can be applied to optimizing patient care workflows.

At the very least, anticipate questions from the hiring team about your industry transition and prepare examples that demonstrate how your transferable skills can add value in the new context. Be prepared to discuss how you plan to bridge any knowledge gaps and show enthusiasm for learning industry-specific regulations and practices.

Action:

Read industry-specific publications or reports to familiarize yourself with current trends and challenges.
Identify and list key trends, regulations, or standards relevant to the industry you're targeting.
Connect with two to three professionals in the target industry via LinkedIn to network and learn more.

5. Do my values align with the company’s culture?

Finding a company that's a good cultural fit is often as important as skills and experience. Misalignment with a company's core values and culture can lead to dissatisfaction and a lack of fulfillment. Before you start applying for jobs, take some time to reflect on your own values, work style preferences, and what you consider to be an ideal work environment.

To pinpoint the cultural vibe of a company you're applying to, consider the following:

Company website and social media: Look beyond the job description. Examine the company's mission statement, values, and how they present themselves online. What values do they emphasize?
Employee reviews: Sites like Glassdoor can offer insights into the company culture from current and former employees. Look for patterns in the reviews.
News and press releases: These can give you an idea of the company's priorities and how they handle various situations.
Interview process: Pay attention to how the company conducts its hiring process. Is it efficient and respectful of your time? This can be indicative of their overall organizational culture.
Questions asked during interviews: Are they focused solely on technical skills, or do they also inquire about your work style, collaboration preferences, and career goals?
Office environment: If you have an on-site interview, observe the workspace. Is it open-plan or more traditional? How do employees interact?
Communication style: Notice how quickly the company responds to your emails and the tone of their communications. This can reflect their internal communication culture.

Action:

List the company's stated values (usually found on their website) and write a brief example of how you've demonstrated each in your career.
Prepare three specific questions about company culture to ask during your interview.
Write down three of your own core values and be prepared to discuss how they align with the company's culture.

6. How can I differentiate myself from other applicants?

In a competitive job market, it's important to evaluate how your profile stacks up against other candidates. Highlighting your unique value proposition can make you a clearer frontrunner. Your unique value proposition is the combination of skills, experiences, and qualities that set you apart from other candidates and make you particularly well-suited for the role.

Remember, your unique value proposition isn't just about being different; it's about being different in a way that's valuable to the employer. It should highlight not just what makes you unique, but why that uniqueness matters in the context of the job you're applying for.

For example, if you're applying for a data scientist position at a competitive tech company, you might be up against candidates with PhDs, extensive publications, and experience at leading firms. To stand out, you could highlight unique projects where you've driven significant business impact, your ability to communicate complex data insights to non-technical stakeholders, or any relevant interdisciplinary skills.

Action:

Write a one-paragraph, unique value proposition statement highlighting your distinctive skills and experiences.
Identify three quantifiable achievements from your career and frame them in terms of business impact.
Create a brief "elevator pitch" that succinctly communicates your unique qualifications for the role.

What happens in the face of rejection?

Remember, rejection is not a reflection of your worth as an engineer. It's an opportunity to learn, adapt, and refine your approach. By incorporating both pre-mortem analysis and post-rejection reflection into your job search, you can turn setbacks into stepping stones for future success.

If faced with rejection, consider these steps for a thorough retrospective:

Request feedback: If possible, ask the hiring manager or recruiter for specific feedback on your application or interview performance.
Review your application: Assess whether your resume, cover letter, and portfolio (if applicable) effectively highlighted your most relevant skills and experiences.
Reflect on the interview process: Consider your performance in technical and behavioral interviews. Did you provide clear, concise examples that demonstrated your skills? Were there areas where you struggled?
Assess your preparation: Evaluate how well you researched the company and articulated your interest in the specific role.
Evaluate your unique value proposition: Did you effectively communicate what sets you apart from other candidates and connect your unique skills to the company's needs?
Consider external factors: Acknowledge that sometimes rejections are due to factors beyond your control, but focus on what you can improve.
Develop an improvement plan: Based on your analysis, identify areas for improvement and set specific, actionable goals. For example, "I will improve my coding interview skills by completing one LeetCode challenge daily for 30 days" or "I will practice mock system design interviews weekly."
Stay resilient: Maintain a growth mindset and view each application and interview as a learning experience. Keep your job search active and don't put all your hopes on one opportunity.

By systematically reviewing your experience and making targeted improvements, you can enhance your candidacy for future opportunities. Remember, many successful professionals have faced rejection on their career paths. The key is to learn, adapt, and persist in your job search journey.

This article was originally published on LeadDev.com on Jul 30th, 2024.

If you liked this post, 🗞 subscribe to my newsletter and follow me on 𝕏!

How to develop your skillset for the AI era

Jesal Gadhia — Sat, 06 Apr 2024 20:24:54 +0000

How can engineers ensure their skills keep up with AI advancements?

Over the last few years, we've watched AI take giant leaps in coding – from AI-assisted coding tools like GitHub Co-Pilot to the first AI engineer named Devin. This explosion of AI capabilities has sparked endless debates, with the burning question on everyone's mind being: does software engineering still have a future?

The short answer? Yes, but with a catch. The coding aspect of software engineering might become more automated, with AI taking over the bulk of code generation in the future. However, software engineering is more than code.

Looking ahead, engineers who can zero in on business results, flex into other roles, and consistently deliver value will continue to be in demand despite AI as their impact extends beyond their coding ability.

Think like an entrepreneur

Simply being skilled in engineering won't cut it anymore. The real magic happens when you connect engineering efforts directly to business outcomes. Embracing an entrepreneurial mindset equips engineers with a broad view that transcends the immediate technical challenges. In a world where AI can write code but cannot grasp the entire organization's nuances, its people, culture, and the broader environment it operates within, having an expansive vision becomes a critical advantage.

Engineers who think like entrepreneurs understand that success in the AI era demands more than technical excellence. They have an intrinsic drive to create customer value, ensuring their engineering solutions are not just technically sound but also strategically aligned with business goals. They're the bridge between technical feasibility and product strategy, using their skills to steer product decisions in the right direction.

How can you become an intrapreneur?

1. Understand the business

Why? This will help you see where your projects fit within the larger picture and how they can influence the bottom line.
How? Dive into your company’s OKRs and KPIs to understand the measures of success. Participate in all-hands meetings, review annual reports, do competitor research, and stay informed on company strategy.

2. Build customer empathy

Why? Putting yourself in the customer's shoes will help you make informed engineering decisions that resonate with user needs and business goals.
How? Explore business intelligence (BI) tools, user experience (UX) research, and analytics platforms to understand customer sentiment and user behavior. Additionally, listen to recorded sales calls, if available, to get a first-hand pulse of what the customer is thinking and how they are reacting to the product.

3. Incorporate insights into decision-making

Why? Combining business context and data-driven insights in your decision-making process ensures that your engineering efforts are both technically sound and strategically aligned with business objectives.
How? When proposing new features or improvements, connect it back to the business objectives, user feedback, and data analysis that support your suggestion. Present your ideas in a way that highlights their potential impact on both user satisfaction and business success.

By embracing these practices, you will become adept at navigating uncertainty, prioritizing projects that offer the highest return on investment, and persuasively advocating for your ideas.

Flex across functions

Hand-in-hand with a business-centric approach is the need to wear multiple hats. The more versatile an engineer is, the more indispensable they become.

In a world where specific tasks may eventually be automated by AI, human engineers' ability to synthesize information across domains, connect disparate ideas, and apply insights in new contexts sets them apart.

How can you cultivate versatility?

1. Collaborate on the product roadmap

Why? Gaining insight into your company's product goals and understanding how technology can drive these objectives is crucial as it broadens your perspective beyond just the technical aspects of your work.
How? Volunteer to join roadmap planning sessions and offer a technical perspective on proposed features. Help with prioritization by assessing the technical effort required versus the projected business impact.

2. Contribute to design systems

Why? Understanding design principles is crucial for creating technically sound products that provide a great user experience. A basic knowledge of UX/UI principles, color theory, and user-centric design can significantly improve the collaboration between engineering and design teams, leading to more cohesive and user-centric products.
How? Work closely with designers to understand the core components of your product's UI/UX. Offer to develop a shared library of UI components or a style guide that codifies design principles and patterns.

3. Mentor across functions

Why? Sharing your expertise and insights across different functions not only helps others grow but also establishes trust, builds communication skills, and positions you as someone they can go to in the future.
How? Initiate informal coffee chats or lunch meetings with peers in different roles. Offer mentorship or advice, focusing on how their roles contribute to the business's success. This could involve guiding a designer on how to consider technical constraints in their designs or helping a marketing colleague understand the possibilities of the product.

By actively engaging in these practices, you'll be well on your way to becoming a more versatile engineer. This cross-functional agility ensures that engineers can adapt to changes, foresee the implications of decisions across domains, and innovate at the intersections of disciplines, enabling them to provide value that no AI point solution can.

Carve your niche

Grounded in a rich understanding of the broader context and enhanced by a diverse skill set, building specialization will ensure that engineers can bring unique insights, creativity, and solutions that AI cannot. It's the intersection of depth and breadth in an engineer's expertise that will define their irreplaceability in an AI-driven world.

This is where Roger Martin's Doctrine of Relentless Utility comes into play, a career strategy that focuses on finding your niche and monopolizing it.

As you become more adept at navigating between different roles and perspectives, you'll be better positioned to uncover unique opportunities where your particular blend of skills and interests intersect with unmet needs within your team or organization.

Aligning what you're good at with areas where you can make a significant impact allows you to establish a distinctive role that plays to your strengths and passions. This strategy promotes an active, value-driven approach, looking for ways to contribute beyond the usual scope of your role.

For instance, let's say you have a knack for simplifying complex technical concepts into digestible content. Your niche could be bridging the gap between advanced technical knowledge and non-technical stakeholders or clients. Build on this by developing and sharing insights through blogs, workshops, or internal documentation that make these concepts accessible and engaging.

Another example could be finding yourself at ease automating repetitive tasks that bog down your team. Your niche is identifying efficiency drains within your team's workflow by creating custom scripts or leveraging existing tools to automate these tasks. This way, you’ll be freeing up time for your team to focus on more impactful projects.

By adopting the doctrine of relentless utility, you're not just filling a role; you're carving out an area of focus where you can truly excel and provide unmatched value.

Remember, the niche you carve out today may not remain your niche tomorrow. Continuously explore, learn, and iterate, allowing your niche to evolve alongside your skills, interests, and the changing needs of your team. This ongoing journey of adaptation ensures your contributions remain impactful and aligned with both personal growth and team objectives.

Closing thoughts

We find ourselves at a juncture similar to the one brought about by the printing press and the typewriter. The fears are familiar – concerns about job displacement, the devaluation of human expertise, and the uncertainties of a shifting professional landscape.

Yet, if history has taught us anything, it’s that with disruption comes opportunity. Just as the printing press opened up the world of knowledge and the typewriter transformed the professional landscape, AI will redefine software engineering. It will challenge us to elevate our roles beyond coding, to think like entrepreneurs, and to build a more multi-faced identity.

The future of software engineering isn't about clinging to the past but one where AI helps us build unimaginable things.

This article was originally published on LeadDev.com on April 4th, 2024.

If you liked this post, 🗞 subscribe to my newsletter and follow me on 𝕏!

Lessons in Leadership Excellence

Jesal Gadhia — Wed, 27 Dec 2023 21:20:48 +0000

Personal Insights on Building Exceptional Teams

In a world where 75% of cross-functional teams are dysfunctional, mastering the art of leadership is not just a skill but a necessity.

I’ve been fortunate enough to lead teams in various environments, from small start-ups to large organizations, and along the way, I’ve received consistently positive feedback on my teams' camaraderie, cohesion, and performance. That and hearing many times from my direct reports that I’ve been one of the best managers they’d had made me wonder what exactly was working well.

I don’t think I was super deliberate about it, to be honest. I just did what felt right intuitively over time. Maybe my experience being on the engineer manager pendulum allowed me to keep my ear to the ground and do what I would have wanted to see from my manager as an IC.

In any case, this is my attempt at writing down some of my implicit strategies into an explicit guidebook for myself and others to reference and follow. This is not meant to be a comprehensive resource such as MGMT Accelerator or The 15 Commitments of Conscious Leadership, which I highly recommend, but just things I found putting the most amount of emphasis on. I’ll also note that you don’t need to be a manager to find this relevant. Effective leadership is a key skill for everyone to master.

So, after doing some reflection, I was able to boil down this essence of effective leadership to six critical elements, which I like to call the ‘6 Cs’. In this post, I’ll share these key elements, each backed by practical strategies and real-life lessons that can redefine how you lead. So, without further adieu, let’s dive in.

1. Communication

Transparent and authentic communication is the cornerstone of effective leadership. It’s not merely about conveying information but about involving team members in decision-making processes, even when ideas are still in their formative stages. This practice fosters a sense of ownership and collaboration.

This also extends to offering constructive feedback, where I aim for open and candid conversations without the proverbial “shit sandwich.” This authentic management style fosters workplace transparency, clarity, and directness. It empowers employees to clearly identify areas where they can make the most meaningful contributions to the team’s success.

Some professionals say you need to have a praise-to-criticism ratio of 3:1, 5:1, or even 7:1. Others advocate the “feedback sandwich” — opening and closing with praise, sticking some criticism in between. I think venture capitalist Ben Horowitz got it right when he called this approach the “shit sandwich.” Horowitz suggests that such a technique might work with less-experienced people, but I’ve found the average child sees through it just as clearly as an executive does. — Kim Malone Scott, Radical Candor

Key practices:

Hold regular one-on-one meetings with a running agenda to discuss your team’s ideas and concerns.
Establish a culture of active listening where team members feel heard and valued.
Leverage collaborative tools that allow team members to contribute to decisions and ask questions async.
Establish a career plan document where you keep track of goals, ideas, and growth conversations and review it at least on a monthly basis.

Questions to measure success:

Do you regularly cascade relevant strategy and business updates to your team members from the company leadership?
Do your team members feel like they are part of shaping the ideas from the early stages and are involved in the decision-making process?
Do your team members regularly receive ongoing feedback rather than just surprising feedback during formal touchpoints like performance review cycles?

2. Connection

Connection is built on the foundation of authentic and transparent communication. Team cohesion and trust are something I value deeply, and I aim to create an environment that values inclusivity, empowerment, and psychological safety.

A team is not a group of people who work together. A team is a group of people who trust each other.— Simon Sinek, Leaders Eat Last

I’ve used many rituals and practices, such as retrospectives, all hands, happy hours, pulse surveys, etc., with the aim of building connection and trust. No matter the ritual, it comes down to ensuring that every team member’s voice is heard and respected.

Creating opportunities for the team to bond outside of work is also particularly effective. This includes team happy hours, get-to-know sessions where team members share their passions and interests, and even occasional team retreats to strengthen in-person connections.

Key practices:

Schedule regular team retrospectives to examine what’s going well and what can be improved. Commit to action items based on the feedback and continue to iterate.
Foster mentorship and peer support within your team.
Encourage cross-functional collaboration to build relationships with other teams.
Share stories of successful team collaborations and their impact on projects.

Questions to measure success:

Do your team members believe they have agency in shaping how the team operates?
Is there a mechanism in place to continually examine team processes and iterate based on feedback?
Does the team regularly interact with each other in informal settings such as happy hours or retreats?

3. Composure

Amidst challenges and crises, maintaining composure is essential. Using the team as an outlet for venting emotions is tempting, but this can be counterproductive. Instead, I’ve learned to adopt a methodical approach: analyzing the situation, acknowledging emotions, outlining a clear plan, and providing transparent updates.

In my experience, team members look to their leaders as beacons of stability during turbulent times. They appreciate leaders who don’t just react emotionally but engage in constructive problem-solving. In fact, research suggests that for almost 70% of people, their manager has more impact on their mental health than their therapist or their doctor.

When interpersonal disagreements or conflicts arise, lean into curiosity to understand why. This is often not an easy thing to do. In her book “Dare to Lead,” Brown uses the term “rumble” to describe a discussion or conversation that is defined by a commitment to lean into vulnerability, stay curious and generous, and be fearless in owning our parts.

A rumble is a discussion, conversation, or meeting defined by a commitment to lean into vulnerability, to stay curious and generous, to stick with the messy middle of problem identification and solving, to take a break and circle back when necessary, to be fearless in owning our parts, and, as psychologist Harriet Lerner teaches, to listen with the same passion with which we want to be heard. More than anything else, when someone says, “Let’s rumble,” it cues me to show up with an open heart and mind so we can serve the work and each other, not our egos.―Brené Brown, Dare to Lead: Brave Work. Tough Conversations. Whole Hearts

She believes that during moments when we’re pulled between our fear and our call to courage, we need shared language, skills, tools, and daily practices that can support us through the rumble.

Key practices:

Establish a culture of practicing non-violent communication or The Story Rumble to help the team effectively and compassionately navigate difficult conversations and conflicts.
Create space to practice mindfulness exercises as a team to help manage stress during challenging times.
Run premortems ahead of major milestones and launches to avoid last-minute surprises and stressful fire drills.
Establish an on-call rotation with predefined roles and responsibilities and detailed runbooks to handle emergencies.
Encourage the team to take time off after intense periods and plan for their absence to allow them to recharge without pressure or guilt.

Questions to measure success:

Does your team feel that they have reliable support during challenging times?
Do you maintain composure and engage in constructive problem-solving rather than venting emotions during crises?

4. Curiosity

Fostering a culture of continuous learning has been a personal passion of mine. I’ve encouraged my team to embark on a journey of exploration, whether through experimentation, reading, or mentorship, to build their personal mastery. This means holding brown bag sessions and providing opportunities for team members to work on experimental ideas, fostering a culture of perpetual learning.

I’ve also seen the importance of extending this curiosity to understand our customers and the business deeply. To truly solve their problems, the team needs to immerse themselves in the customer’s perspective and become students of the business.

The highest quality of thinking cannot emerge without learning. Learning can’t happen without mistakes.― Liz Wiseman, Multipliers: How the Best Leaders Make Everyone Smarter

Key practices:

Organize regular “knowledge-sharing” or “brown bag” sessions where team members present on topics of interest or discuss industry-related books or articles.
Set aside time for team members to explore new technologies, ideas, or tools relevant to their work.
Create opportunities for the team to interact directly or indirectly with customers by fielding support cases or listening in on sales calls.

Questions to measure success:

Do you create space for the team to experiment and fail in their learning journey without it being held against them?
Does the team interact with its customers on a regular basis?

5. Clarity

Clarity serves as the guiding light in decision-making. It’s not just about relying on data; it’s about relying on the right data.

I have found setting ambitious yet achievable goals, defining clear KPIs, and diligently tracking progress have created an immense amount of clarity for the team. In addition, we created a long-term roadmap, acknowledging its inherent uncertainty, and broke it down into detailed plans every quarter to create a seamless runway for execution.

This clarity extends to written communication, where everything from team working norms to schedules and expectations is documented. Having an operating playbook in place ensures that new team members can seamlessly integrate into the team and hit the ground running.

Ideas are easy. Execution is everything. — John Doerr, Measure What Matters

Key practices:

Leverage thinking tools and frameworks to help you and your team solve problems, make decisions, and understand systems.
Create a team charter that outlines norms, expectations, and communication guidelines.
Use project management tools to track progress transparently and provide visibility into goals.
Create an organized repository for regularly updated documentation, making it easy to access important information.
Ensure new team members have a clear checklist of things they can accomplish or learn in the first 30, 60, and 90 days.

Questions to measure success:

Does your team possess a clear operating playbook, ensuring seamless onboarding for newcomers?
Are goals and KPIs well-defined, and is progress diligently tracked?

6. Celebration

Recognizing and celebrating achievements, no matter their scale, injects joy into the workplace. Acknowledging accomplishments through company-wide accolades or simple ‘kudos’ fosters a sense of pride and motivation.

69% of employees state they would work harder if they felt their efforts were better recognized — Zippia, 35+ Powerful Leadership Statistics [2023]: Things All Aspiring Leaders Should Know

Leadership is not about being in charge. It is about taking care of those in your charge. — Simon Sinek, Leaders Eat Last

Key practices:

Establish a practice to regularly recognize people behind their exceptional contributions and celebrate their impact.
Host regular team celebrations for project milestones, both big and small, and send out swag and small tokens of appreciation.
Establish a public forum for team members to express gratitude and recognition for their colleagues.
Establish a public forum for sharing success stories highlighting their positive impact.

Questions to measure success:

Is team spirit consistently high, and do team members eagerly anticipate celebrating successes together?
Are creative ways employed to foster team bonding, even in distributed work settings?

Closing Thoughts

I hope you found valuable takeaways from these 6 Cs. These principles aren’t just personal opinions; they’re insights drawn from practical experiences and even backed by research at BetterUp, which happens to be my former employer and a place where I’d like to think I earned my stripes for leadership:

We found that managers who can strengthen their coaching and problem solving skills, be more authentic with their teams, and work to recognize employee contributions are riding a wave of effectiveness to the bank: Their organizations experience a +400% return on assets, +30% EPS 5-year growth, +17% boost in innovation, and more. — Mindsets on the move: what effective management looks like today, BetterUp Briefing

You could use these 6 Cs as a way to conduct a health check for your team and see where you need to focus your energies. I would love to hear your thoughts and experiences as you apply the 6 Cs to your leadership journey — please feel free to share your feedback and stories!

If you liked this post, 🗞 subscribe to my newsletter and follow me on 𝕏!

How to make your ideas stick

Jesal Gadhia — Fri, 08 Dec 2023 16:00:00 +0000

Effective proposal writing techniques for engineering leaders

As an engineering leader, you’ve likely faced the challenge of writing proposals for ideas you’re passionate about, whether it’s drafting design docs, request for comments (RFCs), memos, or even slide decks. The format might change depending on context, but the fundamental challenge remains: how do we make our ideas resonate?

Throughout my career, I’ve witnessed numerous promising ideas fall by the wayside or need a lot of back and forth due to a lack of clear articulation. The culprit? Often, a missing ingredient from the six crucial elements of a persuasive proposal.

Knowing how to craft proposals that communicate and inspire action will ensure that your next proposal isn’t just read, but acted upon.

1. Establish context

At the heart of a compelling proposal lies the establishment of mutual understanding. Begin by elucidating the “why.” Resist the urge to dive straight into the solution.

For instance, if you’re proposing a shift in system architecture, listing the current system’s flaws might seem intuitive. However, the real starting point should be “Why now?” What future changes do you foresee that necessitate this shift? Relate your proposal to your organization’s KPIs, OKRs, or overarching strategy.

Drawing inspiration from Stephen Covey’s The 7 Habits of Highly Effective People, it’s paramount to prioritize understanding before seeking to be understood. By ensuring your proposal resonates with your audience’s concerns, you pave the way for consensus.

Example of establishing context in a proposal:

With the rapid growth of our user base and the increasing complexity of our application, our existing architecture is becoming a bottleneck. Recent post-mortem analyses of outages have pointed to scalability and tight coupling issues.

Our company’s OKRs aim for a 20% increase in user engagement and a 30% growth in active users over the next year. To achieve this, we need an architecture that is resilient, scalable, and can be iterated upon quickly.

2. Identify the problem

Once mutual understanding is established, it’s imperative to identify the core issue or opportunity your proposal targets.

Begin by articulating the current state, grounding your discussion with metrics and data. Subsequently, spotlight the primary areas of concern with the existing situation and, based on this, formulate your hypothesis.

Overall, you should seek to emphasize the gravity and ramifications of the problem, positioning yourself as the authority spotlighting an urgent issue.

Example of identifying the problem in a proposal:

Current state:

Our monolithic application has been the backbone of our services for the past five years. During this time, we’ve seen:

User growth: A 300% increase in active users.

Feature expansion: The introduction of 50+ new features, increasing the complexity of our codebase.

Deployment frequency: An average of one thousand deployments per month, with each deployment introducing an average of three new features or improvements.

While this growth and activity indicate a thriving product, they’ve also introduced challenges:

1. Latency concerns: Our API response times have gradually increased over the past year. During high-traffic periods, we’ve observed a 30% increase in API response times compared to the previous year, impacting user experience.

2. Tight coupling: The intertwined nature of our services has become more evident. 20% of new feature releases in the past six months have resulted in regressions in unrelated areas, indicating a lack of modularity and separation of concerns.

3. Deploy time: The complexity of our application has affected our deployment times. Each deployment now takes an average of one hour, a 25% increase from the previous year. This affects our ability to swiftly deliver value to our users and maintain a reasonable MTTR.

Hypothesis: By migrating to a microservices architecture, we can achieve faster deployment cycles, better scalability, and independent service evolutions, leading to improved uptime and stable user experience.

3. Present options

When proposing solutions, it’s vital to consider various options. As Chip Heath and Dan Heath advise in Decisive: How to Make Better Choices in Life and Work, follow the W.R.A.P process: widen your options, reality-test your assumptions, attain distance before deciding, and prepare to be wrong.

The W.R.A.P process will help you avoid common pitfalls in decision-making, such as the framing trap, confirming-evidence trap, overconfidence trap, and forecasting trap.

This also builds credibility as an honest analysis rather than a biased proposal. This stage is about demonstrating your critical thinking skills.

Example of presenting options in a proposal:

Option 1: Incremental refactoring

Pros: Lower immediate risk, can be done during regular sprints.

Cons: Longer overall transition period, potential for accumulating tech debt.

Option 2: Big bang rewrite

Pros: Clean slate, can design with best practices from the start.

Cons: High risk, requires significant resources, longer time to see benefits.

Option 3: Hybrid approach (starting with core services and expand outwards)

Pros: Balances risk and allows for learning and iteration.

Cons: Requires careful planning and coordination.

4. Create a roadmap

With a recommended solution, it’s time to detail the implementation. Envision your proposal as a meticulously planned sequence.

Create a timeline detailing the steps, assign responsibilities, and set expectations for each stage.

Like a conductor leading an orchestra, your proposal should coordinate a unified effort toward the intended result.

Example of outlining a roadmap in a proposal:

Phase 1: Identify and isolate core services.

Duration: 1 month

Responsible: Architecture guild

Phase 2: Develop the first set of microservices and integrate them with the existing system.

Duration: 3 months

Responsible: Teams A and B

Phase 3: Gradual rollout to a subset of users and gather feedback.

Duration: 1 month

Responsible: QA team, DevOps, and user feedback group

Phase 4: Full transition and decommissioning of old components.

Duration: 2 months

Responsible: All engineering teams

5. Highlight risks

Analyze potential risks and challenges associated with the proposal, as well as the likelihood and impact of those risks. Provide strategies or plans to mitigate these risks, showing preparedness and foresight.

Examples of highlighting risk in a proposal:

Risk: Integration challenges with existing systems

Likelihood: High

Impact: Moderate

Mitigation: Begin with a pilot project focusing on one or two core services to understand integration pain points. Use this pilot to develop best practices for subsequent integrations.

Risk: Potential downtime during transition

Likelihood: Medium

Impact: High

Mitigation: Schedule migrations during off-peak hours and ensure rollback strategies are in place. Communicate potential downtimes to stakeholders well in advance.

Risk: Team’s learning curve with new architecture

Likelihood: High

Impact: Medium

Mitigation: Invest in training sessions and workshops focused on microservices best practices. Pair experienced team members with those less familiar during the initial phases.

Risk: Increased initial costs

Likelihood: Medium

Impact: Medium

Mitigation: While there might be an upfront cost due to the transition, the long-term benefits in terms of scalability, maintainability, and faster deployment cycles are expected to offset these initial expenses. Regular cost-benefit analyses will be conducted to ensure we remain on track.

Risk: Complexity in monitoring and logging across services

Likelihood: Low

Impact: High

Mitigation: Adopt centralized logging and monitoring solutions that provide a holistic view of the entire system. This will aid in quicker issue detection and resolution.

6. Prompt action

Summarize the proposal’s main points and clearly state what you’re asking stakeholders to do next, whether it’s providing feedback, giving approval, or allocating resources.

Include any additional information, data, charts, or references that support the proposal but might be too detailed for the main body. This provides a deeper dive for those interested.

Example concluding aspect of a proposal:

In light of the challenges posed by our current monolithic architecture and the company’s ambitious growth targets, transitioning to a microservices approach emerges as a strategic imperative. The hybrid approach, which combines the strengths of incremental refactoring and a more comprehensive rewrite, offers a balanced and pragmatic path forward.

We’ve identified potential risks associated with this transition and have proposed robust mitigation strategies to address them. The success of this initiative, however, hinges on a collaborative effort, cross-functional alignment, and shared commitment.

Next Steps:

Feedback round: We invite stakeholders to review this proposal and provide feedback by [specific date].

Pilot execution: Post feedback; if there are no strong objections or concerns, we will commence the pilot project.

We’re at a pivotal juncture, and the decisions we make now will shape our product and capabilities for years to come. Let’s collaborate, innovate, and steer our organization towards a future-ready architecture.

Final thoughts

In conclusion, crafting a persuasive proposal goes beyond organizing information in a digestible pyramid structure. It’s about weaving a narrative that resonates with the organization’s core objectives, prospecting a path forward, anticipating risks, and compelling stakeholders to take action.

When writing your next proposal, see it as more than a document. It’s a powerful call to action, a catalyst for change, and a testament to your leadership acumen. The quality of your proposals can shape the trajectory of your projects, teams, and, in the grander scheme, your evolution as an engineering leader.

This article was originally published on LeadDev.com on Nov 29th, 2023.

If you liked this post, 🗞 subscribe to my newsletter and follow me on 𝕏!

The Role of a System Architect

Jesal Gadhia — Tue, 08 Aug 2023 19:06:02 +0000

Unraveling growth challenges: Could a system architect be the solution?

Serverless Architecture from Classic Programmer Paintings

Most startups that go through hyper-growth tend to face the inevitable Bottlenecks of Scaleups, as masterfully described by ThoughtWorks.

Having navigated the hyper-growth journey myself, I’ve witnessed firsthand how these technical bottlenecks can impede business progress, causing churn and chaos.

Bottlenecks of Scaleups

In light of these challenges, I wondered if a system architect's role could hold the key to identifying and circumventing these bottlenecks and reducing complexity like this infamous Amazon Deathstar.

Amazon internal service dependency visualization

To that end, this article explores the benefits, operating philosophy, guiding principles, job description, career progression, success criteria, and implementation approaches for the role of a system architect.

Benefits of having a System Architect

My hypothesis is rooted in the belief that a clear and purposeful approach to technical architecture can prevent false starts and accumulating tech debt.

Picture a system architect whose sole responsibility is to observe the big picture and facilitate knowledge sharing across teams, breaking down silos for better collaboration.

This individual would need to possess exceptional skills in designing and managing the overall technical architecture of a system. Their role includes assisting teams in defining system components, services, and interactions.

The benefits of having a system architect would ultimately translate into:

Improved alignment and coordination — Foster a common goal and architecture across teams, leading to more efficient and effective development.
Enhanced technical expertise — Offer a deep understanding of system design and architecture, guiding and mentoring other engineering team members to elevate the organization’s overall technical capabilities.
Improved quality and performance — Co-create a well-designed system architecture to boost performance and reliability, resulting in quicker time-to-value, better user experiences, and heightened customer satisfaction.
Reduced technical debt — Proactively manage system architecture to prevent tech debt buildup, saving time and resources in the long run.

Harmonizing With the Agile Philosophy

You may be curious how this role fits into the agile methodology that many organizations follow. On the surface, the role of an architect may be antithetical to being agile.

However, several sources discuss the role of an architect in an agile environment. For example, Agile Architecture describes it as:

The role of the architect in an agile project is to provide guidance and direction to the team on the technical aspects of the project. The architect is not a dictator, but rather a facilitator and coach, who helps the team find the best solutions to their technical challenges.

Similarly, the Scrum Alliance discusses the role, stating that:

[the architect] is responsible for guiding the team to make technical decisions that align with the project’s goals, scope, and delivery schedule. They should be able to communicate the technical vision to the team and help the team understand how their work contributes to that vision.

Large-Scale Scrum (LeSS) Framework’s view on the architecture role is more philosophically aligned with how I see the role. The following statements on the Architecture & Design page in the chapter “Technical Excellence” shape their opinion about software architects:

The sum of all the source code is the true design blueprint or software architecture.
The real software architecture evolves (better or worse) every day of the product, as people do programming.
The real living architecture needs to be grown daily through programming by master programmers.
A software architect who is not in touch with the evolving source code of the product is out of touch with reality.
Every programmer is some kind of architect–whether wanted or not. Every act of programming is some kind of architectural act–good or bad, small or large, intended or not.

LeSS promotes the idea that architects are regular team members. They should participate in hands-on engineering and especially mentoring through design workshops and pair programming. LeSS warns against architecture astronauts (a.k.a. PowerPoint architects):

These are the people I call Architecture Astronauts. It’s very hard to get them to write code or design programs, because they won’t stop thinking about the architecture… They tend to work for really big companies that can afford to have lots of unproductive people with really advanced degrees that don’t contribute to the bottom line.

#211 — The Architect by Comic Agile

Guiding Principles

Now that we have some grounding philosophy behind a system architect’s role in an agile organization, let’s discuss the guiding principles of this role.

For this, we don’t need to reinvent the wheel. Principles proposed by Agile Architecture hold strong:

Value people — You must recognize the value and importance of people and their expertise. No dictators or PowerPoint architects.
Communicate! — Communicate effectively with all stakeholders and tailor information to their needs.
Less is more — Minimize complexity and strive for simplicity in design and communication.
Embrace change — Be prepared for change and welcome it as an opportunity for competitive advantage.
Choose the right solution for the enterprise — Customer centricity. Make sure the solution is right by verifying it with the customer.
Deliver quality — Foster a culture of craftspersonship and sustainable development at a pace that can be maintained indefinitely.
Model and document in an Agile fashion — Leverage proven and effective modeling/mapping and documentation strategies.

Carving the Job Description

Equipped with the guiding philosophy and principles, if you were to recruit someone for this role, let’s see what the job description could look like. A system architect typically collaborates with other teams in several ways:

Guide system design and architecture — Provide guidance and expertise to other teams on how to design and build systems that align with the overall architecture of the organization. This can involve working closely with teams to understand their needs and requirements and recommending the best approaches.
Review and provide feedback for technical designs — Review and provide timely feedback for RFCs proposed by teams. This can involve evaluating the proposed designs to ensure they align with the system’s overall architecture and do not introduce any potential issues or technical debt.
Help resolve technical challenges — Help teams resolve technical challenges during development. This may involve providing guidance on approaching a particular problem or working with the team to develop a solution.
Participate in planning and estimation — Participate in planning and estimation meetings with other teams. This can involve providing input on the technical feasibility of proposed features and estimates for the time and resources required to implement them.

Overall, the specific ways in which a system architect collaborates with other teams will vary depending on the needs and goals of the initiative and the members working on it.

Enterprise Architecture Footprints

Navigating Career Progression

Now that we have a job description, one may wonder how someone would become an architect and what would their career trajectory look like.

Various avenues exist for senior engineers to advance in their careers. On the individual contributor (IC) path, adding a system architect role alongside roles like SRE, Data Engineer, and Staff Engineer is a viable option.

Illustrations below help visualize what that may look like courtesy of CodeCapsule.

Career Growth: What Paths After Senior Engineer

The Skills Map of Senior Tech Career Progression

Note the key difference between an architect and staff/principal engineers: while the latter focus on ensuring their teams’ work functions correctly, architects take a broader view, looking after how all the pieces fit together.

Before you ask, yes, staff and principal engineers also do think about how their work fits into the broader ecosystem, but most often, they don’t consider that their full-time concern.

This role sits at the intersection of ‘What’ (is it we are trying to do/solve) and ‘How’ (are we going to approach the design), and ‘Why’ (are we doing it this way) as described by Peter Cripps.

On Thinking Architecturally

Becoming an architect also has to do with having increased communication and relationship skills, on top of solid tech skills, as ‌architects have to deal with and influence larger groups of people across multiple teams and organizations.

Measuring Success

Introducing the role of a system architect into an organization is a strategic move aimed at improving the technical architecture, fostering collaboration, and overcoming growth challenges.

However, to validate the effectiveness of this role, it's essential to measure its impact and quantify the value it brings to the organization.

Here are some potential KPIs that can help gauge the effectiveness of the role:

Technical debt reduction: Measure the decrease in technical debt over time as a result of proactive architectural decisions made by the system architect. (Related: How Google Measures and Manages Tech Debt)
System performance and reliability: Evaluate the performance and reliability metrics of the system after the system architect makes architectural improvements.
Collaboration and team satisfaction: Conduct surveys or feedback sessions to gauge how well the system architect facilitates collaboration and team communication.
Time-to-Market reduction: Determine how architectural improvements by the system architect have contributed to faster product development and time-to-market.
Resource optimization: Assess how architectural decisions have led to better resource utilization, whether regarding hardware, cloud services, or personnel.
Impact on productivity: Analyze the impact of architectural improvements on the productivity of engineering teams.

Measuring success should not be a one-time activity. Adopting a continuous improvement approach is crucial, wherein feedback from stakeholders, teams, and customers is regularly collected and incorporated into the system architect’s strategies.

Feedback loops enable the system architect to adjust their approach based on real-world experiences and emerging challenges. This will help protect the organization against a gatekeeper or an ivory tower architect, which would be antithetical to the role’s intention.

Considerations for Implementation

Differentiating from EMs and Senior/Staff/Principal Engineers

While engineering managers (EMs) and engineers are usually involved in making architectural decisions, their focus is typically on the domain or squad they operate in. The system architect’s distinctiveness lies in their primary focus on enabling effective cross-squad collaboration and communication, particularly regarding cross-cutting technical implications.

Reporting structure

A role like a system architect often reports directly to engineering leadership, offering broad oversight and visibility across multiple teams. As such, it may be positioned under a Director or VP, or above, depending on the organization’s hierarchy.

Determining the number of architects

Given the depth of domain knowledge required, complex domains may require experienced architects. Hiring for this role may entail identifying existing engineers with the right skills and interest in stepping into the architect role. This may affect the pace of filling the role based on the current team’s availability and skills.

In Closing

In conclusion, I hope this exploration helps gives you a better sense of how this role could be introduced and the benefits it could bring.

I’ll leave you with this timeless video by KRAZAM. Enjoy!

If you liked this post, 🗞 subscribe to my newsletter and follow me on 𝕏!

How to Run Effective Sprint Planning Sessions

Jesal Gadhia — Sun, 30 Jul 2023 05:33:28 +0000

A Step-by-Step Sprint Planning Guide for Engineering Leaders

Sprint Planning Meeting from Classic Programmer Paintings

If you’ve been an engineer for some amount of time in the tech industry, you have probably experienced the highs and lows of sprint planning meetings. Maybe it has felt a bit like this painting by Marie Bashkirtseff. I’ve personally been in the trenches, both as a participant and as a leader, and have witnessed the wide array of approaches and their varying levels of effectiveness. It became apparent to me that the success of sprint planning lies in finding a structured and efficient approach that aligns the team and maximizes productivity.

Sprint planning is the backbone of Scrum-based agile project management, providing the framework for teams to organize their work and establish clear goals for each iteration. Yet, all too often, I’ve seen these meetings devolve into chaos, resulting in misaligned priorities and a waste of valuable time. Recognizing the need for a better way, I embarked on a journey to refine and iterate upon a process that consistently delivers exceptional results.

Through countless iterations and learnings, I’ve developed a process that has significantly improved the effectiveness of sprint planning while keeping the meeting overhead low. This process has helped my teams regain focus, enhance collaboration, and achieve remarkable outcomes.

In this article, I aim to share my insights and provide you with a step-by-step guide to streamline your own sprint planning, ensuring that it becomes a powerful tool for success.

Principles

Before we go any further, I will state this is an opinionated view on sprint planning and it follows these four underlying principles:

Bulk of the sprint planning happens 2 weeks leading up to the sprint start: Allotting adequate time for gathering requirements, aligning priorities, and preparing for the upcoming work is vital. This two-week timeframe allows for thorough preparation and sets the stage for a successful sprint.
Sprint planning session is mainly for high-level sprint review and alignment: The sprint planning session itself should primarily focus on reviewing the high-level aspects of the upcoming sprint and ensuring alignment with the team’s goals. Detailed discussions and refinement of tasks should occur prior to the planning session.
Each team member is an owner and driver of their work stream: Engineers are not “ticket takers” but leaders who take responsibility for problem definition and own end-to-end execution. Consequently, each team member is responsible for creating tickets and breaking down work into manageable chunks, ensuring a proactive and self-driven approach.
Tickets in the sprint are conscious commitments to deliver: The team takes a proactive and accountable approach to managing workload. Promptly flag any potential delays, emergent work, or setbacks and involve the stakeholders to renegotiate commitments in alignment with the evolving circumstances. It’s crucial to avoid scope creep during the sprint. If new tasks or requirements emerge mid-sprint, renegotiate commitments for existing work rather than merely piling new tasks on top.

Prerequisites

Before diving into the tactical sprint planning process, there are a few prerequisites that need to be in place:

Long-term roadmap for your team: Having a clear vision of your team’s long-term goals and initiatives is essential. This roadmap will serve as a guide for prioritizing work during sprint planning, ensuring that each sprint contributes to the broader objectives. If you don’t have this, at minimum having OKRs is important so that everyone is aligned at a high level on how the day-to-day work ladders up to more overarching business (or mission) objectives.
JIRA plan of Epics: Use a project management tool like JIRA to create a plan and map out Epics based on your long-term roadmap. Breaking down large-scale goals into actionable initiatives upfront facilitates better planning and execution.
High-level visual roadmap: Develop a low-fidelity visual roadmap that outlines drivers & their work streams per sprint for the entire quarter. Fill out as much as you have clarity on upfront. Keep this updated as the rest of the quarter progresses with the goal of having a light of sight at least 1–2 sprints out. This holistic view of the work to be accomplished provides clarity and alignment for the rest of the team and an easy-to-consume visual for stakeholders outside the team.

Here’s an example of a high-level visual roadmap using Confluence Roadmap Planner

Tactical Steps

Now, let’s dive into the tactical steps to streamline your sprint planning process:

What a typical schedule of ceremonies looks like for my teams

Part 1: Two weeks prior to the sprint start

During the current sprint planning session, take the following steps:

Create the future sprint: Create the upcoming sprint in the JIRA backlog. This will serve as a staging ground for the upcoming sprint. You could also just pre-create all six sprints for the quarter ahead of time to serve as buckets to allocate work to in advance.
Prospect: Start moving any follow-up tickets or future work from your backlog into that upcoming sprint as you are reviewing the current sprint.

Part 2: One week prior to the sprint start

One week before the start of the sprint, begin populating the sprint with tickets from the backlog. Follow these guidelines:

Align on upcoming priorities: Leverage rituals such as 1:1 meetings, working sessions, and stand-ups during the week to discuss and align with team members on the upcoming priorities. Determine who will be working on each task. Each team member should be responsible for populating their sprint with tickets async.
Ensure sprint backlog is fully populated: 1–2 days prior to the sprint start date, ensure that the upcoming sprint is fully populated with tickets. Each ticket should have an assignee and be appropriately story-pointed. This can happen individually async but you can hold optional sprint planning office hours for any last-minute discussion and alignment.
Add thematic focus to the visual roadmap: Update the high-level visual roadmap with the thematic focus of each individual’s sprint. This step provides a clear understanding of each team member’s responsibilities and aligns them with the overall goals. This can be done during office hours or asynchronously, ensuring that everyone is on the same page.

Part 3: On the day of the sprint start

During the sprint kickoff/review meeting, follow these steps:

Complete the current sprint: Close the current sprint and ensure that any remaining tickets are completed or appropriately moved to the following sprint.
Review the completed sprint: Review the sprint report briefly at a high level to get a sense of where you landed. This is a good opportunity to celebrate achievements and identify any challenges or carryover work. If possible, review the results of shipped work from the previous sprint and track progress against your OKRs. This is also an opportunity to reiterate the team’s thematic goals and their connection to longer-term objectives.
Conduct a sprint review per person: Share the high-level visual roadmap and dive into a sprint review for each team member. Use the following checklist:
1. Ensure thematic alignment: Confirm that the main thematic focus of the tickets aligns with the roadmap at a high level.
2. Story point verification: Verify that all tickets are appropriately story-pointed, reflecting their complexity and effort. Bonus: See my previous post on how to estimate effort.
3. Epic alignment: Ensure that the majority of tickets belong to an epic aligned with a roadmap initiative. This ensures that the work being done is driving the team towards its long-term goals.
4. Anticipate bandwidth constraints: Prospect forward and cnsider any time off, on-call duties, or other events that could impact a team member’s bandwidth. Update the visual roadmap accordingly to avoid overallocation and underestimation.
5. Load balancing: If an individual’s sprint load appears excessive, either move some tickets to the future sprint or load balance the workload with another team member.
Repeat for all team members: Go through the checklist above for each team member, ensuring that no unassigned tickets remain in the sprint.
Start the sprint: With all preparations in place, kick off the sprint.

That’s a Wrap

If you made it this far, congratulations on equipping yourself with the knowledge to run effective sprint planning sessions! Remember, implementing these practices is just the beginning of your journey toward optimized sprint planning and team success.

As you put these principles and tactical steps into action, you may wonder, “How do I know if this is working?”. One key indicator of progress is the improvement in sprint planning accuracy. Keep a close eye on the alignment between planned and actual outcomes i.e. planning accuracy or variance, and assess whether the commitments made during sprint planning are being met consistently.

Source: Head First Agile

Furthermore, don’t underestimate the power of feedback. Retrospective meetings at the end of each sprint cycle provide an excellent opportunity to discuss what went well and what can be improved. By actively soliciting and incorporating feedback, you can fine-tune your approach over time and drive continuous improvement.

Happy sprint planning!

From Chaos to Clarity: The Essential Guide to Personal Knowledge Management

Jesal Gadhia — Sat, 31 Dec 2022 05:47:09 +0000

Are you tired of constantly struggling to remember important information? Do you feel overwhelmed by the endless stream of inputs coming at you every day? You're not alone. In today's digital world, we're bombarded with an average of 74 GB of data per day - that's the equivalent of watching 16 movies! (source)

But what if there was a way to take control of all this information and use it to your advantage? Enter Personal Knowledge Management (PKM).

📔 What is Personal Knowledge Management?

(Source: RoamBrain.com)

PKM is the active and intentional process of managing the information that comes your way on a daily basis. It involves finding, organizing, and storing information in a way that makes it easy for you to access and use when you need it. It also involves using various tools and techniques to help you understand, analyze, and apply the information you've collected.

🤔 Why should you care about PKM?

(Source: jasongilbertson.com)

In this blog post, we'll explore the benefits of PKM and how it can help you create clarity, build a "second brain," and improve your efficiency and effectiveness. PKM is an ongoing process that requires discipline and effort to maintain, but it can provide significant benefits, including:

Connecting ideas for new insights. Atomic bi-directional notes allow you to connect ideas and make new insights possible. Insights don't happen in a vacuum - they're the result of making new connections.
Creating future value. Even if you don't use your notes for a current project, you're preparing knowledge for future projects. Just as data gives organizations an advantage, your personal repository of knowledge will give you a leg up over time.
Tackling complex problems. It can be tough to keep all the balls in the air when you're tackling complex problems. Implementing a PKM system allows you to focus on a small part of the problem and then step back and look at it with a bird's eye view.

🔢 "Levels" of P.K.M

Are you already a pro at taking notes? That's great, but there are different levels to this practice. You can evaluate where you are in your PKM journey using this guide from Tiago Forte's The 4 Levels of Personal Knowledge Management:

Level 1: Storing Information

You use apps to receive, edit, and send information.
You organize your files using default folders and subfolders.
You take notes on your smartphone or mobile device for practical tasks or occasionally from sources such as books or podcasts, but don't do much with them.

Level 2: Managing Knowledge

You capture ideas and creative inspiration from both your own thoughts and external sources.
You use digital note-taking as a significant part of your daily life, capturing information from a wide variety of sources.
Your notes begin to work as a thought partner, reminding you of things you've forgotten and surfacing connections between ideas.
You refine your knowledge management tools and perform small experiments to discover better ways of doing things.

Level 3: Enabling Action

You shift your focus from ideas to action, using your insights to tangibly improve your learning, health, career, business, and society.
You become more selective about the information you consume, preferring only high-quality, substantive sources that relate to your goals.
You use your notes to take on more complex projects and become more productive, creative, and relaxed.
You apply creativity to the workings of your system and integrate it deeply into your thinking for leverage.
The benefits of your system extend to others, such as through a website, blog, social media feed, podcast, or product.

Level 4: Personal Knowledge Mastery

You have achieved a high level of proficiency in managing your personal knowledge.
You are able to use your knowledge effectively to achieve your goals and are constantly learning and adapting to new information and situations.
You regularly review and revise your system to ensure it is still effective and efficient.
Your system becomes a part of your identity and personal brand, and you are able to share your knowledge and insights with others in a way that is meaningful and impactful.

🤓 Building Your PKM System

Now you may be thinking, alright, how do I level up my game? What do I need to do? First things first, you need the right tool for the job.

(Source: NoteApps.Info)

As you probably know, there are tons of apps in the market. Which one do you choose? Is there even a difference?

The app you choose is going to determine how successful you are going to be in organizing your knowledge. You don't want something super complex where all you think about the tool and not why you are using it and on the other hand you don't want something so simple that it lacks the essentials.

🧐 Understand Your Needs

In order to narrow down the search out of the sea of apps, I recommend thinking about a set of requirements that are important to you based on your usage and thinking patterns. For example, consider the following questions:

How do you prefer to take notes? Do you write in bullet points or prose?
Do you care about referencing other notes and seeing backlinks?
Do like the concept of a daily planner?
Do you care about consolidated notes, calendar, and task management?
Will you use it on the go and on multiple devices?

For me, thinking through some of those questions yielded the following list of requirements:

🔗 Requirement #1: Bi-directional - The ability to create connections between different blocks of knowledge and easily reference and backlink them.

📋 Requirement #2: Outline-based - The ability to organize thoughts and ideas in a hierarchical, bullet point format.

📅 Requirement #3: Temporal - The ability to take daily notes and keep a journal of events and activities.

✅ Requirement #4: Action Oriented - The ability to create and track tasks, ensuring that important actions don't fall through the cracks.

📲 Requirement #5: Multi-device syncing - Mobile support for on-the-go note taking as well as multi-device syncing.

🧪 Experiment

A good way to narrow down your options is to use a review site like NoteApps.Info to compare different tools and see which ones seem most promising.

Once you've identified a few tools that you'd like to try, it's a good idea to give each one a test run for a few days to see how it fits into your daily routine. Keep in mind that it may take some trial and error before you find a tool that feels productive and comfortable to use.

For example, I tried all of the following tools for a few days before finding one that worked for me:

Roam Research: This is the leader in the space with a cult following. It offers powerful bi-directional linking and hierarchical outlining capabilities. No free version.
Obsidian: Similar to Roam but with a ton of extensibility and plugin ecosystem. Multi-device syncing is a paid add-on.
RemNote: Similar to Roam but slightly bent towards students and educational use-cases.
logseq: Essentially an open-source version of Roam focused on privacy and data portability. Lacking robust multi-device syncing.
Amplenote: Most low-key out of the ones. Has the most essential features with robust multi-device syncing and mobile support. No plugins or ability to customize the UX. Generous free tier.

I ultimately decided to stick with Amplenote given it checked all my requirements and has proven to be super reliable for me.

💡 Strategies for Success

So now that you know the benefits of PKM, how do you get started and make it stick? Here are some strategies that can help you achieve success:

Start small and focus on one area at a time. It can be overwhelming to try to tackle everything at once. Start with one area of your life, such as taking meeting notes, remembering ideas, writing blog posts, planning a vacation, etc and build from there.
Find a tool that works for you. There are many different tools available for PKM, from simple note taking apps to more advanced knowledge graph apps. Take the time to experiment with different options and see what sticks.
Use your notes to reflect on your learning and progress. PKM isn't just about capturing and organizing information - it's also about making use of it to improve your learning and development. Take time to review and reflect on your notes regularly, and use them to identify patterns, make connections, and track your progress over time.
Be consistent and make PKM a habit. It takes time to build a strong PKM system, so be patient and consistent in your efforts. Make it a habit to capture and organize information on a regular basis.

📑 Additional Resources & Examples

How to Build a System for Lifelong Learning - I highly recommend this post by Jason Gilbertson where he takes you through his knowledge management system to serve as an inspiration.
Building a Second Brain - If you prefer something more guided and structured look no further than Tiago Forte's BASB course.
Andy's Notes - One of the most remarkable examples out there. Andy uses the concept of atomic, evergreen notes, and combines it with a fun way to navigate the content as you open up each note like a book. Each note title = hyperlink.
KasperZutterman/Second-Brain - A curated list of awesome Public Zettelkastens 🗄️ / Second Brains 🧠 / Digital Gardens 🌱

📣 Share The Knowledge

I hope you found this post helpful and thought-provoking. If you have any additional tips or strategies for effective Personal Knowledge Management, please share them in the comments below. And if you found this post valuable, please consider sharing it on social media or tagging a friend or colleague who might benefit from reading it.

How To Own Your Growth As A Software Engineer

Jesal Gadhia — Thu, 14 Jul 2022 02:24:53 +0000

An engineer's journey from a junior to a lead is full of learnings and requires frequent adaption and agility to succeed. But, as this Tweet mentions, what ends up needing most of the time and focus at each engineering level tends to be different.

// Detect dark theme var iframe = document.getElementById('tweet-1527665482391859200-609'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1527665482391859200&theme=dark" }

What got you here will not get you there.

The best and most successful engineers realize this and adapt to meet the need at the moment rather than clinging to what got them so far. They are what Liz Wiseman calls Impact Players — a book and a mental model I highly recommend.

In this post, I will take you through a mental model of thinking about your growth as an engineer, techniques to advocate for yourself, and advice on organizing your thoughts and communicating well.

The Rigor/Relevance Framework®

So the first question that you may have is, how should I operate? How do I know when to focus on learning, building, or something else?

Here comes a structured tool you can leverage called The Rigor/Relevance Framework®, developed by the International Center for Leadership in Education to examine curriculum, instruction, and assessment.

As you might guess, they typically use this framework in education settings to chart a student's growth of knowledge and how they apply it. It is an excellent lens to view an engineer's growth because great engineers are lifelong learners who follow a similar trajectory!

You can use this framework as a guide to track where you are now and where you might go next as a software engineer. We can examine the expectations at each level and how you can grow your impact and reach the next one.

Quadrant A (Acquisition) — Junior Engineer

You are an early stage engineer as an intern, your first job out of school, or switching industries.

Your manager will probably expect you to gather and store bits of knowledge and information. They will also expect you to remember or understand this acquired knowledge. Your team mentor, manager, or tech lead will guide you in completing your tasks.

You can exceed expectations if you understand and leverage existing patterns & practices, as well as execute tasks with minimal guidance. For more, see Samuel Taylor's brilliant advice on how to join a team and learn a codebase.

Quadrant B (Application) — Mid-Level Engineer

You are a mid-career engineer with a few years of experience and maybe in your second or third job as an engineer.

Your manager will expect you to use acquired knowledge to solve problems, design solutions, and complete work more or less independently.

You can exceed expectations if you can quickly ramp up and contribute to a new team/domain and apply your knowledge to new and unpredictable situations.

Quadrant C (Assimilation) — Senior Engineer

You are a veteran engineer who has worked in the industry for about a decade. You are the driver of your work streams and a mentor to others.

Your manager will expect you to extend and refine your acquired knowledge to expertly and routinely analyze and solve problems and create unique solutions.

You can exceed expectations by consistently stepping into a leadership role and helping your team take complex projects to the finish line by working closely with stakeholders, identifying solutions, defining the scope, estimating timelines, and measuring success. You know how to be what Keith Rabois calls a Barrel.

Quadrant D (Adaptation) — Staff+ Engineer

You are a technical craftsperson and have been in leadership roles in the industry. You typically lead teams and companies on strategic and mission-critical initiatives. In addition, you are a compassionate mentor that can guide individuals and teams to achieve greater success.

Your manager will expect you to think in complex ways and apply your acquired knowledge and skills. However, even when confronted with perplexing unknowns, you can use your experience to create solutions and take action that further develops your skills.

You can exceed expectations by practicing imagination, prioritizing the right things at the right time, and executing your vision by inspiring and leading others. See Will Larson's guides for reaching and succeeding at Staff-plus roles for a deep dive.

The map is not the territory.

Organizations essentially split those four quadrants above into different levels at varying granularity, but the core concepts usually remain the same. Therefore, you can think of the above levels as a map for your career progression.

Although these leveling guides are helpful, it is essential to remember that even the best maps are imperfect. In mathematician Alfred Korzybski's words, the map is not the territory. That's because they are reductions of what they represent. If a map were to describe the territory with perfect fidelity, it would no longer be a reduction and thus would no longer be helpful to us.

That is all to say, don't read the leveling guides too literally or get fixated on ticking the boxes. Instead, take the time to explore your interests, learn and savor the journey. Personally, I took several detours along the way, such as pursuing entrepreneurship and switching back and forth from IC to management to maximize my learnings & impact rather than worrying about titles or showing linear growth on my resume. For more, see Charity Majors' thoughts on engineering levels.

Also, it's essential to remember that titles are usually a byproduct of all your achievements. They are a lagging indicator of your impact. Therefore, you should never wait for titles to work on your skills and find your flow state. Psychologist Mihaly Csikszentmihalyi describes the flow state as:

a constant balancing act between anxiety, where the difficulty is too high for the person's skill, and boredom, where the difficulty is too low

Do you have the right opportunities, mindsets, and support to achieve that flow state and create a greater impact? If not, what needs to change?

Be your advocate

Following and meeting the expectations of a leveling guide alone sometimes is not enough. It's also crucial that you become your advocate and authentically tell your story. For one, make sure you have a way to remember, celebrate, share, and learn from all your accomplishments.

Reflecting on your journey will help you spot trends, key strengths, and areas of improvement. I've found it easier to jot down my thoughts right after passing a milestone, such as at the end of the quarter or after completing a project. I've also found it helpful to track my time using tools like Toggl, RescueTime, etc., to understand and visualize my time pie.

You can use those reflections as a springboard to share your achievements & insights more broadly. I know that may not always come naturally to some engineers, but putting yourself out there will help others, amplify your work and unlock new opportunities. It is also a great opportunity to practice gratitude and recognize others around you.

Related to the points above, see Julia Evan's excellent blog post on how to get your work recognized and Jason Roberts's blog post on increasing your luck surface area.

Communicate well

Another critical skill often overlooked is an engineer's ability to write and communicate. Make sure you invest in that skill as early as possible in your career. It will pay long-term dividends unlike any other. In fact, your ability to advocate for yourself hinges on your ability to communicate well.

Clear and concise communication will also help you think through complex concepts more clearly. It is also a huge plus when working at a company that values written communication, such as Amazon's, with a culture of six-page memos, which Jeff Bezos talked about in one of their shareholder letters.

To build this skill gradually, I've used Personal Knowledge Management (PKM) tools, such as Notion, Roam, Remote, etc., to create a daily writing practice and a way to organize thoughts. You can then leverage those raw thoughts and outlines to write polished documents. Set a goal for yourself to write a certain amount of external blog posts or internal memos. Start small, maybe 2 - 3 per year, and slowly increase the cadence.

For more on writing well, check out Heinrich Hartmann's Writing for Engineers guide and Michael A. Covington's presentation on How to Write More Clearly, Think More Clearly, and Learn Complex Material More Easily.

Final thoughts

To recap, the core tenets of owning your growth as an engineer covered here were:

What got you here will not get you there; do the job that's needed of you
Understand what it'll take to exceed expectations and have a plan to outperform
Advocate for yourself through storytelling and sharing your work
Learn to organize your thoughts and communicate articulately

I would love to hear if that resonates and if there is any other advice you've found to be pivotal in your growth journey.

This post was originally published on the BetterUp Product Blog.

The Art of Crafting Effective Pull Requests

Jesal Gadhia — Sun, 08 Aug 2021 15:34:57 +0000

Pull Requests (PRs) are a great way of submitting contributions to a project, especially when there are multiple developers working on it at the same time. If done well, they are a medium through which we can receive feedback and increase visibility for the changes we are shipping.

There are many elements to what makes for a good PR. They end up determining the quality of the end deliverable as well as telling a story of how a certain feature or change came about.

In this post, I'll cover some of the basics of what to optimize for at each stage of the PR's journey to production. My hope is that you'll be able to use this to maximize feedback quality and minimizing time to merge. So let's dive in.

Scoping

The first step in the journey to crafting a PR is to create a separate branch for your changes. At this stage, scoping your changes is key. First, think of how big of a change-set you want to introduce in a single PR. Try to address one issue or build one feature within the PR. The larger it is, the more complex it is to review and the more likely it will be delayed. Remember that reviewing PRs is taking time from someone else's day!

Once you've settled on the scope, ensure that each change is broken up into small logical commits. Having small commits is valuable because of several reasons. First, it allows the reviewer to see the progression of the changes and understand the PR in its current state. It is also very valuable if you are using git bisect to debug an issue or trying to roll back a change in the future.

Drafting

The next step in the journey is to open up a draft PR. This is not always necessary but it's useful if you want to see the full CI run with all the tests, assuming you are not running the entire test suite locally. This will give you early signals on whether your changes are interacting with the rest of the codebase in unexpected ways or not.

Take a moment to review your changes and see if everything makes sense from the perspective of the reviewer. If you are satisfied with the state of the PR, you could tag reviewers for early feedback if necessary.

If you do decide to tag reviewers, ask for specific feedback, calling out the fact that the PR is still work-in-progress. Guide your reviewers with what you want feedback on. This will ensure they don't spend time and energy giving you feedback on things that are likely to change, or things that you are already aware of.

Publishing

If your draft PR is in good shape, you are now ready to tag reviewers. When it comes to picking reviewers, pick no more than 2 - 3 primary reviewers. If you tag a large number of reviewers, chances are you might see a bystander effect.

Before tagging reviewers, ensure that:

Tests are all green. You don’t want the reviewers to spend their valuable time pointing out the basics.
You've clearly articulated the purpose of the PR as well as the broader context for the change. Link back to relevant Slack threads or tickets. This is important even if you think the reviewers will have the full context for the sake of posterity.
You've done a self-review. I would recommend running through Maslow's pyramid of code review to see if your work meets those criteria. You should be the first person to critically review your work before anyone else does.
If there are specific follow-ups that you have in mind, describe those as well so the reviewers know what to expect down the line.
You have provided related artifacts to the PR in the form of screenshots, videos or logs. This will help your reviewers better understand the impact of your changes.

Feedback

When you receive feedback from the reviewers keep in mind:

Tone can easily get lost in async communication. So assume the best intent and don't take the feedback personally. Offer clarification and context for your decisions.
If something isn't clear, ask clarifying questions, don't try to read the tea leaves. If things are still not clear in async conversation, request some time to chat on a call. Once you’ve had a chance to discuss, circle back and summarize the discussed points in the PR to preserve context.
Whether you decide to take the feedback or not, be sure to acknowledge the feedback you've received. This demonstrates that you value the reviewers' time and effort.
Explicitly re-request review once you are done addressing the feedback. Don't assume the reviewer will know when you are done. Bonus points if you follow-up with a comment pointing out the specific commit that addressed their feedback.

Releasing

Once you have gone through the feedback cycles and gotten the approval of reviewers remember to come back and merge your PR. The longer you wait, the greater the chances of encountering merge conflicts or regressions. So be sure to merge as quickly as possible.

If you are releasing a PR that will perform a large data operation or change something in the critical path, ensure that you are available to follow that PR through the release process and mitigate any errors or hiccups along the way.

If all goes well and your changes are in production, remember to follow-up with the stakeholders and update them on the progress.

This post was originally published on the BetterUp Product Blog.

Techniques for Effective Software Development Effort Estimation

Jesal Gadhia — Sun, 16 May 2021 19:31:15 +0000

How long do you think it'll take?

As builders and creative people, we are all too familiar with that question. Getting estimates right is incredibly difficult and it's a skill that we learn slowly over time as we gain more experience building and shipping projects.

So why is this simple exercise so difficult? Oftentimes it's because we forget to ask the right questions and make assumptions that may not be correct. Let's examine what are these questions that we should be asking and break them down into phases.

Scoping — What is being requested and when is it needed?

Don't assume what you think of as “done” is the same as what the party asking for an estimate would call, “done”. It is important to explicitly call out the timeline and specific deliverables before doing the exercise of estimation.

So part of that is first understanding what you are being asked to estimate. Make sure what you have in mind is an acceptable outcome for the stakeholder. If you don't already have it, make a list of user personas and stories to align on the requirements with the stakeholder and decide on what will be in scope.

Secondly, understand the user group that should be targeted as part of the delivery timeline. For example, will the product be shipped in phases such as internal, friends & family, early access, general availability, etc? If so what does our estimate aim for? Be explicit about which release phase you are estimating for.

Technical Exploration — How will it be built?

To provide a good estimate there has to be some level of understanding of the existing system and how to go about making changes in it.

You can never know exactly all the steps you may need to take but there has to be a certain degree of confidence. Anything below 70% confidence would warrant a technical exploration or a spike to get a better understanding of the required effort.

If you are going to touch a particular aspect of the system, take the opportunity to leave it in a better state than you found it in. This is a good time to identify if there are any long-standing hotspots or technical debt that could be addressed as part of this task. Even small incremental improvements will help keep the system maintainable over the long run.

Capacity Planning — What is the level of effort?

The next step in formulating an estimate is to get a handle on the capacity. For example, based on the technical exploration, you may think something might require one week of effort. This is the most common step where the estimation effort derails.

We are not done. We still have to further refine that and ask, “Is it one week of an average engineer's time? Or is it specifically your time?”

If you are estimating for yourself, does that account for all the meetings you have to attend? Are there any holidays coming up? Do you have any other competing priorities or commitments? Estimate the time you may have to focus on those things and add that to the estimate.

Also, does this account for time to deal with any potential hiccups or areas of high ambiguity that you may still have to be fleshed out? Figure out your confidence level after the technical exploration, then account for some additional time based on the percent of ambiguity that remains. It might be helpful to go through a Risks & Mitigations exercise here where you can list out all the areas of risks and potential actions to mitigate them.

Deployment Process — What will it take to be deployed?

We are in the final stretch, but the exercise is not over yet! Now that we are getting a better handle on the actual engineering time, let's start to think about the process of shipping the work.

What are the review phases that you will encounter? Will the changes have to go through peer reviews? If so, what kind of cycle time can we expect from the reviewers? There may be a review/feedback cycle to each change that is shipped. How long will that take approximately?

Will there be any other reviews outside of peer reviews? Will this have to go through a design review to ensure the final product matches the designs? Will it have to go through any compliance audits such as Privacy, Security, Legal, etc? Try to gather the rough turnaround time for those.

Final Thoughts

Providing time-based estimates is always hard. It involves a slew of factors and varies from person to person. That's one of the reasons why many teams have transitioned to a practice of assigning points to a task known as "story points", which is an agile development practice.

Instead of time spent, the team would assign a relative complexity of a task on a point scale. The scale can be anything from Fibonacci sequence to t-shirt sizes. Over time, the team builds a better understanding of how their story points map to difficulty of a task which in turn can be used to inform timelines.

No matter which framework you decide to use, like developing any skill, estimation requires continuous practice, refinement, and learning. You'll get better and better if you treat it as a skill that can be developed over time.

This post was originally published on the BetterUp Product Blog.

Merging SimpleCov results with parallel Rails specs on Semaphore CI

Jesal Gadhia — Tue, 03 Nov 2020 23:58:13 +0000

If you are running Rails specs on parallel machines with Knapsack Pro, one challenge you will run into is combining the code coverage results generated by SimpleCov.

This post will show you how to generate a report of the total combined code coverage after all the tests have executed. Here's what the pipeline diagram looks like at a high level with unrelated sections blurred out:

Configure

SimpleCov

First lets look at the config file used for SimpleCov below. Note no minimum_coverage configuration for failing the build. This is because each node most likely will not meet the minimum coverage threshold on its own so it could lead to the build failing erroneously.

Also note, before_queue hook for Knapsack Pro. This is the important piece, it will set a command name based on the CI node index so that the results are recorded against it.

.simplecov:

  SimpleCov.start do
    add_filter %r{^/config/}
    add_filter %r{^/db/}
    add_filter %r{^/spec/}

    add_group 'Admin', 'app/admin'
    add_group 'Controllers', 'app/controllers'
    add_group 'Helpers', 'app/helpers'
    add_group 'Jobs', 'app/jobs'
    add_group 'Libraries', 'lib/'
    add_group 'Mailers', 'app/mailers'
    add_group 'Models', 'app/models'
    add_group 'Policies', 'app/policies'
    add_group 'Serializers', 'app/serializers'
  end
  Rails.application.eager_load!

  KnapsackPro::Hooks::Queue.before_queue do
    SimpleCov.command_name("rspec_ci_node_#{KnapsackPro::Config::Env.ci_node_index}")
  end

So now when SimpleCov creates a .resultset.json it will have a specific key depending on which CI node it was run in like the example below. This will be useful down the line when it comes to combining the results.

{
  "rspec_ci_node_0": {
    "coverage": { ... }
  }
}

{
  "rspec_ci_node_1": {
    "coverage": { ... }
  }
}
...

Semaphore CI

Below is the relevant portions of the Semaphore CI configuration. It runs the Rails tests and then uploads the coverage results as a Semaphore workflow artifact. After all the parallel tests have completed, it will run a job to collate the coverage results from all the machines.

.semaphore/semaphore.yml:

- name: "Rails Tests"
    task:
      jobs:
        - name: Rails
          parallelism: 10
          commands:
            - ./.semaphore/helpers/rails_tests.sh
      epilogue:
        always:
          commands:
            - ./.semaphore/helpers/upload_test_artifacts.sh
      secrets:
        - name: docker-hub
        - name: knapsack-pro-rails

- name: "Code Coverage"
  dependencies: ["Rails Tests"]
  task:
    env_vars:
      - name: SEMAPHORE_RAILS_JOB_COUNT
        value: "10"
    jobs:
      - name: Collate Results
        commands:
          - ./.semaphore/helpers/calc_code_coverage.sh
    secrets:
      - name: docker-hub

Below is the bash file which executes the Rails tests on each parallel machine. It sets up the Rails environment and then runs Knapsack Pro in Queue Mode.

.semaphore/helpers/rails_tests.sh:

#!/bin/bash
set -euo pipefail

docker-compose -f docker-compose.semaphore.yml --no-ansi run \
  -e KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true \
  -e KNAPSACK_PRO_TEST_FILE_PATTERN="spec/**/*_spec.rb" \
  ci bash -c "bin/rake ci:setup db:create db:structure:load knapsack_pro:queue:rspec['--no-color --format progress --format RspecJunitFormatter --out tmp/rspec-junit/rspec.xml']"

This is the bash file which is responsible for uploading the SimpleCov results from each machine. It compresses the coverage directory and uploads it to Semaphore.

.semaphore/helpers/upload_test_artifacts.sh:

if [ -d "tmp/rspec-junit" ]
then
  echo "Pushing rspec junit results"
  artifact push job tmp/rspec-junit --destination semaphore/test-results/
fi

if [ -d "coverage" ]
then
  echo "Pushing simplecov results"
  tar czf coverage_${SEMAPHORE_JOB_INDEX}.tgz -C coverage .
  artifact push workflow coverage_${SEMAPHORE_JOB_INDEX}.tgz
fi

Lastly, this is the bash file for collating all the results from Semaphore. It will download the coverage artifacts from each parallel machine and run a rake task which will collate them and then upload the results into a combined total_coverage.tgz file as shown below:

.semaphore/helpers/calc_code_coverage.sh:

#!/bin/bash
set -euo pipefail

for i in $(eval echo "{1..$SEMAPHORE_RAILS_JOB_COUNT}"); do
  artifact pull workflow coverage_$i.tgz;
  mkdir coverage_$i
  tar -xzf coverage_$i.tgz -C coverage_$i
done

docker-compose -f docker-compose.semaphore.yml --no-ansi run ci bash -c "bin/rake coverage:report"
tar czf total_coverage.tgz -C coverage .
artifact push workflow total_coverage.tgz

This coverage:report rake task will simply call SimpleCov.collate which will go through the coverage results in each folder and combine them into a single .resultset.json shown below

lib/task/coverage_report.rake:

namespace :coverage do
  desc 'Collates all result sets generated by the different test runners'
  task report: :environment do
    require 'simplecov'

    SimpleCov.collate Dir['coverage_*/.resultset.json']
  end
end

.resultset.json

{
  "rspec_ci_node_0, rspec_ci_node_1, rspec_ci_node_2, rspec_ci_node_3, rspec_ci_node_4, rspec_ci_node_5, rspec_ci_node_6, rspec_ci_node_7, rspec_ci_node_8, rspec_ci_node_9": {
    "coverage": { ... }
}

Summary

Finally here's what your Semaphore workflow artifacts will look like. It will have a compressed coverage file generated on each machine and a total coverage file that we created at the very end:

This approach can also be easily ported over to other CI providers by simply changing the artifact push and artifact pull commands to S3 or another CI specific artifact upload command.

I hope this article was useful to you. Let me know if you have any questions or feedback.

This post was originally published on my blog. If you liked this post, please share it on social media and follow me on Twitter!