DEV Community: Yaw Joseph Etse

Embracing Followership and Technical Fluency in Modern Leadership

Yaw Joseph Etse — Thu, 21 Mar 2024 11:54:03 +0000

It’s always fun when extremely talented engineers bring up the topic of continuing down an individual contributor path versus entering formally into people leadership.

I recently had the chance to talk about the usual trade-offs in conversations about people leadership vs individual contributions. Topics like, do you think you can effectively lead through others, or you’re a brilliant engineer are you sure you want to manage people instead of doubling down on your expertise?

Naturally, The conversation generally shifts to an equally fascinating topic: quantifying effectiveness. For individual contributors, effectiveness can be measured in various ways, such as algorithmic efficiency or the robustness of systems and platforms they develop.

In contrast, measuring performance and effectiveness in leadership introduces a different kind of subjectivity. It tends to lead into an opportunity where I quote from a concept from my favorite leadership book (at this point I should use an Amazon federal link, hah), that followership is a useful measure of leadership effectiveness because it’s both simple and quantifiable.

Consider this: how many people you lead would choose to follow you if you switched teams, organizations, or even companies? This question underpins the argument in Nine Lies About Work, where the author challenges the conventional wisdom of universally definable and measurable leadership. Instead, the book suggests that “followership” is the real measure of leadership effectiveness, proposing that the essence of leadership is not a singular trait or capability.

Coincidentally, our friends at Quotient recently analyzed a Microsoft study that offers complementary insights, identifying key attributes of effective engineering managers from both managers’ and engineers’ perspectives.

These attributes include fostering a positive work environment, enabling autonomy, and nurturing talent. Notably, the study suggests de-emphasizing technical expertise in favor of qualities that promote team cohesion, psychological safety, and personal growth, reflecting a shift in how leadership effectiveness is perceived in the technical domain.

However, I have reservations about the interpretations presented in the Microsoft research paper, especially the assertion that effective managers place less emphasis on technical leadership. This discrepancy might stem from the definition of ‘technical’ prowess, particularly within software and machine learning engineering.
From my experience, there’s a correlation between effective leadership and technical acumen. However, I’d reinterpret ‘technical’ abilities to include the capacity to engage in multiple “languages of abstraction.”

Exceptional leadership, especially in technical disciplines, involves the versatility to engage scientifically with data and empirical evidence, mathematically to argue points with precision, and logically to construct cogent arguments by effortlessly forming valid arguments (any casual reference to using modus ponens or hypothetical syllogism is a win). This abstract multi-lingual fluency enables leaders to connect with their teams on multiple intellectual levels, fostering a deeper appreciation for the leader’s technical capabilities.

I think effective leadership within most engineering domains, hinges as much on technical fluency as it does on vision, empathy, and adaptability. It’s this synthesis of skills and the ability to create an environment where folk would happily follow — that distinguishes truly effective leaders.

Use Cases Define the Choice of Privacy-Enhancing Technologies

Yaw Joseph Etse — Fri, 19 Jan 2024 04:24:37 +0000

I love the recent papers taking a deeper dive into synthetic data that highlight how a one-size-fits-all approach to data access and privacy strategies is a mirage. This is best highlighted from the paper "On the Inadequacy of Similarity-based Privacy Metrics: Reconstruction Attacks against 'Truly Anonymous Synthetic Data'" by Georgi Ganev and Emiliano De Cristofaro. I would consider the paper required reading for anyone who is seriously working with synthetic data and was brought to my attention through Damien Desfontaines' amazing post, which effectively summarizes the key insights.

"Synthetic data, when not underpinned by robust privacy guarantees like Differential Privacy (DP), can lead to significant privacy breaches, especially concerning outliers" (Ganev & De Cristofaro, 2023). The quote is a great call out and highlights the challenges for real-world and product environment usage of synthetic data for various applications.

The paper highlights that most synthetic data products claim compliance with regulations like GDPR, HIPAA, or CCPA, yet rarely use DP. Instead, many companies use empirical heuristics to ensure privacy, which can break the end-to-end DP pipeline and negate its privacy protections.

The authors identify major disadvantages of commonly used privacy metrics and filters. They introduce a novel reconstruction attack, ReconSyn, which exposes the vulnerabilities of these metrics. ReconSyn recovers at least 78% of underrepresented train data records (outliers) with perfect precision across various models and datasets.

The paper identifies eight major issues with using similarity-based privacy metrics (SBPMs), including the lack of theoretical guarantees, treating privacy as a binary property, and the absence of worst-case analysis. These limitations present severe vulnerabilities to privacy attacks.

Synthetic data's appeal lies in its presumed privacy and utility, especially for software and model testing by creating a safe playground without exposing sensitive real-world data. However, synthetic data can be less useful in deep analytics and model training. The paper highlights that synthetic data generated from highly sensitive information often falls short of providing adequate privacy unless it incorporates stringent privacy-preserving methods like DP.

This distinction leads us to a broader narrative in the realm of privacy-enhancing technologies (PETs). It’s often not practical to choose a single PET, such as trusted execution environments or homomorphic encryption, because in practice they all have their ideal use cases. The intent behind their use should be the primary driver of technology decisions. Sometimes, synthetic data suffices; other times, more robust controls are necessary.

The intent and use case should always be at the forefront of any data protection strategy. For instance, safeguarding highly sensitive data in scenarios where accuracy is paramount might call for homomorphic encryption or DP. Conversely, in scenarios with lower privacy risks or less sensitive data contexts, synthetic data could be a viable and efficient option.

Ganev and De Cristofaro advocate for a nuanced approach to data privacy, stating, "A critical examination of current privacy metrics and the adoption of empirically driven methods is essential for ensuring real privacy in synthetic data generation" (Ganev & De Cristofaro, 2023).

This perspective should inform the selection of appropriate PETs based on specific use cases, balancing the dual demands of utility and privacy.

References:

Ganev, G., & De Cristofaro, E. (2023). On the Inadequacy of Similarity-based Privacy Metrics: Reconstruction Attacks against 'Truly Anonymous Synthetic Data'. - https://arxiv.org/abs/2312.05114v1

Balancing Innovation and Privacy: Navigating LLM Augmentation with RAG and RA-DIT

Yaw Joseph Etse — Fri, 12 Jan 2024 11:27:29 +0000

The advancements in Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Dual Instruction Tuning (RA-DIT) are compelling, especially their implications for intellectual property, data privacy and data governance. These generative model augmentation techniques prompt important data security and privacy-related questions in most enterprise settings.

While RAG merges retrieval-based and generative models to improve text retrieval and generation, suitable for tasks like text summarization and content creation (https://spotintelligence.com/2023/10/19/retrieval-augmented-generation-rag/) RA-DIT, on the other hand, adds retrieval capabilities to LLMs through two fine-tuning stages, refining both the model's use of retrieved data and the retriever's relevance (https://ar5iv.labs.arxiv.org/html/2310.01352).

The two methods differ fundamentally. RAG integrates retrieval with generation, potentially raising privacy concerns in cases where sensitive data is involved. RA-DIT focuses on upgrading existing models for better retrieval, possibly offering more control over sensitive data.

Integrating privacy-enhancing technologies (PETs) like differential privacy and homomorphic encryption could transform LLM use in privacy-sensitive areas.

However, selecting the right PET depends on a sound data production and consumption strategy. For example Differential privacy could potentially obscure individual data points in both RAG and RA-DIT processes, while homomorphic encryption would allow for computations on encrypted data, keeping sensitive information secure.

Other considerations should include how data will be shared externally, how the data will be used internally, the importance of data accuracy, and whether the challenges are related to modeling and analysis or software engineering. Data access patterns and user expectations, whether they require static data dumps or API interfacing, could aLos influence the PET application strategy.

While these technologies promise enhanced LLM accuracy and context-rich outputs, balancing technical and ethical considerations is crucial. Differential privacy might reduce accuracy, and homomorphic encryption could increase computational demands.

Understanding how RAG and RA-DIT, coupled with PETs, can create a secure, well governed LLM framework and implementation strategy. This approach could improve LLMs in sensitive sectors and accelerate AI adoption, with data privacy and intellectual property as a foundational design element. The potential of these technologies in enhancing auditability, explainability, and governance in LLMs, particularly when privacy is central, is immense.

Leaking sensitive data via membership inference attacks on machine learning models

Yaw Joseph Etse — Fri, 12 Jan 2024 11:25:30 +0000

The paper, "Membership Inference Attacks against Machine Learning Models," shows how easy it is to expose how models can inadvertently leak sensitive information about the data they were trained on.

Leaking via membership inference attacks (essentially allow an adversary to determine if a particular data record was used in the training set of a machine learning model) is another reason why the work in the differential privacy space is so interesting and valuable. It’s extremely common to deal with confidential and sensitive data in most enterprise settings, and the assurance that this data cannot be reverse-engineered or exposed is critical.

The authors of the paper does a nice job walking through multiple experiments and evaluations, demonstrating that machine learning models, especially those that are overfitted, are susceptible to these types of attacks. The authors show that the models behave differently when queried with data they have seen before, compared to unseen data. This difference in behavior can be exploited to infer membership information.

A key contribution of the paper is the introduction of a novel technique called "shadow training." This technique involves training models (referred to as shadow models) to mimic the behavior of the target model, using data that is similar to the target model’s training data. The shadow models are then used to generate a dataset for training an attack model, which learns to distinguish between the target model’s outputs on its training and test data. This attack model can then be used to infer membership information about new data records.

The implications of this are far-reaching. Whenever the usefulness of privacy enhancing technologies comes up, it’s appropriate to raise the risks associated with membership inference attacks and take steps to mitigate these risks. This includes being mindful of the trade-offs between model accuracy and vulnerability to such attacks, and implementing strategies to prevent overfitting.

Some questions that arise from this research, and that I believe warrant further exploration, include:

How can we effectively measure the susceptibility of our models to membership inference attacks?
What are the best practices for implementing shadow training in a real-world scenario, and how can we ensure its effectiveness?
Are there specific types of data or model architectures that are more prone to these attacks, and how can we safeguard against this?

I would be interested in engaging with more researchers who are currently delving deeper into these questions and to explore potential collaborative efforts to address these vulnerabilities.

Rethinking Performance Management - Embracing Subjectivity Over Objectivity

Yaw Joseph Etse — Fri, 12 Jan 2024 04:46:08 +0000

I was recently asked about my thoughts on performance management and to share the resources I find helpful. So, as we wrap up another performance management (PM) season, I'd like to reiterate what I've mentioned privately: it’s crucial to reflect on the inherent shortcomings of most PM processes, especially as they pertain to engineering culture. The key point is the importance of acknowledging and embracing the subjectivity at the heart of performance management.

One of the most compelling arguments comes from "Nine Lies About Work" by Marcus Buckingham and Ashley Goodall. They assert, "People can reliably rate their own experience" but are notably unreliable at rating others. They explain, "Your rating of a team member on something called 'performance' is unreliable because your definition of performance is unique to you" (Buckingham & Goodall, 2019). This challenges traditional, often rigid, performance evaluation models and suggests a more introspective approach.

Similarly, "No Rules Rules" by Reed Hastings and Erin Meyer presents the intriguing concept of "The Keeper Test." It's a straightforward yet powerful tool: "If a person on your team were to quit tomorrow, would you try to change their mind? Or would you accept their resignation, perhaps with a little relief? If the latter, you should give them a severance package now, and look for a star, someone you would fight to keep" (Hastings & Meyer, 2020). This approach simplifies performance assessment to a singular, yet profound question, focusing on the real value an individual brings to the team.

"Working Backwards" by Colin Bryar and Bill Carr, though not directly addressing PM, offers valuable insights into creating environments where performance is naturally high. Amazon’s emphasis on clarity, autonomy, and direct, candid communication is a testament to creating a culture where high performance is a byproduct of the work environment itself.

In most large corporate settings managing low performance consumes disproportionate time, energy, and resources. The goal should be to cultivate high-performing teams where excellence is the norm, not the exception.

As Quotient's Research-Driven Engineering Leadership aptly notes, measuring engineering productivity requires a nuanced understanding of what performance means in a technical and creative domain (RDEL, 2023).

In conclusion, embracing the subjectivity of performance management can lead to more genuine, effective, and meaningful assessments. By focusing on individual experiences, aligning personal goals with team missions, and empowering team members with context and autonomy, organizations can create a dynamic, responsive, and ultimately more effective approach to managing performance.

References:

Buckingham, M., & Goodall, A. (2019). Nine Lies About Work: A Freethinking Leader's Guide to the Real World.
Hastings, R., & Meyer, E. (2020). No Rules Rules: Netflix and the Culture of Reinvention.
Bryar, C., & Carr, B. (2021). Working Backwards: Insights, Stories, and Secrets from Inside Amazon.
RDEL. (2023). Research-Driven Engineering Leadership: The Most Popular Posts of the Year. Blog Post.

Improving Engineering Culture by Building Products, Not Projects

Yaw Joseph Etse — Fri, 12 Jan 2024 04:42:03 +0000

I love this tweet thread from Paweł Huryn about how projects are not products. Especially when building software, a strong engineering culture is often the cornerstone of building great software. However, what is less understood is the crucial role of product management in fostering this culture. Distinguishing product management from project management is key, as the former is foundational in nurturing an environment where engineers thrive.

The Pitfalls of Mistaking Projects for Products

Many initiatives, labeled as 'products,' are in reality projects in disguise. This mislabeling is a red flag and can derail the potential of an engineering team. A project-centric approach starts with a comprehensive Product Requirements Document (PRD), focuses on meeting these requirements, and often lacks the involvement of a UX Designer. The roadmap is driven by timelines and features, rather than outcomes and innovation.

This approach misses out on essential aspects of product development:

Limited Exploration: When the goal is merely to implement listed features, there's little room for innovation.
Absence of Testing: The lack of emphasis on testing ideas before implementation can lead to misaligned products.
Lack of Strategy: Without a clear product strategy, efforts become dispersed, trying to please all customers, which dilutes the product's essence.
Insufficient Analytics: The absence of product analytics means being blind to user interactions and needs.
Transactional Relationships: Viewing the work as a customer-vendor transaction, whether internally or externally, restricts the creative and collaborative potential of the team.

Cultivating a Product-Centric Engineering Culture

A product-centric approach pivots around a cross-functional team empowered to solve problems, not just implement solutions. The collaboration between Product Managers (PMs), Designers, and Engineers in product discovery is continuous and fluid.

Key characteristics of a product-centric approach include:

Empowerment and Collaboration: Teams are empowered to find the best solutions, with PMs, Designers, and Engineers working in unison.

Outcome-Based Roadmap: The focus shifts to outcomes rather than just outputs, with a roadmap that adapts and evolves.
Risk Management: Managing and mitigating value, usability, feasibility, and viability risks becomes a priority.
Incremental Shipping and Learning: Products are shipped incrementally, allowing for real-time learning and adjustments.
Strategic Tradeoffs: Understanding and making strategic tradeoffs highlights the importance of a unique value proposition and market understanding.

Pre-Launch Strategies for a Product-Centric Approach

Before launching a product, defining the market, value proposition, business model, and initial strategy is crucial. Testing the concept with an MVP prototype, defining a go-to-market strategy, and relying on customer interviews and behavioral data are all part of this phase.

To build a thriving engineering culture, it's essential to shift from a project management to a product management mindset. This shift not only enhances the product's market fit and innovation but also fosters an environment where engineers feel more engaged and empowered. The essence lies in understanding that building a product is not just about ticking off a list of features but about continuously exploring, learning, and adapting to create something truly valuable.