DEV Community: Semernitskaya

7 myths about Software Engineer work in 2022

Semernitskaya — Tue, 31 May 2022 19:16:04 +0000

In my article, I would like to discuss some myths about software engineer work that are considered to be true and can affect the confidence and self-esteem of people who are just starting their careers.

Disclaimer

This list of myths is only my opinion and is based mainly on my experience and stories of engineers I know. Of course, we all have different backgrounds and career paths, so it's normal if my opinion is different from yours.
The main goal of this article is to show people who are at the begging of their career in IT that some of their concerns/insecurities about themself are not true. Because the target audience for my article is beginners - some of my sentences can be pretty obvious to more experienced developers.
The article's header contains the year (2022) to emphasize that some of my ideas do not apply to the past and probably won't be valid in the future (who knows?)

1. You don't need communication skills

It's a common myth popularized by many movies about hackers: a software engineer is a recluse, who works alone, hates people, and is definitely a tech genius (we'll talk about it later). But the truth is that for many roles in IT companies nowadays you have to communicate with other human beings: discuss tasks and possible solutions with your team, collect business requirements, negotiate terms and priorities.
The good news is that there are various roles and ways of working in different companies and even in different teams inside the same company, so you will be able to find a position that matches your level of sociability.

2. It's a fancy job

We all have seen inspiring photos of IT specialists working with their MacBooks in cozy cafes. We all have heard about big salaries, flexible hours and other perks. Is this all true? Yes, but it's challenging to get all these preferences when you are just starting your career and don't have significant experience and many projects in your CV. Sometimes you can get only some of these perks but not the others.
If you are just at the beginning of your career journey, it's important to remember that it's completely normal to change companies and look for a perfect fit for you (money-wise, culture-wise, etc).

3. You must love math

I love math, you probably love it too, or probably you hate math. In both cases, you can be a good software developer.
Let me explain: it's well known that Computer Science is closely connected with math, but let's be honest, most software engineers don't use math in their day to day work; in addition, some companies can have separate departments that create and improve different math models and algorithms used in the company. So if you are good at math - that will be your strength for sure, but if you are not - there are many positions where you can work successfully without it.

4. You must know algorithms (very well)

Algorithm interview questions are a super controversial topic that causes a lot of debates among engineers. Some people argue that you can't be a good software developer without knowing all algorithms from Cormen's book, other think that algorithms are useless in day-to-day engineering work most of the time and how dare someone asks to rotate a binary tree during the interview.
The truth, as usual, lies somewhere in the middle: there are some positions where a good knowledge of algorithms and their optimization are essential, and there are other job roles where you need entirely different skills to succeed. Usually, it's enough to have a basic understanding of different data structures and the complexity of some common algorithms.
It is also important to remember that most of the algorithms are already implemented in libraries for many programming languages. So if, for example, you need to find the shortest path in a graph - you should start by searching for a library that allows you to represent your data in the form of a graph and has all the algorithms you need.

5. Your work has to be your hobby

There are many posts from people who are writing their own mobile apps or other pet-projects in their free time as a hobby. Sometimes such side projects can become successful businesses and primary income sources for the developers. However, even if you can't earn money from your pet-projects - they are always an excellent way to study something and try technologies you want to try.
But your pet-projects or other studying activities should not necessarily be your hobby and things you do to relax and have a rest. If you need to spend time with your family or walk with your dog in the park or binge watch some crazy anime series on Netflix (guilty) to feel rested - just do it. You can always have dedicated time that you spend on self-development, reading tech articles and learning new frameworks.
Do not try to convince yourself that this is your hobby if it is not, otherwise you may soon face burnout or even depression.

6. You have to be a young prodigy

You've probably heard career stories from people who started their career at 13 as a software developer, wrote their first compiler at 16 and became CTO of a successful startup in their 20s. I genuinely believe that they are super talented people passionate about their job.
However, if you have a longer career path or if you realised what you would like to do in your life much later - this is also completely normal and doesn't make you a bad specialist. We all have different life experiences. So it's okay if as a teenager you wanted to be a rock star instead of trying to hack your school website or writing your first computer game - you still can become a great professional.

7. You should be an engineer to work in IT

Despite all I mentioned above: you don't have to be a young genius, you may not be as great at math to be a successful engineer, etc. - this profession is still not for everyone. Sometimes, as a developer you, have to spend hours looking for some complicated bug in your code (or even in someone else's code) - for someone it will be exciting and rewarding, for someone it will be boring and exhausting.
If you are a backend engineer - you probably won't be able to see the results of your work in some visual form, e.g. mobile app or website. For me personally, this is completely okay - I enjoy seeing my services working well, having great performance and matching all the business requirements - but again, it depends on your type of personality.
It's important to remember that there are many other professions in IT, not only engineering positions, and you can make a career in any of them (QA, designers, managers, scientists, tech writers, developer advocates etc.). So you can always find something that will suit your skills, interests and character.

Conclusion

In my article, I tried to dispel some myths about the work of software engineers and working in IT in general. Unfortunately, there are many more myths, some of them are almost harmless, others are more harmful. Regardless of our expertise level and job title, we all need to exercise our critical thinking, stay open-minded and be ready to learn to be less affected by various cognitive biases and stereotypes.

Images:
Hackers (1995)
Confused Math Lady meme
Twitter
Silicon Valley (2014)

Details of how Snowflake Streams work

Semernitskaya — Mon, 28 Feb 2022 18:10:05 +0000

Snowflake is a multicloud Data Warehouse that can be used for building cloud data platform in your company. To keep track of data changes in a table, Snowflake has introduced the Table Streams feature.

Snowflake Table Streams (or Snowflake Streams for short) can be used for implementing Change Data Capture (CDC) architecture design pattern. See this article for more details about CDC pattern.
In my post I'm going to explain some details about Snowflake Streams and show how they can be created and used.

Snowflake Streams definition and types

Definition from the Snowflake documentation:

A stream object records data manipulation language (DML) changes made to tables, including inserts, updates, and deletes, as well as metadata about each change, so that actions can be taken using the changed data. ... A table stream ... makes a “change table” available of what changed, at the row level, between two transactional points of time in a table. This allows querying and consuming a sequence of change records in a transactional fashion.

A stream itself does not contain any table data (this can be important if you have GDPR or similar requirements for you system). A stream stores only an offset for the source table and not any actual table columns or data.

When queried, a stream accesses and returns the historic data in the same shape as the source table (i.e. the same column names and ordering) with the following additional columns:

METADATA$ACTION - DML operation type: INSERTED or DELETED
METADATA$ISUPDATE - Shows if the operation was part of an UPDATE statement. UPDATE operation is represented as a combination of DELETE and INSERT records in the stream with METADATA$ISUPDATE = TRUE
METADATA$ROW_ID - Unique and immutable row ID from the source table

Snowflake Streams types

Stream type	Supporting tables	Is insert only	Comment
Standard	Standard tables, directory tables	NO	Tracks all DML changes to the source table, including inserts, updates, and deletes (including table truncates)
Append-only	Standard tables	YES	Tracks row inserts only. Update and delete operations (including table truncates) are not recorded
Insert-only	External tables	YES	Tracks row inserts only. They do not record delete operations that remove rows from an inserted set

Snowflake Streams creation

Stream can be created using CREATE statement, by default Standard stream will be created.

Use APPEND_ONLY = TRUE to create Append-only stream, use INSERT_ONLY = TRUE to create Insert-only stream
Use SHOW_INITIAL_ROWS = TRUE if you need new stream to include records presented in the table before it was created
You can create several streams for the same table and then read from them and advance offsets independently (see the next section for more details about stream offset). Because stream doesn't store actual data - you can create as many streams as you need for a table without significant cost rise. Example:

-- create a table
create or replace table members (
  id number(8) not null,
  name varchar(255) default null,
  fee number(3) null
);

-- create a Standard stream 
create or replace stream member_check on table members;

-- create an Append-only stream
create or replace stream member_check_append on table members append_only=true;

Use SHOW STREAMS command to show streams you have an access to. Example:

SHOW STREAMS;

Use STREAM_HAS_DATA option to check if stream has data or not. Example:

SELECT SYSTEM$STREAM_HAS_DATA('member_check');

Insert-only stream is only supported for external tables. It can be created for a regular table but in this case this stream will act like a standard stream. Example:

-- for the members table

-- this stream will act like Standard stream
create or replace stream member_check_insert on table members insert_only=true;

Snowflake Streams querying. Stream offset

We can query a stream the same way we query a view or a table. All the changes in the source table are visible in the stream only after they are committed. The stream returns all of the columns from the table and its metadata columns. Example:

-- for the members table

-- insert values
insert into members (id,name,fee)
values
(1,'Joe',0),
(2,'Jane',0),
(3,'George',0),
(4,'Betty',0),
(5,'Sally',0);
commit;

-- check streams
select * from member_check;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
1   Joe 0   INSERT  FALSE   e585f6fd43b0c68492d11c3e9af24198134c011d
2   Jane    0   INSERT  FALSE   878c5702cfe33d390310ec55f0003f320210f721
3   George  0   INSERT  FALSE   b8f4e37b5f8ba669656e8b7e961c473f872d10e9
4   Betty   0   INSERT  FALSE   116dff9927a916f986f4701ead7d73f30b9d2ff7
5   Sally   0   INSERT  FALSE   f739b6f1785779f728a52dfd1db7b2036c44ce96
**/

select * from member_check_append;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
1   Joe 0   INSERT  FALSE   e585f6fd43b0c68492d11c3e9af24198134c011d
2   Jane    0   INSERT  FALSE   878c5702cfe33d390310ec55f0003f320210f721
3   George  0   INSERT  FALSE   b8f4e37b5f8ba669656e8b7e961c473f872d10e9
4   Betty   0   INSERT  FALSE   116dff9927a916f986f4701ead7d73f30b9d2ff7
5   Sally   0   INSERT  FALSE   f739b6f1785779f728a52dfd1db7b2036c44ce96
**/

As it was mentioned above the stream only stores the offset for the source table, then it uses the versioning history for the source table to show changed records.

Just querying the data from the stream will not change the offset. If we run the same SELECT statements from the code snippet above - the same 5 records will be returned.

A stream advances its offset only when it is used in a DML transaction (advancing offset for the stream can be compared to committing offset for Kafka topic for example). The standard approach here is to read changed records from the stream and insert them into some other table in a DML transaction. Example:

-- for the members table

-- create history table
create or replace table members_history (
  id number(8) not null,
  name varchar(255) default null
);

-- advance the offset of the stream member_check_append
begin;
insert into members_history 
select id, name from member_check_append;
commit;

-- check streams again
-- member_check remains the same, because we haven't advanced its offset yet
select * from member_check;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
1   Joe 0   INSERT  FALSE   e585f6fd43b0c68492d11c3e9af24198134c011d
2   Jane    0   INSERT  FALSE   878c5702cfe33d390310ec55f0003f320210f721
3   George  0   INSERT  FALSE   b8f4e37b5f8ba669656e8b7e961c473f872d10e9
4   Betty   0   INSERT  FALSE   116dff9927a916f986f4701ead7d73f30b9d2ff7
5   Sally   0   INSERT  FALSE   f739b6f1785779f728a52dfd1db7b2036c44ce96
**/

-- member_check_append is empty because we've advanced the offset for this stream
select * from member_check_append;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
**/

NOTE
Just putting SELECT from the stream between BEGIN and COMMIT statements won't change the offset of the stream. Example:

-- for the members table

begin;
-- won't advance the offset
select * from member_check;
commit;

If you need to advance stream offset to the current table version but don't want to consume the stream data in a DML transaction - you can use one of two approaches:

Recreate the stream (using the CREATE OR REPLACE command)
Insert the current change data into a dummy table. In the INSERT statement, query the stream but include a WHERE clause that filters out all of the change data. Example:

-- for the members table

-- create utility table
create or replace table utility (
  id number(8) not null
);

-- advance the offset of the stream member_check
begin;
insert into utility select id from member_check where false;
commit;

-- check member_check stream again 
-- member_check is empty because we've advanced offset for this stream
select * from member_check;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
**/

A stream becomes stale when its offset is outside of the data retention period for its source table. When a stream becomes stale, the historical data for the source table is no longer accessible, including any unconsumed change records. To track new change records for the table, recreate the stream. To prevent a stream from becoming stale, consume the stream records within a transaction during the retention period for the table. See more details and related settings in the Snowflake documentation.

Additional details of Snowflake Streams

On DDL changes (e.g. ADD COLUMN, DROP COLUMN) we won’t get any new records in the stream, even adding a new column with a default value won’t cause any UPDATE-s. Existing / new records in the stream will include changes in columns. Example:

-- create a table
create or replace table test (
  id number(8) not null
);

-- create a Standard stream 
create or replace stream test_stream on table test;

-- insert values
insert into test (id)
values (1), (2);
commit;

-- check the stream
select * from test_stream;
/**
ID  METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
1   INSERT  FALSE   71e6b8e0f43b2c14330ce1a3142d2e8001a93c61
2   INSERT  FALSE   c97db7c45da5efeb895cbd811f210a506600f620
**/

-- add a new column with a default value
alter table test add column fee number default 10 not null;

-- check the stream again => no new updates in the stream, a new column is visible
select * from test_stream;
/**
ID  FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
1   10  INSERT  FALSE   71e6b8e0f43b2c14330ce1a3142d2e8001a93c61
2   10  INSERT  FALSE   c97db7c45da5efeb895cbd811f210a506600f620
**/

For the Standard stream type if a record was inserted and then deleted between two stream reads - it won’t be visible at all. Example:

-- for the members table

-- insert a new record and then delete it
insert into members (id,name,fee)
values (6, 'Mike', 0);
commit;
delete from members where id = 6;
commit;

-- check streams
-- member_check is empty because it's Standard stream
select * from member_check;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
**/

-- member_check_append contains a new record because it's Append-only stream
select * from member_check_append;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
6   Mike    0   INSERT  FALSE   d98b3586efc9e5b17ff13956bdb304094894a4ec
**/

Updates are not visible in the stream if there is no actual data change. Example:

-- for the members table

-- update fee column with the same value
update members set fee = 0 where id = 3;
commit;

-- check the stream => no new values
select * from member_check;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
**/

-- update fee column with the new value
update members set fee = 10 where id = 3;
commit;

-- check the stream => update is visible
select * from member_check;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
3   George  10  INSERT  TRUE    640590445de028a556d79d6361a8aa59fc3301d3
3   George  0   DELETE  TRUE    640590445de028a556d79d6361a8aa59fc3301d3
**/

MERGE operations are visible in the stream similar to simple CREATE / UPDATE operations
If the row was updated multiple times before we could read it - in this case, the stream will return only the net changes, i.e. only the last value is returned. Example:

-- for the members table

-- update fee column several times, update name column
update members set fee = 20 where id = 3;
update members set fee = 30 where id = 3;
update members set name = 'Anna' where id = 3;
commit;

-- check the stream => the last version of the record is visible
select * from member_check;
/**
ID  NAME    FEE METADATA$ACTION METADATA$ISUPDATE   METADATA$ROW_ID
3   Anna    30  INSERT  TRUE    640590445de028a556d79d6361a8aa59fc3301d3
3   George  0   DELETE  TRUE    640590445de028a556d79d6361a8aa59fc3301d3
**/

Conclusion

Snowflake Streams is a tool that lets you implement CDC pattern in your Data Warehouse without using external ELT tools, or simply allow consumers to track changes in some of your tables without incurring significant cost. At the same time, you need to be aware of the limitations of streams, some of which I described in my article, for example how DDL changes can affect your stream, or the possibility to get a stale stream, or the way changes are accumulated in the stream between two reads.

Cover image: Shutterstock/Mariia Tagirova

Moving reflection-based code from Java 8 to 11 and 17

Semernitskaya — Fri, 18 Feb 2022 10:18:43 +0000

During the migration of one of the libraries from Java 8 to Java 11 and then to Java 17 I ran into a number of problems with the reflection functionality that was used in this library. In my article I'm going to describe these problems and how they can be solved.

Java 8

In the library reflection mechanism was used to access private fields, methods and constructors of JDK internal classes, for example, to manipulate annotations in runtime. Here are some examples of reflection usages without actual manipulation code:

import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.Map;

public class ReflectionUsage {

    public static void main(String[] args) throws NoSuchMethodException, NoSuchFieldException, ClassNotFoundException {
        Method declaredAnnotationsMethod = Field.class.getDeclaredMethod("declaredAnnotations");
        declaredAnnotationsMethod.setAccessible(true);

        Field declaredAnnotationsField = Field.class.getDeclaredField("declaredAnnotations");
        declaredAnnotationsField.setAccessible(true);

        Class<?> annotationDataClass = Class.forName("java.lang.Class$AnnotationData");
        Constructor<?> annotationDataConstructor = annotationDataClass.getDeclaredConstructor(Map.class, Map.class, int.class);
        annotationDataConstructor.setAccessible(true);

        Method annotationDataMethod = Class.class.getDeclaredMethod("annotationData");
        annotationDataMethod.setAccessible(true);
    }
}

Java 11

After I migrated the library to Java 11, I started getting warnings like the followings:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by ...ReflectionUsage ... to method java.lang.reflect.Field.declaredAnnotations()
WARNING: Please consider reporting this to the maintainers of ...ReflectionUsage
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

These warnings are caused by the modular structure introduced in Java 9: reflective access to internal JDK modules is considered to be Illegal, but in Java 9-11 such access is still allowed - it only produces warnings.

To see all warnings produced due to Illegal access of the code, you can use the option:

--illegal-access=warn

Java 17

InaccessibleObjectException

After I migrated the library to Java 17, I started seeing runtime errors InaccessibleObjectException instead of Illegal access warnings. An example of InaccessibleObjectException stack trace:

Exception in thread "main" java.lang.reflect.InaccessibleObjectException: Unable to make private java.util.Map java.lang.reflect.Field.declaredAnnotations() accessible: module java.base does not "opens java.lang.reflect" to unnamed module @...
    at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
    at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
    at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
    at java.base/java.lang.reflect.Method.setAccessible(Method.java:193)
    at ...ReflectionUsage.main(ReflectionUsage.java:13)

To resolve these exceptions it's required to use the option:

--add-opens

From the Oracle article:

If you have to allow code on the class path to do deep reflection to access nonpublic members, then use the --add-opens runtime option.
Some libraries do deep reflection, meaning setAccessible(true), so they can access all members, including private ones. You can grant this access using the --add-opens option on the java command line. No warning messages are generated as a result of using this option.

So I used the following options for the library:

--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED

NOTE
In Java 16 and prior it's also possible to fix this error by using the option:

--illegal-access=permit

but its support has been removed in Java 17

NoSuchFieldException

Fix with --add-opens option helped me with InaccessibleObjectException but then I started getting NoSuchFieldException when trying to access private field declaredAnnotations from java.lang.reflect.Field class:

Exception in thread "main" java.lang.NoSuchFieldException: declaredAnnotations
    at java.base/java.lang.Class.getDeclaredField(Class.java:2610)
    at ...ReflectionUsage.main(ReflectionUsage.java:15)

for the sentence:

Field declaredAnnotationsField = Field.class.getDeclaredField("declaredAnnotations");

NoSuchFieldException was caused by the feature "Filtering for classes with security sensitive fields" introduced in Java 12 JDK-8210522:

Core reflection has a filtering mechanism to hide security and integrity sensitive fields and methods from Class getXXXField(s) and getXXXMethod(s).
The filtering mechanism has been used for several releases to hide security sensitive fields such as System.security and Class.classLoader.
This CSR proposes to extend the filters to hide fields from a number of highly security sensitive classes in java.lang.reflect and java.lang.invoke.

I found two possible workarounds to access private fields of java.lang.reflect.Field class (in my case I needed access to declaredAnnotations field, but the same approach can be used to access other fields that are not available because of JDK-8210522)

Workaround #1: use VarHandle (source):

var lookup = MethodHandles.privateLookupIn(Field.class, MethodHandles.lookup());
VarHandle declaredAnnotationsField = lookup.findVarHandle(Field.class, "declaredAnnotations", Map.class);

Workaround #2: invoke getDeclaredFields0 method for java.lang.reflect.Field class using reflection (source):

Method getDeclaredFields0 = Class.class.getDeclaredMethod("getDeclaredFields0", boolean.class);
getDeclaredFields0.setAccessible(true);

Field[] fields = (Field[]) getDeclaredFields0.invoke(Field.class, false);
Field declaredAnnotationsField;
for (Field field : fields) {
    if ("declaredAnnotations".equals(field.getName())) {
        declaredAnnotationsField = field;
        declaredAnnotationsField.setAccessible(true);
        break;
    }
}

NOTE
Main goal of JDK-8210522 feature is to prevent reflective access to several internal fields, so workarounds described above can stop working with any new JDK release so if you have the same problem - probably you need to think about re-writing your code to use a different approach.

Conclusion

In this article I've covered only reflection related aspects of migration Java applications to the newest LTS releases of Java (JDK 11, JDK 17), but there are certainly many more potential issues and problems during this upgrade.

Cover image: Photo by Andrew Ly on Unsplash