DEV Community: Philippe Arteau

Fixing Jest import failure

Philippe Arteau — Mon, 29 Jul 2024 18:25:18 +0000

If you are reading this post, you likely encounter this error:

> 4 | import someConfig from '../config/SomeConfigs.json' with { type: 'json' }; 
    |                                                                ^
  5 |
  6 | /**

Add @babel/plugin-syntax-import-attributes (https://github.com/babel/babel/tree/main/packages/babel-plugin-syntax-import-attributes) to the 'plugins' section of your Babel config to enable parsing.

If you already added the plugin for this syntax to your config, it's possible that your config isn't being loaded.
You can re-run Babel with the BABEL_SHOW_CONFIG_FOR environment variable to show the loaded configuration:

The issue in my case was to create a babel configuration file for Jest to use when transpiling the different source files.

babel.config.js (or babel.config.cjs)

module.exports = api => {
    const isTest = api.env('test');

      return {
        plugins: [
          "@babel/plugin-syntax-import-attributes",
        ],
      }

  };

Source

I took inspiration from this blog post which described issues with React dependencies in Jest unit tests.

This complete example uses the conditional configuration suggested in Jest documentation.

module.exports = api => {
  const isTest = api.env('test');
  // You can use isTest to determine what presets and plugins to use.

  return {
    // ...
  };
};

Solving FlashBoy "Error opening COM port"

Philippe Arteau — Wed, 23 Feb 2022 23:01:44 +0000

This is a short post to help Flash Boy ^{1 2} users find the right software and drivers. Flash Boy is a GameBoy cartridge dumper. The manufacturer does not have any product page therefore it can be time consuming to find the software needed or documentation.

If you are reading this, you are probably unable to use the device. Here are the error messages you could get:

"Device not found!"
"Check COM port connection."
"Error opening COM port."

After a few trials and errors, I found that FlashBoy dumpers seem to have multiple versions with different firmware and required different softwares.

Software options

Here is the list of flasher program available.

1) GBX Driver 2.0

This UI tool is developed by Tengu Development. Tengu is also the manufacturer shown when the device is mounting the "fake" drive (GBX).

As of 2021, this is the software to interact with the latest hardware.

Download GBX Driver 2.0 for Flash Boy

2) GB Cart Flasher version 1.1

Software release by Kraku and Chroost. It seems to support older version of the FlashBoy.

This is the first software I tested because it appears in this Youtube Flash Boy review.

Download GB Cart Flasher version 1.1

3) FlashBoy Cyclone

This software is dedictated to the Cyclone version of the FlashBoy. I don't have this specific device to validate that its validity.

Download FlashBoy Cyclone

4) FlashGBX

Last but not least, FlashGBX is the recommended software for the GBxCart RW from insidegadgets. While GBxRW is not manufactured by Backlight ENG.LAB (the company behind Flash Boy), it is my recommendation if you haven't bought a dumper/flasher yet. It can read the ROM and RAM just like the previous, but it also to write the ROM and RAM of numerous cartridges types including cheap reproduction cartridge (~4 USD) found on Aliexpress or eBay.

Download FlashGBX

Advanced Scripting for Orange Data Mining

Philippe Arteau — Tue, 01 Feb 2022 16:25:47 +0000

Have you ever heard of Orange Data Mining? It is a powerful data mining framework that includes Python API and a visual GUI.

This blog post is for people looking to create add new attributes dynamically to your data set. It will also provide a good example of scripting in Orange.

The Problem

I have a CSV dataset with an attribute that has multi-values possible and I want to create a new attribute for every value present in the multi-values attribute.

Here is a simple example that is easier to understand than my actual dataset.

Initial dataset

country	flag_colors
italy	green,white,red
united kingdom	red,blue,white
russia	white,blue,red
canada	red,white
brazil	green,blue,yellow
germany	black,red,yellow

Target dataset

country	...	green	white	red	blue	yellow	black
italy	...	1	1	1	0	0	0
united kingdom	...	0	1	1	1	0	0
russia	...	0	1	1	1	0	0
canada	...	0	1	1	0	0	0
brazil	...	1	0	0	1	1	0
germany	...	0	0	1	0	1	1

Custom Script Solution

I have annotated my solution.

from Orange.data import Table, Domain, ContinuousVariable, DiscreteVariable
import copy

attributes_to_expand = ['colors']
separator = "|" #Separator can be changed to , ; - 

# Structure used to build the new model
attributes_to_keep = []
class_vars_to_keep = []
metas_to_keep = []
variables=[] # Old and new variables
all_values = dict() #values to keep for each multi-values columns

# Building a list of all the known values for each column to expand
for data in in_data:
    for att_exp in attributes_to_expand:
        values = data[att_exp].value.split(separator)
        for v in values:
            if(not(att_exp in all_values)):
                all_values[att_exp] = set() #One new attribute per value maximum
            all_values[att_exp].add(v)

# Keeping existing metadata and class variables (target)
for orig_meta in in_data.domain.metas:
    metas_to_keep.append(orig_meta)
for orig_class_var in in_data.domain.class_vars:
    class_vars_to_keep.append(orig_class_var)

# Keeping non-multi-values variables
for orig_var in in_data.domain.variables:
    if(orig_var.name in attributes_to_expand or orig_var in class_vars_to_keep):
        continue
    variables.append(copy.copy(orig_var))
    attributes_to_keep.append(orig_var)


# Adding the list of all the new variables
for att_exp in attributes_to_expand:

    for v in all_values[att_exp]:
        variables.append(ContinuousVariable(att_exp+"="+v))

# Output Table construction

## Domain describes the variables of our dataset
domain = Domain(variables,class_vars=class_vars_to_keep,metas=metas_to_keep)

## Table include both the domain definition and the data
table = Table.from_domain(domain,len(in_data))

## Rebuilding the data line by line
for index,data in enumerate(in_data):
    # Variables that we keep as-is
    for att in class_vars_to_keep:
        table[index][att] = data[att].value
    for att in attributes_to_keep:
        table[index][att] = data[att].value
    for meta in metas_to_keep:
        table[index][meta] = data[meta].value

    # New variables
    for att_exp in attributes_to_expand:
        for v in all_values[att_exp]:
            values_for_current_line = data[att_exp].value.split(separator)
            value_is_present = v in values_for_current_line
            table[index][att_exp+"="+v] = True if value_is_present else False

# making the new dataset available to linked widget
out_data = table

Tips for Building Scripts

Don't copy objects such as Domain's variable from in_data to out_data. Altering in_data can cause side effects.
Don't forget to class variables and metadatas. The information could be helpful later in your pipeline.
print() debugging is your friend. There is also an interactive console that is very helpful.
Some shortcuts are available [Ctrl]-R: Run script, [Ctrl]-R: Save script, [Ctrl]-/: Comment line ... There is also a Vim mode for the purist. 😉
Look at existing transformations before building your own

Conclusion

Transforming multi-values columns into multiple attributes is very helpful to apply machine learning algorithm to classify and do prediction based on samples data.

Maybe this small script will save you some time...

References

Official Orange Documentation: Python Scripting

Security Puzzle (Log4J Edition)

Philippe Arteau — Mon, 10 Jan 2022 16:16:00 +0000

Apache Log4J has been in the spotlight for its JNDI feature introducing a major security flaw in december.

Here is the function for JNDI lookups modified to avoid unexpected remote class being loaded. This function was used to fix potential RCE in version 2.15.

Can you see why the validation was insufficient?

private final DirContext context; //javax.naming.directory.DirContext

[...]

@SuppressWarnings("unchecked")
public <T> T lookup(final String name) throws NamingException {
public synchronized <T> T lookup(final String name) throws NamingException {

try {
    URI uri = new URI(name);
    if (uri.getScheme() != null) {
        if (!allowedProtocols.contains(uri.getScheme().toLowerCase(Locale.ROOT))) {
            LOGGER.warn("Log4j JNDI does not allow protocol {}", uri.getScheme());
            return null;
        }
        if (LDAP.equalsIgnoreCase(uri.getScheme()) || LDAPS.equalsIgnoreCase(uri.getScheme())) {
            if (!allowedHosts.contains(uri.getHost())) {
                LOGGER.warn("Attempt to access ldap server not in allowed list");
                return null;
            }
            Attributes attributes = this.context.getAttributes(name);
            if (attributes != null) {
                // In testing the "key" for attributes seems to be lowercase while the attribute id is
                // camelcase, but that may just be true for the test LDAP used here. This copies the Attributes
                // to a Map ignoring the "key" and using the Attribute's id as the key in the Map so it matches
                // the Java schema.
                Map<String, Attribute> attributeMap = new HashMap<>();
                NamingEnumeration<? extends Attribute> enumeration = attributes.getAll();
                while (enumeration.hasMore()) {
                    Attribute attribute = enumeration.next();
                    attributeMap.put(attribute.getID(), attribute);
                }
                Attribute classNameAttr = attributeMap.get(CLASS_NAME);
                if (attributeMap.get(SERIALIZED_DATA) != null) {
                    if (classNameAttr != null) {
                        String className = classNameAttr.get().toString();
                        if (!allowedClasses.contains(className)) {
                            LOGGER.warn("Deserialization of {} is not allowed", className);
                            return null;
                        }
                    } else {
                        LOGGER.warn("No class name provided for {}", name);
                        return null;
                    }
                } else if (attributeMap.get(REFERENCE_ADDRESS) != null
                        || attributeMap.get(OBJECT_FACTORY) != null) {
                    LOGGER.warn("Referenceable class is not allowed for {}", name);
                    return null;
                }
            }
        }
    }
} catch (URISyntaxException ex) {
    // This is OK.
}
return (T) this.context.lookup(name);

}

Ref: Changeset

What is/are the weakness in the code above?

A) Incomplete blacklist / whitelist
B) Improper URL parsing
C) Unexpected Nullpointer
D) Exception handling
E) Time of Check, Time of Use

Answer (SPOILER!)

It is both a TOCTOU and a URL parsing issue!

TOCTOU explanation: https://www.gosecure.net/blog/2021/12/21/log4j-2-15-toctou-vulnerability-illustrated-by-gosecure-researchers/

DNS re-parsing issue: https://www.lunasec.io/docs/blog/log4j-zero-day-severity-of-cve-2021-45046-increased/

Did you identify either weakness?

Bypassing ModSecurity WAF

Philippe Arteau — Thu, 21 Oct 2021 02:10:12 +0000

Being able to bypass Web Application Firewall (WAF) depends on your knowledge about their behavior. Here is a cool technique that involve expressions that are ignored in MySQL SQL parser (MySQL <= 5.7). This post summarizes the impact on libinjection. The libinjection library is used by WAF such as ModSecurity and SignalScience. For more details on AWS Cloudfront impact, read the original GoSecure article.

Scientific expression in MySQL

When MySQL sees 1.e(abc), it will ignore the 1.e( portion because the following characters do not form a valid numeric value.

This behavior can be abused to fool libinjection tokenizer. Libinjection internally tokenizes the parameter and identifies contextual section types such as comments and strings. Libinjection sees the string “1.e” as an unknown SQL keyword and concludes that it is more likely to be an English sentence than code. When libinjection is unaware of an SQL function the same behavior can be exhibited.

Attack in action

Here is a demonstration of modsecurity’s capability to block a malicious pattern for SQL injection. A forbidden page is returned which is the consequence of detection.

*Blocked!*

In the following image, you can see the original request being slightly modified to bypass modsecurity and libinjection.

*WAF Bypass!*

Complete payload

Now, how do we go beyond or 1=1 payloads? The NUMBER.e expression can be inserted at plenty of locations without breaking the SQL query. The following payload demonstrated read on arbitrary SQL tables.

' 1.e(ascii 1.e(substring(1.e(select password from users limit 1 1.e,1 1.e) 1.e,1 1.e,1 1.e)1.e)1.e) = 70 or'1'='2

The same payload was indented below for readability. 1.e( and 1.e, are ignored from the parser.

1.e(ascii 
1.e(substring(
1.e(select password from users limit 1 1.e,1 1.e) 1.e,1 1.e,1 1.e)1.e)1.e) = 70 #(first char = 70)
or'1'='2

Conclusion

Most bypass techniques evolve using special encoding to obfuscate the malicious requests (URL encoding, Unicode multibytes, XML entities, etc). This technique will work if the system behind the WAF is doing an unexpected decoding. This new technique is not encoding related. For this reason, it will be a versatile trick until most system upgrade their MySQL instances.

*Full story on [GoSecure blog](https://www.gosecure.net/blog/2021/10/19/a-scientific-notation-bug-in-mysql-left-aws-waf-clients-vulnerable-to-sql-injection/) (More details on AWS Cloudfront)*

How to solve: "Symbol is declared in module X which does not export package Y"

Philippe Arteau — Thu, 18 Feb 2021 20:21:43 +0000

If you are here, I assume that you have encounter a message like one of these.

Symbol is declared in module 'java.desktop' which does not export package 'sun.awt.image'
Symbol is declared in module 'java.xml' which does not export package 'com.sun.org.apache.xerces.internal.xni.parser'
Symbol is declared in module 'java.base' which does not export package 'sun.net.www.protocol.http'

What is happening?

You are accessing a class in a package that was not intended to be used by external library. The class is part of an internal package that is subject to change. When updating the library, it is likely that it will break your application.

These modules restrictions were introduce in Java 9.

Option 1: Long-term fix

If you are working on a project that has long term stability in mind, you should look for an equivalent function that is public. This may required an additional library.

Here is a simple example in Kotlin where Strings.isNullOrEmpty was use by accident. The function call can simply be replace by CharSequence.isNullOrEmpty().

import jdk.internal.joptsimple.internal.Strings.isNullOrEmpty

fun main() {
    print("Enter your name : ")
    val userName = readLine()!!
    if (isNullOrEmpty(userName)) { //FIX: userName.isNullOrEmpty()
        println("Hello, $userName")
    }
}

Example taken from stackoverflow.com

"I know what am doing"

You may still want to use internal API regardless of the recommendation from the compiler. Here are the solution for Java and Kotlin.

Option 2: Java fix

In order to still use the internal API, you will need to pass additional arguments to the compiler (--add-exports ...). This can be configured in your Maven or Gradle build.

Here is an example of Maven configuration:

<build>
    <plugins>
        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.9</source>
                <target>1.9</target>
                <compilerArgs>
                    <!-- Few examples ... only pick the ones needed -->
                    <arg>--add-exports</arg><arg>java.xml/com.sun.org.apache.xerces.internal.impl.dtd=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.xml/com.sun.org.apache.xerces.internal.xni.parser=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.net.www.protocol.http=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>jdk.unsupported/sun.misc=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.ssl=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.util=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/com.sun.net.ssl.internal.ssl=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.jca=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.net.util=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/jdk.internal.misc=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.internal.interfaces=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.provider.certpath=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.internal.spec=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.validator=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.action=ALL-UNNAMED</arg>
                    <arg>--add-exports</arg><arg>java.base/sun.security.x509=ALL-UNNAMED</arg>
                </compilerArgs>
            </configuration>
        </plugin>

Option 3: Kotlin fix

Kotlin does not have a compiler option to turn off specific module restrictions. You can however add an annotation at the beginning of your file to disable module restriction completely.

@file:Suppress("JAVA_MODULE_DOES_NOT_EXPORT_PACKAGE")

import com.sun.org.apache.xerces.internal.util.SAXInputSource
import com.sun.org.apache.xerces.internal.xni.parser.XMLErrorHandler
import com.sun.org.apache.xerces.internal.xni.parser.XMLParseException
[...]

Conclusion

While those new module restrictions may seems drastic, they are great safe guard that will help long term maintainability. Before adding exception, take the time to investigate if you really need the internal API.

Weird Unicode Behaviors

Philippe Arteau — Tue, 04 Aug 2020 20:40:58 +0000

So you know about Unicode codepoint and encoding (UTF-8, UTF-16), but are you aware that few standard conversions have surprising outcomes? In this short post, I'll list some of the most surprising behaviors.

Case Mapping

Case mapping is the behavior behind the uppercase and lowercase functions in your favorite language. Unexpected behaviors can sometimes lead to bugs, some of them affecting software security.

While the strings “go\u017Fecure” and “gosecure” are not equal, a code that applies the uppercase transformation to both strings could mistakenly interpret both strings as being equal.

Here is a demonstration in Python.

>>> "GO\u017FECURE" == "GOSECURE"
False
>>> "GO\u017FECURE".upper() == "GOSECURE"
True

The same behavior applies to Java

>>>"ADM\u0131N".toUpperCase().equals("ADMIN")
$1 ==> true

This behavior occurs because the characters ı (U+0131) and ſ (U+017F) are converted to an ASCII characters as part of Unicode specification. Aside from a few exceptions, you can assume that your language apply these transformation by default.

Normalization

The purpose of normalization is to simplify expressions to allow matching equal or equivalent “meaning”.

Here is a demonstration using normalization functions in Ruby. The five unicode characters become six unicode characters (ASCII only).

irb(main):003:0> "\u216E\u32CE\uFF0E\u209C\u2134".unicode_normalize(:nfkc)
=> "DeV.to"

API can sometimes hide those transformations. For example in C#, the class Uri normalize the hostname from URI entered.

> Console.Write(new Uri("https://faceboo\u212A.com").Host == "facebook.com");
True

Here is a list of APIs with such behavior (see Section Auditing Source Code).

Testing your application

If you have an application that susceptible to issues related to Unicode, you can use this simple cheat sheet.
This cheat sheet can be used by developers to build regression test cases to make sure no characters are being misinterpreted.

If you are curious, you can read the full article with a security focus.