DEV Community: Allen D. Ball

Ganymede Kernel 2.0.0.20220717 Released

Allen D. Ball — Sun, 17 Jul 2022 00:00:00 +0000

Ganymede Kernel 2.0.0.20220717 is released.

The Ganymede Kernel is a Jupyter Notebook Java kernel based on JShell combined with an integrated Apache Maven-like POM, support for JVM languages such as Groovy, Javascript, and Kotlin, and support for Apache Spark and Scala binary distributions.

This release improves the Ganymede Kernel’s robustness by introducing the Ganymede Kernel REST protocol to communicate between the kernel and the JShell instance.

This release adds support for:

The %%java magic with no code will display the context with pointers to the respective type’s javadoc
New magics: mustache (handlebars), spark-session, and sql
New renderers for jOOQ Formattable and Tablesaw Table
Mustache scripting with JMustache
Updated the Apache Maven Resolver to 1.8.1 and support the use of “LATEST” in the POM artifact version specification
Added Ganymede and JDK javadoc to help links
Support Apache Spark and Hive 3.1.3 with runtime support for Hive
Use PicoCLI for magic command line parsing and improved usage messages

Binary and source downloads are available from https://github.com/allen-ball/ganymede/releases/tag/v2.0.0.20220717.

Quickstart if Java 11 (or later) and Jupyter are already installed:

$ curl -sL https://github.com/allen-ball/ganymede/releases/download/v2.0.0.20220717/ganymede-2.0.0.20220717.jar -o ganymede-kernel.jar
$ java -jar ganymede-kernel.jar --install

Please see the project page for detailed installation instructions and its features and usage.

Configuring and Using GitHub Readme Instant Preview

Allen D. Ball — Tue, 27 Jul 2021 00:00:00 +0000

GitHub will render project README and other markdown files to HTMLwhen displaying projects on its website. Those files must be commited and pushed before the developer can see the results which can result in extraneous and unwanted commits. This post discusses installing and using the GitHub Readme Instant Preview (GRIP) tool for previewingGitHub-flavored Markdown locally. It provides a GRIP-specific configuration file to enable authentication to GitHub to prevent the developer from bumping up against GitHub’sthrottlesfor unauthenticated users.

Installation

The GRIP tool is available for installation from many popular package managers. To install on MacOS with Home Brew:

$ brew install grip

Or on Debian:

$ apt install grip

To name a few.

Usage

Usage is straightforward: Change the working directory to the project directory containing the README.md or other markdown and invoke grip.

$ grip
 * Serving Flask app "grip.app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://localhost:6419/ (Press CTRL+C to quit)

And browse to http://localhost:6419/ (or the URL specified in the output). The output of GRIP run in the local project is shown below for comparison to https://github.com/allen-ball/spring-boot-web-server.

As configured (albeit, without configuration), GRIP will invoke theGitHub API without authentication. GitHub will limit unauthenticated requests to as little as 60 per hour which could impact intensive read/review cycles (especially if the developer is sumltaneously using other GitHub resources in an unauthenticated manner). The next section discusses configuring authentication.

Authentication

GRIP invokes the ${HOME}/.grip/settings.py script (if it exists) to determine the USERNAME and PASSWORD to use for authentication. The script below offers a more secure option than simply assigning those raw values in the script.

GitHub currently recommends (and sometimes requires) the developercreate a personal access tokenand also provides a process forcaching GitHub credentials in Git(for Mac, Windows, and Linux). After completing these two procedures, the developer’s credentials are available for GitHub https operations through the git credential fill command. The ${HOME}/.grip/settings.py script below (and available as thisGist) retrieves the cached credentials for GRIP.

# settings.py
# https://github.com/joeyespo/grip

# Uses "git credential fill" to populate USERNAME and PASSWORD

def git_credential_fill():
    import subprocess
    argv = ["git", "credential", "fill"]
    process = subprocess.Popen(argv, text = True, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
    output = process.communicate(input = "protocol=https\nhost=github.com\n")[0].strip()
    map = dict(item.split("=") for item in output.splitlines())

    return (map["username"], map["password"])

(USERNAME, PASSWORD) = git_credential_fill()

Notice GRIP prints Using credentials message when configured.

$ grip
 * Using credentials: allen-ball
 * Serving Flask app "grip.app" (lazy loading)
 * Environment: production
...

Ganymede Kernel 1.1.0.20210614 Released

Allen D. Ball — Mon, 14 Jun 2021 00:00:00 +0000

Ganymede Kernel 1.1.0.20210614 is released.

This release adds support for:

Markdown (CommonMark) with Handlebars, FreeMarker, and Velocity) templating languages in addition to Thymeleaf templates
Apache Spark 3.1.2
Rendering TableModels

Binary and source downloads are available from https://github.com/allen-ball/ganymede/releases/tag/v1.1.0.20210614.

Quickstart if Java 11 (or later) and Jupyter are already installed:

$ curl -sL https://github.com/allen-ball/ganymede/releases/download/v1.1.0.20210614/ganymede-kernel-1.1.0.20210614.jar -o ganymede-kernel.jar
$ java -jar ganymede-kernel.jar --install

Please see the project page for detailed installation instructions and its features and usage.

Ganymede Kernel 1.0.0.20210422 Released

Allen D. Ball — Wed, 28 Apr 2021 00:00:00 +0000

Ganymede Kernel 1.0.0.20210422 is released.

Binary and source downloads are available from https://github.com/allen-ball/ganymede/releases/tag/v1.0.0.20210422.

Quickstart if Java 11 (or later) and Jupyter are already installed:

$ curl -L https://github.com/allen-ball/ganymede/releases/download/v1.0.0.20210422/ganymede-kernel-1.0.0.20210422.jar > ganymede-kernel.jar
$ java -jar ganymede-kernel.jar --install

Please see the project page for detailed installation instructions and its features and usage.

Spring Boot Part 7: Spring Security, Basic Authentication and Form Login, and Oauth2

Allen D. Ball — Sun, 01 Nov 2020 04:26:04 +0000

(This post has been updated at blog.hcf.dev with a later version of Spring Boot.)

This article explores integrating Spring Security into a Spring Boot application. Specifically, it will examine:

Managing users' credentials (IDs and passwords) and granted authorities
Creating a Spring MVC Controller with Spring Method Security and Thymeleaf (to provide features such as customized menus corresponding to a user's grants)
Creating a REST controller with Basic Authentication and Spring Method Security

The MVC application and REST controller will each have functions requiring various granted authorities. E.g., a "who-am-i" function may be executed by a "USER" but the "who" function will require 'ADMINISTRATOR" authority while "logout" and "change password" will simply require the user is authenticated. The MVC application will also use the Spring Security Thymeleaf Dialect to provide menus in the context of the authorities granted to the user.

After creating the baseline application, this article will then explore integrating OAuth authentication.

Source code for the series and for this part are available on Github.

Application

The following subsections outline creating and running the baseline application.

Prerequisites

A PasswordEncoder must be configured. This is straightforward:¹

PasswordEncoderConfiguration

@Configuration
@NoArgsConstructor @ToString @Log4j2
public class PasswordEncoderConfiguration {
    @Bean
    public PasswordEncoder passwordEncoder() {
        return PasswordEncoderFactories.createDelegatingPasswordEncoder();
    }
}

The returned DelegatingPasswordEncoder will decrypt most formats known to Spring and will encrypt using a BCryptPasswordEncoder for storage.

Users' credentials and granted authorities are stored in a database (configured at runtime) and accessed through JPA. The Credential @Entity and JpaRepository are shown below.

Credential

@Entity
@Table(catalog = "application", name = "credentials")
@Data @NoArgsConstructor
public class Credential {
    @Id @Column(length = 64, nullable = false, unique = true)
    @NotBlank @Email
    private String email = null;

    @Lob @Column(nullable = false)
    @NotBlank
    private String password = null;
}

CredentialRepository

@Repository
@Transactional(readOnly = true)
public interface CredentialRepository extends JpaRepository<Credential,String> {
}

The implementations of Authority and AuthorityRepository are nearly identical with the password property/column replaced with grants, a AuthoritiesSet (Set<Authorities>) with a @Converter-annotated AttributeConverter to convert to and from a comma-separated string of Authorities (Enum) names for storing in the database.

Authorities

public enum Authorities { USER, ADMINISTRATOR };

The generated tables are:

mysql> DESCRIBE credentials;
+----------+-------------+------+-----+---------+-------+
| Field    | Type        | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| email    | varchar(64) | NO   | PRI | NULL    |       |
| password | longtext    | NO   |     | NULL    |       |
+----------+-------------+------+-----+---------+-------+
2 rows in set (0.00 sec)

mysql> DESCRIBE authorities;
+--------+--------------+------+-----+---------+-------+
| Field  | Type         | Null | Key | Default | Extra |
+--------+--------------+------+-----+---------+-------+
| email  | varchar(64)  | NO   | PRI | NULL    |       |
| grants | varchar(255) | NO   |     | NULL    |       |
+--------+--------------+------+-----+---------+-------+
2 rows in set (0.00 sec)

The CredentialRepository and AuthorityRepository are injected into a UserDetailsService implementation to provide UserDetails.

UserServicesConfiguration - UserDetailsService @bean

@Configuration
@NoArgsConstructor @ToString @Log4j2
public class UserServicesConfiguration {
    @Autowired private CredentialRepository credentialRepository = null;
    @Autowired private AuthorityRepository authorityRepository = null;

    @Bean
    public UserDetailsService userDetailsService() {
        return new UserDetailsServiceImpl();
    }
    ...
    @NoArgsConstructor @ToString
    private class UserDetailsServiceImpl implements UserDetailsService {
        @Override
        @Transactional(readOnly = true)
        public UserDetails loadUserByUsername(String username) throws UsernameNotFoundException {
            User user = null;

            try {
                Optional<Credential> credential = credentialRepository.findById(username);
                Optional<Authority> authority = authorityRepository.findById(username);

                user =
                    new User(username,
                             credential.get().getPassword(),
                             authority.map(t -> t.getGrants().asGrantedAuthorityList())
                             .orElse(AuthorityUtils.createAuthorityList()));
            } catch (UsernameNotFoundException exception) {
                throw exception;
            } catch (Exception exception) {
                throw new UsernameNotFoundException(username);
            }

            return user;
        }
    }
    ...
}

Separate WebSecurityConfigurer instances will be configured for the RestControllerImpl (/api/**) and ControllerImpl (/**) but each will share the same super-class where the PasswordEncoder and UserDetailsService configured above will be injected and configured.

WebSecurityConfigurerImpl

@Configuration
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true)
@NoArgsConstructor(access = PRIVATE) @Log4j2
public abstract class WebSecurityConfigurerImpl extends WebSecurityConfigurerAdapter {
    @Autowired private UserDetailsService userDetailsService = null;
    @Autowired private PasswordEncoder passwordEncoder = null;

    @Override
    public void configure(AuthenticationManagerBuilder auth) throws Exception {
        auth.userDetailsService(userDetailsService)
            .passwordEncoder(passwordEncoder);
    }
    ...
}

The WebSecurityConfigurer for the REST controller (WebSecurityConfigurerImpl.API) must be ordered before the configurer for the MVC controller (@Order(1)) because otherwise its path-space, /api/**, would be included in that of the MVC controller, /**.

The configuration:

Requires requests are authenticated
Disables Cross-Site Request Forgery checks
Configures Basic Authentication

WebSecurityConfigurerImpl.API

public abstract class WebSecurityConfigurerImpl extends WebSecurityConfigurerAdapter {
    ...
    @Configuration
    @Order(1)
    @NoArgsConstructor @ToString
    public static class API extends WebSecurityConfigurerImpl {
        @Override
        protected void configure(HttpSecurity http) throws Exception {
            http.antMatcher("/api/**")
                .authorizeRequests(t -> t.anyRequest().authenticated())
                .csrf(t -> t.disable())
                .httpBasic(t -> t.authenticationEntryPoint(new HttpStatusEntryPoint(HttpStatus.FORBIDDEN)));
        }
    }
    ...
}

The HttpStatusEntryPoint is configured to prevent authentication failures from redirecting to the /error page configured for the MVC controller.

The WebSecurityConfigurerImpl.UI configuration:

Ignores security checks on static assets
Requires requests are authenticated
Configures Form Login
Configures a Logout Handler (alleviating the need to implement a corresponding MVC controller method)

WebSecurityConfigurerImpl.UI

public abstract class WebSecurityConfigurerImpl extends WebSecurityConfigurerAdapter {
    ...
    @Configuration
    @Order(2)
    @NoArgsConstructor @ToString
    public static class UI extends WebSecurityConfigurerImpl {
        private static final String[] IGNORE = {
            "/css/**", "/js/**", "/images/**", "/webjars/**", "/webjarsjs"
        };

        @Override
        public void configure(WebSecurity web) {
            web.ignoring().antMatchers(IGNORE);
        }
        ...
        @Override
        protected void configure(HttpSecurity http) throws Exception {
            http.antMatcher("/**")
                .authorizeRequests(t -> t.anyRequest().authenticated())
                .formLogin(t -> t.loginPage("/login").permitAll())
                .logout(t -> t.logoutRequestMatcher(new AntPathRequestMatcher("/logout"))
                              .logoutSuccessUrl("/").permitAll());
            ...
        }
    }
    ...
}

The outline of the MVC is shown below. (The individual methods will be described in detail in a subsequent chapter.) There are two things to note:

The template resolver is configured to use "decoupled template logic"
All methods (including a custom /error mapping) return the same view (Thymeleaf template)

ControllerImpl

@Controller
@RequestMapping(value = { "/" })
@NoArgsConstructor @ToString @Log4j2
public class ControllerImpl implements ErrorController {
    private static final String VIEW = ControllerImpl.class.getPackage().getName();
    ...
    @Autowired private SpringResourceTemplateResolver resolver = null;
    ...
    @PostConstruct
    public void init() { resolver.setUseDecoupledLogic(true); }

    @PreDestroy
    public void destroy() { }
    ...
    @RequestMapping(value = { "/" })
    public String root() {
        return VIEW;
    }
    ...
    @RequestMapping(value = "${server.error.path:${error.path:/error}}")
    public String error() { return VIEW; }

    @ExceptionHandler
    @ResponseStatus(value = INTERNAL_SERVER_ERROR)
    public String handle(Model model, Exception exception) {
        model.addAttribute("exception", exception);

        return VIEW;
    }
    ...
}

The common Thymeleaf template is outlined below. <li/> elements provide drop-down menus which are activated by security dialect sec:authorize attributes. A th:switch attribute provides a <section/> "case" element for each supported path. A form is displayed if the "form" attribute is set in the Model. And, if the user is authenticated, their granted authorities are displayed in the right of the footer with the sec:authentication attribute.

application.html

<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org" th:xmlns="@{http://www.w3.org/1999/xhtml}">
  <head>...</head>
  <body>
    <header>
      <nav th:ref="navbar">
        <th:block th:ref="container">
          ...
          <div th:ref="navbar-menu">
            ...
            <ul th:ref="navbar-end">
              <li th:ref="navbar-item" sec:authorize="hasAuthority('ADMINISTRATOR')">
                <button th:text="'Administrator'"/>
                <ul th:ref="navbar-dropdown">...</ul>
              </li>
              <li th:ref="navbar-item" sec:authorize="hasAuthority('USER')">
                <button th:text="'User'"/>
                <ul th:ref="navbar-dropdown">...</ul>
              </li>
              <li th:ref="navbar-item" sec:authorize="isAuthenticated()">
                <button sec:authentication="name"/>
                <ul th:ref="navbar-dropdown">...</ul>
              </li>
              <li th:ref="navbar-item" sec:authorize="!isAuthenticated()">
                <a th:text="'Login'" th:href="@{/login}"/>
              </li>
            </ul>
          </div>
        </th:block>
      </nav>
    </header>
    <main th:unless="${#ctx.containsVariable('exception')}"
          th:switch="${#request.servletPath}">
      <section th:case="'/who'">...</section>
      <section th:case="'/who-am-i'">...</section>
      <section th:case="'/error'">...</section>
      <section th:case="*">
        <th:block th:if="${#ctx.containsVariable('form')}">
          <th:block th:insert="~{${#execInfo.templateName + '/' + form.class.simpleName}}"/>
        </th:block>
        <p th:if="${#ctx.containsVariable('exception')}" th:text="${exception}"/>
      </section>
    </main>
    <main th:if="${#ctx.containsVariable('exception')}">
      <section>...</section>
    </main>
    <footer>
      <nav th:ref="navbar">
        <div th:ref="container">
          ...
          <span th:ref="right">
            <th:block sec:authorize="isAuthenticated()">
              <span sec:authentication="authorities"/>
            </th:block>
          </span>
        </div>
      </nav>
    </footer>
    ...
  </body>
</html>

Bootstrap attributes are added through the "decoupled template logic" expressed in src/main/resources/templates/application.th.xml. (The mechanics of decoupled template logic are not discussed further in this article.)

The outline of the REST controller is shown below. The common exception handler returns an HTTP 403 code for security-related exceptions. This combined with the Basic Authentication entry point setting in the WebSecurityConfigurer prevents the REST controller from redirecting in the event of an Exception.

RestControllerImpl

@RestController
@RequestMapping(value = { "/api/" }, produces = APPLICATION_JSON_VALUE)
@NoArgsConstructor @ToString @Log4j2
public class RestControllerImpl {
    ...
    @ExceptionHandler({ AccessDeniedException.class, SecurityException.class })
    public ResponseEntity<Object> handleFORBIDDEN() {
        return new ResponseEntity<>(HttpStatus.FORBIDDEN);
    }
}

Runtime Environment

The POM (pom.xml) has a similar spring-boot:run profile to that described in part 1 of this series. The relevant parts of the application.properties file are shown below.²

application.properties

spring.jpa.defer-datasource-initialization: true
spring.jpa.format-sql: true
spring.jpa.hibernate.ddl-auto: create
spring.jpa.open-in-view: true
spring.jpa.show-sql: false

spring.sql.init.enabled: true
spring.sql.init.data-locations: file:data.sql
...

While the above specifies the contents of data.sql is to be loaded to the Spring data source, it does not configure the data source. Two additional profiles are provide to configure an hsqldb or mysql data source. The hsqldb profile and application properties are shown below.

    <profile>
      <id>hsqldb</id>
      <dependencies>
        <dependency>
          <groupId>org.hsqldb</groupId>
          <artifactId>hsqldb</artifactId>
          <scope>runtime</scope>
        </dependency>
      </dependencies>
      <build>
        <plugins>
          <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
              <profiles combine.children="append">
                <profile>hsqldb</profile>
              </profiles>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>

application-hsqldb.properties

spring.datasource.driver-class-name: org.hsqldb.jdbc.JDBCDriver
spring.datasource.url: jdbc:hsqldb:mem:testdb;DB_CLOSE_DELAY=-1
spring.datasource.username: sa
spring.datasource.password:

The mysql profile requires the embedded MySQL server described in Spring Embedded MySQL Server and packaged in the starter described in part 6 of this series.

    <profile>
      <id>mysql</id>
      <repositories>...</repositories>
      <dependencies>
        <dependency>
          <groupId>ball</groupId>
          <artifactId>ball-spring-mysqld-starter</artifactId>
          <version>2.1.2.20210415</version>
        </dependency>
      </dependencies>
      <build>
        ...
      </build>
    </profile>

application-mysql.properties

spring.jpa.hibernate.naming.implicit-strategy: default
spring.jpa.hibernate.naming.physical-strategy: org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl

spring.datasource.driver-class-name: com.mysql.cj.jdbc.Driver
spring.datasource.url: jdbc:mysql://localhost:${mysqld.port}/application?serverTimezone=UTC&createDatabaseIfNotExist=true
spring.datasource.username: root

mysqld.home: target/mysql
mysqld.port: 3306

Depending on the desired database selection, run either mvn -Pspring-boot:run,hsqldb or mvn -Pspring-boot:run,mysql to start the application server.

data.sql defines two users: user@example.com who is granted "USER" authority and admin@example.com who is granted "USER" and "ADMINISTRATOR" authorities.

data.sql

INSERT INTO credentials (email, password)
       VALUES ('admin@example.com', '{noop}abcdef'),
              ('user@example.com', '{noop}123456');
INSERT INTO authorities (email, grants)
       VALUES ('admin@example.com', 'ADMINISTRATOR,USER'),
              ('user@example.com', 'USER');

The result of executing the above SQL is shown below.

mysql> SELECT * FROM credentials;
+-------------------+--------------+
| email             | password     |
+-------------------+--------------+
| admin@example.com | {noop}abcdef |
| user@example.com  | {noop}123456 |
+-------------------+--------------+
2 rows in set (0.00 sec)

mysql> SELECT * FROM authorities;
+-------------------+--------------------+
| email             | grants             |
+-------------------+--------------------+
| admin@example.com | ADMINISTRATOR,USER |
| user@example.com  | USER               |
+-------------------+--------------------+
2 rows in set (0.00 sec)

For purposes of demonstration, the above passwords are exceedingly weak and unencrypted. Passwords may be encrypted outside the application with the htpasswd command:

$ htpasswd -bnBC 10 "" 123456 | tr -d ':\n' | sed 's/$2y/$2a/'
$2a$10$PJO7Bxx9u9JHnZ0lhHJ2dO5WwWwGrDvBdy82mV/KHUw/b1Us1yZS6

Whose output may be used to set (UPDATE) user@example.com's password to {BCRYPT}$2a$10$PJO7Bxx9u9JHnZ0lhHJ2dO5WwWwGrDvBdy82mV/KHUw/b1Us1yZS6.

The next section discusses the MVC controller.

MVC Controller

Navigating to http://localhost:8080/login/ will present:

The portion of the Thymeleaf template that generates the navbar buttons and drop-down menus is shown below. The sec:authorize expressions are evaluated to determine if Thymeleaf renders the corresponding HTML.

            <ul th:ref="navbar-end">
              <li th:ref="navbar-item" sec:authorize="hasAuthority('ADMINISTRATOR')">
                <button th:text="'Administrator'"/>
                <ul th:ref="navbar-dropdown">
                  <li><a th:text="'Who'" th:href="@{/who}"/></li>
                </ul>
              </li>
              <li th:ref="navbar-item" sec:authorize="hasAuthority('USER')">
                <button th:text="'User'"/>
                <ul th:ref="navbar-dropdown">
                  <li><a th:text="'Who Am I?'" th:href="@{/who-am-i}"/></li>
                </ul>
              </li>
              <li th:ref="navbar-item" sec:authorize="isAuthenticated()">
                <button sec:authentication="name"/>
                <ul th:ref="navbar-dropdown">
                  <li><a th:text="'Change Password'" th:href="@{/password}"/></li>
                  <li><a th:text="'Logout'" th:href="@{/logout}"/></li>
                </ul>
              </li>
              <li th:ref="navbar-item" sec:authorize="!isAuthenticated()">
                <a th:text="'Login'" th:href="@{/login}"/>
              </li>
            </ul>

In the case of an unauthenticated client, only the "Login" button is rendered (as shown in the image above). The resulting HTML (with the decoupled template logic applied) is shown below.

            <ul class="navbar-nav text-white bg-dark">



              <li class="navbar-item dropdown">
                <a href="/login" class="btn navbar-link text-white bg-dark">Login</a>
              </li>
            </ul>

The controller methods to present the Login form, present the Change Password form, and handle the change password POST method are shown below. The default Spring Security login POST method is used and does not have to be implemented here. The @PreAuthorize annotations on the change password methods enforce that the client must be authenticated to use those functions. No logout method needs to be implemented because a logout handler was configured in the WebSecurityConfigurer.

ControllerImpl - Login and Change Password Methods

public class ControllerImpl implements ErrorController {
    ...
    @Autowired private CredentialRepository credentialRepository = null;
    @Autowired private PasswordEncoder encoder = null;
    ...
    @RequestMapping(method = { GET }, value = { "login" })
    public String login(Model model, HttpSession session) {
        model.addAttribute("form", new LoginForm());

        return VIEW;
    }

    @RequestMapping(method = { GET }, value = { "password" })
    @PreAuthorize("isAuthenticated()")
    public String password(Model model, Principal principal) {
        Credential credential =
            credentialRepository.findById(principal.getName())
            .orElseThrow(() -> new AuthorizationServiceException("Unauthorized"));

        model.addAttribute("form", new ChangePasswordForm());

        return VIEW;
    }

    @RequestMapping(method = { POST }, value = { "password" })
    @PreAuthorize("isAuthenticated()")
    public String passwordPOST(Model model, Principal principal, @Valid ChangePasswordForm form, BindingResult result) {
        Credential credential =
            credentialRepository.findById(principal.getName())
            .orElseThrow(() -> new AuthorizationServiceException("Unauthorized"));

        try {
            if (result.hasErrors()) {
                throw new RuntimeException(String.valueOf(result.getAllErrors()));
            }

            if (! (Objects.equals(form.getUsername(), principal.getName())
                   && encoder.matches(form.getPassword(), credential.getPassword()))) {
                throw new AccessDeniedException("Invalid user name and password");
            }

            if (! (form.getNewPassword() != null
                   && Objects.equals(form.getNewPassword(), form.getRepeatPassword()))) {
                throw new RuntimeException("Repeated password does not match new password");
            }

            if (encoder.matches(form.getNewPassword(), credential.getPassword())) {
                throw new RuntimeException("New password must be different than old");
            }

            credential.setPassword(encoder.encode(form.getNewPassword()));
            credentialRepository.save(credential);
        } catch (Exception exception) {
            model.addAttribute("form", form);
            model.addAttribute("errors", exception.getMessage());
        }

        return VIEW;
    }
    ...
}

Once authenticated, the user management drop-down is rendered and the Login button is not. In addition, because user@example.com has been granted "USER" authority, the User drop-down is rendered to HTML, also.

The change password form is straightforward.

And, as an aside, changed passwords are stored encrypted (as expected).

mysql> SELECT * FROM credentials;
+-------------------+----------------------------------------------------------------------+
| email             | password                                                             |
+-------------------+----------------------------------------------------------------------+
| admin@example.com | {noop}abcdef                                                         |
| user@example.com  | {bcrypt}$2a$10$UXRB6BbmcbHfXkWDTk755ewgWsENMgFZoJ.JcIoiIjuRyGhOpEaNS |
+-------------------+----------------------------------------------------------------------+
2 rows in set (0.00 sec)

The user dropdown expanded below:

The /who-am-i method adds the client's Principal (injected as a parameter by Spring) to the Model so it may be presented in the Thymeleaf template (as long as the client has the "USER" authority).

ControllerImpl - /who-am-i

public class ControllerImpl implements ErrorController {
    ...
    @RequestMapping(value = { "who-am-i" })
    @PreAuthorize("hasAuthority('USER')")
    public String whoAmI(Model model, Principal principal) {
        model.addAttribute("principal", principal);

        return VIEW;
    }
    ...
}

When selecting "User->Who Am I?" the application shows something similar to:

The application Thymeleaf template contains to display the method parameter Principal:³

      <section th:case="'/who-am-i'">
        <p th:text="${principal}"/>
      </section>

Clients that have been granted "ADMINISTRATOR" authority will be presented the Administrator drop-down menu.

The /who method will list the Principals of currently registered sessions and is only available to clients granted "ADMINISTRATOR" authority.

ControllerImpl - /who

public class ControllerImpl implements ErrorController {
    ...
    @Autowired private SessionRegistry registry = null;
    ...
    @RequestMapping(value = { "who" })
    @PreAuthorize("hasAuthority('ADMINISTRATOR')")
    public String who(Model model) {
        model.addAttribute("principals", registry.getAllPrincipals());

        return VIEW;
    }
    ...
}

The method implementation is straightforward with the injected SessionRegistry. The SessionRegistry implementation bean must be configured as part of the WebSecurityConfigurer.

WebSecurityConfigurerImpl.UI - SessionRegistry

public abstract class WebSecurityConfigurerImpl extends WebSecurityConfigurerAdapter {
    ...
    public static class UI extends WebSecurityConfigurerImpl {
        ...
        @Override
        protected void configure(HttpSecurity http) throws Exception {
            ...
            http.sessionManagement(t -> t.maximumSessions(-1).sessionRegistry(sessionRegistry()));
        }
        ...
        @Bean
        public SessionRegistry sessionRegistry() {
            return new SessionRegistryImpl();
        }
        ...
    }
    ...
}

A client with ADMINISTRATION authority may navigate to /who:

While a client without (even if authenticated) will be denied:

The next section discusses the REST controller.

REST Controller

Similar to the corresponding @Controller method described in the previous section, the /api/who-am-i method returns the client's Principal (injected as a parameter by Spring) if the client has the "USER" authority.

RestControllerImpl - /who-am-i

public class RestControllerImpl {
    ...
    @RequestMapping(method = { GET }, value = { "who-am-i" })
    @PreAuthorize("hasAuthority('USER')")
    public ResponseEntity<Principal> whoAmI(Principal principal) throws Exception {
        return new ResponseEntity<>(principal, HttpStatus.OK);
    }
    ...
}

Invoking without authentication returns HTTP/1.1 403.⁴

$ curl -is http://localhost:8080/api/who-am-i
HTTP/1.1 403
Set-Cookie: JSESSIONID=6EBD3FEED11F2499F6915E98E02D1C26; Path=/; HttpOnly
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Length: 0
Date: Sat, 17 Oct 2020 05:25:04 GMT

While supplying credentials for a user that is granted "USER" is successful.

$ curl -is --basic -u user@example.com:123456 http://localhost:8080/api/who-am-i
HTTP/1.1 200
Set-Cookie: JSESSIONID=C0F5D3A67AA59521A223B5F87B2915FC; Path=/; HttpOnly
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sat, 17 Oct 2020 05:25:44 GMT

{
  "authorities" : [ {
    "authority" : "USER"
  } ],
  "details" : {
    "remoteAddress" : "0:0:0:0:0:0:0:1",
    "sessionId" : null
  },
  "authenticated" : true,
  "principal" : {
    "password" : null,
    "username" : "user@example.com",
    "authorities" : [ {
      "authority" : "USER"
    } ],
    "accountNonExpired" : true,
    "accountNonLocked" : true,
    "credentialsNonExpired" : true,
    "enabled" : true
  },
  "credentials" : null,
  "name" : "user@example.com"
}

The /api/who method returns the list of all Principals logged in (defined as having active sessions in the UI) if the client has "ADMINISTRATOR" authority.

RestControllerImpl - /who

public class RestControllerImpl {
    @Autowired private SessionRegistry registry = null;
    ...
    @RequestMapping(method = { GET }, value = { "who" })
    @PreAuthorize("hasAuthority('ADMINISTRATOR')")
    public ResponseEntity<List<Object>> who() throws Exception {
        return new ResponseEntity<>(registry.getAllPrincipals(), HttpStatus.OK);
    }
    ...
}

Invoking with an authenticated client without "ADMINISTRATOR" authority granted returns HTTP/1.1 403 (as expected).⁵

$ curl -is --basic -u user@example.com:123456 http://localhost:8080/api/who
HTTP/1.1 403
...

While supplying credentials for a user that is granted "ADMINISTRATOR" is successful.

$ curl -is --basic -u admin@example.com:abcdef http://localhost:8080/api/who
HTTP/1.1 200
Set-Cookie: JSESSIONID=8E283763FD321382417C89B609DE9EDC; Path=/; HttpOnly
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sat, 17 Oct 2020 05:27:39 GMT

[ {
  "password" : null,
  "username" : "user@example.com",
  "authorities" : [ {
    "authority" : "USER"
  } ],
  "accountNonExpired" : true,
  "accountNonLocked" : true,
  "credentialsNonExpired" : true,
  "enabled" : true
}, {
  "password" : null,
  "username" : "admin@example.com",
  "authorities" : [ {
    "authority" : "ADMINISTRATOR"
  }, {
    "authority" : "USER"
  } ],
  "accountNonExpired" : true,
  "accountNonLocked" : true,
  "credentialsNonExpired" : true,
  "enabled" : true
} ]

The next chapter will examine OAuth integration.

OAuth

The following subsections will:

Run an experiment by configuring the application as described in the first chapter for OAuth authentication
Change the application implementation to allow Form Login and OAuth authenitcation

Experiment

This section will examine the behavior Spring Security's default settings for authentication. An OAuth provider must be configured. This can easily be done on GitHub:

Navigate to GitHub and login
Select "Settings" from the right-most profile drop-down menu
Click "Developer Settings" from the left column
Click "OAuth Apps" from the left column
Click "Register a new application." Fill in the form with the following values:
- Homepage URL: http://localhost:8080/
- Authorization callback URL: http://localhost:8080/login/oauth2/code/github
Note the Client ID and Secret as it will be required in the application configuration

The POM oauth profile enables the Spring Boot oauth profile. In addition, the required Spring Security dependencies for OAuth are added: An OAuth 2.0 client and support for Javascript Object Signing and Encryption (JOSE).

  <profiles>
    ...
    <profile>
      <id>oauth</id>
      <build>
        ...
      </build>
    </profile>
    ...
  </profiles>
  <dependencies verbose="true">
    ...
    <dependency>
      <groupId>com.okta.spring</groupId>
      <artifactId>okta-spring-boot-starter</artifactId>
      <version>2.0.1</version>
    </dependency>
    ...
    <dependency>
      <groupId>org.springframework.security</groupId>
      <artifactId>spring-security-oauth2-client</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.security</groupId>
      <artifactId>spring-security-oauth2-jose</artifactId>
    </dependency>
    ...
  </dependencies>

Allowing the application to be run from Maven with either mvn -Pspring-boot:run,hsqldb,oauth or mvn -Pspring-boot:run,mysql,oauth.

The WebSecurityConfigurer is changed to use OAuth Login instead of Form Login (with default configuration):

WebSecurityConfigurerImpl.UI (Default OAuth2 Customizer)

public abstract class WebSecurityConfigurerImpl extends WebSecurityConfigurerAdapter {
    ...
    public static class UI extends WebSecurityConfigurerImpl {
        ...
        @Override
        protected void configure(HttpSecurity http) throws Exception {
            http.antMatcher("/**")
                .authorizeRequests(t -> t.anyRequest().authenticated())
                /* .formLogin(t -> t.loginPage("/login").permitAll()) */
                .oauth2Login(Customizer.withDefaults())
                .logout(t -> t.logoutRequestMatcher(new AntPathRequestMatcher("/logout"))
                              .logoutSuccessUrl("/").permitAll());
            ...
        }
        ...
    }
    ...
}

Finally, the OAuth client prperties must be configured in the profile-specific application properties YAML file:⁶

application-oauth.yml

spring:
  security:
    oauth2:
      client:
        registration:
          github:
            client-id: dad3306da38eb7be68a1
            client-secret: 8a5394b2e29037b9bdf17e51af472020f85bfca6

Running the application now offers OAuth login:

Clicking GitHub will redirect for authorization:

If granted, the application will successfully login. However, the Principal name will be unrecognizable as well as the granted authorities:

If invoked, the Change Password function fails with:

Because the authenticated Principal has no corresponding Credential database record.

A naive implementation to integrate Form Login and OAuth2 Login is configured by simply enabling both:

        @Override
        protected void configure(HttpSecurity http) throws Exception {
            http.antMatcher("/**")
                .authorizeRequests(t -> t.anyRequest().authenticated())
                .formLogin(t -> t.loginPage("/login").permitAll())
                .oauth2Login(Customizer.withDefaults())
                .logout(t -> t.logoutRequestMatcher(new AntPathRequestMatcher("/logout"))
                              .logoutSuccessUrl("/").permitAll());
            ...
        }

However, only the OAuth2 Login page is available (illustrated previously). A successful integration will need to create a custom OAuth2 Login page compatible with (and the same as) the Form Login Page.

The next subsection will adjust the implementation to:

Use user e'mail as Principal name
Integrate Form Login and OAuth2 Login into a single custom login page
Manage granted authorities for OAuth2-authenticated users
Not offer the Change Password function to users logged in through OAuth2

Implementation

This section will adjust the implementation as outlined at the end of the previous section. The first step is to provide the OAuth 2.0 security client registrations and configured providers. The processes to configure Google and Okta Client IDs is very similar to the one for GitHub and must be configured on their respective sites. The authorization callback URI must be http://localhost:8080/login/oauth2/code/google and http://localhost:8080/login/oauth2/code/okta, respectively. A redacted example is shown below.

---
spring:
  security:
    oauth2:
      client:
        registration:
          github:
            client-id: XXXXXXXXXXXXXXXXXXXX
            client-secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
            scope: user:email
          google:
            client-id: XXXXXXXXXXXXXXXXXXXX
            client-secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
          okta:
            client-id: XXXXXXXXXXXXXXXXXXXX
            client-secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
            client-name: Okta
        provider:
          github:
            user-name-attribute: email
          google:
            user-name-attribute: email
          okta:
            issuer-uri: https://DOMAIN.okta.com/oauth2/default
            user-name-attribute: email

Note that the providers are configured to use the user's e'mail address as the Principal name (user-name-attribute: email).

The next step is to integrate the OAuth Login page with the custom Form Login page. Simply calling HttpSecurity.oauth2Login(Customizer.withDefaults()) will attempt to configure a ClientRegistrationRepository bean but that will fail if no spring.security.oauth2.client.registration.* properties are configured. The implementation tests if the bean is configured before attempting the HttpSecurity method call.

WebSecurityConfigurerImpl.UI (Custom Login Page)

public abstract class WebSecurityConfigurerImpl extends WebSecurityConfigurerAdapter {
    ...
    public static class UI extends WebSecurityConfigurerImpl {
        ...
        @Autowired private OidcUserService oidcUserService = null;
        ...
        @Override
        protected void configure(HttpSecurity http) throws Exception {
            ...
            try {
                ClientRegistrationRepository repository =
                    getApplicationContext().getBean(ClientRegistrationRepository.class);

                if (repository != null) {
                    http.oauth2Login(t -> t.clientRegistrationRepository(repository)
                                           .userInfoEndpoint(u -> u.oidcUserService(oidcUserService))
                                           .loginPage("/login").permitAll());
                }
            } catch (Exception exception) {
            }
            ...
        }
        ...
    }
    ...
}

Both the OAuth2UserService and OidcUserService beans are configured by configuring the Open ID Connect (OIDC) OidcUserService -- OIDC is built on top of OAuth 2.0 to provide identity services. The OidcUserService delegates to the OAuth2UserService for retrieving Oauth 2.0-specific information.

UserServicesConfiguration - OAuth2UserService and OidcUserService Beans

@Configuration
@NoArgsConstructor @ToString @Log4j2
public class UserServicesConfiguration {
    ...
    @Bean
    public OAuth2UserService<OAuth2UserRequest,OAuth2User> oAuth2UserService() {
        return new OAuth2UserServiceImpl();
    }

    @Bean
    public OidcUserService oidcUserService() {
        return new OidcUserServiceImpl();
    }
    ...
    private static final List<GrantedAuthority> DEFAULT_AUTHORITIES =
        AuthorityUtils.createAuthorityList(Authorities.USER.name());

    @NoArgsConstructor @ToString
    private class OAuth2UserServiceImpl extends DefaultOAuth2UserService {
        private final DefaultOAuth2UserService delegate = new DefaultOAuth2UserService();

        @Override
        public OAuth2User loadUser(OAuth2UserRequest request) throws OAuth2AuthenticationException {
            String attribute =
                request.getClientRegistration().getProviderDetails()
                .getUserInfoEndpoint().getUserNameAttributeName();
            OAuth2User user = delegate.loadUser(request);

            try {
                Optional<Authority> authority = authorityRepository.findById(user.getName());

                user =
                    new DefaultOAuth2User(authority.map(t -> t.getGrants().asGrantedAuthorityList())
                                          .orElse(DEFAULT_AUTHORITIES),
                                          user.getAttributes(), attribute);
            } catch (OAuth2AuthenticationException exception) {
                throw exception;
            } catch (Exception exception) {
                log.warn("{}", request, exception);
            }

            return user;
        }
    }

    @NoArgsConstructor @ToString
    private class OidcUserServiceImpl extends OidcUserService {
        { setOauth2UserService(oAuth2UserService()); }

        @Override
        public OidcUser loadUser(OidcUserRequest request) throws OAuth2AuthenticationException {
            String attribute =
                request.getClientRegistration().getProviderDetails()
                .getUserInfoEndpoint().getUserNameAttributeName();
            OidcUser user = super.loadUser(request);

            try {
                Optional<Authority> authority = authorityRepository.findById(user.getName());

                user =
                    new DefaultOidcUser(authority.map(t -> t.getGrants().asGrantedAuthorityList())
                                        .orElse(DEFAULT_AUTHORITIES),
                                        user.getIdToken(), user.getUserInfo(), attribute);
            } catch (OAuth2AuthenticationException exception) {
                throw exception;
            } catch (Exception exception) {
                log.warn("{}", request, exception);
            }

            return user;
        }
    }
}

Both the OAuth2UserService and OidcUserService map granted authorities for this application.

A @ControllerAdvice is implemented to add two attributes to the Model:

oauth2, a List of configured ClientRegistrations
isPasswordAuthenticated, indicating if the Principal was
authenticated with a password

ControllerAdviceImpl

@ControllerAdvice
@NoArgsConstructor @ToString @Log4j2
public class ControllerAdviceImpl {
    @Autowired private ApplicationContext context = null;
    private List<ClientRegistration> oauth2 = null;
    ...
    @ModelAttribute("oauth2")
    public List<ClientRegistration> oauth2() {
        if (oauth2 == null) {
            oauth2 = new ArrayList<>();

            try {
                ClientRegistrationRepository repository =
                    context.getBean(ClientRegistrationRepository.class);

                if (repository != null) {
                    ResolvableType type =
                        ResolvableType.forInstance(repository)
                        .as(Iterable.class);

                    if (type != ResolvableType.NONE
                        && ClientRegistration.class.isAssignableFrom(type.resolveGenerics()[0])) {
                        ((Iterable<?>) repository)
                            .forEach(t -> oauth2.add((ClientRegistration) t));
                    }
                }
            } catch (Exception exception) {
            }
        }

        return oauth2;
    }

    @ModelAttribute("isPasswordAuthenticated")
    public boolean isPasswordAuthenticated(Principal principal) {
        return principal instanceof UsernamePasswordAuthenticationToken;
    }
}

The LoginForm is modified to include configured OAuth 2.0 authentication options (if configured):

LoginForm.html

<div>
  <div>
    <div>
      <form th:object="${form}">
        <input type="email" th:name="username" th:placeholder="'E\'mail Address'"/>
        <label th:text="'E\'mail Address'"/>
        <input type="password" th:name="password" th:placeholder="'Password'"/>
        <label th:text="'Password'"/>
        <button type="submit" th:text="'Login'"/>
        <th:block th:if="${! oauth2.isEmpty()}">
          <hr/>
          <a th:each="client : ${oauth2}" th:href="@{/oauth2/authorization/{id}(id=${client.registrationId})}" th:text="${client.clientName}"/>
        </th:block>
      </form>
    </div>
  </div>
  <div>
    <div>
      <p th:if="${param.error}">Invalid username and password.</p>
      <p th:if="${param.logout}">You have been logged out.</p>
    </div>
  </div>
</div>

And the Change Password menu option is only offered if the client was authenticated with a password (by testing the isPasswordAuthenticated Model attribute:

application.html - Change Password

              ...
              <li th:ref="navbar-item" sec:authorize="isAuthenticated()">
                <button sec:authentication="name"/>
                <ul th:ref="navbar-dropdown">
                  <li th:if="${isPasswordAuthenticated}">
                    <a th:text="'Change Password'" th:href="@{/password}"/>
                  </li>
                  <li><a th:text="'Logout'" th:href="@{/logout}"/></li>
                </ul>
              </li>
              ...

The end result for the login page is shown below:

With a successful (Google) login:

[1] Implementing a PasswordEncoder is discussed in detail in Spring PasswordEncoder Implementation. ↩

[2] This article has been updated for Spring Boot 2.5.x. spring.jpa.defer-datasource-initialization has been added and spring.datasource.initialization-mode and spring.datasource.data were used instead of the spring.sql.init.* properties. ↩

[3] It's important to note that the equivalent value for Principal is available in the security dialect as ${#authentication} which references an implementation of Authentication. The use of ${#authentication} will be explored further in the OAuth discussion in the next chapter. ↩

[4] Recall the discussion of setting HttpSecurity basic authentication entry point. ↩

[5] Recall the discussion RestControllerImpl's @ExceptionHandler method. ↩

[6] YAML is not required but YAML lends itself to expressing the configuration compactly. ↩

java.util.concurrent.ForkJoinPool Example

Allen D. Ball — Wed, 12 Aug 2020 02:37:46 +0000

The combination of RecursiveTask implementations running in a ForkJoinPool allows tasks to be defined that may spawn subtasks that in turn may run asynchronously. The ForkJoinPool manages efficient processing of those tasks.

This article presents a simple calculator application to evaluate a formula defined as a List presents a single-threaded recursive solution, and then converts that solution to use RecursiveTasks executed in a ForkJoinPool.

Recursive Solution

A formula to be computed is defined as follows:

    public enum Operator { ADD, MULTIPLY; }

    public static List<?> FORMULA =
        List.of(Operator.ADD,
                List.of(Operator.MULTIPLY, 1, 2, 3), 4, 5,
                List.of(Operator.ADD, 6, 7, 8,
                        List.of(Operator.MULTIPLY, 9, 10, 11)));

A formula is either a Number or a List consisting of an Operator followed by other formulae. For illustration, the Lisp equivalent of FORMULA would be:

(+ (* 1 2 3) 4 5
   (+ 6 7 8
      (* 9 10 11)))

A Task class is defined to solve formulae:

    public static class Task {
        private final Object formula;

        public Task(Object formula) { this.formula = formula; }

        public Integer compute() {
            Integer result = null;

            if (formula instanceof Number) {
                result = ((Number) formula).intValue();
            } else {
                List<?> list = (List<?>) formula;
                Operator operator = (Operator) list.get(0);
                List<Task> subtasks =
                    list.subList(1, list.size())
                    .stream()
                    .map(Task::new)
                    .collect(toList());
                IntStream operands =
                    subtasks.stream()
                    .map(Task::compute)
                    .mapToInt(Integer::intValue);

                switch (operator) {
                case ADD:
                    result = operands.sum();
                    break;

                case MULTIPLY:
                    result = operands.reduce(1, (x, y) -> x * y);
                    break;
                }
            }

            System.out.println(formula + " -> " + result);

            return result;
        }
    }

The compute() method evaluates the formula by creating another Task instance to evaluate each operand recursively by calling compute(). The actual mechanics are to create a List of Tasks and then map the Stream of Tasks to an IntStream by calling Task.compute() which is evaluated based on the operator.

The static main(String[]) function simply instantiates a Task and calls compute().

    public static void main(String[] argv) {
        Task task = new Task(FORMULA);

        System.out.println("Result: " + task.compute());
    }

Which generates the following output:

1 -> 1
2 -> 2
3 -> 3
[MULTIPLY, 1, 2, 3] -> 6
4 -> 4
5 -> 5
6 -> 6
7 -> 7
8 -> 8
9 -> 9
10 -> 10
11 -> 11
[MULTIPLY, 9, 10, 11] -> 990
[ADD, 6, 7, 8, [MULTIPLY, 9, 10, 11]] -> 1011
[ADD, [MULTIPLY, 1, 2, 3], 4, 5, [ADD, 6, 7, 8, [MULTIPLY, 9, 10, 11]]] -> 1026
Result: 1026

The next chapter details how to convert this solution to use ForkJoinPool to enable some level of parallel processing.

ForkJoinPool Solution

The Task class is modified to extend RecursiveTask<Integer>. When a subtask is instantiated, RecursiveTask.fork() is called to asynchronously execute this task in the pool the current task is running in. In the IntStream, RecursiveTask.join() is called to wait for the subtask to complete (if it hasn't already) and return the result of the compute() method.

    public static class Task extends RecursiveTask<Integer> {
        ...
        @Override
        public Integer compute() {
            Integer result = null;

            if (formula instanceof Number) {
                ...
            } else {
                ...
                List<Task> subtasks =
                    list.subList(1, list.size())
                    .stream()
                    .map(Task::new)
                    .peek(RecursiveTask::fork)
                    .collect(toList());
                IntStream operands =
                    subtasks.stream()
                    .map(RecursiveTask::join)
                    .mapToInt(Integer::intValue);
                ...
            }

            System.out.println(Thread.currentThread() + "\t"
                               + formula + " -> " + result);

            return result;
        }
    }

The executing Thread is included in the output to demonstrate the behavior. The main(String[]) function creates a ForkJoinPool, the Task to compute FORMULA, and uses the pool to invoke the Task. The result is obtained through Task.join() to wait for the computation to complete.

    public static int N = 10;

    public static void main(String[] argv) {
        ForkJoinPool pool = new ForkJoinPool(N);
        RecursiveTask<Integer> task = new Task(FORMULA);

        pool.invoke(task);

        System.out.println("Result: " + task.join());
    }

Output using 10 threads (N = 10):

Thread[ForkJoinPool-1-worker-31,5,main] 2 -> 2
Thread[ForkJoinPool-1-worker-19,5,main] 8 -> 8
Thread[ForkJoinPool-1-worker-23,5,main] 4 -> 4
Thread[ForkJoinPool-1-worker-17,5,main] 3 -> 3
Thread[ForkJoinPool-1-worker-9,5,main]  1 -> 1
Thread[ForkJoinPool-1-worker-27,5,main] 5 -> 5
Thread[ForkJoinPool-1-worker-3,5,main]  7 -> 7
Thread[ForkJoinPool-1-worker-21,5,main] 6 -> 6
Thread[ForkJoinPool-1-worker-17,5,main] 11 -> 11
Thread[ForkJoinPool-1-worker-23,5,main] 10 -> 10
Thread[ForkJoinPool-1-worker-31,5,main] 9 -> 9
Thread[ForkJoinPool-1-worker-5,5,main]  [MULTIPLY, 1, 2, 3] -> 6
Thread[ForkJoinPool-1-worker-31,5,main] [MULTIPLY, 9, 10, 11] -> 990
Thread[ForkJoinPool-1-worker-13,5,main] [ADD, 6, 7, 8, [MULTIPLY, 9, 10, 11]] -> 1011
Thread[ForkJoinPool-1-worker-19,5,main] [ADD, [MULTIPLY, 1, 2, 3], 4, 5, [ADD, 6, 7, 8, [MULTIPLY, 9, 10, 11]]] -> 1026
Result: 1026

And with 1 pool thread (N = 1):

Thread[ForkJoinPool-1-worker-3,5,main]  1 -> 1
Thread[ForkJoinPool-1-worker-3,5,main]  2 -> 2
Thread[ForkJoinPool-1-worker-3,5,main]  3 -> 3
Thread[ForkJoinPool-1-worker-3,5,main]  [MULTIPLY, 1, 2, 3] -> 6
Thread[ForkJoinPool-1-worker-3,5,main]  4 -> 4
Thread[ForkJoinPool-1-worker-3,5,main]  5 -> 5
Thread[ForkJoinPool-1-worker-3,5,main]  6 -> 6
Thread[ForkJoinPool-1-worker-3,5,main]  7 -> 7
Thread[ForkJoinPool-1-worker-3,5,main]  8 -> 8
Thread[ForkJoinPool-1-worker-3,5,main]  9 -> 9
Thread[ForkJoinPool-1-worker-3,5,main]  10 -> 10
Thread[ForkJoinPool-1-worker-3,5,main]  11 -> 11
Thread[ForkJoinPool-1-worker-3,5,main]  [MULTIPLY, 9, 10, 11] -> 990
Thread[ForkJoinPool-1-worker-3,5,main]  [ADD, 6, 7, 8, [MULTIPLY, 9, 10, 11]] -> 1011
Thread[ForkJoinPool-1-worker-3,5,main]  [ADD, [MULTIPLY, 1, 2, 3], 4, 5, [ADD, 6, 7, 8, [MULTIPLY, 9, 10, 11]]] -> 1026
Result: 1026

Which (unsurprisingly) calculates the formulae in the same order as the recursive solution.

Summary

Single threaded recursive solutions may be converted to RecursiveTask¹ implementations and invoked through a ForkJoinPool to enable efficient processing of subtasks.

[1] Implementation class of ForkJoinTask. ↩

Spring Boot Part 6: Auto Configuration and Starters

Allen D. Ball — Mon, 20 Jul 2020 06:23:29 +0000

Spring Boot allows for the creation of "starters:" Convenient dependency descriptors that often provide some specific but complex functionality. Examples from Spring Boot discussed in this series of articles include spring-boot-starter-web, spring-boot-starter-thymeleaf, and spring-boot-starter-actuator.

This article describes some reusable code targeted for development and testing (a REST controller @ /jig/bean/{name}.json and /jig/bean/{name}.xml) and the steps necessary to create a Spring Boot starter. It also describes a starter for the embedded MySQL server process described in "Spring Embedded MySQL Server".

Complete javadoc is provided.

Theory of Operation

As discussed in part 1, the spring-boot-starter-actuator may be configured. One of its sevices may be used to get the list (with attributes) of the configured beans. Sample partial output below:

$ curl -is -X GET http://localhost:5001/actuator/beans/
HTTP/1.1 200
Content-Type: application/vnd.spring-boot.actuator.v3+json
Transfer-Encoding: chunked
Date: Sun, 19 Jul 2020 20:22:10 GMT

{
  "contexts" : {
    "application" : {
      "beans" : {
        ...
        "discoveryService" : {
          "scope" : "singleton",
          "type" : "upnp.DiscoveryService",
          "resource" : "file [/Users/ball/upnp-media-server/target/classes/upnp/DiscoveryService.class]",
          "dependencies" : [ "mediaServer" ]
        },
        ...
      },
      "parentId" : null
    }
  }
}

The ball-spring-jig-starter will provide the following two REST APIs to retrieve a specific bean's value and demonstrate serialization to JSON or XML as appropriate. JSON partial output:

$ curl -is -X GET http://localhost:5000/jig/bean/discoveryService.json
HTTP/1.1 200
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 19 Jul 2020 20:23:38 GMT

{
  "uuid:00000000-0000-1010-8000-0024BEF18BCC" : {
    "expiration" : 1595191949391,
    "ssdpmessage" : {
      "params" : { },
      "entity" : null,
      "locale" : null,
      "inetAddress" : "10.0.1.9",
      "st" : "uuid:00000000-0000-1010-8000-0024BEF18BCC",
      "location" : "http://10.0.1.9:52323/dmr.xml",
      "usn" : "uuid:00000000-0000-1010-8000-0024BEF18BCC",
      "protocolVersion" : {
        "protocol" : "HTTP",
        "major" : 1,
        "minor" : 1
      },
      ...
    },
    ...
  },
  ...
}

And corresponding XML output:

$ curl -is -X GET http://localhost:5000/jig/bean/discoveryService.xml
HTTP/1.1 200
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sun, 19 Jul 2020 20:29:47 GMT

<DiscoveryService>
  <uuid:00000000-0000-1010-8000-0024BEF18BCC>
    <expiration>1595192286661</expiration>
    <ssdpmessage>
      <params/>
      <entity/>
      <locale/>
      <inetAddress>10.0.1.9</inetAddress>
      <st>uuid:00000000-0000-1010-8000-0024BEF18BCC</st>
      <location>http://10.0.1.9:52323/dmr.xml</location>
      <usn>uuid:00000000-0000-1010-8000-0024BEF18BCC</usn>
      <protocolVersion>
        <protocol>HTTP</protocol>
        <major>1</major>
        <minor>1</minor>
      </protocolVersion>
      ...
    </ssdpmessage>
  </uuid:00000000-0000-1010-8000-0024BEF18BCC>
  ...
</DiscoveryService>

The implementation is described in the next section.

Implementation

The steps to create the "jig" starter:

Implement the REST controller
Create a project and POM for the starter artifact
Create the auto-configuration class(es)
Link the auto-configuration class(es) into the starter's META-INF/spring.factories resource¹

The BeanRestController implementation is shown below.

ball.spring.jig.BeanRestController

@RestController
@RequestMapping(value = { "/jig/bean/" })
@ResponseBody
@NoArgsConstructor @ToString @Log4j2
public class BeanRestController implements ApplicationContextAware {
    private ApplicationContext context = null;

    @Override
    public void setApplicationContext(ApplicationContext context) {
        this.context = context;
    }

    @RequestMapping(method = { GET }, value = { "{name}.json" }, produces = APPLICATION_JSON_VALUE)
    public Object json(@PathVariable String name) throws Exception {
        return context.getBean(name);
    }

    @RequestMapping(method = { GET }, value = { "{name}.xml" }, produces = APPLICATION_XML_VALUE)
    public Object xml(@PathVariable String name) throws Exception {
        return context.getBean(name);
    }

    @ExceptionHandler({ NoSuchBeanDefinitionException.class, NoSuchElementException.class })
    @ResponseStatus(value = NOT_FOUND, reason = "Resource not found")
    public void handleNOT_FOUND() { }
}

Its implementation is straightforward: A method each to look-up the requested bean and then serialize to JSON and XML.

In the project for the starter, add the AutoConfiguration class.

ball.spring.jig.autoconfigure.AutoConfiguration

@Configuration
@ConditionalOnClass({ BeanRestController.class })
@Import({ BeanRestController.class })
@NoArgsConstructor @ToString @Log4j2
public class AutoConfiguration {
}

It is critical that @Configuration classes are added through @Import annotations and not @ComponentScan Neither the starter author nor the integrator will be able to predict what components will or will not be included in a scan.

It is a good practice to include a "conditional-on" annotation (e.g., @ConditionalOnClass to test any required dependency software has been configured and/or is on the class path.

Finally, META-INF/spring.factories must be configured in the starter JAR to notify Spring Boot to add the AutoConfiguration.

META-INF/spring.factories

org.springframework.boot.autoconfigure.EnableAutoConfiguration: ball.spring.jig.autoconfigure.AutoConfiguration

Creating a starter for the embedded MySQL process described in "Spring Embedded MySQL Server" is equally straightforward. Its AutoConfiguration class is shown below:

ball.spring.mysqld.autoconfigure.AutoConfiguration

@Configuration
@ConditionalOnClass({ MysqldConfiguration.class })
@Import({ EntityManagerFactoryComponent.class, MysqldConfiguration.class })
@NoArgsConstructor @ToString @Log4j2
public class AutoConfiguration {
}

With it corresponding META-INF/spring.factories:

META-INF/spring.factories

org.springframework.boot.autoconfigure.EnableAutoConfiguration: ball.spring.mysqld.autoconfigure.AutoConfiguration

Summary

Creating a Spring Boot starter is straightfoward: Create a project/POM to host the starter's dependencies and auto-configuration class(es), add the annotated auto-configuration class(es), and configure the starter JAR's META-INF/spring.factories with the auto-configuration class(es). That starter's functionality can then be added to a Spring Boot application simply by including a single dependency in the application POM.

[1] Many Spring Boot components provide separate auto-configuration and starter artifacts to support all use cases. The author feels these example implementations do not benefit from separate auto-configuration artifacts. ↩

Java Multi-Release JARs

Allen D. Ball — Sun, 19 Jul 2020 17:39:18 +0000

"Adding Support to Java InvocationHandler Implementations for Interface Default Methods" describes how to implement an InvocationHandler to invoke default interface methods. This mechanism is critical to the FluentNode implementation described in "Java Interface Facades". The first article also notes that the MethodHandles.Lookup method used with Java 8 would not work with Java 9 which means the whole API will not work on Java 9 and subsequent JVMs.

This article describes the Java 9-specific solution, refactoring the InvocationHandler implementation to separate and compartmentalize the Java 8 and Java 9-specific solution logic, and introduces "JEP 238: Multi-Release JAR Files" to deliver a Java 8 and Java 9 (and later) solutions simultaneously in the same JAR.

Theory of Operation

As described in JEP 238, multi-release JARs provide a means to provide alternate versions of classes that can take advantage of specific platform features. Alternate classes are stored in the JAR within the hierarchy described by /META-INF/versions/${java.specification.version}/. (The implementation hierarchy is shown in detail at the end of this article.) For archivers and class-loaders that are not multi-release aware (e.g., Java 8), these additional classes are ignore. However, for Java 9 and subsequent environments, these additional classes are loaded if the version is less than or equal to the JVM's ${java.specification.version} (with the latest taking precedence).

The class with the Java 8-specific code will be refactored to implement a super-interface with a single method embodying that code. That existing code base will be compiled as-is to create a JAR suitable for Java 8 JVMs. In addition, a Java 9 version of the super-interface will be compiled for a Java 9 environment and saved to the JAR beneath the /META-INF/versions/9/ hierarchy.

Implementation

DefaultInvocationHandler.invoke(Object,Method,Object[]) is re-factored to implement DefaultInterfaceMethodInvocationHandler:

DefaultInvocationHandler

@NoArgsConstructor @ToString
public class DefaultInvocationHandler implements DefaultInterfaceMethodInvocationHandler {
    ...
    @Override
    public Object invoke(Object proxy, Method method, Object[] argv) throws Throwable {
        Object result = null;
        Class<?> declarer = method.getDeclaringClass();

        if (method.isDefault()) {
            result = DefaultInterfaceMethodInvocationHandler.super.invoke(proxy, method, argv);
        } else if (declarer.equals(Object.class)) {
            result = method.invoke(this, argv);
        } else {
            result = invokeMethod(this, true, method.getName(), argv, method.getParameterTypes());
        }

        return result;
    }
    ...
}

With the Java 8 implementation:

DefaultInterfaceMethodInvocationHandler (Java 8)

public interface DefaultInterfaceMethodInvocationHandler extends InvocationHandler {
    @Override
    default Object invoke(Object proxy, Method method, Object[] argv) throws Throwable {
        Constructor<MethodHandles.Lookup> constructor =
            MethodHandles.Lookup.class.getDeclaredConstructor(Class.class);

        constructor.setAccessible(true);

        Class<?> declarer = method.getDeclaringClass();
        Object result =
            constructor.newInstance(declarer)
            .in(declarer)
            .unreflectSpecial(method, declarer)
            .bindTo(proxy)
            .invokeWithArguments(argv);

        return result;
    }
}

While the Java 9 implementation is:

DefaultInterfaceMethodInvocationHandler (Java 9)

public interface DefaultInterfaceMethodInvocationHandler extends InvocationHandler {
    @Override
    default Object invoke(Object proxy, Method method, Object[] argv) throws Throwable {
        Class<?> declarer = method.getDeclaringClass();
        Object result =
            MethodHandles.lookup()
            .findSpecial(declarer, method.getName(),
                         methodType(method.getReturnType(), method.getParameterTypes()), declarer)
            .bindTo(proxy)
            .invokeWithArguments(argv);

        return result;
    }
}

The Java 9 alternative is saved to ${basedir}/src/main/java9/ball/lang/reflect/DefaultInterfaceMethodInvocationHandler.java within the Maven project. Compiling with the maven-compiler-plugin is straightforward as the following profile demonstrates the incremental configuration.¹

Parent POM: Java 9 Profile

<project ...>
  ...
  <profiles>
  ...
    <profile>
      <id>[+] src/main/java9</id>
      <activation>
        <file><exists>${basedir}/src/main/java9</exists></file>
      </activation>
      <build>
        <pluginManagement>
          <plugins>
            <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-compiler-plugin</artifactId>
              <executions>
                <execution>
                  <id>jdk9</id>
                  <goals>
                    <goal>compile</goal>
                  </goals>
                  <configuration>
                    <release>9</release>
                    <compileSourceRoots>
                      <compileSourceRoot>${project.basedir}/src/main/java9</compileSourceRoot>
                    </compileSourceRoots>
                    <multiReleaseOutput>true</multiReleaseOutput>
                  </configuration>
                </execution>
              </executions>
            </plugin>
          </plugins>
        </pluginManagement>
      </build>
    </profile>
  ...
  </profiles>
  ...
</project>

To complete the JAR's configuration, the Multi-Release flag must be set in the JAR's manifest.

META-INF/MANIFEST.MF

Manifest-Version: 1.0
...
Multi-Release: true
...

The resulting JAR hierarchy is depicted below.

Muti-Release JAR Hierarchy

ball-util.jar
├── META-INF
│   ├── MANIFEST.MF
│   ├── ...
│   └── versions
│       └── 9
│           └── ball
│               └── lang
│                   └── reflect
│                       └── DefaultInterfaceMethodInvocationHandler.class
└── ball
    ├── ...
    ├── lang
    │   ├── ...
    │   └── reflect
    │       ├── DefaultInterfaceMethodInvocationHandler.class
    │       ├── DefaultInvocationHandler.class
    │       └── ...
    ├── ...
    ...

Summary

Multi-release JARs provide a solution for a single JAR to support a range of Java platform versions. Developers should be aware that the solution does present some challenges: The compiled versioned class files cannot be run outside the JAR (making testing difficult) and class-loader implementation details will effect resource discovery and loading² (to name a few). However, as illustrated with this use-case, it is a powerful tool for creating JARs that support a wide range of Java platforms.

[1] The Parent POM has similar profiles for Java 10 through 14. ↩

[2] Please see the discussion in "JEP 238: Multi-Release JAR Files". ↩

Spring Boot Part 5: voyeur, A Non-Trivial Application

Allen D. Ball — Fri, 26 Jun 2020 23:08:01 +0000

This series of articles examines Spring Boot features. This fifth article in the series presents a non-trivial application which probes local hosts (with the help of the nmap command) to assist in developing UPNP and SSDP applications.

Complete source and javadoc are available on GitHub. Additional artifacts (including their respective source and javadoc JARs) are available from the Maven repository at:

    <repository>
      <id>repo.hcf.dev-RELEASE</id>
      <name>hcf.dev RELEASE Repository</name>
      <url>https://repo.hcf.dev/maven/release/</url>
      <layout>default</layout>
    </repository>

Specific topics covered herein:

@Service implementations with @Scheduled updates
- @Autowired
UI @Controller
- Populates Model
- Thymeleaf temlates and decoupled logic
@RestController implementation

Theory of Operation

The following subsections describe the components.

@Service Implementations

The voyeur package defines a number of (annotated) @Services:

`@Service`	Description
`ArpCache`	`Map` of `InetAddress` to hardware address periodically updated by reading `/proc/net/arp` or parsing the output of `arp -an`
`NetworkInterfaces`	`Set` of `NetworkInterface`s
`Nmap`	`Map` of XML output of the `nmap` command for each `InetAddress` discovered via `ARPCache`, `NetworkInterfaces`, and/or `SSDP`
`SSDP`	SSDP hosts discovered via `SSDPDiscoveryCache`

Each of these services implement a Set or Map, which may
be @Autowired into other components, and periodically update
themselves with a @Scheduled method. The
Nmap service is examined in detail.

First, the @PostConstruct method (in addition to
performing other initialization chores) tests to determine if the
nmap command is available:

...
@Service
@NoArgsConstructor @Log4j2
public class Nmap extends InetAddressMap<Document> ... {
    ...
    private static final String NMAP = "nmap";
    ...
    private boolean disabled = true;
    ...
    @PostConstruct
    public void init() throws Exception {
        ...
        try {
            List<String> argv = Stream.of(NMAP, "-version").collect(toList());

            log.info(String.valueOf(argv));

            Process process =
                new ProcessBuilder(argv)
                .inheritIO()
                .redirectOutput(PIPE)
                .start();

            try (InputStream in = process.getInputStream()) {
                new BufferedReader(new InputStreamReader(in, UTF_8))
                    .lines()
                    .forEach(t -> log.info(t));
            }

            disabled = (process.waitFor() != 0);
        } catch (Exception exception) {
            disabled = true;
        }

        if (disabled) {
            log.warn("nmap command is not available");
        }
    }
    ...
    public boolean isDisabled() { return disabled; }
    ...
}

If the nmap command is successful, its version is logged. Otherwise, disabled is set to true and no further attempt is made to run the nmap command in other methods.

The @Scheduled update() method is invoked every 30 seconds and ensures a map entry exists for every InetAddress previously discovered by the NetworkInterfaces, ARPCache, and SSDP components and then queues a Worker Runnable for any value whose output is more than INTERVAL (60 minutes) old. The @EventListener (with ApplicationReadyEvent guarantees the method won't be called before the application is ready (to serve requests).

public class Nmap extends InetAddressMap<Document> ... {
    ...
    private static final Duration INTERVAL = Duration.ofMinutes(60);
    ...
    @Autowired private NetworkInterfaces interfaces = null;
    @Autowired private ARPCache arp = null;
    @Autowired private SSDP ssdp = null;
    @Autowired private ThreadPoolTaskExecutor executor = null;
    ...
    @EventListener(ApplicationReadyEvent.class)
    @Scheduled(fixedDelay = 30 * 1000)
    public void update() {
        if (! isDisabled()) {
            try {
                Document empty = factory.newDocumentBuilder().newDocument();

                empty.appendChild(empty.createElement("nmaprun"));

                interfaces
                    .stream()
                    .map(NetworkInterface::getInterfaceAddresses)
                    .flatMap(List::stream)
                    .map(InterfaceAddress::getAddress)
                    .filter(t -> (! t.isMulticastAddress()))
                    .forEach(t -> putIfAbsent(t, empty));

                arp.keySet()
                    .stream()
                    .filter(t -> (! t.isMulticastAddress()))
                    .forEach(t -> putIfAbsent(t, empty));

                ssdp.values()
                    .stream()
                    .map(SSDP.Value::getSSDPMessage)
                    .filter(t -> t instanceof SSDPResponse)
                    .map(t -> ((SSDPResponse) t).getInetAddress())
                    .forEach(t -> putIfAbsent(t, empty));

                keySet()
                    .stream()
                    .filter(t -> INTERVAL.compareTo(getOutputAge(t)) < 0)
                    .map(Worker::new)
                    .forEach(t -> executor.execute(t));
            } catch (Exception exception) {
                log.error(exception.getMessage(), exception);
            }
        }
    }
    ...
    private Duration getOutputAge(InetAddress key) {
        long start = 0;
        Number number = (Number) get(key, "/nmaprun/runstats/finished/@time", NUMBER);

        if (number != null) {
            start = number.longValue();
        }

        return Duration.between(Instant.ofEpochSecond(start), Instant.now());
    }

    private Object get(InetAddress key, String expression, QName qname) {
        Object object = null;
        Document document = get(key);

        if (document != null) {
            try {
                object = xpath.compile(expression).evaluate(document, qname);
            } catch (Exception exception) {
                log.error(exception.getMessage(), exception);
            }
        }

        return object;
    }
    ...
}

Spring Boot's ThreadPoolTaskExecutor is injected. To guarantee more than one thread is allocated the application.properties contains the following property:

spring.task.scheduling.pool.size: 4

The Worker implementation is given below.

    ...
    private static final List<String> NMAP_ARGV =
        Stream.of(NMAP, "--no-stylesheet", "-oX", "-", "-n", "-PS", "-A")
        .collect(toList());
    ...
    @RequiredArgsConstructor @EqualsAndHashCode @ToString
    private class Worker implements Runnable {
        private final InetAddress key;

        @Override
        public void run() {
            try {
                List<String> argv = NMAP_ARGV.stream().collect(toList());

                if (key instanceof Inet4Address) {
                    argv.add("-4");
                } else if (key instanceof Inet6Address) {
                    argv.add("-6");
                }

                argv.add(key.getHostAddress());

                DocumentBuilder builder = factory.newDocumentBuilder();
                Process process =
                    new ProcessBuilder(argv)
                    .inheritIO()
                    .redirectOutput(PIPE)
                    .start();

                try (InputStream in = process.getInputStream()) {
                    put(key, builder.parse(in));

                    int status = process.waitFor();

                    if (status != 0) {
                        throw new IOException(argv + " returned exit status " + status);
                    }
                }
            } catch (Exception exception) {
                remove(key);
                log.error(exception.getMessage(), exception);
            }
        }
    }
    ...

Note that the InetAddress will be removed from the Map if the Process fails.

UI @Controller, Model, and Thymeleaf Template

The complete UIController implementation is given below.

@Controller
@NoArgsConstructor @ToString @Log4j2
public class UIController extends AbstractController {
    @Autowired private SSDP ssdp = null;
    @Autowired private NetworkInterfaces interfaces = null;
    @Autowired private ARPCache arp = null;
    @Autowired private Nmap nmap = null;

    @ModelAttribute("upnp")
    public Map<URI,List<URI>> upnp() {
        Map<URI,List<URI>> map =
            ssdp().values()
            .stream()
            .map(SSDP.Value::getSSDPMessage)
            .collect(groupingBy(SSDPMessage::getLocation,
                                ConcurrentSkipListMap::new,
                                mapping(SSDPMessage::getUSN, toList())));

        return map;
    }

    @ModelAttribute("ssdp")
    public SSDP ssdp() { return ssdp; }

    @ModelAttribute("interfaces")
    public NetworkInterfaces interfaces() { return interfaces; }

    @ModelAttribute("arp")
    public ARPCache arp() { return arp; }

    @ModelAttribute("nmap")
    public Nmap nmap() { return nmap; }

    @RequestMapping(value = {
                        "/",
                        "/upnp/devices", "/upnp/ssdp",
                        "/network/interfaces", "/network/arp", "/network/nmap"
                    })
    public String root(Model model) { return getViewName(); }

    @RequestMapping(value = { "/index", "/index.htm", "/index.html" })
    public String index() { return "redirect:/"; }
}

The @Controller populates the Model with five attributes and implements the root method¹ to serve the UI request paths. The superclass implements getViewName() which creates a view name based on the implementing class's package which translates to classpath:/templates/voyeur.html, a Thymeleaf template to generate a pure HTML5 document. Its outline is shown below.

<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org" th:xmlns="@{http://www.w3.org/1999/xhtml}">
  <head>
    ...
  </head>
  <body>
    ...
    <header>
      <nav th:ref="navbar">
        ...
      </nav>
    </header>
    <main th:unless="${#ctx.containsVariable('exception')}"
          th:switch="${#request.servletPath}">
      <section th:case="'/error'">
        ...
      </section>
      <section th:case="'/upnp/devices'">
        ...
      </section>
      <section th:case="'/upnp/ssdp'">
        ...
      </section>
      ...
    </main>
    <main th:if="${#ctx.containsVariable('exception')}">
      ...
    </main>
    <footer>
      <nav th:ref="navbar">
        ...
      </nav>
    </footer>
    <script/>
  </body>
</html>

The template's <header/> <nav/> implements the menu and references the paths specified in the UIController.root().

      <nav th:ref="navbar">
        <th:block th:ref="container">
          <div th:ref="navbar-brand">
            <a th:text="${#strings.defaultString(brand, 'Home')}" th:href="@{/}"/>
          </div>
          <div th:ref="navbar-menu">
            <ul th:ref="navbar-start"></ul>
            <ul th:ref="navbar-end">
              <li th:ref="navbar-item">
                <button th:text="'UPNP'"/>
                <ul th:ref="navbar-dropdown">
                  <li><a th:text="'Devices'" th:href="@{/upnp/devices}"/></li>
                  <li><a th:text="'SSDP'" th:href="@{/upnp/ssdp}"/></li>
                </ul>
              </li>
              <li th:ref="navbar-item">
                <button th:text="'Network'"/>
                <ul th:ref="navbar-dropdown">
                  <li><a th:text="'Interfaces'" th:href="@{/network/interfaces}"/></li>
                  <li><a th:text="'ARP'" th:href="@{/network/arp}"/></li>
                  <li><a th:text="'Nmap'" th:href="@{/network/nmap}"/></li>
                </ul>
              </li>
            </ul>
          </div>
        </th:block>
      </nav>

The template is also structured to produce a <main/> node with a <section/> node corresponding to the request path if there is no exception variable in the context (normal operation). The th:switch and th:case attributes are used to create a <section/> corresponding to each ${#request.servletPath}. The <section/> specific to the /network/nmap path is shown below:

    <main th:unless="${#ctx.containsVariable('exception')}"
          th:switch="${#request.servletPath}">
      ...
      <section th:case="'/network/nmap'">
        <table>
          <tbody>
            <tr th:each="key : ${nmap.keySet()}">
              <td>
                <a th:href="@{/network/nmap/{ip}.xml(ip=${key.hostAddress})}" th:target="_newtab">
                  <code th:text="${key.hostAddress}"/>
                </a>
                <p><code th:text="${nmap.getPorts(key)}"/></p>
              </td>
              <td>
                <p th:each="product : ${nmap.getProducts(key)}" th:text="${product}"/>
              </td>
            </tr>
          </tbody>
        </table>
      </section>
      ...
    </main>

The template generates a <table/> with a row (<tr/>) for each key in the Nmap. Each row consists of two columns (<td/>):

The InetAddress of the host with a link to the nmap command output² and a list of open TCP ports
The services/products detected

The getPorts(InetAddress) and getProducts(InetAddress) methods are provided to avoid XPath calculations within the Thymeleaf template.

...
@Service
@NoArgsConstructor @Log4j2
public class Nmap extends InetAddressMap<Document> ... {
    ...
    public Set<Integer> getPorts(InetAddress key) {
        Set<Integer> ports = new TreeSet<>();
        NodeList list = (NodeList) get(key, "/nmaprun/host/ports/port/@portid", NODESET);

        if (list != null) {
            for (int i = 0; i < list.getLength(); i += 1) {
                ports.add(Integer.parseInt(list.item(i).getNodeValue()));
            }
        }

        return ports;
    }
    ...
}

The getProducts(InetAddress) implementation is similar with an XPathExression of /nmaprun/host/ports/port/service/@product.

The UIController instance combined with the Thymeleaf template described so far will only generate pure HTML5 with no style markup. This implementation uses Thymeleaf's Decoupled Template Logic feature and can be found at classpath:/templates/voyeur.th.xml.³ The decoupled logic for the table described in this section is shown below.

<?xml version="1.0" encoding="UTF-8"?>
<thlogic>
  ...
  <attr sel="body">
    ...
    <attr sel="main" th:class="'container'">
      <attr sel="table" th:class="'table table-striped'">
        <attr sel="tbody">
          <attr sel="tr" th:class="'row'"/>
          <attr sel="tr/td" th:class="'col'"/>
        </attr>
      </attr>
    </attr>
    ...
  </attr>
</thlogic>

The UIController superclass provides one more feature: To inject the proprties defined in classpath:/templates/voyeur.model.properties into the Model.

brand = ${application.brand:}

stylesheets: /webjars/bootstrap/css/bootstrap.css
style:\
body { padding-top: 60px; margin-bottom: 60px; }\n\
@media (max-width: 979px) { body { padding-top: 0px; } }
scripts: /webjars/jquery/jquery.js, /webjars/bootstrap/js/bootstrap.js

The design goal of this implementation was to commit all markup logic to the *.th.xml resource allowing only the necessity to modify the decoupled logic and the model properties to use an alternate framework. This goal was defeated in this implementation because different frameworks support to different degrees HTML5 elements. A partial Bulma implementation is available in https://github.com/allen-ball/voyeur/tree/trunk/src/main/resources/templates-bulma which demonstrates the HTML5 differences.

The /network/nmap request is shown rendered in the image in the Introduction of this article.

nmap Output @RestController

nmap XML output can be served by implementing a @RestController. The Nmap class is annotated with @RestController and @RequestMapping to requests for /network/nmap/ and the nmap(String) method provides the XML serialized to a String.

@RestController
@RequestMapping(value = { "/network/nmap/" }, produces = MediaType.APPLICATION_XML_VALUE)
...
public class Nmap ... {
    ...
    @RequestMapping(value = { "{ip}.xml" })
    public String nmap(@PathVariable String ip) throws Exception {
        ByteArrayOutputStream out = new ByteArrayOutputStream();

        transformer.transform(new DOMSource(get(InetAddress.getByName(ip))),
                              new StreamResult(out));

        return out.toString("UTF-8");
    }
    ...
}

Packaging

The spring-boot-maven-plugin has a repackage goal which may be used to create a self-contained JAR with an embedded launch script. That goal is used in the project pom to create and attach a self-contained JAR atifact.

<project ...>
  ...
  <build>
    <pluginManagement>
      <plugins>
        ...
        <plugin>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-maven-plugin</artifactId>
          <executions>
            <execution>
              <goals>
                <goal>build-info</goal>
                <goal>repackage</goal>
              </goals>
            </execution>
          </executions>
          <configuration>
            <attach>true</attach>
            <classifier>bin</classifier>
            <executable>true</executable>
            <mainClass>${start-class}</mainClass>
            <embeddedLaunchScriptProperties>
              <inlinedConfScript>${basedir}/src/bin/inline.conf</inlinedConfScript>
            </embeddedLaunchScriptProperties>
          </configuration>
        </plugin>
        ...
      </plugins>
    </pluginManagement>
    ...
  </build>
  ...
</project>

Please see the project GitHub page for instructions on how to run the JAR.

Summary

This article discusses aspects of the voyeur application and provides specific examples of:

@Service implementation and @Autowired components with @Scheduled methods
@Controller implementation, Model population, and Thymeleaf templates and decoupled logic
@RestController implementation

[1] A misleading method name at best. ↩

[2] The @RestController is described in the next subsection. ↩

[3] The common application properties do not provide an option to enable this functionality. It is enabled in the UIController suprclass by configuring the injected SpringResourceTemplateResolver. ↩

Java Stream Tree Walker

Allen D. Ball — Wed, 17 Jun 2020 22:26:07 +0000

This article discusses walking a tree or graph of nodes with a Java 8 Stream implementation. The implementation is codified in a Spliterator The strategy described herein can be a useful alternative to implementing a class-hierarchy specific visitor because only a single per-type method need be defined (often as a Java lambda).

Please refer to this article for an in-depth discussion of creating Spliterators.

API Definition

The implementaion provides the following API:

public class Walker<T> ... {
    ...
    public static <T> Stream<T> walk(T root, Function<? super T,Stream<? extends T>> childrenOf)
    ...
}

The Function provides the subordinate ("children") nodes of the argument node. For example, for Java Classes to obtain inner (declared) Classes:

    t -> Stream.of(t.getDeclaredClasses()

or for Files:

    t -> t.isDirectory() ? Stream.of(t.listFiles()) : Stream.empty()

Implementation

The API provides a static method to create a Stream from the implemented Spliterator:

    public static <T> Stream<T> walk(T root, Function<? super T,Stream<? extends T>> childrenOf) {
        return StreamSupport.stream(new Walker<>(root, childrenOf), false);
    }

The complete Spliterator implementation is:

public class Walker<T> extends Spliterators.AbstractSpliterator<T> {
    private final Stream<Supplier<Spliterator<T>>> stream;
    private Iterator<Supplier<Spliterator<T>>> iterator = null;
    private Spliterator<? extends T> spliterator = null;

    private Walker(T node, Function<? super T,Stream<? extends T>> childrenOf) {
        super(Long.MAX_VALUE, IMMUTABLE | NONNULL);

        stream =
            Stream.of(node)
            .filter(Objects::nonNull)
            .flatMap(childrenOf)
            .filter(Objects::nonNull)
            .map(t -> (() -> new Walker<T>(t, childrenOf)));
        spliterator = Stream.of(node).spliterator();
    }

    @Override
    public Spliterator<T> trySplit() {
        if (iterator == null) {
            iterator = stream.iterator();
        }

        return iterator.hasNext() ? iterator.next().get() : null;
    }

    @Override
    public boolean tryAdvance(Consumer<? super T> consumer) {
        boolean accepted = false;

        while (! accepted) {
            if (spliterator == null) {
                spliterator = trySplit();
            }

            if (spliterator != null) {
                accepted = spliterator.tryAdvance(consumer);

                if (! accepted) {
                    spliterator = null;
                }
            } else {
                break;
            }
        }

        return accepted;
    }
    ...
}

A Stream of Spliterator Suppliers is created at instantiation (and the Supplier will supply another Walker). The first time trySplit() is called the Stream is converted to an Iterator and the next Spliterator is generated. The tryAdvance(Consumer) will exhaust the last created Spliterator before calling tryAdvance() to obtain another. The first Spliterator consists solely of the root node.

Examples

To print the inner defined classes of Collections:

        Walker.<Class<?>>walk(java.util.Collections.class,
                              t -> Stream.of(t.getDeclaredClasses()))
            .limit(10)
            .forEach(System.out::println);

Yields:

class java.util.Collections
class java.util.Collections$UnmodifiableCollection
class java.util.Collections$UnmodifiableSet
class java.util.Collections$UnmodifiableSortedSet
class java.util.Collections$UnmodifiableNavigableSet
class java.util.Collections$UnmodifiableNavigableSet$EmptyNavigableSet
class java.util.Collections$UnmodifiableRandomAccessList
class java.util.Collections$UnmodifiableList
class java.util.Collections$UnmodifiableMap
class java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet

And to print the directory structure (sorted):

        Walker.walk(new File("."),
                    t -> t.isDirectory() ? Stream.of(t.listFiles()) : Stream.empty())
            .filter(File::isDirectory)
            .sorted()
            .forEach(System.out::println);

Yields:

.
./src
./src/main
./src/main/resources
./target

This presents a flexible solution and can be extended with disparate objects. For example, a method to walk XML Nodes might be:

    public static Stream<Node> childrenOf(Node node) {
        NodeList list = node.getChildNodes();

        return IntStream.range(0, list.getLength()).mapToObj(list::item);
    }

Alternate Entry Point

It is straightforward to offer an alternate entry point where multiple root nodes are supplied by defining a corresponding constructor:

public class Walker<T> extends Spliterators.AbstractSpliterator<T> {
    ...
    private Walker(Stream<T> nodes, Function<? super T,Stream<? extends T>> childrenOf) {
        super(Long.MAX_VALUE, IMMUTABLE | NONNULL);

        stream = nodes.map(t -> (() -> new Walker<T>(t, childrenOf)));
    }
    ...
    public static <T> Stream<T> walk(Stream<T> roots, Function<? super T,Stream<? extends T>> childrenOf) {
        return StreamSupport.stream(new Walker<>(roots, childrenOf), false);
    }
    ...
}

Summary

Streams created from the Spliterator implementation described herein provide a versatile means for walking any tree with a minimum per-type implementation while offering all the filtering and processing power of the Stream API.

AWS VPC Set-Up with Ansible

Allen D. Ball — Mon, 16 Mar 2020 21:28:27 +0000

A critical first step to creating an Amazon Web Services (AWS) AWS Elastic Compute Cloud (EC2) configuration is to configure a Virtual Private Cloud (VPC) Often, the default configuration is sufficient for most administrators' needs but some solutions require an IP address space different than the Amazon default. This article presents the Ansible boilerplate for configuring an alternative IP address space specified by a minimum of parameters: region, VPC CIDR block (e.g., 10.1.0.0/16) and subnet mask size (e.g., 20). The boilerplate calculates subnet CIDR blocks (e.g., 10.1.0.0/20, 10.1.16.0/20, etc...) for each availability zone within the region.

The solution leverages Ansible's ipaddr filter interface to the netaddr Python package with its extended loop variables. The solution also makes extensive use of the Ansible JSON query filter to parse the results of the AWS modules.

Theory of Operation

The implementation:

Requires the specification of:

- AWS profile (for autheniticaion and use as a project-level name)
- AWS region (where the VPC will be deployed)
- VPC CIDR block
- subnet mask size (in bits)

Creates the VPC in the specified region (with the same name as the profile for idempotent operation)
For each availability zone within the region, creates a unique subnet within the VPC's CIDR block of the specified size
Creates an Internet Gateway and connects to each of the subnets

The implementation also demonstrates how host IP address may be calculated relative to an availability zone's subnet.

Implementation

The Ansible controller must have the netaddr Python package installed.

$ pip install --upgrade netaddr

In these examples the administrator has configured the AWS CLI environment variables AWS_PROFILE and AWS_DEFAULT_REGION to provide the necessary parameters. The CIDR block and subnet mask size are also specified for the project as Ansible facts.

- name: Role Parameters
  set_fact:
    profile: "{{ lookup('env', 'AWS_PROFILE') }}"
    cidr_block: 10.1.0.0/16
    subnet_mask_size: 20

- name: aws_region_info
  aws_region_info:
    filters:
      region_name: "{{ lookup('env', 'AWS_DEFAULT_REGION') }}"
  register: aws_region_info

The aws_region_info module is invoked to verify the AWS_DEFAULT_REGION parameter.

    "aws_region_info": {
        "changed": false,
        "failed": false,
        "regions": [
            {
                "endpoint": "ec2.us-west-1.amazonaws.com",
                "opt_in_status": "opt-in-not-required",
                "region_name": "us-west-1"
            }
        ]
    }

Once verified, the region is set as a fact for clarity and ease of reference in subsequent module invocations.

- name: region
  set_fact:
    region: "{{ aws_region_info.regions[0].region_name }}"

The VPC may be created with the above parameters. Note that the AWS_PROFILE value is used to name the VPC for idempotent operation.

- name: "{{ profile }} VPC"
  ec2_vpc_net:
    name: "{{ profile }}"
    region: "{{ region }}"
    cidr_block: "{{ cidr_block }}"
  register: ec2_vpc_net

    "ec2_vpc_net": {
        "changed": true,
        "failed": false,
        "vpc": {
            "cidr_block": "10.1.0.0/16",
            "cidr_block_association_set": [
                {
                    "association_id": "vpc-cidr-assoc-ffffffffffffa4c3",
                    "cidr_block": "10.1.0.0/16",
                    "cidr_block_state": {
                        "state": "associated"
                    }
                }
            ],
            "classic_link_enabled": false,
            "dhcp_options_id": "dopt-ffffffffffff49b8",
            "id": "vpc-ffffffffffff9ace",
            "instance_tenancy": "default",
            "is_default": false,
            "owner_id": "999999999999",
            "state": "available",
            "tags": {
                "Name": "PROFILE"
            }
        }
    }

The ec2_vpc_net.vpc is set as a fact for ease of reference in subsequent modules.

- name: vpc
  set_fact:
    vpc: "{{ ec2_vpc_net.vpc }}"

The aws_az_info module is invoked to retrieve the availability zones for the region:

- name: aws_az_info
  aws_az_info:
    filters:
      region_name: "{{ aws_region_info.regions[0].region_name }}"
  register: aws_az_info

    "aws_az_info": {
        "availability_zones": [
            {
                "group_name": "us-west-1",
                "messages": [],
                "network_border_group": "us-west-1",
                "opt_in_status": "opt-in-not-required",
                "region_name": "us-west-1",
                "state": "available",
                "zone_id": "usw1-az1",
                "zone_name": "us-west-1a"
            },
            {
                "group_name": "us-west-1",
                "messages": [],
                "network_border_group": "us-west-1",
                "opt_in_status": "opt-in-not-required",
                "region_name": "us-west-1",
                "state": "available",
                "zone_id": "usw1-az3",
                "zone_name": "us-west-1c"
            }
        ],
        "changed": false,
        "failed": false
    }

The necessary parameters to create the subnets have been accumulated. The JSON query availability_zones[].zone_name is applied to the results of aws_az_info resulting in the string array (['us-west-1a', 'us-west-1c']) of availability zone names. This array is iterated over to calculate the subnet CIDR blocks based on vpc.cidr_block, subnet mask size, and the array index (ansible_loop.index0).

- name: "{{ profile }} VPC Subnets"
  vars:
    json: "{{ aws_az_info }}"
    query: "availability_zones[].zone_name"
    availability_zones: "{{ json | json_query(query) }}"
  ec2_vpc_subnet:
    vpc_id: "{{ vpc.id }}"
    az: "{{ item }}"
    cidr: >-
      {{ vpc.cidr_block | ipsubnet(subnet_mask_size, ansible_loop.index0) }}
  loop: "{{ availability_zones }}"
  loop_control:
    extended: yes

- name: ec2_vpc_subnet_info
  ec2_vpc_subnet_info:
    filters:
      vpc-id: "{{ vpc.id }}"
  register: ec2_vpc_subnet_info

The descriptions of the subnets are retrieved with ec2_vpc_subnet_info:

    "ec2_vpc_subnet_info": {
        "changed": false,
        "failed": false,
        "subnets": [
            {
                "assign_ipv6_address_on_creation": false,
                "availability_zone": "us-west-1c",
                "availability_zone_id": "usw1-az3",
                "available_ip_address_count": 4091,
                "cidr_block": "10.1.16.0/20",
                "default_for_az": false,
                "id": "subnet-ffffffffffffeb5e",
                "ipv6_cidr_block_association_set": [],
                "map_public_ip_on_launch": false,
                "owner_id": "999999999999",
                "state": "available",
                "subnet_arn": "arn:aws:ec2:us-west-1:999999999999:subnet/subnet-ffffffffffffeb5e",
                "subnet_id": "subnet-ffffffffffffeb5e",
                "tags": {},
                "vpc_id": "vpc-ffffffffffff9ace"
            },
            {
                "assign_ipv6_address_on_creation": false,
                "availability_zone": "us-west-1a",
                "availability_zone_id": "usw1-az1",
                "available_ip_address_count": 4091,
                "cidr_block": "10.1.0.0/20",
                "default_for_az": false,
                "id": "subnet-ffffffffffff57fd",
                "ipv6_cidr_block_association_set": [],
                "map_public_ip_on_launch": false,
                "owner_id": "999999999999",
                "state": "available",
                "subnet_arn": "arn:aws:ec2:us-west-1:999999999999:subnet/subnet-ffffffffffff57fd",
                "subnet_id": "subnet-ffffffffffff57fd",
                "tags": {},
                "vpc_id": "vpc-ffffffffffff9ace"
            }
        ]
    }

With the above information, an Internet Gateway may be created and added to the route tables for the subnets.

- name: "{{ profile }} VPC IGW"
  ec2_vpc_igw:
    vpc_id: "{{ vpc.id }}"
  register: ec2_vpc_igw

- name: "{{ profile }} VPC IGW Route Table"
  vars:
    json: "{{ ec2_vpc_subnet_info }}"
    query: "subnets[].id"
    subnets: "{{ json | json_query(query) }}"
  ec2_vpc_route_table:
    vpc_id: "{{ vpc.id }}"
    tags:
      Name: Internet
    subnets: "{{ subnets }}"
    routes:
      - dest: 0.0.0.0/0
        gateway_id: "{{ ec2_vpc_igw.gateway_id }}"

The following snippet demonstrates how a subnet may be found for an availability zone and a host IP may be calculated relative to that subnet.

- name: availability_zone
  set_fact:
    availability_zone: "{{ region }}a"

- name: "subnet ({{ availability_zone }})"
  vars:
    json: "{{ ec2_vpc_subnet_info }}"
    query: "subnets[?availability_zone=='{{ availability_zone }}'] | [0]"
    subnet: "{{ json | json_query(query) }}"
  set_fact:
    subnet: "{{ subnet }}"

- name: ENI
  vars:
    index: 4
    private_ip_address: "{{ subnet.cidr_block | ipmath(index) }}"
  ec2_eni:
    subnet_id: "{{ subnet.id }}"
    private_ip_address: "{{ private_ip_address }}"
  register: eni

Summary

The Ansible boilerplate discussed herein can configure an AWS VPC with a minimum of parameters specified by the administrator.

Boilerplate

The complete boilerplate suitable for cut-and-paste is provide below.

- name: Role Parameters
  set_fact:
    profile: "{{ lookup('env', 'AWS_PROFILE') }}"
    cidr_block: 10.1.0.0/16
    subnet_mask_size: 20

- name: aws_region_info
  aws_region_info:
    filters:
      region_name: "{{ lookup('env', 'AWS_DEFAULT_REGION') }}"
  register: aws_region_info

- name: aws_az_info
  aws_az_info:
    filters:
      region_name: "{{ aws_region_info.regions[0].region_name }}"
  register: aws_az_info

- name: region
  vars:
    region: "{{ aws_region_info.regions[0].region_name }}"
  set_fact:
    region: "{{ region }}"

- name: "{{ profile }} VPC"
  ec2_vpc_net:
    name: "{{ profile }}"
    region: "{{ region }}"
    cidr_block: "{{ cidr_block }}"
  register: ec2_vpc_net

- name: vpc
  vars:
    vpc: "{{ ec2_vpc_net.vpc }}"
  set_fact:
    vpc: "{{ vpc }}"

- name: "{{ profile }} VPC Subnets"
  vars:
    json: "{{ aws_az_info }}"
    query: "availability_zones[].zone_name"
    availability_zones: "{{ json | json_query(query) }}"
  ec2_vpc_subnet:
    vpc_id: "{{ vpc.id }}"
    az: "{{ item }}"
    cidr: >-
      {{ vpc.cidr_block | ipsubnet(subnet_mask_size, ansible_loop.index0) }}
  loop: "{{ availability_zones }}"
  loop_control:
    extended: yes

- name: ec2_vpc_subnet_info
  ec2_vpc_subnet_info:
    filters:
      vpc-id: "{{ vpc.id }}"
  register: ec2_vpc_subnet_info

- name: "{{ profile }} VPC IGW"
  ec2_vpc_igw:
    vpc_id: "{{ vpc.id }}"
  register: ec2_vpc_igw

- name: "{{ profile }} VPC IGW Route Table"
  vars:
    json: "{{ ec2_vpc_subnet_info }}"
    query: "subnets[].id"
    subnets: "{{ json | json_query(query) }}"
  ec2_vpc_route_table:
    vpc_id: "{{ vpc.id }}"
    tags:
      Name: Internet
    subnets: "{{ subnets }}"
    routes:
      - dest: 0.0.0.0/0
        gateway_id: "{{ ec2_vpc_igw.gateway_id }}"

CentOS In-Place Upgrade

Allen D. Ball — Mon, 16 Mar 2020 06:03:29 +0000

CentOS 8.0 was released on September 24th, 2019 and 8.1 on January 15, 2020. This article describes how a CentOS 7 may be upgraded in place. The steps are captured in an Ansible role published on GitHub.

Theory of Operation

The steps to migrate a CentOS 7 instance to CentOS 8.1 are:

Replace yum with dnf

a. Prepare yum installation

b. Install dnf

c. Remove yum
Use dnf to upgrade

a. Configure CentOS 8.1 packages

b. Install CentOS 8.1 (userland)

c. Install CentOS 8.1 kernel

Step #1 is necessitated because dnf has replaced yum in CentOS/RHEL 8 systems. The implementation is discussed in the next session.

Implementation

The following subsection describe the major steps of the upgrade.

Replace `yum` with `dnf`

The first step is to replace yum with dnf. The implementation uses the fact that /usr/bin/yum is a regular file when yum is installed and a symbolic link to dnf-3 when dnf is installed.

- name: /usr/bin/yum
  stat: path=/usr/bin/yum
  register: yum

The condition yum.stat.isreg is defined and yum.stat.isreg, if true, indicates yum was the package manager when Anisble was invoked. The Ansible script takes advantage of this observation to provide idempotent operation. package-cleanup(1) is installed from epel-release and used to remove locally installed RPMs.

- name: epel-release
  package:
    name: epel-release
    state: latest
  when:
    - yum.stat.isreg is defined and yum.stat.isreg

- name: yum-utils
  package:
    name: yum-utils
    state: latest
  when:
    - yum.stat.isreg is defined and yum.stat.isreg

- name: package-cleanup
  command:
    cmd: "{{ item }}"
  loop:
    - package-cleanup --leaves
    - package-cleanup --orphans
  when:
    - yum.stat.isreg is defined and yum.stat.isreg

The author's use case is to upgrade a fresh install of CentOS 7. However, if the upgrade is to be performed on a configured system, then rpmconf should be invoked to determine if any configuration files need to be preserved and/or migrated:

# yum -y install rpmconf
# rpmconf -a

dnf is installed with yum and then yum is removed with the corresponding dnf request, /etc/yum is removed, and dnf is updated. It is critical that these steps are completed so the system is not left in an inconsistent state without a functioning yum or dnf.

- name: dnf
  package:
    name: dnf
    state: latest
  when:
    - yum.stat.isreg is defined and yum.stat.isreg

- name: yum -> dnf
  shell: |-
    dnf -y remove yum yum-metadata-parser
    rm -rf /etc/yum
    dnf -y upgrade
  args:
    warn: false
  when:
    - yum.stat.isreg is defined and yum.stat.isreg

dnf is now installed and available to use for an in-place upgrade.

Upgrade CentOS

CentOS 8 requires 3 CentOS RPMs plus the latest epel-release (obtained via RPM) which are installed explicitly with dnf. The conditional ansible_distribution_major_version is version(releasever, "lt") is leveraged to provide idempotent operation and avoid re-running once CentOS 8 is installed.

- name: centos_packages
  vars:
    target: 8.1-1.1911.0.8.el8
    arch: "{{ ansible_architecture }}"
    releasever: "{{ target | regex_replace('^([0-9]+)[.].*$', '\\1') }}"
    BaseOS: "http://mirror.centos.org/centos/{{ releasever }}/BaseOS"
    Packages: "{{ BaseOS }}/{{ arch }}/os/Packages"
  set_fact:
    releasever: "{{ releasever }}"
    centos_packages:
      - "{{ Packages }}/centos-gpg-keys-{{ target }}.noarch.rpm"
      - "{{ Packages }}/centos-release-{{ target }}.{{ arch }}.rpm"
      - "{{ Packages }}/centos-repos-{{ target }}.{{ arch }}.rpm"
      - "https://dl.fedoraproject.org/pub/epel/epel-release-latest-{{ releasever }}.noarch.rpm"

The system is now ready for the actual upgrade. The CentOS Upgrade script has to run until reboot or the system will be left in an inconsistent state. Unfortunately python is replaced (and moved) so the Anisble client loses communication during the process eliminating the possibility of using the Anisble reboot module. Instead, the administrator should reinvoke the Ansible play once the update and reboot are complete.

- name: warn
  debug:
    msg: >-
      Warning: CentOS Upgrade will install kernel and initiate reboot
  when:
    - ansible_distribution_major_version is version(releasever, "lt")

- name: CentOS Upgrade
  shell: |-
    dnf -y install {{ centos_packages | join(" ") }}
    dnf clean all
    rpm -e $(rpm -q kernel)
    rpm -e --nodeps sysvinit-tools
    dnf -y --releasever={{ releasever }} --allowerasing --setopt=deltarpm=false distro-sync
    dnf -y install kernel-core
    dnf -y groupupdate "Core" "Minimal Install"
    shutdown -r now
  args:
    warn: false
  when:
    - ansible_distribution_major_version is version(releasever, "lt")

Post Upgrade Steps

The script allows for enabling the CentOS-Plus repository and updating installed packages after the reboot.

- name: Enable CentOS-Plus repository
  ini_file:
    dest: /etc/yum.repos.d/CentOS-centosplus.repo
    create: no
    section: centosplus
    option: enabled
    value: "1"

- name: package update
  package:
    name: "*"
    state: latest

Summary

CentOS 7 installation may be upgraded to CentOS 8 in-place once yum is replaced by dnf.

DEV Community: Allen D. Ball

Ganymede Kernel 2.0.0.20220717 Released

Configuring and Using GitHub Readme Instant Preview

Installation

Usage

Authentication

Ganymede Kernel 1.1.0.20210614 Released

Ganymede Kernel 1.0.0.20210422 Released

Spring Boot Part 7: Spring Security, Basic Authentication and Form Login, and Oauth2

Application

Prerequisites

Runtime Environment

MVC Controller

REST Controller

OAuth

Experiment

Implementation

java.util.concurrent.ForkJoinPool Example

Recursive Solution

ForkJoinPool Solution

Summary

Spring Boot Part 6: Auto Configuration and Starters

Theory of Operation

Implementation

Summary

Java Multi-Release JARs

Theory of Operation

Implementation

Summary

Spring Boot Part 5: voyeur, A Non-Trivial Application

Theory of Operation

@Service Implementations

UI @Controller, Model, and Thymeleaf Template

nmap Output @RestController

Packaging

Summary

Java Stream Tree Walker

API Definition

Implementation

Examples

Alternate Entry Point

Summary

AWS VPC Set-Up with Ansible

Theory of Operation

Implementation

Summary

Boilerplate

CentOS In-Place Upgrade

Theory of Operation

Implementation

Replace yum with dnf

Upgrade CentOS

Post Upgrade Steps

Summary

Replace `yum` with `dnf`