Spring Boot Part 5: voyeur, A Non-Trivial Application

This series of articles examines Spring Boot features. This fifth article in the series presents a non-trivial application which probes local hosts (with the help of the nmap command) to assist in developing UPNP and SSDP applications.

Complete source and javadoc are available on GitHub. Additional artifacts (including their respective source and javadoc JARs) are available from the Maven repository at:

    <repository>
      <id>repo.hcf.dev-RELEASE</id>
      <name>hcf.dev RELEASE Repository</name>
      <url>https://repo.hcf.dev/maven/release/</url>
      <layout>default</layout>
    </repository>

Specific topics covered herein:

• @Service implementations with @Scheduled updates

  • @Autowired

• UI @Controller

  • Populates Model
  • Thymeleaf temlates and decoupled logic

• @RestController implementation

Theory of Operation

The following subsections describe the components.

@Service Implementations

The voyeur package defines a number of (annotated) @Services:

@Service	Description
ArpCache	Map of InetAddress to hardware address periodically updated by reading /proc/net/arp or parsing the output of arp -an
NetworkInterfaces	Set of NetworkInterfaces
Nmap	Map of XML output of the nmap command for each InetAddress discovered via ARPCache, NetworkInterfaces, and/or SSDP
SSDP	SSDP hosts discovered via SSDPDiscoveryCache

Each of these services implement a Set or Map, which may
be @Autowired into other components, and periodically update
themselves with a @Scheduled method. The
Nmap service is examined in detail.

First, the @PostConstruct method (in addition to
performing other initialization chores) tests to determine if the
nmap command is available:

...
@Service
@NoArgsConstructor @Log4j2
public class Nmap extends InetAddressMap<Document> ... {
    ...
    private static final String NMAP = "nmap";
    ...
    private boolean disabled = true;
    ...
    @PostConstruct
    public void init() throws Exception {
        ...
        try {
            List<String> argv = Stream.of(NMAP, "-version").collect(toList());

            log.info(String.valueOf(argv));

            Process process =
                new ProcessBuilder(argv)
                .inheritIO()
                .redirectOutput(PIPE)
                .start();

            try (InputStream in = process.getInputStream()) {
                new BufferedReader(new InputStreamReader(in, UTF_8))
                    .lines()
                    .forEach(t -> log.info(t));
            }

            disabled = (process.waitFor() != 0);
        } catch (Exception exception) {
            disabled = true;
        }

        if (disabled) {
            log.warn("nmap command is not available");
        }
    }
    ...
    public boolean isDisabled() { return disabled; }
    ...
}

If the nmap command is successful, its version is logged. Otherwise, disabled is set to true and no further attempt is made to run the nmap command in other methods.

The @Scheduled update() method is invoked every 30 seconds and ensures a map entry exists for every InetAddress previously discovered by the NetworkInterfaces, ARPCache, and SSDP components and then queues a Worker Runnable for any value whose output is more than INTERVAL (60 minutes) old. The @EventListener (with ApplicationReadyEvent guarantees the method won't be called before the application is ready (to serve requests).

public class Nmap extends InetAddressMap<Document> ... {
    ...
    private static final Duration INTERVAL = Duration.ofMinutes(60);
    ...
    @Autowired private NetworkInterfaces interfaces = null;
    @Autowired private ARPCache arp = null;
    @Autowired private SSDP ssdp = null;
    @Autowired private ThreadPoolTaskExecutor executor = null;
    ...
    @EventListener(ApplicationReadyEvent.class)
    @Scheduled(fixedDelay = 30 * 1000)
    public void update() {
        if (! isDisabled()) {
            try {
                Document empty = factory.newDocumentBuilder().newDocument();

                empty.appendChild(empty.createElement("nmaprun"));

                interfaces
                    .stream()
                    .map(NetworkInterface::getInterfaceAddresses)
                    .flatMap(List::stream)
                    .map(InterfaceAddress::getAddress)
                    .filter(t -> (! t.isMulticastAddress()))
                    .forEach(t -> putIfAbsent(t, empty));

                arp.keySet()
                    .stream()
                    .filter(t -> (! t.isMulticastAddress()))
                    .forEach(t -> putIfAbsent(t, empty));

                ssdp.values()
                    .stream()
                    .map(SSDP.Value::getSSDPMessage)
                    .filter(t -> t instanceof SSDPResponse)
                    .map(t -> ((SSDPResponse) t).getInetAddress())
                    .forEach(t -> putIfAbsent(t, empty));

                keySet()
                    .stream()
                    .filter(t -> INTERVAL.compareTo(getOutputAge(t)) < 0)
                    .map(Worker::new)
                    .forEach(t -> executor.execute(t));
            } catch (Exception exception) {
                log.error(exception.getMessage(), exception);
            }
        }
    }
    ...
    private Duration getOutputAge(InetAddress key) {
        long start = 0;
        Number number = (Number) get(key, "/nmaprun/runstats/finished/@time", NUMBER);

        if (number != null) {
            start = number.longValue();
        }

        return Duration.between(Instant.ofEpochSecond(start), Instant.now());
    }

    private Object get(InetAddress key, String expression, QName qname) {
        Object object = null;
        Document document = get(key);

        if (document != null) {
            try {
                object = xpath.compile(expression).evaluate(document, qname);
            } catch (Exception exception) {
                log.error(exception.getMessage(), exception);
            }
        }

        return object;
    }
    ...
}

Spring Boot's ThreadPoolTaskExecutor is injected. To guarantee more than one thread is allocated the application.properties contains the following property:

spring.task.scheduling.pool.size: 4

The Worker implementation is given below.

    ...
    private static final List<String> NMAP_ARGV =
        Stream.of(NMAP, "--no-stylesheet", "-oX", "-", "-n", "-PS", "-A")
        .collect(toList());
    ...
    @RequiredArgsConstructor @EqualsAndHashCode @ToString
    private class Worker implements Runnable {
        private final InetAddress key;

        @Override
        public void run() {
            try {
                List<String> argv = NMAP_ARGV.stream().collect(toList());

                if (key instanceof Inet4Address) {
                    argv.add("-4");
                } else if (key instanceof Inet6Address) {
                    argv.add("-6");
                }

                argv.add(key.getHostAddress());

                DocumentBuilder builder = factory.newDocumentBuilder();
                Process process =
                    new ProcessBuilder(argv)
                    .inheritIO()
                    .redirectOutput(PIPE)
                    .start();

                try (InputStream in = process.getInputStream()) {
                    put(key, builder.parse(in));

                    int status = process.waitFor();

                    if (status != 0) {
                        throw new IOException(argv + " returned exit status " + status);
                    }
                }
            } catch (Exception exception) {
                remove(key);
                log.error(exception.getMessage(), exception);
            }
        }
    }
    ...

Note that the InetAddress will be removed from the Map if the Process fails.

UI @Controller, Model, and Thymeleaf Template

The complete UIController implementation is given below.

@Controller
@NoArgsConstructor @ToString @Log4j2
public class UIController extends AbstractController {
    @Autowired private SSDP ssdp = null;
    @Autowired private NetworkInterfaces interfaces = null;
    @Autowired private ARPCache arp = null;
    @Autowired private Nmap nmap = null;

    @ModelAttribute("upnp")
    public Map<URI,List<URI>> upnp() {
        Map<URI,List<URI>> map =
            ssdp().values()
            .stream()
            .map(SSDP.Value::getSSDPMessage)
            .collect(groupingBy(SSDPMessage::getLocation,
                                ConcurrentSkipListMap::new,
                                mapping(SSDPMessage::getUSN, toList())));

        return map;
    }

    @ModelAttribute("ssdp")
    public SSDP ssdp() { return ssdp; }

    @ModelAttribute("interfaces")
    public NetworkInterfaces interfaces() { return interfaces; }

    @ModelAttribute("arp")
    public ARPCache arp() { return arp; }

    @ModelAttribute("nmap")
    public Nmap nmap() { return nmap; }

    @RequestMapping(value = {
                        "/",
                        "/upnp/devices", "/upnp/ssdp",
                        "/network/interfaces", "/network/arp", "/network/nmap"
                    })
    public String root(Model model) { return getViewName(); }

    @RequestMapping(value = { "/index", "/index.htm", "/index.html" })
    public String index() { return "redirect:/"; }
}

The @Controller populates the Model with five attributes and implements the root method¹ to serve the UI request paths. The superclass implements getViewName() which creates a view name based on the implementing class's package which translates to classpath:/templates/voyeur.html, a Thymeleaf template to generate a pure HTML5 document. Its outline is shown below.

<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org" th:xmlns="@{http://www.w3.org/1999/xhtml}">
  <head>
    ...
  </head>
  <body>
    ...
    <header>
      <nav th:ref="navbar">
        ...
      </nav>
    </header>
    <main th:unless="${#ctx.containsVariable('exception')}"
          th:switch="${#request.servletPath}">
      <section th:case="'/error'">
        ...
      </section>
      <section th:case="'/upnp/devices'">
        ...
      </section>
      <section th:case="'/upnp/ssdp'">
        ...
      </section>
      ...
    </main>
    <main th:if="${#ctx.containsVariable('exception')}">
      ...
    </main>
    <footer>
      <nav th:ref="navbar">
        ...
      </nav>
    </footer>
    <script/>
  </body>
</html>

The template's <header/> <nav/> implements the menu and references the paths specified in the UIController.root().

      <nav th:ref="navbar">
        <th:block th:ref="container">
          <div th:ref="navbar-brand">
            <a th:text="${#strings.defaultString(brand, 'Home')}" th:href="@{/}"/>
          </div>
          <div th:ref="navbar-menu">
            <ul th:ref="navbar-start"></ul>
            <ul th:ref="navbar-end">
              <li th:ref="navbar-item">
                <button th:text="'UPNP'"/>
                <ul th:ref="navbar-dropdown">
                  <li><a th:text="'Devices'" th:href="@{/upnp/devices}"/></li>
                  <li><a th:text="'SSDP'" th:href="@{/upnp/ssdp}"/></li>
                </ul>
              </li>
              <li th:ref="navbar-item">
                <button th:text="'Network'"/>
                <ul th:ref="navbar-dropdown">
                  <li><a th:text="'Interfaces'" th:href="@{/network/interfaces}"/></li>
                  <li><a th:text="'ARP'" th:href="@{/network/arp}"/></li>
                  <li><a th:text="'Nmap'" th:href="@{/network/nmap}"/></li>
                </ul>
              </li>
            </ul>
          </div>
        </th:block>
      </nav>

The template is also structured to produce a <main/> node with a <section/> node corresponding to the request path if there is no exception variable in the context (normal operation). The th:switch and th:case attributes are used to create a <section/> corresponding to each ${#request.servletPath}. The <section/> specific to the /network/nmap path is shown below:

    <main th:unless="${#ctx.containsVariable('exception')}"
          th:switch="${#request.servletPath}">
      ...
      <section th:case="'/network/nmap'">
        <table>
          <tbody>
            <tr th:each="key : ${nmap.keySet()}">
              <td>
                <a th:href="@{/network/nmap/{ip}.xml(ip=${key.hostAddress})}" th:target="_newtab">
                  <code th:text="${key.hostAddress}"/>
                </a>
                <p><code th:text="${nmap.getPorts(key)}"/></p>
              </td>
              <td>
                <p th:each="product : ${nmap.getProducts(key)}" th:text="${product}"/>
              </td>
            </tr>
          </tbody>
        </table>
      </section>
      ...
    </main>

The template generates a <table/> with a row (<tr/>) for each key in the Nmap. Each row consists of two columns (<td/>):

1. The InetAddress of the host with a link to the nmap command output² and a list of open TCP ports

2. The services/products detected

The getPorts(InetAddress) and getProducts(InetAddress) methods are provided to avoid XPath calculations within the Thymeleaf template.

...
@Service
@NoArgsConstructor @Log4j2
public class Nmap extends InetAddressMap<Document> ... {
    ...
    public Set<Integer> getPorts(InetAddress key) {
        Set<Integer> ports = new TreeSet<>();
        NodeList list = (NodeList) get(key, "/nmaprun/host/ports/port/@portid", NODESET);

        if (list != null) {
            for (int i = 0; i < list.getLength(); i += 1) {
                ports.add(Integer.parseInt(list.item(i).getNodeValue()));
            }
        }

        return ports;
    }
    ...
}

The getProducts(InetAddress) implementation is similar with an XPathExression of /nmaprun/host/ports/port/service/@product.

The UIController instance combined with the Thymeleaf template described so far will only generate pure HTML5 with no style markup. This implementation uses Thymeleaf's Decoupled Template Logic feature and can be found at classpath:/templates/voyeur.th.xml.³ The decoupled logic for the table described in this section is shown below.

<?xml version="1.0" encoding="UTF-8"?>
<thlogic>
  ...
  <attr sel="body">
    ...
    <attr sel="main" th:class="'container'">
      <attr sel="table" th:class="'table table-striped'">
        <attr sel="tbody">
          <attr sel="tr" th:class="'row'"/>
          <attr sel="tr/td" th:class="'col'"/>
        </attr>
      </attr>
    </attr>
    ...
  </attr>
</thlogic>

The UIController superclass provides one more feature: To inject the proprties defined in classpath:/templates/voyeur.model.properties into the Model.

brand = ${application.brand:}

stylesheets: /webjars/bootstrap/css/bootstrap.css
style:\
body { padding-top: 60px; margin-bottom: 60px; }\n\
@media (max-width: 979px) { body { padding-top: 0px; } }
scripts: /webjars/jquery/jquery.js, /webjars/bootstrap/js/bootstrap.js

The design goal of this implementation was to commit all markup logic to the *.th.xml resource allowing only the necessity to modify the decoupled logic and the model properties to use an alternate framework. This goal was defeated in this implementation because different frameworks support to different degrees HTML5 elements. A partial Bulma implementation is available in https://github.com/allen-ball/voyeur/tree/trunk/src/main/resources/templates-bulma which demonstrates the HTML5 differences.

The /network/nmap request is shown rendered in the image in the Introduction of this article.

nmap Output @RestController

nmap XML output can be served by implementing a @RestController. The Nmap class is annotated with @RestController and @RequestMapping to requests for /network/nmap/ and the nmap(String) method provides the XML serialized to a String.

@RestController
@RequestMapping(value = { "/network/nmap/" }, produces = MediaType.APPLICATION_XML_VALUE)
...
public class Nmap ... {
    ...
    @RequestMapping(value = { "{ip}.xml" })
    public String nmap(@PathVariable String ip) throws Exception {
        ByteArrayOutputStream out = new ByteArrayOutputStream();

        transformer.transform(new DOMSource(get(InetAddress.getByName(ip))),
                              new StreamResult(out));

        return out.toString("UTF-8");
    }
    ...
}

Packaging

The spring-boot-maven-plugin has a repackage goal which may be used to create a self-contained JAR with an embedded launch script. That goal is used in the project pom to create and attach a self-contained JAR atifact.

<project ...>
  ...
  <build>
    <pluginManagement>
      <plugins>
        ...
        <plugin>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-maven-plugin</artifactId>
          <executions>
            <execution>
              <goals>
                <goal>build-info</goal>
                <goal>repackage</goal>
              </goals>
            </execution>
          </executions>
          <configuration>
            <attach>true</attach>
            <classifier>bin</classifier>
            <executable>true</executable>
            <mainClass>${start-class}</mainClass>
            <embeddedLaunchScriptProperties>
              <inlinedConfScript>${basedir}/src/bin/inline.conf</inlinedConfScript>
            </embeddedLaunchScriptProperties>
          </configuration>
        </plugin>
        ...
      </plugins>
    </pluginManagement>
    ...
  </build>
  ...
</project>

Please see the project GitHub page for instructions on how to run the JAR.

Summary

This article discusses aspects of the voyeur application and provides specific examples of:

• @Service implementation and @Autowired components with @Scheduled methods

• @Controller implementation, Model population, and Thymeleaf templates and decoupled logic

• @RestController implementation

[1] A misleading method name at best. ↩

[2] The @RestController is described in the next subsection. ↩

[3] The common application properties do not provide an option to enable this functionality. It is enabled in the UIController suprclass by configuring the injected SpringResourceTemplateResolver. ↩