DEV Community

Cover image for GSOC 2025 Report: AprilTag, FAST, and ORB Detectors
Aditya Kumar
Aditya Kumar

Posted on • Originally published at adityais.dev

GSOC 2025 Report: AprilTag, FAST, and ORB Detectors

#googlesummerofcode2025 #gsoc2025

This project adds three detection/feature modules to kornia-rs ecosystem: a full AprilTag Detector, FAST Corner Detector, and an ORB Detector. The implementations are CPU-focused, modular, well-documented, and include examples, tests, and benchmarks.

Goals

  1. Implement AprilTag detection and decoding pipeline (configurable, mult-family)
  2. Implement newer complete FAST corner detection
  3. Implement ORB Detector
  4. Provide examples, tests, and benchmarks; compare to common libraries when possible

AprilTag Detectors

An AprilTag is a fiducial marker: a flat, square pattern that encodes a small binary code inside a black/white border. Once detected and decoded in an image, a tag lets you compute camera pose relative to the tag if you know the tag's physical size and the camera intrinsics. They are widely used for camera calibration, robot localization, AR markers and many tasks where 2D to 3D reference is useful.

Why AprilTags are useful:

  • Robust detection under many viewing angles and lighting conditions.
  • Provide direct geometric constraints for pose estimation
  • Small payload, easy to print and deploy

The entire AprilTag decoding pipeline

AprilTag Detection output

During the first half of the coding period, I primarily worked on creating the kornia-apriltag crate that focuses on AprilTag Detection. It breaks down the entire decoding pipeline in the following steps:

  1. Adaptive Threshold
  2. Connected Components (aka Union Find)
  3. Gradient Clusters
  4. Quad Fitting
  5. Tag Decoding

The API of kornia-apriltag is a well documented crate which is easy to use while being super customizable. The following code snippet is the easiest way to use the crate:

use kornia_apriltag::*;

let config = DecodeTagsConfig::all();
let mut decoder = AprilTagDecoder::new(config, image_size)?;

let detections = decoder.decode(&img)?;
// Do whatever you want with the detections, or just plot them
Enter fullscreen mode Exit fullscreen mode

But you can go much further by choosing which Tag Families to consider detection and their individual configs and even replace/customize any step of the pipeline as all of the functions are well-documented and exposed publicly.

You can find a complete example in examples/apriltag

PRs associated: #380, #384, #406, #410, #419, #420, #429, #433, #435, #436, #438, #440, #444, #458

Issues associated: #408, #450

Resources

ORB Detectors

ORB (Oriented FAST and Rotated BRIEF) is a feature detection and description algorithm optimized for speed and robustness. It combines FAST for keypoint detection, orientation assignment for each keypoint, and BRIEF-like binary descriptors that are rotation-aware. ORB is widely used for feature matching, visual odometry, and as input to SLAM pipelines.

Basic ORB Detector

ORB Detector on live webcam feed and a static image

The second half of the coding period, I implemented an ORB Detector in the kornia-imgproc crate.

use kornia_imgproc::features::*;

let mut orb_detector = OrbDetector::new(OrbDetectorConfig::default(), image.size())?;

orb_detector.detect(&image)?;
let (descriptors, _) = orb_detector.extract(&image)?;
let Detection { keypoints, .. } = orb_detector.get_detection();
Enter fullscreen mode Exit fullscreen mode

The matching of the descriptors is also as simple as the detection:

// Your detection logic

let matches = match_descriptors(&first_descriptors, &second_descriptors, None, true, None);

// let's plot them
let mut coords = Vec::new();
for &(i1, i2) in matches.iter() {
    let kp1 = &keypoints1[i1];
    let kp2 = &keypoints2[i2];

    coords.push([kp1, kp2]);
}

// Your plotting logic
Enter fullscreen mode Exit fullscreen mode

PRs associated: #457 (UNDER REVIEW)

Resources

FAST Corner Detector

FAST (Features from Accelerated Segment Test) is a high-speed corner detector. For each candidate pixel, FAST compares the intensities of a circle of surrounding pixels to the center pixel. If a contiguous arc of pixels is all brighter (or all darker) than the center by a threshold, then the pixel is classified as a corner.

Key points identified by FAST Corner Detector

During the second half of the coding period while implementing ORB Detector I needed complete implementation of FAST corner detector. So, I implemented a newer complete version of FAST Corner detector in kornia-imgproc crate.

use kornia_imgproc::features::FastDetector;

let mut fast_detector = FastDetector::new(image_size, threshold, arc_length, min_distance)?;
fast_detector.compute_corner_response(&image)?;

let keypoints = fast_detector.extract_keypoints()?;
// Go ahead plot the keypoints
Enter fullscreen mode Exit fullscreen mode

You can find a complete example using FAST Corner Detector in examples/fast_detector.

PRs associated: #460 (UNDER REVIEW)

Resources

Timeline

The following is the list of all the PR/Issue(s) created by me during the GSoC Coding Period.

Goal Classification

  • AprilTag
  • ORB Detector
  • FAST Detector
  • Other
Date Title Type Goal Link
1. Jun 2 Add kornia-apriltag crate PR AprilTag #380
2. Jun 2 Use std::hint::black_box instead of criterion::black_box because it’s deprecated PR Other #381
3. Jun 2 Update ONNX example to use ort 2.0.0-rc.10 PR Other #382
4. Jun 4 Add Adaptive Threshold Binary functions PR AprilTag #384
5. Jun 8 Fix kornia-io docs.rs not building PR Other #388
6. Jun 20 Add Connected Components for AprilTag detection PR AprilTag #406
7. Jun 21 Safe Implementation of Image TileIterator yielding borrowed slices Issue AprilTag #408
8. Jun 21 Implement Zero-copy Image conversion in kornia-py Issue Other #409
9. Jun 23 Add gradient cluster detection PR AprilTag #410
10. Jun 26 Replace clippy-action with direct cargo clippy command PR Other #412
11. Jun 29 Add Quad Fitting PR AprilTag #419
12. Jul 1 Add AprilTag Decoding PR AprilTag #420
13. Jul 5 Remove license-file from smol_vlm example PR Other #424
14. Jul 10 Add support for multiple AprilTag Families PR AprilTag #429
15. Jul 16 Add TagFamilyKind::all method and use it in DecodeTagsConfig PR AprilTag #433
16. Jul 18 Reset sharpening buffer for every Quad PR AprilTag #435
17. Jul 18 Add AprilTag detection example PR AprilTag #436
18. Jul 19 Add AprilTag Benchmark PR AprilTag #438
19. Jul 22 Update apriltag-imgs submodule URL PR AprilTag #440
20. Aug 2 Add Image downscaling support in AprilTag Detection PR AprilTag #444
21. Aug 27 Implement ORB Detector PR ORB Detector #457
22. Aug 28 Add benchmark aprilgrid-rs PR AprilTag #458
23. Aug 29 Implement newer FAST Corner Detector PR FAST Detector #460
24. Aug 30 Add Gaussian Blur support for u8 images Issue Other #461
25. Aug 30 Add HarrisResponse support for u8 images Issue Other #462

Challenges faced

  • My university end-term exams overlapped with the first half of the coding period, and I initially expected this to limit my time. In practice, I became more interested in AprilTags and allocated increasing effort to the project.

  • Since kornia-rs is a low-level Computer Vision library, I need to understand the algorithms/detectors that I implemented well. But the issue was that I was in my 2nd semester and core computer science stuff starts from 3rd semester in my university. Luckily I understood the papers well and understood the code of existing implementations which made it slightly easier.

Learning Outcomes

I learned a lot during these 3 months and had a lot of new experiences. Prior to this, I had not worked in a team; my experience was primarily solo projects and occasional open-source contributions. But during this time, we had weekly sync and regular conversation of what and how to do the next thing. It was truly a wonderful experience.

Top comments (0)