Virtually Infinite Scrolling with Angular

Infinite scrollers have been around for a long time. The concept is simple: as you scroll down the page, additional content is fetched and added to the bottom, making it so the scrolling never ends. Implementing an infinite scroller is straightforward, but you might negatively affect page performance without careful consideration. By the time you've re-fetched content a few times, you could have hundreds of elements in the DOM that aren't even visible all at the same time. Fortunately, there are some patterns you can implement to avoid this, some of which we'll cover in this article using Angular.

Ideally we want to avoid this.

Virtual Scrolling

Virtual Scrolling is a technique used to render a subset of an extensive list of items at any given time, not to be confused with Infinite Scrolling. This technique is advantageous when you have a large dataset that you want to display in a list, but you don't want to render all of the items at once. Instead, you only render the currently visible items in the viewport (and those almost visible), and as the user scrolls, you dynamically swap out the contents of the nodes. Virtual Scrolling can improve performance by reducing the number of DOM elements that need to be rendered and updated at any time.

How Virtual Scrolling works is pretty straightforward: we create a container that is the viewport's height and only render the items currently visible (along with a buffer) within that container at the given scroll depth from memory. As the user scrolls, we update the container to show the next set of items and remove the items that are no longer visible while increasing and decreasing the scroll depth using CSS. Suppose we combine this concept with an infinite scroller. In that case, we can create a scrollable list that can be virtually infinite in length without negatively affecting performance as we increase the maximum depth every time more content is added.

The following sample has a list of items well into the thousands, but as you can see, we only render a maximum of 8 elements. As we scroll, we adjust the CSS to change the scroll height to give the illusion of a long list.

Real World Example

Let's build an example to demonstrate how you might implement a Virtual Scroller in Angular. We'll make a simple application that fetches media content from Reddit's paginated API and displays it in a list. The application will have a search bar that allows you to type in a subreddit name to change the retrieved content list and a way to apply a filter. As you scroll down the page, the service adds additional content to the bottom. Our high-level requirements are as follows:

1. RxJs Observables and the async pipe should drive it.
2. It should reset content whenever the subreddit name or filter changes. It should not reset when we append new content.
3. It should store prior content in memory so we can scroll back up and down without making unnecessary API requests.

We'll need to install the @angular/cdk package, which contains a Virtual Scroller component as part of Angular's Component DevKit. You can do this by running npm i @angular/cdk.

While the following example uses Angular, there's no reason you can't implement a similar pattern in other frameworks, such as React, Vue, or even vanilla JavaScript. I've assembled a small demo of what this might look like under the hood that you can view here.

Setting Up The Service

First, we must set up a service that can fetch content from Reddit's API. We'll use Angular's HttpClient to make the request and set up a few RxJs observables to hold the state for the requested subreddit name and filter. I've omitted some of the code for brevity, but you can find the full implementation here.

@Injectable({
  providedIn: 'root'
})
export class RedditService {
  private static readonly API_BASE = 'https://www.reddit.com/r'
  private static readonly MAX_CONTENT_FETCH = 24
  private static readonly DEFAULT_SUBREDDIT = 'cats'
  private static readonly DEFAULT_PAGE = 't3_'

  /**
   * We'll implement _query$ later in the article.
   */
  private readonly _query$: Observable<IRedditQuery>


  /**
   * BehaviorSubject which houses the current subreddit name.
   */
  private readonly _subRedditName$ = new BehaviorSubject(
    RedditService.DEFAULT_SUBREDDIT
  )

  /**
   * BehaviorSubject which houses the current page.
   */
  private readonly _subRedditPage$ = new BehaviorSubject(
    RedditService.DEFAULT_PAGE
  )

  /**
   * BehaviorSubject which houses the currently set filter.
   */
  private readonly _subRedditFilter$ = new BehaviorSubject(RedditFilter.HOT)

  /**
   * Sets the name of subreddit, for example 'cats' or 'fish'.
   * @param name The name of the subreddit.
   */
  public setSubRedditName(name: string): void {
    /**
    * Whenever the subreddit name changes we reset the page to the default.
    */
    this.setSubRedditPage(RedditService.DEFAULT_PAGE)

    this._subRedditName$.next(name)
  }

  /**
   * Sets the name of the page to display. The Reddit API allows you to specify
   * at which point you want to fetch after. As a result we index the page name
   * after each fetch and use it to fetch the next set of paginated content.
   * @param page The name of the page to fetch content after.
   */
  public setSubRedditPage(page: string): void {
    this._subRedditPage$.next(`${RedditService.DEFAULT_PAGE}${page}`)
  }

  /**
   * Sets the name of the subreddit filter, for example 'hot' or 'new'.
   * @param filter The name of the filter {@see RedditFilter} for options.
   */
  public setSubRedditFilter(filter: RedditFilter): void {
    /** On each filter change reset the page back to default value
     * to prevent lingering pages. */
    this.setSubRedditPage(RedditService.DEFAULT_PAGE)

    this._subRedditFilter$.next(filter)
  }

  /**
   * Gets the sub reddit name option as an observable.
   * @returns An observable that contains the currently set name.
   */
  public getSubRedditName(): Observable<string> {
    return this._subRedditName$.asObservable()
  }

  /**
   * Gets the current page being viewed as an observable.
   * @returns An observable that can contains the currently set page.
   */
  public getSubRedditPage(): Observable<string> {
    return this._subRedditPage$.asObservable()
  }

  /**
   * Gets the sub reddit filter option as an observable.
   * @returns An observable that can translate the currently set filter option.
   */
  public getSubRedditFilter(): Observable<RedditFilter> {
    return this._subRedditFilter$.asObservable()
  }


  /**
   * Gets the current data requested from the Reddit API as an observable.
   * @returns An observable that contains the filtered content from the Reddit API.
   */
  public getQuery(): Observable<IRedditQuery> {
    return this._query$
  }
}

The method for implementing content fetching needs to track specific properties while requesting more data. We are adding a page property to the query string, which is essential to ensure we make a new request for content after the last item in the current set. Additionally, we filter out unwanted content from our stream, such as NSFW (not safe for work) material and content lacking a post hint. This approach guarantees we only provide content we expect to display in our component.

/**
 * Gets data from the Reddit API based on a series of defined filtering options.
 * @returns Returns an observable containing the data formatted as {@see IRedditResult}
*/
private getSubRedditContent({
  name,
  filter,
  page
}: IRedditRequestOptions): Observable<IRedditResult[]> {
  const path = new URL(`${RedditService.API_BASE}/${name}/${filter}/.json`)
  path.searchParams.append(
    RedditRequestParameters.LIMIT,
    RedditService.MAX_CONTENT_FETCH.toString()
  )

  // If page is provided it gets appended to the query to ensure we're not re-fetching the same content.
  if (page) {
    path.searchParams.append(RedditRequestParameters.AFTER, page)
  }

  console.info('✅ Making request to:', path.toString())

  return this.http.get<IRedditResultNatural>(path.toString()).pipe(
    map(result =>
      result.data.children
        .map(item => item.data)
        .filter(
          (item: IRedditResult) =>
            !item.over_18) &&
            item.post_hint &&
            ((item.post_hint === RedditPostHint.LINK &&
              item.secure_media_embed &&
              item.secure_media_embed.media_domain_url) ||
              item.post_hint === RedditPostHint.IMAGE ||
              item.post_hint === RedditPostHint.RICH_VIDEO)
        )
    )
  )
}

You may have noticed in the above examples that we were missing the implementation for the query$ observable. We'll use this as the central machinery for the entire application, which is ultimately what the Virtual Scroller component will subscribe to. The following sample code merges the different observable streams into one before fetching content. We then use the scan operator to integrate the previous stream results with the current one, creating an extensive array of data spanning multiple pages.

The idea behind this is that you can scroll as much as you like, and only in cases where the subreddit name or filter changes do we re-fetch entirely to get a fresh list of content. We also set nextPage as a property of the query$ stream, which matches the last item ID in the current set. This is used later in our component to determine what page of content to fetch when we're nearing the bottom of the Virtual Scroller.

 public constructor(private readonly http: HttpClient) {
    this._query$ = combineLatest([
      this.getSubRedditName(),
      this.getSubRedditFilter(),
    ]).pipe(
      switchMap(([name, filter]) =>
        this.getSubRedditPage().pipe(
          mergeMap(page =>
            this.getSubRedditContent({
              name,
              filter,
              page
            }).pipe(
              catchError(() => {
                /**
                 * If the stream catches an error return an empty array.
                 * For the sake of this demo we'll just assume that the subreddit
                 * has no content if it doesn't exist.
                 */
                return of([])
              })
            )
          ),

          /**
           * If the page observable emits instead of creating a new list it
           * will instead concat the previous results together with the new ones.
           * This is used for infinite scrolling pagination.
           */
          scan(
            (acc, curr) => {
              const newResults = curr.filter(
                item =>
                  !acc.results.some(existingItem => existingItem.id === item.id)
              )

              return {
                results: acc.results.concat(newResults),
                nextPage: curr[curr.length - 1]?.id
              }
            },
            {
              results: [] as IRedditResult[],
              nextPage: undefined
            } as IRedditQuery
          ),
          /**
           * Whenever a user requests a fresh set of content (ie changing the filter, subfilter, subreddit, etc)
           * we start with an empty array to prevent lingering content from previous pages.
           * It also gives instant feedback to the user that new content is being fetched by clearing the list.
           */
          startWith({results: [], nextPage: undefined} as IRedditQuery)
        )
      )
    )
  }

While all this logic may initially seem confusing, the advantage of using RxJs is that you can combine and manipulate data streams, which allows us to perform all of our business logic before it reaches our component. It also means that all the mechanisms are housed outside the component, allowing us to reuse the same data in another part of the application by subscribing to the same observable. I've found that learning RxJs has been the key to getting more comfortable with Angular; I now prefer Angular over any other front-end framework, as the patterns it introduces when working with them together make a lot of sense to me after I became more familiar with them.

Setting Up The Component

The next step is to set up the component to display the content in the Virtual Scroller. We'll use Angular's CdkVirtualScrollViewport component to render the content and set up a method for when the user scrolls near the bottom of the viewport. This method will fetch the next page of content from the Reddit API by informing our subRedditPage$ observable. Below is an example of the initial implementation of our search-results component.

import {
  CdkVirtualScrollViewport,
  ScrollingModule,
} from '@angular/cdk/scrolling'

@Component({
  selector: 'app-search-results',
  standalone: true,
  changeDetection: ChangeDetectionStrategy.OnPush,
  imports: [CommonModule, ScrollingModule],
  schemas: [CUSTOM_ELEMENTS_SCHEMA],
  templateUrl: './search-results.component.html',
})
export class SearchResultsComponent {
  @ViewChild(CdkVirtualScrollViewport)
  public viewPort?: CdkVirtualScrollViewport

  /**
   * Observable used to display post content from Reddit.
   */
  protected readonly query$: Observable<IRedditQuery>

  /**
   * The size of the item.
   */
  public itemSize = 1264

  /**
   * Subscribes to the query observable.
   */
  public constructor(
    private readonly redditService: RedditService
  ) {
    this.query$ = this.redditService.getQuery()
  }

  /**
   * We'll implement this next!
   */
  public onScroll(nextPage?: string): void {}
}

In the template, we'll subscribe to query$ using the async pipe. One thing to note about Virtual Scrollers is that they become vastly more complicated if your content is not all equal heights. If your content is dynamic heights, or you don't know the sizes ahead of time, it can create performance implications, as recalculating can be expensive. In my example, every item is the same height.

<ng-container *ngIf="{query: query$ | async} as data">
  <ul *ngIf="data.query">
    <cdk-virtual-scroll-viewport 
      scrollWindow 
      [itemSize]="itemSize"
      (scrolledIndexChange)="onScroll(data.query.nextPage)">
      <li *cdkVirtualFor="let result of data.query.results" [height]="itemSize">
        <app-media [content]="result" [size]="itemSize"></app-media>
      </li>
    </cdk-virtual-scroll-viewport>
  </ul>
</ng-container>

Finally, we'll set up a method that will be called when the user scrolls near the bottom of the Virtual Scroller to fetch more content; in the prior example, this is leveraging scrollIndexChanged on cdk-virtual-scroll-viewport. This method accepts the following page ID as a parameter, which we set earlier as a property of the query$ stream, and emits to the subRedditPage$ observable. Once this is emitted, the service will fetch the next page of content from the Reddit API
and automatically update our list via the query$ observable.

  public onScroll(nextPage?: string): void {
    if (this.viewPort && nextPage) {
      const end = this.viewPort.getRenderedRange().end
      const total = this.viewPort.getDataLength()

      // If we're close to the bottom, fetch the next page.
      if (end >= total - SearchResultsComponent.FETCH_MINIMUM) {
        this.redditService.setSubRedditPage(nextPage)
      }
    }
  }

We'll also need to wire up other application parts, such as the search bar and the tab controls. Below is a small example of how I did this, it's pretty straightforward as it's just a form that emits the subRedditName property to the setSubRedditName method on the service.

<form (submit)="onSubmit()">
  <input type="text" [(ngModel)]="subRedditName" />
  <button type="submit">Search</button>
</form>

@Component({
  selector: 'app-search',
  templateUrl: './search.component.html',
  changeDetection: ChangeDetectionStrategy.OnPush,
  imports: [CommonModule, ReactiveFormsModule]
})
export class SearchComponent {
  /**
   * An observable containing the selected sub reddit name.
   * Used to push the current Reddit page back to the input placeholder.
   */
  public readonly subRedditName$: Observable<string>

  /**
   * @inheritdoc
   * @param formBuilder Injected form builder from Angular.
   * @param redditService Injected Reddit service.
   */
  public constructor(
    private readonly formBuilder: FormBuilder,
    private readonly redditService: RedditService
  ) {
    this.subRedditName$ = this.redditService.getSubRedditName()
  }

  /**
   * Contains all of the search form data points.
   */
  public searchForm = this.formBuilder.group({
    term: ''
  })

  /**
   * Handles submission of the search field.
   */
  public onSubmit(event: Event): void {
    if (this.searchForm.value.term) {
      this.redditService.setSubRedditName(this.searchForm.value.term.trim())
    }
  }
}

Conclusion

With all this rigged up, you'll have a virtually infinite scroller! Give it a try here. The Reddit API has a rate limit, so you might hit it quickly if you're testing this out. If you do, you can always try again later. I've also done a lot of additional work to mine, so please refer to the GitHub repository if you'd like to learn how to do the same.