Why does the textarea rows count not decrease when content is removed?

When I don't set the rows to state or textAreaRef.current.rows = calculatedRows, the console log properly increases and decreases the calculatedRows. However, as soon as I set the calculatedRows to the state or textAreaRef.current.rows = calculatedRows, it increases the rows but does not decrease them; it stays at the last calculatedRows.

import React, { useState, useRef, useEffect } from "react";
import ReactDOM from "react-dom";
import styled from "styled-components";

const TextArea = styled.textarea`
  padding: 2px;
  resize: none;
  overflow-y: hidden;
  line-height: 1.5;
`;

function App() {
  const [val, setVal] = useState("");
  const textAreaRef = useRef(null);
  const [rows, setRows] = useState(1);

  const resizeTextArea = () => {
    const lineHeight = parseInt(
      window.getComputedStyle(textAreaRef.current).lineHeight
    );
    const calculatedRows = Math.floor(
      textAreaRef.current.scrollHeight / lineHeight
    );
    console.log(calculatedRows);
    setRows(calculatedRows);
  };

  useEffect(() => {
    resizeTextArea();
  }, [val]);

  const onChange = (e) => {
    setVal(e.target.value);
  };

  return (
    <div>
      <TextArea ref={textAreaRef} value={val} onChange={onChange} rows={rows} />
    </div>
  );
}

const rootElement = document.getElementById("root");
ReactDOM.render(<App />, rootElement);

I expected the number of rows in the textarea to decrease when content is removed, but it only increases. I've tried using setRows(calculatedRows) and textAreaRef.current.rows = calculatedRows to update the number of rows but encountered the same issue.