How to remove event listener inside useCallback hook

I want to create a generic react hook that will add a scroll event to the element and return a boolean indicating that the user has scrolled to the top of the element.
Now, the problem is this element might not be visible right away. Hence I'm not able to use useEffect. As I understand in that situation it is advised to use useCallback

So I did, and it works:

function useHasScrolled() {
  const [hasScrolled, setHasScrolled] = useState(false);
  const ref = useRef(null);
  const setRef = useCallback((element) => {
    const handleScroll = (e) => {
      setHasScrolled(e.target.scrollTop !== 0);
    };
    if (element) {
      element.addEventListener("scroll", handleScroll);
    }

    ref.current = element;
  }, []);
  return {
    hasScrolled,
    scrollingElementRef: setRef
  };
}

I can use my hook like this:

  const { hasScrolled, scrollingElementRef } = useHasScrolled();
  ....
  return <div ref={scrollingElementRef}>....

However, the problem is, I don't know how to remove the event listener. With the useEffect hook, it's pretty straightforward - you just return the cleanup function.

Here's the codesandbox, if you want to check the implementation: https://codesandbox.io/s/pedantic-dhawan-83fdw3

Expected behavior - when node is removed from DOM - event listeners will be also removed and collected by GC.

But

Codesandbox example is a bit tricky, React treats

<div>Loading...</div>

and

<div className="scrollingDiv" ref={scrollingElementRef}>
  <h1>Hello, I've finally loaded!</h1>
  <Lorem />
</div>

as a same div, same object, just with different props (className and children), so when div.scrollingDiv is replaced by conditional rendering to div(loading) - event listeners are still there and accumulating.

This behavior can be fixed as is by using keys.

{loading ? (
  <div key="div1">Loading...</div>
) : (
  <div key="div2" className="scrollingDiv" ref={scrollingElementRef}>
    <h1>Hello, I've finally loaded!</h1>
    <Lorem />
  </div>
)}

In that way event listeners will be removed as expected.

Another solution is to add 1 more useRef and useEffect to the custom hook to store and execute actual unsubscribe function:

function useHasScrolled() {
  const [hasScrolled, setHasScrolled] = useState(false);
  const ref = useRef(null);
  const unsubscribeRef = useRef(null);

  const setRef = useCallback((element) => {
    const eventName = "scroll";
    const handleScroll = (e) => {
      setHasScrolled(e.target.scrollTop !== 0);
    };

    if (unsubscribeRef.current) {
      unsubscribeRef.current();
      unsubscribeRef.current = null;
    }

    if (element) {
      element.addEventListener(eventName, handleScroll);

      unsubscribeRef.current = () => {
        console.log("removeEventListener called on: ", element);
        element.removeEventListener(eventName, handleScroll);
      };
      ref.current = element;
    } else {
      unsubscribeRef.current = null;
      ref.current = null;
    }
  }, []);

  useEffect(() => {
    return () => {
      if (unsubscribeRef.current) {
        unsubscribeRef.current();
        unsubscribeRef.current = null;
      }
    };
  }, []);

  return {
    hasScrolled,
    scrollingElementRef: setRef
  };
}

That code will work without adding key.

Utility code for Chrome dev console to count scroll listeners:

Array.from(document.querySelectorAll('*'))
  .reduce(function(pre, dom){
    var clks = getEventListeners(dom).scroll;
    pre += clks ? clks.length || 0 : 0;
    return pre
  }, 0)

Updated codesandbox: https://codesandbox.io/s/angry-einstein-6fb1u4?file=/src/App.js