How do standard library containers know what's before the past-the-end iterator?

I'm trying to implement my own list and map, and I've been having trouble with the end() method since these containers aren't contiguous in memory.

After some research, I've ended up implementing the end() as a method that returns an iterator that holds nullptr. (https://stackoverflow.com/questions/44767368/). That is enough for most of my implementation, as it verifies that list::begin() == list::end() when list is empty (empty list has no nodes so head points to nullptr).

However, I've seen that it's possible to call operator--() on past-the-end iterators to point to the actual last element of the container.

int main() {
  {
    std::map<int, char> m = {{1, 'a'}, {2, 'b'}, {3, 'c'}};
    auto it = m.end();
    --it;
    std::cout << it->first << "\n";  // output: 3
    std::cout << it->second << "\n"; // output: c
  }
  {
    std::list<int> l{ 1, 2, 8};
    auto it = l.end();
    --it;
    std::cout << *it << "\n"; // output: 8
  }
}

How is this achieved? The implementation of operator--() of _List_iterator in the gcc changes the current node to point to the previous one:

      _Self&
      operator--() _GLIBCXX_NOEXCEPT
      {
	_M_node = _M_node->_M_prev;
	return *this;
      }

But how is this possible if the node is out of the list? How does that node even exist? Does this mean that an extra node is allocated? Are you actually allowed to call operator--() to a past-the-end iterator?

These are the begin() and end() methods of gcc's std::list, in case it helps:

     /**
       *  Returns a read-only (constant) iterator that points to the
       *  first element in the %list.  Iteration is done in ordinary
       *  element order.
       */
      _GLIBCXX_NODISCARD
      const_iterator
      begin() const _GLIBCXX_NOEXCEPT
      { return const_iterator(this->_M_impl._M_node._M_next); }

      /**
       *  Returns a read/write iterator that points one past the last
       *  element in the %list.  Iteration is done in ordinary element
       *  order.
       */
      _GLIBCXX_NODISCARD
      iterator
      end() _GLIBCXX_NOEXCEPT
      { return iterator(&this->_M_impl._M_node); }

> But how is this possible if the node is out of the list?

Simply, _M_node->_M_prev points to the last node.

> How does that node even exist?

In the same way that any node exists. In the case of standard list, it would be created with the provided allocator.

> Does this mean that an extra node is allocated?

That seems like a reasonable conclusion. This pattern is called "sentinel node".

> Are you actually allowed to call operator--() to a past-the-end iterator?

Yes.

The standard containers use a bit of a dirty trick (or clever, depending on the way you look at it). (Simplifying the code somewhat).

You start out with an 'base' node type. This is just your forward and backwards pointers.

struct ListNodeBase {
  ListNodeBase* next;
  ListNodeBase* prev;
};

The list node extends the base node type with the actual data to store. This separation between the pointers and the data is kinda useful!

template<typename T>
struct ListNode : public ListNodeBase {
  T data;
};

Iteration across the linked list ONLY uses ListNodeBase. To access the data, the objects are upcast to ListNode.

template<typename T>
struct ListIter {
   ListNodeBase* current;

   // Here however we upcast to the node type to access the data. 
   T& operator*() const
      { return static_cast< ListNode<T>* >(current)->data; }
   T* operator->() const
      { return &static_cast< ListNode<T>* >(current)->data; }

   // just your bog standard list traversal.. 
   // again, this only operates on the 
   ListIter<T>& operator++() {
      current = current->next;
      return *this;
   }

   ListIter<T>& operator--() {
      current = current->prev;
      return *this;
   }
};

The trick though is that there is one node of ListNodeBase (which is the list container), and the rest of the nodes are ListNode<T>.

template<typename T> 
struct list : private ListNodeBase {

   typedef ListIter<T> iterator;


   list() 
   {
     // when empty, store pointers to itself 
     this->next = this;
     this->prev = this;
   }

   iterator begin() 
     { return (iterator){ this->next }; }

   iterator end() 
     { return (iterator){ this }; }
};

In other words, the list container is the sentinel node (It just doesn't store any data, because it uses the base node type, and node the actual node).