In order to write new classes, a deeper understanding is required of how the auto-overlay package works. In fact, two kinds of overlays are automatically created, updated and destroyed when auto-overlays are active: the auto-overlays themselves, and “match” overlays, used to mark text that matches an auto-overlay regexp.
For overlay classes that only require one regexp to fully define an
overlay (the word and line classes are the only standard
classes like this1), the
auto-overlays are always matched with the corresponding match
overlay. For classes that require two regexp matches to define the start
and end of an overlay (all other standard classes), each edge of an
auto-overlay can be matched with a match overlay. The match overlays
define where the edge of the auto-overlay is located. There will always
be at least one matched edge, since an auto-overlay is only created when
a regexp match is found, but it is possible for the second edge to not
yet be matched (for many classes, the unmatched edge will be located at
the beginning or end of the buffer).
If a match overlay delimits the start of an auto-overlay, the match
overlay is stored in the auto-overlay's start property. The match
overlay is also stored in the start property for auto-overlays
that only require a single match. If a match overlay delimits the end of
an auto-overlay, the match overlay is stored in the auto-overlay's
end property. Conversely, a “link” to the auto-overlay is
always stored in the match overlay's parent property2.
Whenever a buffer is modified, the lines containing the modifications are scanned for new regexp matches. If one is found, a new match overlay is created covering the matching text, and then passed as an argument to the appropriate “parse” function3 for its class. This deals with creating or updating the auto-overlays as appropriate. If the text within a match overlay is modified, the match overlay checks whether the text it covers still matches the regexp. If it no longer matches, the match overlay is passed as an argument to the appropriate “suicide” function for its class, which deals with updating or deleting its parent auto-overlay (and possibly more besides).
To summarise, the core of the auto-overlays package deals with searching for regexp matches, and creating or deleting the corresponding match overlays. It then hands over the task of creating, updating or deleting the auto-overlays themselves to class-specific functions, which implement the correct behaviour for that class.
[1] Although the self class only requires
one regexp definition, the auto-overlays themselves require two matches
to that same regexp to set the start and end of the overlay.
[2] The “parent” terminology is admittedly very poor, and is a relic of a previous incarnation of the auto-overlays package, when it made more sense.
[3] More bad terminology.