Both the 1.0 and 1.1 specs say:

The optional links member within each resource object contains links related to the resource.

If present, this links object MAY contain a self link that identifies the resource represented by the resource object.

Specifically, this wording does not prohibit additional members in the links object. (For that, the spec would have to either say something like "MAY ONLY contain", or add a sentence saying that the links object "MAY NOT contain additional members".) However, in #1019 (comment), @ethanresnick stated that this is forbidden.

I have implemented the now two-and-a-half year old JSON:API framework Felicity. Based on my understanding of the spec, it has always allowed custom links, which we use in many of our APIs. I have essentially used custom links in cases where representing state transitions via resource creation or normal PATCH update seemed too heavy-handed. For example, in an API representing a directory tree, folder resources have a makeAllDescendantsInheritAccess link that supports POST to make all child files/folders inherit its access rules. Its presence or absence indicates whether the operation is currently allowed by the requesting user. We also have a download link on file resources that serves the actual file contents (i.e., does not return a JSON:API response). I have many more examples if need be.

I just wanted to let you know that custom links are easily interpreted as allowed by the current spec, and that these exist in the wild. The spec may need clarification on this point if you really want them to be forbidden (which may cause breaking changes in APIs in the wild, preventing them from targeting a new JSON:API version and still be spec compliant).