Cancellative hypergraphs and Steiner triple systems

A triple system is cancellative if it does not contain three distinct sets A, B, C such that the symmetric difference of A and B is contained in C. We show that every cancellative triple system 7-t that satisfies a particular inequality between the sizes of 7-t and its shadow must be structurally close to the balanced blowup of some Steiner triple system. Our result contains a stability theorem for cancellative triple systems due to Keevash and Mubayi as a special case. It also implies that the boundary of the feasible region of cancellative triple systems has infinitely many local maxima, thus giving the first example showing this phenomenon. (c) 2024 Elsevier Inc. All rights reserved.