In mathematics, coset enumeration is the problem of counting the cosets of a subgroup H of a group G given in terms of a presentation. As a by-product, one obtains a permutation representation for G on the cosets of H. If H has a known finite order, coset enumeration gives the order of G as well.

For small groups it is sometimes possible to perform a coset enumeration by hand. However, for large groups it is time-consuming and error-prone, so it is usually carried out by computer. Coset enumeration is usually considered to be one of the fundamental problems in computational group theory.

The original algorithm for coset enumeration was invented by John Arthur Todd and H. S. M. Coxeter. Various improvements to the original Todd–Coxeter algorithm have been suggested, notably the classical strategies of V. Felsch and HLT (Haselgrove, Leech and Trotter). A practical implementation of these strategies with refinements is available at the ACE website.^[1] The Knuth–Bendix algorithm also can perform coset enumeration, and unlike the Todd–Coxeter algorithm, it can sometimes solve the word problem for infinite groups.

The main practical difficulties in producing a coset enumerator are that it is difficult or impossible to predict how much memory or time will be needed to complete the process. If a group is finite, then its coset enumeration must terminate eventually, although it may take arbitrarily long and use an arbitrary amount of memory, even if the group is trivial. Depending on the algorithm used, it may happen that making small changes to the presentation that do not change the group nevertheless have a large impact on the amount of time or memory needed to complete the enumeration. These behaviours are a consequence of the unsolvability of the word problem for groups.

A gentle introduction to coset enumeration is given in Rotman's text on group theory.^[2] More detailed information on correctness, efficiency, and practical implementation can be found in the books by Sims^[3] and Holt et al.^[4]