I know of a number of existing library classification systems (DDC, LC, CC), but what area of library science is concerned with creating new classification systems, and how one judges the suitability of such a system?
As said by Joe, the general area is Knowledge Organization and information science has strong roots in knowledge organization with the International Federation for Information and Documentation (1895-2002). The International Society for Knowledge Organization (ISKO) should provide some references, but my personal impression is that ISKO somehow lost touch to the technical development during the 1990s. For recent research in knowledge organization I better recommend the Networked Knowledge Organization Systems/Services group (NKOS). Within the field of information science, knowledge organization has lost importance with the rise of information retrieval and the shift from documentation science to information science.
Classifications are just one kind of the systems, which knowledge organization deals with. Despite the long tradition of classification systems, I found that knowledge about these systems is often low and practical application is poor. For instance one should know about basic terms such as facet, precombination, and postcombination, instead of the simplified image of a classification as a tree. In theory, revision and evaluation should be applied regularly to classifications. In practice some libraries even fail to cleanly track classification changes and to create a simple histogram or tree-map with distribution of their holdings in the classification. One reason is the lack of availability of the classification system in a machine-readable form. For instance the SKOS ontology better fits to thesauri than to specialized classification systems and many classifications are hidden in proprietary library systems.
Maybe someone else can suggest a good textbook are best practice guide to work with classifications. With a theoretical point of view I recommend Sorting things out: classification and its consequences
You're asking for a classification for design of classification systems?
With this whole 'iSchool' push to get away from libraries, it's possible some will call it ontology as that's what it's called in computer science, but in LIS, it's knowledge organization, which tends to be a little more broad, as it concerns itself with other types of controlled vocabularies
Ontology in LIS tends to be a specific type of controlled vocabulary in which you can define any arbitrary relationships between the entries (eg, in biology, you might define which animals eat which other animals in the region you're studying)
As for judging for suitability, in many ways, it's like judging databases -- can I ask it the questions that I need to have answered? But there's also 'will I get useful answers?'
For this, you need to have:
One important thing to remember is that classification systems don't have to be fixed ... there are people that maintain them : clarify entries, split them, merge them, or otherwise deal with language changing or other changing needs of the users of the system.
(and for our final in "Construction of Index Languages and Thesauri", I remember we had to judge a thesauri ... I took a sampling of concepts that in the field, and then checked to see if they existed within the system. Which isn't quite full use of the system, but that's really the only way to really see if it works for you)