Can ChatGPT perform COSMIC functional size measurement accurately?
In previous experiments, I asked both ChatGPT and Gemini to determine the functional size of a functional process using the COSMIC method.
To further examine the accuracy of ChatGPT in COSMIC functional size measurement, I conducted a small comparison using examples from the Course Registration System case study published by the COSMIC Group.
The requirements in this case study are described in sufficient detail for COSMIC measurement, and the corresponding measurement results have already been established by COSMIC measurement experts.
Two requirements were selected and submitted to ChatGPT for measurement. The results were then compared with the reference measurements.
Case 1: Search for a Course
The first requirement concerns searching for a course.
After entering the requirement into ChatGPT, the following measurement result was obtained:
In this case, the result produced by ChatGPT was consistent with the result provided in the COSMIC case study.
This example suggests that when the functional requirements are sufficiently detailed and relatively straightforward, ChatGPT may be able to identify the required COSMIC data movements correctly.
Case 2: Delete a Student Schedule
The second requirement concerns deleting a student’s course schedule.
The requirement was again submitted to ChatGPT for measurement.
This time, however, the COSMIC reference measurement contained two more function points than the result generated by ChatGPT.
A closer comparison showed that ChatGPT failed to identify two separate Exit (X) data movements:
- displaying the student’s schedule; and
- sending information to another system, the Course Catalog System.
Instead, ChatGPT grouped these outputs together with the error/confirmation messages and treated them as a single Exit.
Refining the Instructions
This result suggested that ChatGPT did not fully distinguish the COSMIC rules related to Exit data movements.
I therefore provided an additional instruction:
The returned schedule information needs to be displayed separately and should not be combined with the error/confirmation message. Updating the Course Catalog System also requires information to be sent to another system. Please identify the data movements again.
After receiving this instruction, ChatGPT identified all of the Exits separately.
However, this result was still incorrect.
According to the COSMIC rules applied in the reference case, error/confirmation messages without a clearly distinguishable object of interest should be grouped together rather than counted separately.
I therefore refined the instruction again:
Do not identify the deletion confirmation separately. Group all error/confirmation messages into one Exit, while keeping the display output and the information sent to the external system separate.
ChatGPT recalculated the measurement, but it still treated the successful deletion message as a separate Exit.
This was still inconsistent with the reference measurement because, in this context, the successful deletion message should also be treated as part of the error/confirmation messaging.
I therefore provided one more instruction:
The successful deletion message is also part of the error/confirmation messages and should be grouped with them.
After this clarification, ChatGPT finally produced the same measurement result as the COSMIC reference case.
What Can We Learn from This Comparison?
This small experiment highlights two important points.
First, COSMIC functional size measurement requires requirements to be described at a sufficient level of detail. If the available requirements are too vague, it becomes difficult to identify functional processes, data groups, and data movements reliably.
Second, understanding the COSMIC measurement rules remains essential.
Even though large language models such as ChatGPT are trained on large amounts of information, the quality and accuracy of the underlying knowledge cannot always be verified. As this example shows, ChatGPT was able to reach the correct result only after several increasingly specific instructions about the COSMIC rules.
Therefore, AI tools may be useful as assistants for COSMIC measurement, but their results still require review and validation by people who understand the COSMIC method and its measurement rules.
For professional or research applications, the role of the COSMIC measurement expert remains important.
This article is adapted from an earlier Chinese article written by the author.
What are your thoughts?