Human judgement is needed to judge whether something is real. Can we rely on AI engines to perform this judgement? I think not. Should we allow AI engines to generate questions to assess human knowledge without a human being in the loop? We need to revisit augmented intelligence as a key concept and determine where the human should be inserted in the process to ensure scalability rather than eliminating the human in the process altogether.

ChatGPT and other AI engines can generate millions of items but this doesn’t solve the scalability problem because humans then have the task of reviewing millions of items. On the surface the items seem ok but upon closer review there are often many defects. When one uses the cognitive modeling method of AIG humans review cognitive models rather than the hundreds to thousands and potentially even millions of individual items generated from a single cognitive model.

When determining to use or not use AIG for item development there are both quantitative and qualitative factors to consider. Together these factors contribute to the return on investment (ROI) of operationalizing AIG in an organization. Quantitative Factors
1.     Cost per item
2.     Item survival rate from field testing
3.     Statistical item quality
4.     Scalability potential with and without using AIG
5.     Product diversification potential

Is AIG artificial intelligence?
AIG is the process of using item models to generate items using computer technology.  It can be considered a form of augmented intelligence.  Augmented intelligence is an area within artificial intelligence that deals with how computer systems emulate and extend human cognitive abilities thereby helping to improve human task performance.  It requires the interaction between a computer system and a human in order for the computer system to produce an output or solution.  Augmented intelligence combines the strength of modern computing using computational analysis and data storage with the human capacity for judgment to solve complex unstructured problems.  Augmented intelligence can therefore be characterized as any process or system that improves the human capacity for solving complex problems by relying on a partnership between a machine and a human.  The AIG methodology at MGHL Partners is an augmented intelligence approach to item development.

AIG is a method for producing evidence needed to validate the knowledge, skills, and/or competencies required to solve problems in a particular domain or content area. Given this definition, a human-in-the-loop AIG methodology is so important.