Conducting Legitimate Research Using Various Synthetic Imagery From Artmaking Generative AIs: An Exploration

Introduction

The popularization of artmaking generative AIs, which create visuals from text / image / mixed-modality prompts, enables researchers to tap the generated synthetic image data for learning. Certainly, the current academic literature suggests that there are insights to be had. Some researchers have used generative AI outputs to assess the capabilities of the AI tools, make predictions about the future (Haluza & Jungwirth, 2023), to form strategies, to teach, to assess learners, and other applications. They have surfaced insights about particularly effective magical combinations of text prompts for “prompt engineering.” They have shed light on how artificial neural networks (ANNs) of various types capture visual learning at particular nodes (among billions to trillions of them in various large language models or “LLMs”). Makers of the LLMs have released research papers about the unsupervised pre-training on image and language and code datasets and then the refinement through supervised fine-tuning, and the capture of billions to trillions of parameter weights that enable the regeneration of images and other digital contents based on text / visual / multi-modal prompts to the text-to-image models (TTI). Many such models protect against “forgetting” and “data drift” (such as the unintended changing of parameter values) but also strive to ensure diversities of outputs (protecting against overfitting to the training data and “mode collapse” or “creativity collapse” or “innovation decline”).

Some works have made inferences about how the studied artmaking generative AI system functions and the quality of seeding imagery in the underlying training imageset. Some studies have involved how these technologies are used for teaching and learning, including student receptivity to such technologies and those exploring the fitness of such tools for teaching and learning. Some studies have explored how these technologies affect work sequences (work pipelines). A number have explored the implications of such technologies for various learning domains and professions. Some have explored how generative AI may be used to better communicate complex concepts and practices to the general public, such as in public health, emergency, and hard science contexts. There have been plentiful works as position papers (op eds) that take various stances on how these new technologies should or should not be used.

Various traditional research methods have been used: the gold-standard experimental design (with interventions and pre- and post- testing); action research in classrooms; lab studies including AI analytics; survey research; case studies, and others. Some of the studies have been experiential ones, reported in anecdotal ways by the lead researcher. Some follow the rigorous practices of quantitative research, with high statistical standards to reject null hypotheses.

Some computational imagesets have been used as prompts for research into perception and other applications. Synthetic imagesets have been used to improve various machine vision algorithms. There have been applications of such imagesets to train yet other models, such as those used in the healthcare diagnostics space. Synthetic data, historically, has been generated to achieve various aims, such as to supplement actual data, to test technological systems / research instruments / computational models, to enable simulations, and other applications. In many cases, the actual data may not be available yet. Or the actual data may be protected, too sensitive to use, or difficult to de-identify. The real-world data may be of poor quality and incomplete or difficult to clean. Synthetic data, in typical contexts, should be validated to actual-world data in terms of accuracy and representativeness and realism (to the world).

Where this work is pseudo-novel is in the proposition that synthetic datasets from human-machine collaborations have not been used in-depth for research and that there are many as-yet unexplored approaches. In these early days, the question is how to exploit such machine-generated image data for exploration and research. Are there ways to systematize the work and make the findings more rigorous? (Synthetic datasets are used to train LLMs, in some cases. They are also used as “filler” to test socio-technical and other systems, such as online surveys.)

There are many extant or outstanding questions in terms of trying to extract research value from the synthetic image data from human-machine collaboration via the artmaking generative AI tools. They include the following:

(Un)askable research questions with synthetic AI-generated imagery