Journal papers

Design artifacts provide a mechanism for illustrating design information and concepts, but their effectiveness relies on alignment across design agents in what these artifacts represent. This work investigates the agreement between multi-modal representations of design artifacts by humans and artificial intelligence (AI). Design artifacts are considered to constitute stimuli designers interact with to become inspired (i.e., inspirational stimuli), for which retrieval often relies on computational methods using AI. To facilitate this process for multi-modal stimuli, a better understanding of human perspectives of non-semantic representations of design information, e.g., by form or function-based features, is motivated. This work compares and evaluates human and AI-based representations of 3D-model parts by visual and functional features. Humans and AI were found to share consistent representations of visual and functional similarities, which aligned well to coarse, but not more granular, levels of similarity. Human-AI alignment was higher for identifying low compared to high similarity parts, suggesting mutual representation of features underlying more obvious than nuanced differences. Human evaluation of part relationships in terms of belonging to same or different categories revealed that human and AI-derived relationships similarly reflect concepts of 'near' and 'far'. However, levels of similarity corresponding to 'near' and 'far' differed depending on the criteria evaluated, where 'far' was associated with nearer visually than functionally related stimuli. These findings contribute to a fundamental understanding of human evaluation of information conveyed by AI-represented design artifacts needed for successful human-AI collaboration in design.

Throughout the design process, designers encounter diverse stimuli that influence their work. This influence is particularly notable during idea generation processes that are augmented by novel design support tools that assist in inspiration discovery. However, fundamental questions remain regarding why and how interactions afforded by these tools impact design behaviors. This work explores how designers search for inspirational stimuli using an AI-enabled multi-modal search platform, which supports queries by text and non-text-based inputs. Student and professional designers completed a think-aloud design exploration task using this platform to search for stimuli to inspire idea generation. We identify expertise and search modality as factors influencing design exploration, including the frequency and framing of searches, and the evaluation and utility of search results.

Inspirational stimuli are known to be effective in supporting ideation during early-stage design. However, prior work has predominantly constrained designers to using text-only queries when searching for stimuli, which is not consistent with real-world design behavior where fluidity across modalities (e.g., visual, semantic, etc.) is standard practice. In the current work, we introduce a multi-modal search platform that retrieves inspirational stimuli in the form of 3D-model parts using text, appearance, and function-based search inputs. Computational methods leveraging a deep-learning approach are presented for designing and supporting this platform, which relies on deep-neural networks trained on a large dataset of 3D-model parts. This work further presents the results of a cognitive study (n = 21) where the aforementioned search platform was used to find parts to inspire solutions to a design challenge. Participants engaged with three different search modalities: by keywords, 3D parts, and user-assembled 3D parts in their workspace. When searching by parts that are selected or in their workspace, participants had additional control over the similarity of appearance and function of results relative to the input. The results of this study demonstrate that the modality used impacts search behavior, such as in search frequency, how retrieved search results are engaged with, and how broadly the search space is covered. Specific results link interactions with the interface to search strategies participants may have used during the task. Findings suggest that when searching for inspirational stimuli, desired results can be achieved both by direct search inputs (e.g., by keyword) as well as by more randomly discovered examples, where a specific goal was not defined. Both search processes are found to be important to enable when designing search platforms for inspirational stimuli retrieval.

Divergent thinking, an aspect of creativity, is often studied by measuring performance on the Alternative Uses Test (AUT). There is, however, a gap in creativity research concerning how visual stimuli on the AUT are perceived. Memory and attention researchers have used eye-tracking studies to reveal insights into how people think and how they perceive visual stimuli. Thus, the current work uses eye tracking to study how eye movements are related to creativity. Participants orally listed alternative uses for twelve objects, each visually presented for 2 min in four different views. Using eye tracking, we specifically explored where and for how long participants fixate their eyes at visual presentations of objects during the AUT. Eye movements before and while naming alternative uses were analyzed. Results revealed that naming new instances and categories of alternative uses correlated more strongly with visual fixation toward multiple views than toward single views of objects. Alternative uses in new, previously unnamed categories were also more likely named following increased visual fixation toward blank space. These and other findings reveal the cognitive-thinking styles and eye-movement behaviors associated with naming new ideas. Such findings may be applied to enhance divergent thinking during design.

The challenge of finding alternative uses for retired wind-turbine blades, which have limited disposal options, motivates this work. Two reuse concept-generation activities (CGAs) conducted in German universities revealed difficulties with the parts' large scale and seeing beyond their original use. Existing methods, e.g., using functional analogy, are less applicable, since for safety reasons, these parts should not be reused to fulfill the same function. Therefore, this work explores the use of visual similarity to support reuse-concept generation. A method was developed that (1) finds visually similar images (VSIs) for wind-turbine-blade photos and (2) derives potential-reuse concepts based on objects that are visually similar to wind-turbine blades in these images. Comparing reuse concepts generated from the two methods, VSI produced fewer smaller-than-scale concepts than CGA. While other qualities such as feasibility depend on the specific photo selected, this work provides a new framework to exploit visual similarity to find alternative uses. As demonstrated for wind-turbine blades, this method aids in generating alternative-use concepts, especially for large-scale objects.