Publications

This study presents an AI-driven method for generating preinventive structures - initial precursors to creative design concepts - using the Geneplore model as a theoretical framework. Multimodal AI is leveraged to derive preinventive structures from combinations of components of an existing product. This method is evaluated by comparing AI-generated structures of a product to those reverse identified from real repurposing solutions for the same product (IKEA hacks). The appearance of AI-generated preinventive structures in the repurposed designs suggests that this method can inspire and lead to viable design concepts. Implications extend to sustainable design, creative ideation, and the theory-driven development of design methods that support design in constrained solution spaces. Future work can refine these approaches and investigate broader applications in diverse design contexts.

Recent developments in using Large Language Models (LLMs) to predict and align with neural representations of language can be applied to achieving a future vision of design tools that enable detection and reconstruction of designers’ mental representations of ideas. Prior work has largely explored this relationship during passive language tasks only, e.g., reading or listening. In this work, the relationship between brain activation data (functional imaging, fMRI) during appropriate and novel word association generation and LLM (Llama-2 7b) word representations is tested using Representational Similarity Analysis (RSA). Findings suggest that LLM word representations align with brain activity captured during novel word association, but not when forming appropriate associates. Association formation is one cognitive process central to design. By demonstrating that brain activity during this task can align with LLM word representations, insights from this work encourage further investigation into this relationship during more complex design ideation processes.

Design artifacts provide a mechanism for illustrating design information and concepts, but their effectiveness relies on alignment across design agents in what these artifacts represent. This work investigates the agreement between multi-modal representations of design artifacts by humans and artificial intelligence (AI). Design artifacts are considered to constitute stimuli designers interact with to become inspired (i.e., inspirational stimuli), for which retrieval often relies on computational methods using AI. To facilitate this process for multi-modal stimuli, a better understanding of human perspectives of non-semantic representations of design information, e.g., by form or function-based features, is motivated. This work compares and evaluates human and AI-based representations of 3D-model parts by visual and functional features. Humans and AI were found to share consistent representations of visual and functional similarities, which aligned well to coarse, but not more granular, levels of similarity. Human-AI alignment was higher for identifying low compared to high similarity parts, suggesting mutual representation of features underlying more obvious than nuanced differences. Human evaluation of part relationships in terms of belonging to same or different categories revealed that human and AI-derived relationships similarly reflect concepts of 'near' and 'far'. However, levels of similarity corresponding to 'near' and 'far' differed depending on the criteria evaluated, where 'far' was associated with nearer visually than functionally related stimuli. These findings contribute to a fundamental understanding of human evaluation of information conveyed by AI-represented design artifacts needed for successful human-AI collaboration in design.

As inspirational stimuli can assist designers with achieving enhanced design outcomes, supporting the retrieval of impactful sources of inspiration is important. Existing methods facilitating this retrieval have relied mostly on semantic relationships, e.g., analogical distances. Increasingly, data-driven methods can be leveraged to represent diverse stimuli in terms of multi-modal information, enabling designers to access stimuli in terms of less explored, non-text-based relationships. Toward improved retrieval of multi-modal representations of inspirational stimuli, this work compares human-evaluated and computationally derived similarities between stimuli in terms of non-text-based visual and functional features. A human subjects study (n = 36) was conducted where similarity assessments between triplets of 3D-model parts were collected and used to construct psychological embedding spaces. Distances between unique part embeddings were used to represent similarities in terms of visual and functional features. Obtained distances were compared with computed distances between embeddings of the same stimuli generated using artificial intelligence (AI)-based deep-learning approaches. When used to assess similarity in appearance and function, these representations were found to be largely consistent, with highest agreement found when assessing pairs of stimuli with low similarity. Alignment between models was otherwise lower when identifying the same pairs of stimuli with higher levels of similarity. Importantly, qualitative data also revealed insights regarding how humans made similarity assessments, including more abstract information not captured using AI-based approaches. Toward providing inspiration to designers that considers design problems, ideas, and solutions in terms of non-text-based relationships, further exploration of how these relationships are represented and evaluated is encouraged.

External sources of inspiration can promote the discovery of new ideas as designers ideate on a design task. Data-driven techniques can increasingly enable the retrieval of inspirational stimuli based on nontext-based representations, beyond semantic features of stimuli. However, there is a lack of fundamental understanding regarding how humans evaluate similarity between non-semantic design stimuli (e.g., visual). Toward this aim, this work examines human-evaluated and computationally derived representations of visual and functional similarities of 3D-model parts. A study was conducted where participants (n=36) assessed triplet ratings of parts and categorized these parts into groups. Similarity is defined by distances within embedding spaces constructed using triplet ratings and deep-learning methods, representing human and computational representations. Distances between stimuli that are grouped together (or not) are determined to understand how various methods and criteria used to define non-text-based similarity align with perceptions of 'near' and 'far'. Distinct boundaries in computed distances separating stimuli that are 'too far' were observed, which include farther stimuli when modeling visual vs. functional attributes.

Throughout the design process, designers encounter diverse stimuli that influence their work. This influence is particularly notable during idea generation processes that are augmented by novel design support tools that assist in inspiration discovery. However, fundamental questions remain regarding why and how interactions afforded by these tools impact design behaviors. This work explores how designers search for inspirational stimuli using an AI-enabled multi-modal search platform, which supports queries by text and non-text-based inputs. Student and professional designers completed a think-aloud design exploration task using this platform to search for stimuli to inspire idea generation. We identify expertise and search modality as factors influencing design exploration, including the frequency and framing of searches, and the evaluation and utility of search results.

Designers can benefit from inspirational stimuli when presented during the design process. Encountering external stimuli can also lead designers to negative design outcomes by limiting exploration of the design space and idea generation. Prior work has investigated how specific features of inspirational stimuli can be beneficial or harmful to designers. However, the processes designers use to search for and discover inspirational stimuli leading to these outcomes are less known. The objective of this work is thus to better understand how designers search for inspirational design stimuli. Specifically, we investigate how factors such as designer expertise and search modality (e.g., text vs. visual-based) impact both explicit and implicit features during the search for design stimuli. A cognitive study was completed by novice and expert designers (seven students and eight professionals), who searched for design stimuli using a novel multi-modal search platform while following a think-aloud protocol. The multi-modal search platform enabled search using text and nontext inputs, and provided design stimuli in the form of 3D-model parts. This work presents methods to describe search processes in terms of three levels: activities, behaviors, and pathways, as defined in this paper. Our findings determine that design expertise and search modality influence search behavior. Illustrative examples are presented and discussed of search processes leading designers to both negative and beneficial outcomes, such as designers fixating on specific results or benefiting unexpectedly from unintentional inspirational stimuli. Overall, this work contributes to an improved understanding of how designers search for inspiration, and key factors influencing these behaviors.

In prior work on designers’ search for inspirational stimuli, random discovery of stimuli through passive search processes has been underexplored. This paper primarily investigates how unintentionally discovered stimuli influence design outcomes, and why designers select these stimuli despite not meeting their initial expectations. In the present work, designers’ search for inspirational stimuli is explored through their use of a multi-modal search tool developed by our team. Fifteen designers used the search tool to find inspirational stimuli to solve an open-ended design challenge. During this study, many search results were found not to meet designers’ expectations. Nonetheless, designers incorporated a portion of these unexpected stimuli into their design ideas, resulting in the design outcomes: introduction of novel features, fulfillment of needs in an unanticipated way, and acceptance of readily available stimuli. This work suggests that encounters with unexpected stimuli can be beneficial, suggesting implications for future design tool development.

The successful adoption of artificial intelligence (AI)-enabled tools in engineering design requires an understanding of designers’ mental models of such tools. This work explores how professional and student engineering designers (1) develop mental models of a novel AI-driven engineering design tool and (2) speculate AI-enabled functionalities that can aid them. Student (N = 7) and professional (N = 8) designers completed a task using an AI-enabled tool, and were interviewed to uncover their mental model of the tool and speculations on future AI-enabled functionalities. Both professional and student designers developed accurate mental models of the AI tool, and speculated functionalities that were similarly “near” and “far” in terms of analogical distance from the AI tool’s functionality. These findings suggest that mental models and cross-application of AI tool functionality are readily accessible to designers, offering several implications for widespread adoption of AI-enabled design tools.

Inspirational stimuli are known to be effective in supporting ideation during early-stage design. However, prior work has predominantly constrained designers to using text-only queries when searching for stimuli, which is not consistent with real-world design behavior where fluidity across modalities (e.g., visual, semantic, etc.) is standard practice. In the current work, we introduce a multi-modal search platform that retrieves inspirational stimuli in the form of 3D-model parts using text, appearance, and function-based search inputs. Computational methods leveraging a deep-learning approach are presented for designing and supporting this platform, which relies on deep-neural networks trained on a large dataset of 3D-model parts. This work further presents the results of a cognitive study (n = 21) where the aforementioned search platform was used to find parts to inspire solutions to a design challenge. Participants engaged with three different search modalities: by keywords, 3D parts, and user-assembled 3D parts in their workspace. When searching by parts that are selected or in their workspace, participants had additional control over the similarity of appearance and function of results relative to the input. The results of this study demonstrate that the modality used impacts search behavior, such as in search frequency, how retrieved search results are engaged with, and how broadly the search space is covered. Specific results link interactions with the interface to search strategies participants may have used during the task. Findings suggest that when searching for inspirational stimuli, desired results can be achieved both by direct search inputs (e.g., by keyword) as well as by more randomly discovered examples, where a specific goal was not defined. Both search processes are found to be important to enable when designing search platforms for inspirational stimuli retrieval.

Inspirational stimuli are known to be effective in supporting ideation during the design process. However, minimal prior work has allowed individuals to search using multiple modes of input simultaneously, which is more representative of real design behavior. In the current work, we developed a multi-modal search platform that retrieves 3D model parts based on text, appearance, and function-based search inputs. This work presents the results of an experimental study (n = 21) in which the search platform was used to find parts identified as potentially useful for inspiring solutions to a design challenge. Participants were asked to engage with three different search modalities: search by keywords, by curated 3D parts, and by user-assembled 3D parts in their workspace. When searching by parts that are curated or in their workspace, additional control over the similarity of appearance and function of results in reference to the input was available to participants. The results of this study demonstrate that the modality used affects search behavior, such as in the frequency of searches, how participants engage with retrieved search results, and how broadly the search space is covered. Specific results link interactions with the interface to search strategies participants may have used during the task. Findings suggest that multi-modal search should enable intentional search for desired goals through direct search inputs (e.g., by keyword) and incremental adjustments to features of visually represented search inputs. Moreover, enabling discovery of inexplicitly searched for examples through related information or more randomly encountered examples may assist exploratory search behavior.

Divergent thinking, an aspect of creativity, is often studied by measuring performance on the Alternative Uses Test (AUT). There is, however, a gap in creativity research concerning how visual stimuli on the AUT are perceived. Memory and attention researchers have used eye-tracking studies to reveal insights into how people think and how they perceive visual stimuli. Thus, the current work uses eye tracking to study how eye movements are related to creativity. Participants orally listed alternative uses for twelve objects, each visually presented for 2 min in four different views. Using eye tracking, we specifically explored where and for how long participants fixate their eyes at visual presentations of objects during the AUT. Eye movements before and while naming alternative uses were analyzed. Results revealed that naming new instances and categories of alternative uses correlated more strongly with visual fixation toward multiple views than toward single views of objects. Alternative uses in new, previously unnamed categories were also more likely named following increased visual fixation toward blank space. These and other findings reveal the cognitive-thinking styles and eye-movement behaviors associated with naming new ideas. Such findings may be applied to enhance divergent thinking during design.

Divergent thinking, an aspect of creativity, is often studied by measuring performance on the Alternative Uses Test (AUT). There is however a gap in creativity research concerning how visual stimuli on the AUT are most effectively perceived. Research in memory and attention have used eye-tracking studies to reveal insights into how people think and perceive visual stimuli. Thus, the current work uses eye tracking to study how eye movements are related to creativity. Participants orally listed alternative uses for twelve objects, each visually represented for two minutes in four different views. Using eye tracking, we specifically explored where and for how long people fixate their eyes at objects during the AUT. Eye movements before and while naming alternative uses are studied. Results revealed that naming new instances and categories of alternative uses correlates more strongly with visual fixation towards multiple views than towards a single view of the object. Alternative uses in new, previously unnamed categories are also more likely named following increased visual fixation towards blank space. These and other findings reveal the cognitive-thinking styles and eye-movement behaviors associated with finding new ideas. Such findings may be applied to reduce fixation to existing ideas during design.

The challenge of finding alternative uses for retired wind-turbine blades, which have limited disposal options, motivates this work. Two reuse concept-generation activities (CGAs) conducted in German universities revealed difficulties with the parts' large scale and seeing beyond their original use. Existing methods, e.g., using functional analogy, are less applicable, since for safety reasons, these parts should not be reused to fulfill the same function. Therefore, this work explores the use of visual similarity to support reuse-concept generation. A method was developed that (1) finds visually similar images (VSIs) for wind-turbine-blade photos and (2) derives potential-reuse concepts based on objects that are visually similar to wind-turbine blades in these images. Comparing reuse concepts generated from the two methods, VSI produced fewer smaller-than-scale concepts than CGA. While other qualities such as feasibility depend on the specific photo selected, this work provides a new framework to exploit visual similarity to find alternative uses. As demonstrated for wind-turbine blades, this method aids in generating alternative-use concepts, especially for large-scale objects.

This work is motivated by finding alternative uses for retired wind-turbine blades, which have limited disposal options. Two reuse concept-generation activities (CGA) conducted in German universities revealed difficulties with the parts’ large scale and seeing beyond their original use. Existing methods, e.g., using functional analogy, are less applicable, since for safety reasons, these parts should not be reused in the same function. Therefore, this work explores the use of visual similarity to support reuse-concept generation. A method was developed that 1) finds visually similar images (VSI) for wind-turbine-blade photos; and 2) derives potential-reuse concepts based on objects that are visually similar to wind-turbine blades in these images. Comparing reuse concepts generated from the two methods, VSI produced fewer smaller-than-scale concepts than CGA. While other qualities like feasibility depend on the specific photo selected, this work provides a new framework to exploit visual similarity to find alternative uses. As demonstrated for wind-turbine blades, this method aids in generating alternative-use concepts, especially for large-scale objects.