Calista: A deep learning-based system for understanding and evaluating website aesthetics

It is a fact that the Internet and the Software as a Service industries have been pervasive in all human activity: finding information, marketing and advertisement, entertainment and social networking among others. Studies (Lindgaard et al., 2006, Tractinsky et al., 2006, Tuch et al., 2012) have shown that users establish a lasting impression about a website’s appeal in splits of a second. This first impression is based on how visually appealing is what the user is experiencing and has a great impact on the user’s perception about the website’s quality and usability (Hartmann et al., 2008). A first good impression can greatly increase the trust towards the website and its content (Michailidou et al., 2008, Lindgaard et al., 2011, Reinecke et al., 2013) and decrease bounce rates (Robins and Holmes, 2008). For an e-commerce website, this could mean increased conversion rates, while for a news portal increase in daily visits. Thus, the aesthetics of a website is a very important trait that should be taken seriously during the design process (Lindgaard et al., 2011, Lindgaard and Dudek, 2003).

Aesthetics is a topic of major interest within the field of human–computer interaction and especially in the context of webpage design (Tractinsky, 2013). When designing a website, having a good understanding of what users perceive as visually appealing is of utmost significance. Web designers follow some guidelines that have prevailed (Galitz, 2007), but the final outcome is subjective in terms of aesthetics and may differ from the preferences of the user target group (Park et al., 2004). This happens because the designers’ work is influenced by their personal experience and taste. Companies deal with the designer bias problem by getting feedback through user groups and questionnaires. However, this procedure is costly and time-consuming, since enough data must be collected and analyzed. As a consequence, a growing need emerges for tools which can assess website aesthetics automatically.

A number of previous works have tried to quantify website aesthetics by proposing various aesthetics-relevant hand-crafted features, inspired by the photography or psychology literature. For instance, Wu et al. (2011) generate manual features by considering some webpage aspects such as the layout’s block size and color characteristics (e.g. hue, brightness). However, such approaches exhibit some known limitations (Lu et al., 2014). First, they are manually defined, thus a possibility exists that some features are not taken into account. Second, these hand-crafted features comprise mere approximations of psychological rules and empirical observations due to the complexity of computationally implementing them.

With the evolution of deep learning techniques, research shows that by employing convolutional neural networks (CNNs) one can result in state-of-the-art performance for photo aesthetics assessment (Talebi and Milanfar, 2018, Lu et al., 2015a, Kong et al., 2016, Kao et al., 2015). Additionally, it has become evident that replacing hand-crafted features with an end-to-end automatic learning system for aesthetics-related tasks leads to more efficient solutions (Lu et al., 2015b, Lu et al., 2015a). Although a webpage is a special kind of image with differences in its composition, Dou et al. (2019) show that a CNN-based method significantly outperforms previous works that utilize hand-crafted features. However, research on the enhancement of automated website aesthetics evaluation is still limited and immature, since most previous works mainly focus on providing qualitative design guidelines rather than quantitative methods for prediction.

Another point that should be stressed is that the majority of research on website aesthetics assessment exploits the dataset introduced in Reinecke and Gajos (2014). In this work the authors conducted a survey where a webpage was displayed to users and asked them to score it on a scale of 1 to 9. Asking users to provide a numerical rating on a scale is a commonly used method to collect data, e.g. assign a star from 1 to 5 for a book or a score from 1 to 10 for an art painting. Although more straightforward, this approach has a number of limitations, which are discussed in the next paragraph. On the contrary, research dictates that collecting data through pairwise comparisons comprises a more reliable alternative to numerical ratings and overcomes the aforementioned limitations (Ammar and Shah, 2011, Perez-Ortiz and Mantiuk, 2017). In the pairwise comparison case, two items are displayed at a time, selected from a set of items, and the users are asked to choose the one they prefer most.

Given that website aesthetics evaluation is a demanding process for non-expert survey participants, comparison-based data collection is a more appropriate direction for this task. A limitation introduced by ratings is that users may perceive the scores differently which leads to calibration issues (Tsukida and Gupta, 2011, Ammar and Shah, 2011). For instance, users A and B might have had exactly the same enjoyment out of using a product, but the user A’s score “3” might practically correspond to the user B’s score “4”. This does not apply to the comparison-based data collection method since users who share the same impression cannot interpret the given task differently. Second, rating-based surveys demand more concentration from users since they should remember their previous answers and use it as reference. Users may forget their prior answers and mistakenly assign a higher/lower score to an item (Stewart et al., 2005), which may contradict their previous ratings. In contrast, pairwise comparisons are independent of each other and users do not have to remember their previous answers. During each comparison, the items that should be used as reference are displayed, guaranteeing that users use the same reference to avoid calibration issues. Third, pairwise comparisons are recognized as easier and faster for humans to make, as well as a more “natural” task for the human brain (Stewart et al., 2005, Barnett, 2003), which renders them more well suited to non-expert survey participants (Perez-Ortiz and Mantiuk, 2017). Finally, pairwise comparisons provide higher sensitivity and lower measurement error in comparison with ratings (Shah et al., 2016).

In this work, we propose a novel method aiming to further bridge the gap between human–computer interaction research, web design practices and advancements in machine learning. We introduce “Calista”, a deep learning-based system capable of automatically evaluating the visual aesthetics of a website from the perspective of the average user. We address the limitations of previous works in two ways. First, we exploit the strength of convolutional neural networks, which process visual information and automatically extract semantically meaningful features. Second, we introduce a novel strategy to crowdsource pairwise comparison data concerning the user preferences and incorporate them in the training pipeline. Our vision through this work is to establish the first steps towards providing designers and developers with a reliable tool to quantify their web designs’ appealingness to users and customers as well as to facilitate the flow of web design processes in terms of time and cost efficiency. Our main contributions are:

The rest of the paper is organized as follows. In Section 2, we present related work on computational website and photo aesthetics. In Section 3, we discuss the datasets and the methods that we implemented, while in Section 4, we present the results from the experiments. Finally, in Section 5, we conclude the paper and propose suggestions for future work. All the datasets, the source code for training the models and for the applications discussed in this paper are publicly available at https://github.com/calista-ai.