Multimodal game bot detection using user behavioral characteristics

Abstract

As the online service industry has continued to grow, illegal activities in the online world have drastically increased and become more diverse. Most illegal activities occur continuously because cyber assets, such as game items and cyber money in online games, can be monetized into real currency. The aim of this study is to detect game bots in a massively multiplayer online role playing game (MMORPG). We observed the behavioral characteristics of game bots and found that they execute repetitive tasks associated with gold farming and real money trading. We propose a game bot detection method based on user behavioral characteristics. The method of this paper was applied to real data provided by a major MMORPG company. Detection accuracy rate increased to 96.06 % on the banned account list.

Background

A game bot is an automated program that plays a given game on behalf of a human player. Game bots can earn much more game money and items than human users because the former can play without requiring a break. Game bots also disturb human users because they consistently consume game resources. For instance, game bots defeat all monsters quite rapidly and harvest items, such as farm produce and ore, before human users have an opportunity to harvest them. Accordingly, game bots cause complaints from human users and damage the reputation of the online game service provider. Furthermore, game bots can cause inflation in a game’s economy and shorten the game’s lifecycle, which defeats the purpose for which game companies develop such games (Lee et al. 2016).

Several studies for detecting game bots have been proposed in academia and industry. These studies can be classified into three categories: client-side, network-side, and server-side. Most game companies have adopted client-side detection methods that analyze game bot signatures as the primary measure against game bots. Client-side detection methods use the bot program’s name, process information, and memory status. This method is similar to antivirus programs that detect computer viruses (Mohaisen and Alrawi 2014). Client-side detection methods can be readily detoured by game bot developers, in addition to degrading the computer’s performance. For this reason, many countermeasures that are based on this approach, such as commercial anti-bot programs, are not currently preferred. Network-side detection methods, such as network traffic monitoring or network protocol change analysis, can cause network overload and lag in game play, a significant annoyance in the online gaming experience. To overcome these limitations of the client-side and network-side detection methods, many online game service providers employ server-side detection methods. Server-side detection methods are based on data mining techniques that analyze log data from game servers. Most game servers generate event logs whenever users perform actions such as hunting, harvesting, and chatting. Hence, these in-game logs facilitate data analysis as a possible method for detecting game bots. Online game companies analyze user behaviors or packets at the server-side, and then online game service providers can selectively block those game bot users that they want to ban without deploying additional programs on the client-side. For that, most online game service providers prefer server-side detection methods. In addition, some online game companies introduced big data analysis system approaches that make use of data-driven profiling and detection (Lee et al. 2016). Such approaches can analyze over 600 TB of logs generated by game servers and do not cause any side-effects, such as performance degradation or conflict with other programs.

The literature is rich of various works on the problem of game bot detection that we review in the following. Table 1 summarizes and compares various server-side detection schemes. We present key server-side detection methods classified into six analysis categories: action frequency, social activity, gold farming group, sequence, similarity, and moving path.

Table 1 Previous research on server-side detection

Action frequency analysis uses the fact that the frequencies of particular actions by game bots are much higher than that of human users. To this end, Chen and Hong (2007) studied the dynamics of certain actions performed by users. They showed that idle and active times in a game are representative of users and discriminative of users and bots. Thawonmas and Kashifuji (2010) utilized the information on action frequencies, types, and intervals in MMORPG log data. To detect game bots, Park et al. (2010) selected six game features, namely map changes, counter-turn, rest states, killing time, experience point, and stay in town. Chung et al. (2013) were concerned with various game play styles and classified them into four player types: killers, achievers, explorers, and socializers. Zhang et al. (2015) clarified user behaviors based on game playing time. While this approach provides high accuracy, it is limited in several ways. First, they only focus on observations of short time window, thus they are easy to evade. Second, some of such work focuses only on a limited feature space, thus the approach is prone to confusing bots with “hardcore” users (users who use the game for long times; who are increasingly becoming a phenomenon in the online gaming communities).

Social activity analysis uses the characteristics of the social network to differentiate between human users and game bots. Varvello and Voelker (2010) proposed a game bot detection method emphasizing on the social connections of players in a social graph. Our previous study chose chat logs that reflect user communication patterns and proposed a chatting pattern analysis framework (Kang et al. 2012). Oh et al. (2013) used the fact that game bots and human users tend to form respective social networks in contrasting ways and focused on the in-game mentoring network. Our other previous work found that the goal of game bot parties is different from that of human users parties, and proposed a party log-based detection method (Kang et al. 2013). This approach is however limited to detecting misbehavior in party play and cannot detect misbehavior in single play games.

Gold farming group analysis uses the virtual economy in online games and traces abnormal trade networks formed by gold farmers, merchants, bankers, and buyers. To characterize each player, Itsuki et al. (2010) used four types of statistics: total action count, activity time, total chat count, and the amount of virtual currency managed in a given period of time. Seo and Kim (2011) analyzed gold farming group connection patterns using routing and source location information. Kwon et al. (2013) investigated gold farming networks and detected the entire network structure of gold farming groups. This work, while distantly related, is not concerned with the detection of bots, but with understanding the unique roles each bot plays in the virtual underground ecosystem given a valid detection.

Sequence analysis uses iterated sequence datasets from login to logout. Ahmad et al. (2009) studied activity sequence features, defined as the number of times a given player engages in an activity, such as the number of monsters killed and the number of times the player was killed. Platzer (2011) used the combat sequence each avatar produces. Lee et al. (2015) examined the full action sequence of users on big data analysis platform. While such technique has been shown to work in the past, such feature lacks context, and might be easily manipulated by bot settings.

Similarity analysis uses the fact that game bots have a strong regular pattern because they play to earn in-game money. Kwon and Kim (2011) derived vectors using the frequency of each event and calculated the vector’s cosine similarity with a unit vector. Game bots repeatedly do the same series of actions, therefore their action sequences have high self-similarity. Lee et al. (2016) employed self-similarity measures to detect game bots. They proposed the self-similarity measure and tested it in three major MMORPGs (“Lineage”, “Aion” and “Blade&Soul”). Their scheme requires a lot of data of certain behavior for establishing self-similarity.

Moving path analysis uses the fact that game bots have pre-scheduled moving paths, whereas human users have various moving patterns. Thawonmas et al. (2007) provided a method for detecting landmarks from user traces using the weighted entropy of the distribution of visiting users in a game map. They presented user clusters based on transition probabilities. To identify game bots and human users, van Kesteren et al. (2009) took advantage of the difference in their movement patterns. Mitterhofer et al. (2009) detected the players controlled by a script with repeated movement patterns. Pao et al. (2010) used the entropy values of a user’s trace and a series of location coordinates. They employed a Markov chain model to describe the behavior of the target trajectory. Pao et al. (2012) applied their method to various types of trajectories, including handwriting, mouse, and game traces, in addition to the traces of animal movement. However, their feature also can be evaded and noised by adaptive bots that integrate human-like moving behavior.

Contribution.

To this end, we collaborated with NCSOFT, Inc., one of the largest MMORPG service companies in South Korea, in order to analyze long-term user activity logs and understand discriminative features for high fidelity bot detection. In this paper, we propose a game bot detection framework. Our framework utilizes multimodal users’ behavioral characteristic analysis and feature extraction to improve the accuracy of game bot detection. We adopted some features discovered in the prior literature in confirmed in our analysis, as well as some new features discovered in this study. We combine those features in a single framework to achieve better accuracy and enable robust detection. An additional contribution of this work is also the exploration of characteristics of the misclassified users and bots, highlighting plausible explanations that are in line with users and bots features, as well as the game operations.

Methods

Before elaborating on the framework and workflow of our method, we first highlight the dataset and ethnical guidelines used for obtaining and analyzing it.

Dataset.

To perform this study, we rely on a real-world dataset obtained from the operation of Aion, a popular game. Our Aion dataset contains all in-game action logs for 88 days, between April 9th and July 5th of 2010. During this period, there were 49,739 characters that played more than 3 h. Among these players, 7702 characters were game bots, identified and labeled by the game company. The banned list was provided by the game company to serve as the ground truth, and each banned user has been vetted and verified by human labor and active monitoring.

Ethnical and privacy considerations.

In order to perform this study we follow best practices in ensuring users privacy and complying with ethical guidelines. First, the privacy of users in the data is ensured by anonymizing all personal identifiable information. Furthermore, consent of users is taken into account by ensuring that data analysis is within the scope of end user license agreement (EULA): upon joining Aion, users grant NCSOFT, Inc. the full permission to use and share user data for analysis purpose with parties of NCSOFT’s choosing. One of such parties was our research group, and for research purpose only.

Framework and workflow

Our proposed framework for game bot detection is shown in Fig. 1. We posed the problem of identifying game bots as a binary classification problem. At a high-level, our method starts with a data collection phase, followed by a data exploration phase (including feature extraction), a machine learning phase, and a validation phase. In the following we highlight each of those phases.

Data collection.

In the data collection phase, we gathered a dataset that combines in-game logs and chat contents.

Data exploration.

We then performed data exploration in order to comprehend the characteristics of the dataset using data preprocessing, feature extraction, feature representation, exploration, and selection for best discriminating between bots and normal users. In the feature representation procedure, we followed standard methods for unifying data and reducing its dimensionality. For example, we quantized each network measure into three clusters with low, medium, and high values using the k-means clustering algorithm. In the feature exploration phase, we selected the components of the data vectors and pre-pocessed them. For example, we determined seven activities as social interactions and quantified the diversity of social interactions by the Shannon diversity entropy. In the feature selection phase, we selected significant features with the best-first search, greedy-stepwise search, and information gain ranking filter to avoid overfitting and reduce the features (thus improving the performance).

Machine learning.

In the machine learning phase, we choose algorithms (e.g., decision tree, random forest, logistic regression, and naïve Bayes) and parameters (e.g., k-fold cross-validation parameters, specific algorithm parameters, etc.), and feed the data collected using the selected features in their corresponding representation. We further build models (using the data fed) and establish baselines by computing various performance metrics.

Evaluation.

In the evaluation phase, we summarize the performance of each classifier with the banned account list provided by the game company as a ground truth, by providing various performance measures, such as the accuracy, precision, recall, and F-measure.

Fig. 1

Game bot detection framework based on user behavioral characteristics

Used features and their gap.

As indicated in Table 2, we classified the features we used in our work into personal and social features. Given that the aim of game bots is to earn unfair profits, there is a gap between the values of the personal features of game bots and those of human users. The personal features can be also categorized into player information and actions. The player information features include login frequency, play time, game money, and number of IP address. The player action features contain sitting (an action taken by players to recover their health), earning experience points, obtaining items, earning game money, earning player kill (PK) points, harvesting items, resurrecting, restoring experience points, being killed by a non-player and/or player character (NPC/PC), and using portals. The frequency and ratio of these actions reflects the behavioral characteristics of game bots and human users. For example, game bots sit more frequently than human users to recover health and mana points. Moreover, a player can acquire PK points by defeating players of opposing factions. PK points can be used to purchase various items from vendors. PK points are also used to determine a player’s rank within the game world. In Aion, the more PK points a player has, the higher is the player’s rank. The high ranking player can feel a sense of accomplishment. On the other hand, it is seen that game bots are not interested in rank.

Table 2 Personal and social features

In addition, there is gap between the values of the social features of game bots and those of human users because game bots do not attempt to social as humans. The social features can be categorized into group activities, social interaction diversity, and network measures. The features of group activities include the average duration of party play and number of guild activities. Party play is a group play formed by two or more players in order to undertake quests or missions together. The goals of party play commonly are to complete difficult quests by collaboration and enjoy socialization. Interestingly, some game bots perform party play, but the goal of party play of the game bots is different from that of human users. Their aim is to acquire game money and items faster and more efficiently. Hence, there are the behavioral differences between game bots and human users. The social interaction diversity feature indicates the entropy of party play, friendship, trade, whisper, mail, shop, and guild actions. Game bots concentrate only on particular actions, whereas human users execute multiple tasks as needed to thrive in the online game world. The player’s social interaction network can be represented as a graph with characters as the nodes and interactions between them as the edges. An edge between two nodes (players) in this graph may, for example, highlight the transfer of an item between the two nodes. The features of network measures include the degree, betweenness, closeness, eigenvector centrality, eccentricity, authority, hub, PageRank, and clustering coefficient. The definitions of the network measures are listed in Table 3.

Table 3 Definition of network measures

Results and discussion

In this section we review more concretely the behavioral characteristics of bots and humans based on the various features utilized, and using the aforementioned dataset. We then propose our bot detection mechanism based on discriminative features and by elaborating on details of the high level workflow in the previous section, including the performance evaluation.

Behavioral characteristics

Player information

We compared the distribution of player information features in order to identify the difference between the behavioral characteristics of game bots and human users more concretely. Figure 2 shows how intensively game bots play games. Game bots often connect to the game and spend much longer time playing it than human users. Game bots can play a given game for 24 consecutive hours, whereas human users hardly connect to the game during working hours. Game bots invest significant time in a game until they are blocked. Figure 2c shows the cumulative distribution of the maximum number of items harvested by users per day. It is almost impossible for human users to harvest more than 1000 items per day. Since this is repetitive and hard work, human users are easily exhausted. Nevertheless, 60 % of game bots harvest more than 5000 items a day. This is an obvious characteristic for identifying game bots that we include in our feature set.

Fig. 2

Player information. a Cumulative distribution of the user login frequency. b Cumulative distribution of user play time. c Cumulative distribution of the number of items harvested by users

Fig. 3

Comparison of activity ratios between game bots and human users. The ratios of “earning experience points” and “obtaining items” of game bots are much higher than those of human users

Player actions

We examined the frequency and ratio of player actions to determine the unique characteristics of game bots. Figure 3 presents the ratios of the activities of both game bots and human users. The points in red indicate game bots, and those in blue indicate human users. The ratio of “earning game money” of game bots is nearly similar to that of human users. Remarkably, the ratios of “earning experience points” and “obtaining items” of game bots are much higher than those of human users. The cumulative ratio of “earning experience points”, “obtaining items”, and “earning game money” of game bots is close to 0.5, whereas that of human users is only 0.33. This implies that game bots concentrate heavily on profit-related activities, and human users enjoy various activities. In contrast, the ratio of “earning PK points” of human users is as much as three times that of game bots. This reflects the fact that game bots are not interested in rankings.

Group activities

Figure 4 shows the distribution of the average party play time of game bots and human users. To acquire game money and items, some game bots form a party with other game bots. They can help each other not to be killed by monsters during party play. Consequently, their party play patterns are unusual. A total of 80 % of game bots last longer than 4 h 10 min, whereas 80 % of human users last less than 2 h 20 min. Since difficult missions can normally be completed within 2 h through collaboration, human users do not maintain party play as long as game bots.

Fig. 4

Cumulative distribution of user average party play time. A total of 80 % of game bots last longer than 4 h 10 min, whereas 80 % of human users last less than 2  h 20 min

Social interaction diversity

Figure 5 shows the cumulative distribution of the entropy of social interactions. First, we determined seven activities as social interactions: party, friendship, trade, whisper, mail, shop, and guild. We quantified the diversity of social interactions by calculating the Shannon diversity entropy defined by:

$$\begin{aligned} H' = -\sum _{i=1}^{n}{p_i \ln {p_i}} \end{aligned}$$

(1)

n, number of social interaction types. $p_i$, relative proportion of the ith social interaction type.

The entropy of the social interactions of a player indicates the various activities performed by the player. Figure 5 represents the fact that human users enjoy diverse activities, whereas game bots do not. We notice that game bots are interested in other activities.

Fig. 5

Cumulative distribution of user social interaction diversity. The average entropy of game bot social interaction is much lower than that of human users (0.4299 and 0.8352, respectively)

Network measures

In Table 4, we present the basic directed characteristics of each network of the game bot and human groups from Aion (Son et al. 2012). First, we see that the average degree of the human group is approximately 18 times larger compared with the game bot group in the party network. The reason is that human users form a party with many and unspecified users, whereas game bots play with several specific other game bots. The average degree of the friendship network of the human group is larger by a factor of approximately four compared with the game bot group. This fact indicates that the friendship of game bots is utterly different from that of human users. Game bot friends simply mean other game bots with which to play. The fact that the average degree of the human group is 2.5 times larger than the game bot group is observed in the case of the trade network. However, the average clustering coefficient of the game bot group is approximately five times larger compared with the human group. We assume that game bots have roles (Kwon et al. 2013; Ahmad et al. 2009). For instance, some game bots are responsible for gold farming, while other game bots gather game money and items from gold farmers or sell them for real money (Woo et al. 2011).

Interestingly, in the case of the mail network of the game bots, we discovered nine spammers during the observation period. The number of mail pieces sent by the spammers is 1000 times per person on average. We observed the behavioral characteristics of the spammers in more detail. Hence, we found that they only send mail and stay online for a short period of time in the online game world.

We also observed the existence of five collectors who received items attached to mail from many other game bots. These collectors received items over 6000 times during the observation period. This shows that there are several gold farming groups. In the case of the shop network, we can see the smallest number of nodes of both groups. Players are immobile in the merchant mode, and thus cannot engage in any action that requires movement, such as hunting monsters, harvesting items, etc. Consequently, game bots do not focus on the merchant mode because it can be a waste of time for them.

Table 4 Basic network characteristics of six interaction networks

The triad census

The relative prevalence of each of the 13 triad network motifs given in Fig. 6a indicates the interaction pattern in the networks in more detail (Jeong et al. 2015). For our Aion networks, we show the interaction pattern in Fig. 6b in terms of both the fractions of each motif type and the Z-scores assessed against the null model [Eq. (2), also see Tables 7, 8]. This score is defined as follows:

$$\begin{aligned} Z_i = {N_{i}^{\mathrm {real}} - N_{i}^{\mathrm {random}} \over \sigma _{i}^{\mathrm {random}}}, \end{aligned}$$

(2)

where $N_{i}^{\mathrm {real}}$ is the number of motif i found observed in the network, $N_{i}^{\mathrm {random}}$ is the expected number in the randomized network, and $\sigma _{i}^{\mathrm {random}}$ is the standard deviation of its expected number in the randomized network.

Fig. 6

Network motif analysis of node triplets reveals detailed interaction patterns in directed networks of game bots and human users. a The 13 possible motifs composed of three nodes in a directed network. b The fractions of each motif type in each of the six networks. Those motifs that account for fewer than 18 % of all motifs are not shown. Friendship, whisper, mail, and shop of the game bot group, and friendship and shop of the human group show one dominant motif each, consistent with the high or low reciprocity found in the networks. c A closer look at the (normalized) Z-score triad census of party and trade networks where no dominant motif is evident; the Z-score method is employed to determine significantly over and underrepresented triangular motifs

Findings.

Interestingly, the friendship, whisper, mail, and shop networks of the game bot group, and the friendship and shop networks of the human group, show one predominant motif type. For instance, in the friendship network, type 7 accounts for more than 90 % of the node triplet relationships, which can be attributed to the highly reciprocal nature of the interactions. The opposite reasoning can be applied to shop: low reciprocity reflects the existence of big merchants. Moreover, in the whisper and mail network of the game bot group, type 1 accounts for more than 80 % of the node triplet relationships. This reflects the fact that some game bots send information about the location coordinates of monsters to other game bots in the case of the whisper network.

Some game bots send several mail pieces in the case of the mail network. Comparing the prevalence of motifs against the null models allows us to detect signals discounted by random expectation, and this is done via the Z-scores [Eq.(2)]. This is particularly necessary and illuminating in the case of the other two networks (party and trade) because, by considering the null models, we can see that although multiple motifs can be similarly abundant (Fig. 6b), some can be significantly over or underrepresented, as we can see in Fig. 6. In the case of the human group, the overrepresented motif type 5 [with ${\tilde{Z}}> 0.4$, the normalized version ${\tilde{Z}} \equiv Z_{i} \sqrt{\displaystyle {\Sigma }_{i}({Z_{i}^{2}})}$] is indeed closed triangles, consistent with the relatively high clustering tendencies in the party network. In the case of the game bot group, the overrepresented motif type 13 shows the fact that there is a large gap between the number of motifs observed in the network and the expected number of motifs in the randomized network. This reflects the fact that game bots have their own group for helping and trading with each other.

Network overlap

To determine how pairwise networks are correlated, we studied the network similarities between the game bot and human groups. For example, two networks can show similar clustering values, and yet this does not guarantee at all that nodes connected in one network are connected in another, or that the nodes show similar levels of activity. Thus, we consider here two measures of network overlap. The first is the link overlap between two networks quantified by the Jaccard coefficient. The second is the degree overlap given by the Pearson correlation coefficient between node degrees in network pairs. The results of link and degree overlap for ten network pairs of the game bot and human groups are given in Fig. 7. By examining the link overlap (Fig. 7a), we found that the game bot group has higher Jaccard coefficient in the party-friendship and party-trade pairwise networks. This is a result of the fact that the main activities of game bots are party play and trading items. The friend list offers convenience to a game bot when it wants to form a party group. Game bots gather game money and items collected through party play in an account by trading. Then the account that collects the cyber assets changes the game money and items to real money.

Node degree overlap (Fig. 7b) is another way of seeing the connection between interactions: here, for instance, the party-trade pairwise networks of the human group show a positive Pearson correlation coefficient value that exceeds 0.7, which can be understood by the fact that a party activity, being above all the favorite way of engaging in battles or hunting, often concludes with members trading booties. In contrast, the Pearson correlation coefficient values of the game bot group are extremely low because game bots maintain relationships with a small number of other game bots.

Fig. 7

Pairwise network overlap indicates similarity or dependence between interactions. a Link overlap. The game bot group has higher Jaccard coefficient in the party-friendship and party-trade pairwise networks. b Node overlap that quantifies the node degree overlap between different networks. The human group has high degree overlap between 0.4 and 0.7, whereas the game bot group has degree overlap lower than 0.2 in all networks

Game bot detection

We took a discriminative approach to learning the distinction between game bots and human users in order to detect the game bot and build automatic classifiers that can automatically recognize the distinction. We divided the dataset into training and test sets, built the classifiers through the training dataset, and evaluated the trained classifiers through the test dataset. In addition, we performed tenfold cross-validation to avoid classifiers from being overfitted to the test data. Cross-validation generalizes the classifier trained by the test data to the validation data. Tenfold cross-validation divides the dataset into ten groups, trains the learning model with randomly selected nine groups, and verifies the classifiers from the model with one group. These training and validation processes are repeated ten times.

Feature selection

We compared the bot detection results from our model with the banned account list provided by the game company in order to evaluate the proposed framework upon running our detection method of selected features. We conducted feature selection with the best first, greedy stepwise, and information gain ranking filter algorithms in advance in order to improve the selection process. Feature_Set1 consists of all the features (114) mentioned in “Methods” section. Feature_Set2 is composed of the top 62 features extracted by the information gain ranking filter algorithm. Feature_Set3 is comprised of the six features selected by the best first and greedy stepwise algorithms. Figure 8 shows the classification results using these three feature sets. Feature_Set3 presents lower performance than Feature_Set1 and Feature_Set2. In comparison, Feature_Set2 has almost the same performance as Feature_Set1, although the number of Feature_Set2 is barely half that of Feature_Set1. Thus, we finally selected Feature_Set2 for game bot detection.

Fig. 8

Performance comparison of feature sets. Feature_Set2 has as high performance as Feature_Set1

Classification and evaluation

The results of the users’ behavioral pattern analysis for game bot detection are listed in Table 5. The four classifiers used as training algorithms—decision tree, random forest, logistic regression, and naïve Bayes—are tested on Feature_Set2. The performances are listed in terms of overall accuracy, precision, recall, and F-measure. Random forest outperforms the other models. Its overall accuracy, precision value, recall value, and F-measure with emphasis on precision ($\alpha = 0.9$) are 0.961, 0.956, 0.742, and 0.929, respectively. As can be seen, the recall value is slightly low. We analyzed the characteristics of true positive, false positive, false negative, and true negative cases to inquire into the cause of this phenomenon.

The random forest technique is a well-known ensemble learning method for classification and it constructs multiple decision trees in its training phase to overcome the decision tree’s overfitting problem. The random forest learning is also robust when training with imbalanced data set. It is also useful when training large data with a lot of features. Our data set consists of 85 % of human players and 15 % of game bots—so it is considered as an imbalanced and large data set—and random forests perform well in that context given that the context meets the settings in which random forests are to perform ideally.

Naïve Bayes showed the lowest performance among four classifiers, and that is probably because of its nature as a generative model that requires independence of features. Although we performed feature selection, still there are correlations between selected features used in our experiment. For example, obtaining_items_count, earning_exp_points_count, harvesting_items_max_count, party_eccentricity, play_time and obtaining_items_ratio are less significant features. However, those features are also naturally correlated and they cannot be easily separated because they are all related to essential game behaviors (hunting, harvesting, collaboration, etc., which are all related to high level process). Indeed, such hypothesis is confirmed by removing those features, bringing the performance of the naïve Bayes on par with other algorithms.

Table 5 Precision, recall, and F-measure (0.9) ratios for each classifier

Figure 9 shows the relative similarities and differences of the classification evaluation outcomes (classes): true positive, false positive, false negative, and true negative. To obtain the relative similarity, we normalize all classes by the lowest class value, thus comparing outcomes relatively. Such normalization would bring the lowest class in the evaluation to one. For each class other than the lowest, we calculated the ratio by dividing the values of the other classes by the value of the lowest class. The pattern of the relative similarity is consistent with most features and classes, with the exception of the “mail_between_centrality” and “mail_outdegree” features. It is highly probable that game bots had not been detected yet in the case of false negatives. This also implies that human users temporarily employed a game bot in the case of false positives. To confirm this observation, we analyzed the case of false positives weekly and finally found harvesting and party play game bots.

Fig. 9

Comparison of four cases: true-positive, false-positive, false-negative, and true-negative. The ratios of false-positive cases are exceedingly similar to those of true-positive cases. The ratios of false-negative cases are similar to those of true-negative cases

Conclusions

We proposed a multimodal framework for detecting game bots in order to reduce damage to online game service providers and legitimate users. We observed the behavioral characteristics of game bots and found several unique and discriminative characteristics. We found that game bots execute repetitive tasks associated with earning unfair profits, they do not enjoy socializing with other players, are connected among themselves and exchange cyber assets with each other. Interestingly, some game bots use the mail function to collect cyber assets. We utilized those observations to build discriminative features. We evaluated the performance of the proposed framework based on highly accurate ground truth—resulting from the banning of bots by the game company. The results showed that the framework can achieve detection accuracy of 0.961. Nonetheless, we should consider that the banned list does not include every game bot.

The game company imposes a penalty point on an account that performs abnormal activities, and eventually blocks the account when its cumulative penalty score is quite high. Some game bots can evade the penalty scoring system of the game companies. Hence, the actions of a player are more important than whether the player is banned or not, and we concede that a player is a game bot when the player’s actions are abnormal. We focused on those user behavioral patterns that reflect user status to interpret the false positive cases, and hypothesize that they are game bots not yet blocked, and false negative cases are human users occasionally employing a game bot. Although different from those in the banned list, they behave in the same pattern. We believe that our detection model is more robust by relying on multiple classes of features, and its analyses promise further interesting directions in understanding game bot and their detection.

Appendix

See Tables 6, 7 and 8.

Table 6 Network diameters from 100 randomized network versions

Table 7 Complete frequency distribution for triangular motifs

Table 8 Network diameters from 100 randomized network versions