Beaten by bots – training AI for first-person shooter games

Tough enemies tend to become easier by the end of a game, normally because you learn and they don’t. Robots image from www.shutterstock.com

When you first stare down that tough opponent in a videogame (think Big Daddies in Bioshock or Striders in Half-Life 2), it can seem impossible to beat the computer-controlled character you’re facing.

But come the end of the game and you’re often ploughing through computerised enemies with ease – the bots are no longer the insurmountable challenge they were when you first saw them.

What’s happened is you’ve gone through a process of learning and training as the game progressed, and your character might have gotten stronger too.

Meanwhile, computer-controlled enemies (or “bots) will often stay at a particular skill level throughout the game - even if it’s the game’s hardest difficulty - and they normally stick to a series of rules. That is, they don’t learn.

But what if they could?

FPS and bots

Research in artificial intelligence in videogames (AI) is becoming increasingly popular as we realise the advantages of using games as simulation test-beds.

First-person shooter (FPS) games are among the most popular type of game in today’s market and are known for their combative nature and fast-paced action. Players of these games will take on computer-controlled enemies across different maps, using different weapons, and collecting different items as they go.

The bots are non-playable characters with the ability to move around an environment, avoid obstacles, aim, shoot, change weapons, pick up items, sprint and crouch.

Creating bots with human-like behaviour is one way we can use videogames to research AI techniques. We can then transfer these insights to the fields of physical robotics and real-life simulations in things such as military training simulations.

Researching the creation of interesting behaviours, such as combat strategies, may also help the videogames industry develop more realistic and entertaining characters to play against.

So, what are we doing here?

In researching game AI, my PhD supervisor, Dr Marcus Gallagher and I decided to use a machine-learning algorithm to get bots to learn how to play an FPS. Machine-learning is a branch of AI that enables a machine to learn through experience.

We used reinforcement learning (RL), which allows a bot to learn a problem by interacting with its environment. The environment provides a reward or penalty based on how the bot is performing. These values are used to build a map telling the bot which action is good to perform in the current state of the environment. In our game, the bot receives a reward if it collects an item or kills an enemy. If the bot dies, it gets a penalty.

Our training tool

Screenshot of the interactive training tool software. Michelle McPartland

Our training tool lets the bots roam in a space and interact with each other. As the bots play out a game, users can reward or penalise the bot’s actions to modify their learning and enforce positive behaviours, such as picking up ammo or shooting other bots on sight.

The user can direct the bot by choosing any of seven guide actions. These actions are ranged attack, melee attack, wander, collect health, collect ammo, dodge and hide.

The experiment

We knew that the training system could create bots with different personality types such as a haphazard, beginner style bot and an aggressive, sharpshooter style bot. But we wanted to test the tool on commercial game designers to check its viability for industry use.

Five different designers used the tool to train a bot. We then threw the trained bots into a custom-built free-for-all FPS game to play against each other.

Visual representations of the behaviour mappings from the five user-trained bots. Red values are highly positive actions, black values are highly negative actions. Michelle McPartland

Training results showed that two of our five users (User 3 and User 5) trained bots with unique behaviours. The other three users (User 1, User 2 and User 4) trained similar types of bots:

  • User 1 created a bot that was good at collecting health items.
  • User 2 made a bot whose strength was in collecting ammo items.
  • User 3 created a bot notable for health-collection and combat.
  • User 4 made a bot that was okay at item collection and combat.
  • User 5 created the bot best at combat.

    Video of User 3’s bot versus User 5’s bot.

    Our research shows that interactive training is a viable option for creating FPS bots. Different types of bots were created using the same underlying code and, excitingly, the people training the bots could see their behaviour changing in real-time while they played the game.

    Interactive training can be used by designers during the development of the game to create different types of enemies. The underlying algorithm can also be used during the game to learn different strategies against the player.

    So, what lies ahead?

    We want to implement the tool in a commercial game or engine along with some new features of the tool based on the user testing feedback.

    We will add the ability to pre-define behaviours of a bot prior to training, for example, making the bot collect health when its health is low.

    Multiple controllers that run in parallel will be added. One controller will act as the high level decision maker - "who should I attack?” The other controller will control the movement - “where should I go?” This will allow more complex combat strategies to emerge.

    We believe the training tool has the potential to aid game designers in creating challenging and stimulating FPS bots – bots that can learn on the job and give gamers a real run for their money.

  • The Conversation is a non-profit + your donation is tax deductible. Help knowledge-based, ethical journalism today.