What’s Horde?

Horde is a scalable, cloud-ready, distributed build framework and content-addressable storage platform designed for Unreal Engine.

Goals of the project include:

• Remote execution of arbitrary workloads, including compilation of source code and building assets (like SN-DBS,
  FastBuild, IncrediBuild et al).
• Storage of bulk data for source assets.
• Caching of assets and build artifacts through Unreal Engine’s DDC system.
• Storage for final build artifacts (eg. packaged builds).
• Replication of data between studios on a planet scale.
• A CI/CD system for teams to test, build and publish their projects.

README.md (//Engine/Source/Programs/Horde/README.md

For the purposes of this guide, I’m going to be talking about Horde.Build, which is the CI/CD system mentioned above.

As this system is built explicitly for Unreal it works with projects built with it out of the box. It makes use of BuildGraph to distribute build steps across multiple agents, signficantly speeding up iteration time (for instance, have one agent compile the tools / editor, then have several other agents create build artifacts simultaniously, finally packaging the builds for distribution).

I am still discovering features that I had no idea existed in the system, the developer experience is significantly better than any other CI/CD system I’ve used in the past. For example the automatic bug reporting from build issues along with direct linking to swarm changelists all form within the dashboard.

Whilst you could get Jenkins to do something similar, it’d take quite a bit of work.

Disclaimer

I cannot stress this enough – This is NOT ready to be used in a production environment – if you decide to use it, you assume the risk and you will have to handle debugging on your own.

This guide will not going into securing your instance, it will have no authentication (though it is supported by Horde.Build). It will be setting passwords directly into JSON for ease of setup, however this is VERY unsecure.

I’m willing to offer assistance to the best of my abilities through Twitter (you can find me at https://twitter.com/DGoodayle), but if you intend to use this for a team of 500 people across 30 projects, I would advise waiting until the official release + documentation.

Why do you need continuous intergration in an Unreal Project?

Setting up a CI/CD (Continuous Integration/Continuous Deployment) system for Unreal projects offers several benefits.

First, it helps streamline the development process by automating build, test, and deployment tasks, this automation reduces the chances of human error and allows for more frequent and reliable releases.

Secondly, a CI/CD system ensures that code and content changes are regularly tested against automated tests which can help you catch regressions and track it back to an exact commit. This enabling of early detection of integration issues promotes a culture of quality assurance.

Finally, a well-implemented CI/CD system enables faster feedback loops, allowing developers to iterate and deliver new features more rapidly whilst quickly being able to playtest the build.

My use case

I am currently working on two projects of my own. Sometimes, I have individuals who join me in the collaboration, but they are not engineers. These collaborators will require a build distributed by UnrealGameSync (UGS). I am interested in having a build created every night for the platforms I am targeting. The build should include both Development and Shipping configurations. Additionally, I would like to run automated tests on the features and receive status updates through a slack channel.

I have two build agents, both Windows OS. My P4 Server is running on a Synology NAS inside my house, which I also intend to run the Horde Build System from.

---

Setting up Horde

Prerequisites

Docker

https://docs.docker.com/engine/install/

Head over to the Docker website and grab a build that suits your specific setup and requirements. In my own experience, I utilized Windows Docker Desktop, which proved to be quite effective for my purposes.

By using Docker, we can package the Horde executables in a neat and organized manner, ensuring seamless deployment onto a dedicated server. It is important to note that when constructing your image, Docker must be actively running to successfully complete the process.

Redis

To make it easier, you should have Redis running in a way that you can access it from wherever you use Horde. Since I’m using Horde from a Docker Container, I chose to run Redis and MongoDB in containers as well.

I used this image https://hub.docker.com/_/redis

MongoDB

Also a Docker image https://hub.docker.com/_/mongo, choose whatever flavour you want.

Slack

When I was setting up Horde, I found that it failed to run without a Slack token, so make sure you’ve got a slack instance setup and grab a bot token.

https://api.slack.com/start/quickstart

It must be a bot token – starting with xoxb – in order to work.

Yarn

This is needed for packaging up the frontend

https://classic.yarnpkg.com/lang/en/docs/install/#windows-stable

Test that it’s all working by running this in your terminal

yarn --version

JIRA + P4Swarm

These are optional – I don’t have them set up on my local Horde version, but they allow for deep linking to issues and automated bug creation when there are errors/warnings in a build.

Settings + Docker Files

You will either need to modify OR create the files listed at the path below each header. Make sure you modify them to fit your own needs.

Horde.Build – AppSettings.json

\Engine\Source\Programs\Horde\Horde.Build\appsettings.json

This is the main build server configuration file.

Horde.Agent – AppSettings.json

\Engine\Source\Programs\Horde\Horde.Agent\appsettings.json

The agent needs to be configured to talk to the Horde.Build server and setup its network mounts for sharing build artifacts between others agents.

Horde.Build – Globals.json

\Engine\Source\Programs\Horde\Horde.Build\Data\Config\globals.json

Horde.Build – UE.project.json

\Engine\Source\Programs\Horde\Horde.Build\Data\Config\ue.project.json

This file should be created per-project, in my case I have two project files.

UE-Main.stream.json

\Engine\Source\Programs\Horde\Horde.Build\Data\Config\ue-main.stream.json

The stream JSON file is where we set up the templates for build jobs (think of it like providing the arguments for a BuildGraph script), along with setting up schedules for builds, only building if the last built changelist isn’t the same as the latest.

I strongly recommend going through this file and modifying it to your own use cases, for this demo we’re going to make use of this great example from a talk that Regner Blok-Andersen did at Develop, we’re also going to make use of their BuildGraph file as it’s packed with features.

I’ve slightly modified it for this demo below:
https://gist.github.com/DannyGoodayle/fc68f3a0acecb982cfe692fb22b7564e

Horde Specific Docker File

\Engine\Source\Programs\Horde\Horde.Build\OnlyHordeDockerFile

For my specific setup, I wanted to quickly iterate, so I shrunk the existing DockerFile down to the following and called it OnlyHordeDockerFile saved next to the other docker file.

BuildHorde-DashboardAgentAndBackend.xml

\Engine\Source\Programs\Horde\BuildHorde-DashboardAgentAndBackend.xml

This will build the Horde Build System along with packaging up the Build Agent and the dashboard which your users will end up using. Just save this next to the Horde.sln file for now.

Code Modifications

Potential bug fix

During testing, I noticed that the P4 Service account does not explicitly log in until after it creates a workspace, which is a problem if you do not have P4 environment variables already set up. To bypass this, I modified the code to log in immediately after creating a connection.

	// Horde.Build/Perforce/PerforceService.cs:276
	IPerforceConnection connection = await PerforceConnection.CreateAsync(settings, _logger);
	if (settings.Password != null)
	{
		await connection.LoginAsync(settings.Password, cancellationToken);
	}

Build Graph Scripts

These are copied almost completely from Regner Blok-Andersen’s original gist and slightly modified to make sense in the context of this guide.

https://gist.github.com/DannyGoodayle/bd6e5bfc6b6ab34ebb9ec4bef33d84bf

These should be saved in your projects folder under Build\Graph, for example

Don’t forget to modify the Project-Config.xml to fit your project’s needs, setting the NetworkRootDirectory (which your build agents should have a mount set up for) and the project name, which must match your actual project name. This needs to be committed to your Perforce before you start any jobs in Horde.

---

Building

From the root of your project folder, run this command in a terminal.

.\Engine\Build\BatchFiles\RunUAT.bat BuildGraph -Script="Engine\Source\Programs\Horde\BuildHorde-DashboardAgentAndBackend.xml" -Target="HordeServer"