It is not easy to keep up with the amount of threat intelligence that is produced every day, whether it is open-source or commercial. The number of published reports, tweets, or just internal incident response reports is simply staggering, and it makes the job of threat researchers, security analysts, or any other professional monitoring the threat landscape, simply frustrating, repetitive, and just unattainable in the long-term.
To solve this issue, companies (but also communities of information security professionals) started developing Threat Intelligence Platforms (TIPs) to automatically collect, aggregate, and organize threat intelligence data ingested from multiple sources. While the number of products stabilized over the course of the past years, there are still several different platforms catering to different needs, and, obviously, budgets.
MISP, see Figure 1, is a threat intelligence platform co-financed by the European Union, which has steadily grown to become the open-source default choice despite several valid competitors, such as OpenCTI. MISP comes with a number of side projects like warning lists, i.e., list of false positives, crowd-sourced taxonomies called MISP galaxies, or a well-maintained Python library to interact with the system, which, together, create a thriving ecosystem.
Figure 1: Home page of the MISP project.
In this blog post, we detail the challenges faced when installing MISP in a production environment and present a new approach to deploying MISP in a repeatable, automatable manner using Docker.
Challenges When Installing MISP
In pure open-source spirit, there is no “right” way to install MISP, and while users can always download the entire code base from GitHub and use the provided installation script (which tries to support as many Linux distributions as possible), there are several other ways to install MISP: using a virtual machine, an ansible role, or even via RPM packages.
As they are all developed by users of the platform and not by the core developers of the platform, the level of support and features of these installers are not consistent and can vary (for example, some installation methods have not been updated for the past two years). Further, as MISP requires several components (and backends) to function correctly, installation is just the first step, and users need to take great care in choosing and deploying the right combination of external services. If MISP is also to be deployed in an unattended manner (as is often the case when cloud deployments are involved), it is easy to see why most of the development has lately been focused on creating Docker containers.
Unfortunately, while reviewing several external projects, we could not find an implementation that could satisfy our needs. We wanted our installation to take advantage of all the latest features (Background Workers, to name one) while providing means to automate the configuration of the users, servers, and organizations allowed to interact with the threat intelligence platform once installed. We also wanted the installation process to be as deterministic as possible, forcing database schema updates to take place at installation time rather than at the time of the first login. As the last requirement, we also wanted to be able to create new Docker images for specific commits rather than waiting every time for a new release; this would give us the added benefit of cherry-picking fixes without waiting for a full release cycle.
TAU MISP Docker Images
We are proud to present our latest development efforts in the shape of a MISP Docker image specifically tailored to support easy and repeatable deployments here. While the image is based on the work by Jason Kendall, there are many new features. To name a few:
- Background jobs are now managed by Supervisor rather than by the MISP underlying PHP libraries, making background workers more robust and easier to manage.
- Schema updates are run at installation time in a deterministic manner, without postponing migrations (and potential errors) to the first time a user logs in.
- The web server (Nginx) is now configured to expose (via the /feeds/ path) the local directory used to persist locally generated MISP feeds (more details on this in a future blog post).
- The Docker image can be built specifying the commit hash rather than a version number (useful when a sudden regression is introduced in the code base).
- Active MISP modules, Organizations, and Authentication Keys can all be automatically configured by modifying either the environment file or one of the provided entry point files.
In addition, when compared with the upstream version, our Docker Image also provides the following fixes:
- ARM support: migrate to MariaDB to better comply with MISP schema checks.
- ARM support: migrate to a multi-architecture exim4 image.
- Fix execution and improve support for CRON jobs: jobs are now always run and restarting the entry point does not break execution.
- Fix support for sync servers: it is possible now to configure several additional server synchronizations and run all synchronization tasks automatically.
- Fix Supervisor process control: processes are terminated correctly by Supervisor allowing correct restart and reload of MISP processes.
- Fix enforcement of permissions: permissions are now enforced correctly so all MISP components can write log files.
We are painfully aware that creating yet another fork may lead to further fragmentation, and this is the reason why, in the spirit of collaboration, we also created a number of pull requests to contribute back all main features and fixes.
Running MISP
Pulling or building the Docker image is a straightforward process. First and foremost, it is necessary to clone the repository and move to the newly created directory.
git clone https://github.com/MISP/misp-docker/ && cd docker-misp
There are a number of settings that the user can tune to better fit his/her own needs; some are included in the “docker-compose.yml” file (and thus require building the image rather than pulling it) but the majority can be found inside the environment file. While default values are more than enough to get started, it is still necessary to copy the environment file and remove its temporary prefix.
cp template.env .env
Once that is done, it is possible to just use the docker-compose command to pull the images:
docker-compose pull
or alternatively, build them:
docker-compose build
Once the command is completed, it is time to load all the required components (misp, misp-modules, redis, database, and mail containers) with a simple:
docker-compose up
The first time the containers are started, bootstrapping the system will likely require up to 10 minutes. Unfortunately, several processes are just computationally expensive: generating GPG keys, generating SSL certificates, and initializing the DB, to name a few. When this final step is eventually done, the user should be presented with a log similar to the one displayed in Figure 2.
If that is the case, Congratulations! You have a MISP instance fully up and running!
Figure 2: Logs to expect when the container has completed the initialization phase.
You can now use your favorite web browser and point it to https://localhost to finally log in. Use the default credentials (username: ‘[email protected]’ and password: ‘admin’), and if everything went as expected you will be presented with the events currently stored in your instance (see Figure 3). While the result might appear a tad underwhelming, this is just because MISP is currently waiting to ingest and process any threat intelligence feeds the user might decide to add to the threat intelligence platform. We will give some examples in a future blog post.
Figure 3: MISP up and running.
Conclusions
Threat Intelligence Platforms are becoming more and more a pivotal element of any SOC, Threat Research Team, or security professional aiming to keep up with an ever-evolving threat landscape. MISP emerged as the de-facto open-source TIP of choice due to its thriving ecosystem. In this blog post, we presented our Docker image specifically tailored to allow repeatable, deterministic, and unattended deployments in the cloud as well as in our data center. We also published our main features as pull requests, to avoid introducing unnecessary fragmentation.
