Holmes Processing: Distributed Large-Scale Analysis

Holmes Processing was born out of the need to rapidly process and analyse large volumes data in the computer security community. At its core, Holmes Processing is a catalyst for extracting useful information and generate meaningful intelligence. Furthermore, the robust distributed architecture allows the system to scale while also providing the flexibility needed to evolve.

Holmes Processing Architecture is based on 3 core pillars.

• Resilient: Failures should be gracefully handled and not affect other parts of the system.
• Scalable: The system should be easily able to scale vertically and horizontally.
• Flexible: Components should be interchangeable and new features should be easy to add.

• 1. Introduction
  • 1.1. What is Holmes?
  • 1.2. How does Holmes Work?
• 2. Architecture Design
  • 2.1. Data-Information-Knowledge-Wisdom (DIKW)
  • 2.2. Skald
    • 2.2.1. Planner
    • 2.2.2. Transport
    • 2.2.3. Services
      • 2.2.3.1. Information Extraction
      • 2.2.3.2. Knowledge Generation
      • 2.2.3.3. Functionality
• 3. Quick Start
• 4. Deployment Strategies
• 5. Components
  • 5.1. Holmes Totem
    • 5.1.1. What is Totem?
    • 5.1.2. Architecture Overview
    • 5.1.3. Installation
      • 5.1.3.1. Dependencies
      • 5.1.3.2. Configuration
      • 5.1.3.3. Running
    • 5.1.4. Executing Tasks
    • 5.1.5. Services
      • 5.1.5.1. Overview
      • 5.1.5.2. Analyser Library
      • 5.1.5.3. Structure
        • 5.1.5.3.1. Configuration file
          • 5.1.5.3.1.1. Service Configuration for Holmes Totem
          • 5.1.5.3.1.2. Centralised Service configuration of Holmes Totem
          • 5.1.5.3.1.3. Reading Configuration files for Services.
        • 5.1.5.3.2. Service Logic
          • 5.1.5.3.2.1. Communication
          • 5.1.5.3.2.2. API Endpoints
          • 5.1.5.3.2.3. Scala File
          • 5.1.5.3.2.4. Containerization
      • 5.1.5.4. Output
        • 5.1.5.4.1. HTTP Error Codes
          • 5.1.5.4.1.1. JSON
    • 5.1.6. Extending Holmes Totem
      • 5.1.6.1. Writing Services
        • 5.1.6.1.1. Writing a Service
          • 5.1.6.1.1.1. Overview
          • 5.1.6.1.1.2. Details
            • 5.1.6.1.1.2.1. Files To Add
              • Service Files
                • Dockerfile
                • LICENSE
                • README.md
                • acl.conf
                • service.conf.example
                • watchdog.scala
                • YourServiceREST.scala
                • Service-logic file
                  • Endpoint - ‘/’
                  • Endpoint ‘/analyze?obj=’
                    • Reading configuration file
                    • HTTP Error Codes
            • 5.1.6.1.1.2.2. Files to Edit
              • Config Files
                • totem.conf.example
                • docker-compose.yml.example
                • compose_download_conf.sh
              • Scala Files
                • driver.scala
        • 5.1.6.1.2. Example Service: Hello World
          • 5.1.6.1.2.1. In Golang
            • 5.1.6.1.2.1.1. Install Dependencies
            • 5.1.6.1.2.1.2. Dockerfile
            • 5.1.6.1.2.1.3. helloworld.go
          • 5.1.6.1.2.2. In Python
            • 5.1.6.1.2.2.1. Install Dependencies
            • 5.1.6.1.2.2.2. Dockerfile
            • 5.1.6.1.2.2.3. helloworld.py
  • 5.2. Holmes Totem Dynamic
    • 5.2.1. What is Totem-Dynamic?
    • 5.2.2. Architecture overview
    • 5.2.3. Installation
      • 5.2.3.1. Dependencies
      • 5.2.3.2. Configuration
    • 5.2.4. Executing Tasks
    • 5.2.5. Services
      • 5.2.5.1. Overview
      • 5.2.5.2. Structure
      • 5.2.5.3. Configuration
        • 5.2.5.3.1. Configuration file
        • 5.2.5.3.2. Port Selection
      • 5.2.5.4. API Endpoints
        • 5.2.5.4.1. Endpoint - /
        • 5.2.5.4.2. Endpoint - /analyze?obj=
        • 5.2.5.4.3. Endpoint - /feed?obj=
        • 5.2.5.4.4. Endpoint - /check?taskid=
        • 5.2.5.4.5. Endpoint - /result?taskid=
        • 5.2.5.4.6. Endpoint - /status
      • 5.2.5.5. Output
        • 5.2.5.5.1. HTTP Error Codes
        • 5.2.5.5.2. Results
  • 5.3. Holmes Storage
    • 5.3.1. Overview
    • 5.3.2. Installation
      • 5.3.2.1. Dependencies
        • 5.3.2.1.1. Supported Databases
        • 5.3.2.1.2. Object Stores
        • 5.3.2.1.3. Document Stores
      • 5.3.2.2. Configuration
        • 5.3.2.2.1. RiakCS
        • 5.3.2.2.2. Fake-S3
        • 5.3.2.2.3. Cassandra
      • 5.3.2.3. Installation
      • 5.3.2.4. Best Practices
        • 5.3.2.4.1. Nodetool
        • 5.3.2.4.2. Nodetool Repair
        • 5.3.2.4.3. Nodetool Compact
        • 5.3.2.4.4. Indexing
  • 5.4. Holmes Gateway
    • 5.4.1. Overview
      • 5.4.1.1. Highlights
    • 5.4.2. Installation
      • 5.4.2.1. Dependencies
      • 5.4.2.2. Configuration
      • 5.4.2.3. Installation
    • 5.4.3. Routing Services
    • 5.4.4. Uploading Samples
    • 5.4.5. Requesting Tasks
  • 5.5. Holmes Interrogation
  • 5.6. Holmes Presentation
  • 5.7. Transport
• 6. Toolbox and Helper Scripts
• 7. FAQ