Styx

What are we trying to solve

Styx is the passive sibling of Vader. When Vader allows users to get "on-demand" data from connectors, Styx will just ingest streams and streams of data. Styx will find things when they are happening on the contrary of retro-hunting, Styx find patterns in the current events and flags them on the spot. It's not retro-hunting, it's present-hunting or even future-hunting as we hope to find actors movement when they happen.

Prerequisites

Styx uses a couple of other services to run:

• Kafka for messaging (not implemented yet in the docker, but currently not necessary)
• Dgraph for graph representation of results
• Docker-compose to launch everything

For that purposes, there is a docker-compose.yml file that you can spin up with the following command when in the directory:

docker-compose up -d

Note: for some reasons, OpenVPN blocks the establishment of the docker compose, you can alternatively run Dgraph manually as such:

docker run --rm -it -p 8080:8080 -p 9080:9080 -p 8000:8000 -v ~/dgraph:/dgraph dgraph/standalone:v20.03.0

Install

go get -u gitlab.dcso.lolcat/LABS/styx
cd $GOPATH/src/gitlab.dcso.lolcat/LABS/styx
go build gitlab.dcso.lolcat/LABS/styx/cmd/styxd
docker-compose up -d # or the other docker command
./styxd

#  build the loader helper binary
go build gitlab.dcso.lolcat/LABS/styx/cmd/iocloader
# update the IOC list while the programm is already running
./iocloader

Note: if you have issues with the docker compose, make sure it runs on the same subnet. Check this for inspiration.

Example configuration:

Note: For Pastebin, you will have to authorise your IP address when you login through the web interface.

certstream:
    activated: true

pastebin:
    activated: true

shodan:
    activated: true
    key: "SHODAN_KEY"
    ports:
        - 80
        - 443

Dgraph Interface

You can connect to the Dgraph interface at this default address: http://localhost:8000. There you would be able to run GraphQL+ queries, here to query a node.

query {
    Node(func: eq(uid, 0x23)) {
        uid
            ndata
            modified
            nodeType
            id
    }
}

Or filter node by type, this example works for certstream nodes:

query {
    Node(func: eq(type, "certstream")) {
        uid
            created
            modified
            nodeType
            ndata
            certNode {
      uid
      fingerprint
      cn
      raw {
        uid
        id
      }
      chain {
        uid
        id
      }
      sourceName
      serialNumber
      basicConstrains
      notBefore
      notAfter
    }
  }
}

Example query for pastebin data:

query {
  Node(func: eq(type, "pastebin")) {
    uid
    created
    modified
    nodeType
    ndata
    pasteNode {
      id
      type
      created
      modified
      fullPaste {
        full
        meta {
          full_url
          size
          expire
          title
          syntax
          user
          scrape_url
          date
          key
        }
      }
    }
  }
}

Dgraph also supports full text search, so you can query things like:

query {
  Node(func: allofterms(full, "code")) {
    uid
    created
    modified
    nodeType
    full
  }
}

The following fields have can be used as index for searches:

• id
• type
• sourceName
• cn
• serialNumber
• hostnames
• organization
• full (full text of a pastbin)
• title
• user

By design, each node has a type field so you know which field you should query each time you query something.

Datastructure

Meta

Edges provide an existing relation between two nodes of different origin. They are part of Dgraph features.

Node --[Edge]-- Node

type Node struct {
	ID       string `json:"id"`
	NodeType     string `json:"nodeType"`
	Data     string `json:"data"` // For plain Node, the data is the ID of another typed node or a unique value like a domain or a host name.
	Created  string `json:"created"`
	Modified string `json:"modified"`
}

// Edge defines a relation between two nodes.
type Edge struct {
	ID        string `json:"id"`
	NodeOneID string `json:"nodeOneID"`
	NodeTwoID string `json:"nodeTwoID"`
	Timestamp string `json:"timestamp"`
	Source    string `json:"source"`
}

Certstream

Node -- CertNode -- CertStreamRaw

// CertStreamRaw is a wrapper around the stream function to unmarshall the
// data receive in a Go structure.
type CertStreamRaw struct {
	ID       string           `json:"id"`
	NodeType     string           `json:"nodeType"`
	Data     CertStreamStruct `json:"data"`
	Created  string           `json:"created"`
	Modified string           `json:"modified"`
}

// CertNode represents our custom struct of data extraction from CertStream.
type CertNode struct {
	ID               string     `json:"id"`
	Fingerprint      string     `json:"fingerprint"`
	NotBefore        string     `json:"notBefore"`
	NotAfter         string     `json:"notAfter"`
	CN               string     `json:"cn"`
	SourceName       string     `json:"sourceName"`
	SerialNumber     string     `json:"serialNumber"`
	BasicConstraints string     `json:"basicConstraints"`
	RawUUID          string     `json:"rawUUID"`
	Chain            []CertNode `json:"chainedTo"`
}

Pastebin

Node -- PasteNode -- FullPaste

// PasteNode is a node from PasteBin.
type PasteNode struct {
	ID       string    `json:"id"`
	NodeType string    `json:"nodeType"`
	Data     FullPaste `json:"data"`
	Created  string    `json:"create"`
	Modified string    `json:"modified"`
}

// FullPaste wrapes meta and information from Pastebin.
type FullPaste struct {
	Meta     PasteMeta `json:"meta"`
	Full     string    `json:"full"`
    NodeType string    `json:"nodeType"`
}

Shodan

Node -- ShodanNode -- Node(s) (hostnames and domains)

type ShodanNode struct {
	ID           string           `json:"id"`
	NodeType     string           `json:"nodeType"`
	Data         *shodan.HostData `json:"data"`
	Created      string           `json:"created"`
	Modified     string           `json:"modified"`
}

Balboa (not in Dgraph yet)

Balboa enrichment happens on domains and hostnames extracted from Certstream and Shodan streams and the node is created only if Balboa returns data.

Node -- ShodanNode -- Node (domain) -- BalboaNode

type BalboaNode struct {
	ID           string           `json:"id"`
	NodeType     string           `json:"nodeType"`
	Data         []balboa.Entries `json:"data"`
	Created      string           `json:"created"`
	Modified     string           `json:"modified"`
}