styx/README.md

# Styx

## What are we trying to solve

Styx is the passive sibling of Vader. When Vader allows users to get "on-demand"
data from connectors, Styx will just ingest streams and streams of data. Styx
will find things when they are happening on the contrary of retro-hunting, Styx
find patterns in the current events and flags them on the spot. It's not
retro-hunting, it's present-hunting or even future-hunting as we hope to find
actors movement when they happen.

## Prerequisites

Styx uses a couple of other services to run:

* Kafka for messaging (not implemented yet in the docker, but currently not necessary)
* Dgraph for graph representation of results
* Docker-compose to launch everything

For that purposes, there is a `docker-compose.yml` file that you can spin up
with the following command when in the directory:

```sh
docker-compose up -d
```

*Note*: for some reasons, OpenVPN blocks the establishment of the docker
compose, you can alternatively run Dgraph manually as such:

```sh
docker run --rm -it -p 8080:8080 -p 9080:9080 -p 8000:8000 -v ~/dgraph:/dgraph dgraph/standalone:v20.03.0
```

## Install

```sh
go get -u gitlab.dcso.lolcat/LABS/styx
cd $GOPATH/src/gitlab.dcso.lolcat/LABS/styx
go build gitlab.dcso.lolcat/LABS/styx/cmd/styxd
docker-compose up -d # or the other docker command
./styxd

#  build the loader helper binary
go build gitlab.dcso.lolcat/LABS/styx/cmd/iocloader
# update the IOC list while the programm is already running
./iocloader
```

*Note*: if you have issues with the docker compose, make sure it runs on the
same subnet. Check [this](https://serverfault.com/questions/916941/configuring-docker-to-not-use-the-172-17-0-0-range) for inspiration.

### Example configuration:

*Note*: For Pastebin, you will have to authorise your IP address when you login through the web interface.

```
certstream:
    activated: true

pastebin:
    activated: true

shodan:
    activated: true
    key: "SHODAN_KEY"
    ports:
        - 80
        - 443
```

## Dgraph Interface

You can connect to the Dgraph interface at this default address: http://localhost:8000.
There you would be able to run GraphQL+ queries, here to query a node.

```graphql
query {
    Node(func: eq(uid, 0x23)) {
        uid
            ndata
            modified
            nodeType
            id
    }
}

```

Or filter node by type, this example works for certstream nodes:

```graphql
query {
    Node(func: eq(type, "certstream")) {
        uid
            created
            modified
            nodeType
            ndata
            certNode {
      uid
      fingerprint
      cn
      raw {
        uid
        id
      }
      chain {
        uid
        id
      }
      sourceName
      serialNumber
      basicConstrains
      notBefore
      notAfter
    }
  }
}
```

Example query for pastebin data:

```graphql
query {
  Node(func: eq(type, "pastebin")) {
    uid
    created
    modified
    nodeType
    ndata
    pasteNode {
      id
      type
      created
      modified
      fullPaste {
        full
        meta {
          full_url
          size
          expire
          title
          syntax
          user
          scrape_url
          date
          key
        }
      }
    }
  }
}

```

Dgraph also supports full text search, so you can query things like:

```graphql
query {
  Node(func: allofterms(full, "code")) {
    uid
    created
    modified
    nodeType
    full
  }
}
```

The following fields have can be used as index for searches:

* id
* type
* sourceName
* cn
* serialNumber
* hostnames
* organization
* full (full text of a pastbin)
* title
* user

By design, each node has a `type` field so you know which field you should query
each time you query something.

## Datastructure

### Meta

Edges provide an existing relation between two nodes of different origin. They are part of Dgraph features.

Node --[Edge]-- Node

```go
type Node struct {
	ID       string `json:"id"`
	NodeType     string `json:"nodeType"`
	Data     string `json:"data"` // For plain Node, the data is the ID of another typed node or a unique value like a domain or a host name.
	Created  string `json:"created"`
	Modified string `json:"modified"`
}

// Edge defines a relation between two nodes.
type Edge struct {
	ID        string `json:"id"`
	NodeOneID string `json:"nodeOneID"`
	NodeTwoID string `json:"nodeTwoID"`
	Timestamp string `json:"timestamp"`
	Source    string `json:"source"`
}

```

### Certstream

Node -- CertNode -- CertStreamRaw

```go

// CertStreamRaw is a wrapper around the stream function to unmarshall the
// data receive in a Go structure.
type CertStreamRaw struct {
	ID       string           `json:"id"`
	NodeType     string           `json:"nodeType"`
	Data     CertStreamStruct `json:"data"`
	Created  string           `json:"created"`
	Modified string           `json:"modified"`
}

// CertNode represents our custom struct of data extraction from CertStream.
type CertNode struct {
	ID               string     `json:"id"`
	Fingerprint      string     `json:"fingerprint"`
	NotBefore        string     `json:"notBefore"`
	NotAfter         string     `json:"notAfter"`
	CN               string     `json:"cn"`
	SourceName       string     `json:"sourceName"`
	SerialNumber     string     `json:"serialNumber"`
	BasicConstraints string     `json:"basicConstraints"`
	RawUUID          string     `json:"rawUUID"`
	Chain            []CertNode `json:"chainedTo"`
}

```

### Pastebin

Node -- PasteNode -- FullPaste

```go
// PasteNode is a node from PasteBin.
type PasteNode struct {
	ID       string    `json:"id"`
	NodeType string    `json:"nodeType"`
	Data     FullPaste `json:"data"`
	Created  string    `json:"create"`
	Modified string    `json:"modified"`
}

// FullPaste wrapes meta and information from Pastebin.
type FullPaste struct {
	Meta     PasteMeta `json:"meta"`
	Full     string    `json:"full"`
    NodeType string    `json:"nodeType"`
}
```

### Shodan

Node -- ShodanNode -- Node(s) (hostnames and domains)

```go
type ShodanNode struct {
	ID           string           `json:"id"`
	NodeType     string           `json:"nodeType"`
	Data         *shodan.HostData `json:"data"`
	Created      string           `json:"created"`
	Modified     string           `json:"modified"`
}
```

### Balboa (not in Dgraph yet)

Balboa enrichment happens on domains and hostnames extracted from Certstream
and Shodan streams and the node is created only if Balboa returns data.

Node -- ShodanNode -- Node (domain) -- BalboaNode

```go
type BalboaNode struct {
	ID           string           `json:"id"`
	NodeType     string           `json:"nodeType"`
	Data         []balboa.Entries `json:"data"`
	Created      string           `json:"created"`
	Modified     string           `json:"modified"`
}
```