298 lines
6.6 KiB
Markdown
298 lines
6.6 KiB
Markdown
# Styx
|
|
|
|
## What are we trying to solve
|
|
|
|
Styx is the passive sibling of Vader. When Vader allows users to get "on-demand"
|
|
data from connectors, Styx will just ingest streams and streams of data. Styx
|
|
will find things when they are happening on the contrary of retro-hunting, Styx
|
|
find patterns in the current events and flags them on the spot. It's not
|
|
retro-hunting, it's present-hunting or even future-hunting as we hope to find
|
|
actors movement when they happen.
|
|
|
|
## Prerequisites
|
|
|
|
Styx uses a couple of other services to run:
|
|
|
|
* Kafka for messaging (not implemented yet in the docker, but currently not necessary)
|
|
* Dgraph for graph representation of results
|
|
* Docker-compose to launch everything
|
|
|
|
For that purposes, there is a `docker-compose.yml` file that you can spin up
|
|
with the following command when in the directory:
|
|
|
|
```sh
|
|
docker-compose up -d
|
|
```
|
|
|
|
*Note*: for some reasons, OpenVPN blocks the establishment of the docker
|
|
compose, you can alternatively run Dgraph manually as such:
|
|
|
|
```sh
|
|
docker run --rm -it -p 8080:8080 -p 9080:9080 -p 8000:8000 -v ~/dgraph:/dgraph dgraph/standalone:v20.03.0
|
|
```
|
|
|
|
## Install
|
|
|
|
```sh
|
|
go get -u gitlab.dcso.lolcat/LABS/styx
|
|
cd $GOPATH/src/gitlab.dcso.lolcat/LABS/styx
|
|
go build gitlab.dcso.lolcat/LABS/styx/cmd/styxd
|
|
docker-compose up -d # or the other docker command
|
|
./styxd
|
|
|
|
# build the loader helper binary
|
|
go build gitlab.dcso.lolcat/LABS/styx/cmd/iocloader
|
|
# update the IOC list while the programm is already running
|
|
./iocloader
|
|
```
|
|
|
|
*Note*: if you have issues with the docker compose, make sure it runs on the
|
|
same subnet. Check [this](https://serverfault.com/questions/916941/configuring-docker-to-not-use-the-172-17-0-0-range) for inspiration.
|
|
|
|
### Example configuration:
|
|
|
|
*Note*: For Pastebin, you will have to authorise your IP address when you login through the web interface.
|
|
|
|
```
|
|
certstream:
|
|
activated: true
|
|
|
|
pastebin:
|
|
activated: true
|
|
|
|
shodan:
|
|
activated: true
|
|
key: "SHODAN_KEY"
|
|
ports:
|
|
- 80
|
|
- 443
|
|
```
|
|
|
|
## Dgraph Interface
|
|
|
|
You can connect to the Dgraph interface at this default address: http://localhost:8000.
|
|
There you would be able to run GraphQL+ queries, here to query a node.
|
|
|
|
```graphql
|
|
query {
|
|
Node(func: eq(uid, 0x23)) {
|
|
uid
|
|
ndata
|
|
modified
|
|
nodeType
|
|
id
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
Or filter node by type, this example works for certstream nodes:
|
|
|
|
```graphql
|
|
query {
|
|
Node(func: eq(type, "certstream")) {
|
|
uid
|
|
created
|
|
modified
|
|
nodeType
|
|
ndata
|
|
certNode {
|
|
uid
|
|
fingerprint
|
|
cn
|
|
raw {
|
|
uid
|
|
id
|
|
}
|
|
chain {
|
|
uid
|
|
id
|
|
}
|
|
sourceName
|
|
serialNumber
|
|
basicConstrains
|
|
notBefore
|
|
notAfter
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Example query for pastebin data:
|
|
|
|
```graphql
|
|
query {
|
|
Node(func: eq(type, "pastebin")) {
|
|
uid
|
|
created
|
|
modified
|
|
nodeType
|
|
ndata
|
|
pasteNode {
|
|
id
|
|
type
|
|
created
|
|
modified
|
|
fullPaste {
|
|
full
|
|
meta {
|
|
full_url
|
|
size
|
|
expire
|
|
title
|
|
syntax
|
|
user
|
|
scrape_url
|
|
date
|
|
key
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
Dgraph also supports full text search, so you can query things like:
|
|
|
|
```graphql
|
|
query {
|
|
Node(func: allofterms(full, "code")) {
|
|
uid
|
|
created
|
|
modified
|
|
nodeType
|
|
full
|
|
}
|
|
}
|
|
```
|
|
|
|
The following fields have can be used as index for searches:
|
|
|
|
* id
|
|
* type
|
|
* sourceName
|
|
* cn
|
|
* serialNumber
|
|
* hostnames
|
|
* organization
|
|
* full (full text of a pastbin)
|
|
* title
|
|
* user
|
|
|
|
By design, each node has a `type` field so you know which field you should query
|
|
each time you query something.
|
|
|
|
## Datastructure
|
|
|
|
### Meta
|
|
|
|
Edges provide an existing relation between two nodes of different origin. They are part of Dgraph features.
|
|
|
|
Node --[Edge]-- Node
|
|
|
|
```go
|
|
type Node struct {
|
|
ID string `json:"id"`
|
|
NodeType string `json:"nodeType"`
|
|
Data string `json:"data"` // For plain Node, the data is the ID of another typed node or a unique value like a domain or a host name.
|
|
Created string `json:"created"`
|
|
Modified string `json:"modified"`
|
|
}
|
|
|
|
// Edge defines a relation between two nodes.
|
|
type Edge struct {
|
|
ID string `json:"id"`
|
|
NodeOneID string `json:"nodeOneID"`
|
|
NodeTwoID string `json:"nodeTwoID"`
|
|
Timestamp string `json:"timestamp"`
|
|
Source string `json:"source"`
|
|
}
|
|
|
|
```
|
|
|
|
### Certstream
|
|
|
|
Node -- CertNode -- CertStreamRaw
|
|
|
|
```go
|
|
|
|
// CertStreamRaw is a wrapper around the stream function to unmarshall the
|
|
// data receive in a Go structure.
|
|
type CertStreamRaw struct {
|
|
ID string `json:"id"`
|
|
NodeType string `json:"nodeType"`
|
|
Data CertStreamStruct `json:"data"`
|
|
Created string `json:"created"`
|
|
Modified string `json:"modified"`
|
|
}
|
|
|
|
// CertNode represents our custom struct of data extraction from CertStream.
|
|
type CertNode struct {
|
|
ID string `json:"id"`
|
|
Fingerprint string `json:"fingerprint"`
|
|
NotBefore string `json:"notBefore"`
|
|
NotAfter string `json:"notAfter"`
|
|
CN string `json:"cn"`
|
|
SourceName string `json:"sourceName"`
|
|
SerialNumber string `json:"serialNumber"`
|
|
BasicConstraints string `json:"basicConstraints"`
|
|
RawUUID string `json:"rawUUID"`
|
|
Chain []CertNode `json:"chainedTo"`
|
|
}
|
|
|
|
```
|
|
|
|
### Pastebin
|
|
|
|
Node -- PasteNode -- FullPaste
|
|
|
|
```go
|
|
// PasteNode is a node from PasteBin.
|
|
type PasteNode struct {
|
|
ID string `json:"id"`
|
|
NodeType string `json:"nodeType"`
|
|
Data FullPaste `json:"data"`
|
|
Created string `json:"create"`
|
|
Modified string `json:"modified"`
|
|
}
|
|
|
|
// FullPaste wrapes meta and information from Pastebin.
|
|
type FullPaste struct {
|
|
Meta PasteMeta `json:"meta"`
|
|
Full string `json:"full"`
|
|
NodeType string `json:"nodeType"`
|
|
}
|
|
```
|
|
|
|
### Shodan
|
|
|
|
Node -- ShodanNode -- Node(s) (hostnames and domains)
|
|
|
|
```go
|
|
type ShodanNode struct {
|
|
ID string `json:"id"`
|
|
NodeType string `json:"nodeType"`
|
|
Data *shodan.HostData `json:"data"`
|
|
Created string `json:"created"`
|
|
Modified string `json:"modified"`
|
|
}
|
|
```
|
|
|
|
### Balboa (not in Dgraph yet)
|
|
|
|
Balboa enrichment happens on domains and hostnames extracted from Certstream
|
|
and Shodan streams and the node is created only if Balboa returns data.
|
|
|
|
Node -- ShodanNode -- Node (domain) -- BalboaNode
|
|
|
|
```go
|
|
type BalboaNode struct {
|
|
ID string `json:"id"`
|
|
NodeType string `json:"nodeType"`
|
|
Data []balboa.Entries `json:"data"`
|
|
Created string `json:"created"`
|
|
Modified string `json:"modified"`
|
|
}
|
|
```
|