Commit graph

18 commits

Author SHA1 Message Date
Christopher Talib 84e4937f85 Major version update
This new work implements the server and the loader in two different
binaries allowing the code while updating the IOC list.

It updates also the documentation to reflect the new changes.
2020-08-24 17:20:07 +02:00
Christopher Talib b1ca4b3c5f Shodan in Dgraph, first part
Implementing first version for shodan node, missing yet some models, but
the overal approach works and can be queried in Ratel.
2020-05-18 16:09:04 +02:00
Christopher Talib e7421931c2 Couple of quickfixes to allow run a dry installion (deactivating ES and balboa) 2020-02-19 15:26:47 +01:00
Christopher Talib 02a014262a Adding Akamai filters for data recieved in Shodan + allow deactivating Balboa search 2020-02-14 11:36:47 +01:00
Christopher Talib 638216e8fa Implementing config variables in the application 2020-02-10 16:11:25 +01:00
Christopher Talib afd99c0a4d Adding balboa enrichment for domains and hostnames + documentation 2020-02-07 17:39:33 +01:00
Christopher Talib 67aecd65c1 Saving edges and node from CertStream traffic.
This work extracts fingerprints and domains from CertStream data
structure received through the stream. It builds nodes and edges and
saves them to the relevant files. It sends this data to Kafka but no
logic is implemented at the exit of the broker yet.
2020-01-29 10:03:52 +01:00
Christopher Talib 7785372e3a Refactoring from utils to models
This work refactors saving and extracting function from the utils
package to the models package as it is a main component of the tool.
`utils` will take care of not related to models functions (such as
finding the files for example).

Also creating unique files for each type of source we are parsing.
2020-01-28 16:02:17 +01:00
Christopher Talib 01f3e3ca1e Only saving objects that have an ID 2020-01-23 15:01:08 +01:00
Christopher Talib 2a04e01457 Remove error management as it break on empty values 2020-01-23 14:38:27 +01:00
Christopher Talib 7cbb7e9180 Cleaning and working write/read to kafka and saving objects 2020-01-23 14:36:24 +01:00
Christopher Talib 2548c19ca4 saving 2020-01-23 13:13:59 +01:00
Christopher Talib 64e1d3d7a4 Cleaning up non used functions 2020-01-22 16:33:07 +01:00
Christopher Talib 0d082374e9 Some temporal refactoring, saving domains and nodes 2020-01-22 15:45:25 +01:00
Christopher Talib d33b293e7c Read/write from Kafka, simpler structure with plain data and not pointers 2020-01-22 15:01:07 +01:00
Christopher Talib 48cc976595 Adding a consummer for Kafka, WIP 2020-01-21 16:50:50 +01:00
Christopher Talib 06f42fc472 Moving SendToKafka function to broker package 2020-01-17 14:02:52 +01:00
Christopher Talib e9c065bcc8 Setting up connector to Kafka and sending consummed and formatted Nodes to Kafka 2020-01-16 15:56:57 +01:00