Commit graph

13 commits

Author SHA1 Message Date
Christopher Talib cc87dab922 Cleaning up and and reorganizing the models package 2020-02-06 17:13:19 +01:00
Christopher Talib 0cd0faa6fa Working pastebin connector with parallel running with certstream
Parallel running of Certstream and Pastebin connectors in go routine
with wait groups to exit them properly.
2020-02-03 10:01:52 +01:00
Christopher Talib 67aecd65c1 Saving edges and node from CertStream traffic.
This work extracts fingerprints and domains from CertStream data
structure received through the stream. It builds nodes and edges and
saves them to the relevant files. It sends this data to Kafka but no
logic is implemented at the exit of the broker yet.
2020-01-29 10:03:52 +01:00
Christopher Talib de0c5bfe83 Setting up linking and and creating nodes already from the input source 2020-01-28 23:52:24 +01:00
Christopher Talib 7785372e3a Refactoring from utils to models
This work refactors saving and extracting function from the utils
package to the models package as it is a main component of the tool.
`utils` will take care of not related to models functions (such as
finding the files for example).

Also creating unique files for each type of source we are parsing.
2020-01-28 16:02:17 +01:00
Christopher Talib e4cc92e100 First work on Pastebin parser 2020-01-26 17:27:20 +01:00
Christopher Talib a0d2761c4e Adding RFC3339 times 2020-01-26 17:26:03 +01:00
Christopher Talib 7cbb7e9180 Cleaning and working write/read to kafka and saving objects 2020-01-23 14:36:24 +01:00
Christopher Talib 2548c19ca4 saving 2020-01-23 13:13:59 +01:00
Christopher Talib d33b293e7c Read/write from Kafka, simpler structure with plain data and not pointers 2020-01-22 15:01:07 +01:00
Christopher Talib e6cbfc6466 Changing IDs to stix compatible format 2020-01-16 12:06:03 +01:00
Christopher Talib 6eaae99668 Extracting structures from CertStream
This work builds an extractor for the data in the CertStream in order to
save it. It builds itself from the previous work, so extensions and
flags can be added to the structures. The work in `utils` is basically a
big extractor for the data taking advantage of the JSONq library.

Currently, there is not refactoring and the "chains" are not saved
because they need additionnal computation which will come in a later
commit.
2020-01-15 14:36:53 +01:00
Christopher Talib b7dce16c9e Adding unpacking in JSON format, and types for certstream format 2020-01-14 16:31:57 +01:00