Commit graph

10 commits

Author SHA1 Message Date
Christopher Talib
e458a327d6 Moving SaveDomains to parser package 2020-01-28 16:05:36 +01:00
Christopher Talib
7785372e3a Refactoring from utils to models
This work refactors saving and extracting function from the utils
package to the models package as it is a main component of the tool.
`utils` will take care of not related to models functions (such as
finding the files for example).

Also creating unique files for each type of source we are parsing.
2020-01-28 16:02:17 +01:00
Christopher Talib
e4cc92e100 First work on Pastebin parser 2020-01-26 17:27:20 +01:00
Christopher Talib
2548c19ca4 saving 2020-01-23 13:13:59 +01:00
Christopher Talib
8ac9d5786b Refactoring and setting clearer filenames in utils/ 2020-01-22 15:49:52 +01:00
Christopher Talib
0d082374e9 Some temporal refactoring, saving domains and nodes 2020-01-22 15:45:25 +01:00
Christopher Talib
d33b293e7c Read/write from Kafka, simpler structure with plain data and not pointers 2020-01-22 15:01:07 +01:00
Christopher Talib
1081e0c728 Full saving of CertStream data
This work saves the entires CertStream into JSON.
2020-01-15 16:07:11 +01:00
Christopher Talib
6eaae99668 Extracting structures from CertStream
This work builds an extractor for the data in the CertStream in order to
save it. It builds itself from the previous work, so extensions and
flags can be added to the structures. The work in `utils` is basically a
big extractor for the data taking advantage of the JSONq library.

Currently, there is not refactoring and the "chains" are not saved
because they need additionnal computation which will come in a later
commit.
2020-01-15 14:36:53 +01:00
Christopher Talib
b7dce16c9e Adding unpacking in JSON format, and types for certstream format 2020-01-14 16:31:57 +01:00