Commit graph

27 commits

Author SHA1 Message Date
Christopher Talib
c7a52c527a Refactoring saving single value into a helper, adding meta names to main nodes and edges 2020-02-07 15:27:14 +01:00
Christopher Talib
93f577cae9 Settting up more verbose logging with calling function and timestamp 2020-02-07 14:53:39 +01:00
Christopher Talib
2813ab0e96 Cleaning up and and reorganizing the models package 2020-02-06 17:16:47 +01:00
Christopher Talib
cc87dab922 Cleaning up and and reorganizing the models package 2020-02-06 17:13:19 +01:00
Christopher Talib
b5caae0974 Shodan connector and saving up and running 2020-02-06 17:01:37 +01:00
Christopher Talib
c2b59808be First implementation of shodan connector, but error on API key on streaming 2020-02-05 14:46:52 +01:00
Christopher Talib
efde1a4a54 Building ID on certnode 2020-02-04 10:58:56 +01:00
Christopher Talib
380e51229b Building nodes, certnode, and edges 2020-02-04 10:57:14 +01:00
Christopher Talib
0cd0faa6fa Working pastebin connector with parallel running with certstream
Parallel running of Certstream and Pastebin connectors in go routine
with wait groups to exit them properly.
2020-02-03 10:01:52 +01:00
Christopher Talib
b9a2e73e33 Working version capturing CertStream saving raw, nodes and edges.
There was some problem with my editor and lost some code, this work is
fixing that and adding new features. Current state of the code:
* capturing CertStream traffic
* saving raw certstream objects in a custom wrapper
* extracting fingerprints and domains from the certstream object
* saving fingerprints and domains nodes and edges between them.
* fingerprint is linked to the raw certstream object with an edge
* saving to files with customizable names (raw in code)

broker:
* kafka connection and test
* no sending of data to it for the moment
2020-01-29 12:47:01 +01:00
Christopher Talib
b9f0ac688c Removing extract.go file and refactoring the adhoc functions to files with name of the service 2020-01-29 11:05:05 +01:00
Christopher Talib
67aecd65c1 Saving edges and node from CertStream traffic.
This work extracts fingerprints and domains from CertStream data
structure received through the stream. It builds nodes and edges and
saves them to the relevant files. It sends this data to Kafka but no
logic is implemented at the exit of the broker yet.
2020-01-29 10:03:52 +01:00
Christopher Talib
de0c5bfe83 Setting up linking and and creating nodes already from the input source 2020-01-28 23:52:24 +01:00
Christopher Talib
7785372e3a Refactoring from utils to models
This work refactors saving and extracting function from the utils
package to the models package as it is a main component of the tool.
`utils` will take care of not related to models functions (such as
finding the files for example).

Also creating unique files for each type of source we are parsing.
2020-01-28 16:02:17 +01:00
Christopher Talib
01f3e3ca1e Only saving objects that have an ID 2020-01-23 15:01:08 +01:00
Christopher Talib
2548c19ca4 saving 2020-01-23 13:13:59 +01:00
Christopher Talib
64e1d3d7a4 Cleaning up non used functions 2020-01-22 16:33:07 +01:00
Christopher Talib
0d082374e9 Some temporal refactoring, saving domains and nodes 2020-01-22 15:45:25 +01:00
Christopher Talib
d33b293e7c Read/write from Kafka, simpler structure with plain data and not pointers 2020-01-22 15:01:07 +01:00
Christopher Talib
48cc976595 Adding a consummer for Kafka, WIP 2020-01-21 16:50:50 +01:00
Christopher Talib
06f42fc472 Moving SendToKafka function to broker package 2020-01-17 14:02:52 +01:00
Christopher Talib
e9c065bcc8 Setting up connector to Kafka and sending consummed and formatted Nodes to Kafka 2020-01-16 15:56:57 +01:00
Christopher Talib
1b25c25aa7 Adding some configuration, flagging inputs as CertStream 2020-01-15 16:46:13 +01:00
Christopher Talib
1081e0c728 Full saving of CertStream data
This work saves the entires CertStream into JSON.
2020-01-15 16:07:11 +01:00
Christopher Talib
b7dce16c9e Adding unpacking in JSON format, and types for certstream format 2020-01-14 16:31:57 +01:00
Christopher Talib
7d741f4cf3 Basic read/write implementation with output in JSON 2020-01-14 14:49:14 +01:00
Christopher Talib
b4e07b254a First commit with gitignore for the binary and initializing go modules 2020-01-14 09:43:12 +01:00