Christopher Talib
5b1bfbc195
Filters for IP and shodan
2020-02-14 20:46:09 +01:00
Christopher Talib
1fc11e7a4e
Adding all CIDR files in filters/data
2020-02-14 12:01:50 +01:00
Christopher Talib
02a014262a
Adding Akamai filters for data recieved in Shodan + allow deactivating Balboa search
2020-02-14 11:36:47 +01:00
Christopher Talib
e963633c89
Allowing the possiblity to activate or deactivate modules
2020-02-14 11:30:59 +01:00
Christopher Talib
bd3f108d12
Shodan ports in config
2020-02-12 16:54:14 +01:00
Christopher Talib
638216e8fa
Implementing config variables in the application
2020-02-10 16:11:25 +01:00
Christopher Talib
7c5b2b714a
Adding configuration documentation and the config file in the gitignore
2020-02-10 14:40:33 +01:00
Christopher Talib
e634636768
Fix readme
2020-02-10 10:36:36 +01:00
Christopher Talib
40bfc4b01d
Beautify readme
2020-02-07 17:50:07 +01:00
Christopher Talib
2991f830bb
Update README with info about Balboa queries
2020-02-07 17:45:37 +01:00
Christopher Talib
afd99c0a4d
Adding balboa enrichment for domains and hostnames + documentation
2020-02-07 17:39:33 +01:00
Christopher Talib
787e2c3d02
Creating balboa package
2020-02-07 17:38:43 +01:00
Christopher Talib
01f7ae70b9
Renaming CertStreamWrapper to CertStreamRaw for more clarity
2020-02-07 15:50:21 +01:00
Christopher Talib
e38b05de66
Update README with more information on the nodes and edges connections
2020-02-07 15:49:42 +01:00
Christopher Talib
c7a52c527a
Refactoring saving single value into a helper, adding meta names to main nodes and edges
2020-02-07 15:27:14 +01:00
Christopher Talib
93f577cae9
Settting up more verbose logging with calling function and timestamp
2020-02-07 14:53:39 +01:00
Christopher Talib
2813ab0e96
Cleaning up and and reorganizing the models package
2020-02-06 17:16:47 +01:00
Christopher Talib
cc87dab922
Cleaning up and and reorganizing the models package
2020-02-06 17:13:19 +01:00
Christopher Talib
b5caae0974
Shodan connector and saving up and running
2020-02-06 17:01:37 +01:00
Christopher Talib
a321b46d0f
Adding chained items to parent object in certstream
2020-02-05 15:57:30 +01:00
Christopher Talib
c2b59808be
First implementation of shodan connector, but error on API key on streaming
2020-02-05 14:46:52 +01:00
Christopher Talib
efde1a4a54
Building ID on certnode
2020-02-04 10:58:56 +01:00
Christopher Talib
380e51229b
Building nodes, certnode, and edges
2020-02-04 10:57:14 +01:00
Christopher Talib
00bfb5eaaf
First implementation of certnode
2020-02-04 10:53:09 +01:00
Christopher Talib
f0f6457dd3
go mod tidy
2020-02-04 10:35:39 +01:00
Christopher Talib
8fc88b2694
Merge branch 'pastebin-poc' into 'master'
...
Working pastebin connector with parallel running with certstream
See merge request LABS/styx!1
2020-02-03 10:02:43 +01:00
Christopher Talib
0cd0faa6fa
Working pastebin connector with parallel running with certstream
...
Parallel running of Certstream and Pastebin connectors in go routine
with wait groups to exit them properly.
2020-02-03 10:01:52 +01:00
Christopher Talib
b9a2e73e33
Working version capturing CertStream saving raw, nodes and edges.
...
There was some problem with my editor and lost some code, this work is
fixing that and adding new features. Current state of the code:
* capturing CertStream traffic
* saving raw certstream objects in a custom wrapper
* extracting fingerprints and domains from the certstream object
* saving fingerprints and domains nodes and edges between them.
* fingerprint is linked to the raw certstream object with an edge
* saving to files with customizable names (raw in code)
broker:
* kafka connection and test
* no sending of data to it for the moment
2020-01-29 12:47:01 +01:00
Christopher Talib
b9f0ac688c
Removing extract.go file and refactoring the adhoc functions to files with name of the service
2020-01-29 11:05:05 +01:00
Christopher Talib
668686fbab
Saving edges and node from CertStream traffic.
...
This work extracts fingerprints and domains from CertStream data
structure received through the stream. It builds nodes and edges and
saves them to the relevant files. It sends this data to Kafka but no
logic is implemented at the exit of the broker yet.
2020-01-29 11:02:19 +01:00
Christopher Talib
67aecd65c1
Saving edges and node from CertStream traffic.
...
This work extracts fingerprints and domains from CertStream data
structure received through the stream. It builds nodes and edges and
saves them to the relevant files. It sends this data to Kafka but no
logic is implemented at the exit of the broker yet.
2020-01-29 10:03:52 +01:00
Christopher Talib
de0c5bfe83
Setting up linking and and creating nodes already from the input source
2020-01-28 23:52:24 +01:00
Christopher Talib
20f34c350a
Moving SaveDomains to parser package
2020-01-28 16:09:35 +01:00
Christopher Talib
e458a327d6
Moving SaveDomains to parser package
2020-01-28 16:05:36 +01:00
Christopher Talib
e39cf58d39
Moving SaveDomains to parser package
2020-01-28 16:03:46 +01:00
Christopher Talib
7785372e3a
Refactoring from utils to models
...
This work refactors saving and extracting function from the utils
package to the models package as it is a main component of the tool.
`utils` will take care of not related to models functions (such as
finding the files for example).
Also creating unique files for each type of source we are parsing.
2020-01-28 16:02:17 +01:00
Christopher Talib
0e1e003ab2
First work on test for connection to CertStream
2020-01-26 17:27:40 +01:00
Christopher Talib
e4cc92e100
First work on Pastebin parser
2020-01-26 17:27:20 +01:00
Christopher Talib
a0d2761c4e
Adding RFC3339 times
2020-01-26 17:26:03 +01:00
Christopher Talib
01f3e3ca1e
Only saving objects that have an ID
2020-01-23 15:01:08 +01:00
Christopher Talib
2a04e01457
Remove error management as it break on empty values
2020-01-23 14:38:27 +01:00
Christopher Talib
7cbb7e9180
Cleaning and working write/read to kafka and saving objects
2020-01-23 14:36:24 +01:00
Christopher Talib
2548c19ca4
saving
2020-01-23 13:13:59 +01:00
Christopher Talib
64e1d3d7a4
Cleaning up non used functions
2020-01-22 16:33:07 +01:00
Christopher Talib
8ac9d5786b
Refactoring and setting clearer filenames in utils/
2020-01-22 15:49:52 +01:00
Christopher Talib
0d082374e9
Some temporal refactoring, saving domains and nodes
2020-01-22 15:45:25 +01:00
Christopher Talib
3bcafa4084
Adding .txt files to gitignore
2020-01-22 15:44:56 +01:00
Christopher Talib
d33b293e7c
Read/write from Kafka, simpler structure with plain data and not pointers
2020-01-22 15:01:07 +01:00
Christopher Talib
48cc976595
Adding a consummer for Kafka, WIP
2020-01-21 16:50:50 +01:00
Christopher Talib
06f42fc472
Moving SendToKafka function to broker package
2020-01-17 14:02:52 +01:00