* Adding Certstream and Shodan matchers
* Insert or skip for new matchers (working without having to drop the DB
and not more duplicate matchers)
* Closing files after using them
* Adding Match model to schema and Node (for unmarshalling purposes)
This work starts to build the matcher logic into styx. For the moment,
the goal is to define IOCs and load them when the Matcher plugin is
activated.
To implement: Then, the matcher will run periodic queries to different
types of nodes and index them to its one Matcher Dgraph Node. So be
targetting a specific IOCs, the user will be able to list the
observation that have been made to it.
Some of keywords are indexed and open for full text search, please refer
to the README for more details.
CertStream, Pastebin and Shodan are running as services and can be
searched.
Next steps: building the matcher and creating edges.
The first work and input to the graph db is set up in this work. It's
for the moment very basic and doesn't cover relations and only works for
certstream data.
There was some problem with my editor and lost some code, this work is
fixing that and adding new features. Current state of the code:
* capturing CertStream traffic
* saving raw certstream objects in a custom wrapper
* extracting fingerprints and domains from the certstream object
* saving fingerprints and domains nodes and edges between them.
* fingerprint is linked to the raw certstream object with an edge
* saving to files with customizable names (raw in code)
broker:
* kafka connection and test
* no sending of data to it for the moment
This work extracts fingerprints and domains from CertStream data
structure received through the stream. It builds nodes and edges and
saves them to the relevant files. It sends this data to Kafka but no
logic is implemented at the exit of the broker yet.
This work extracts fingerprints and domains from CertStream data
structure received through the stream. It builds nodes and edges and
saves them to the relevant files. It sends this data to Kafka but no
logic is implemented at the exit of the broker yet.
This work refactors saving and extracting function from the utils
package to the models package as it is a main component of the tool.
`utils` will take care of not related to models functions (such as
finding the files for example).
Also creating unique files for each type of source we are parsing.
This work builds an extractor for the data in the CertStream in order to
save it. It builds itself from the previous work, so extensions and
flags can be added to the structures. The work in `utils` is basically a
big extractor for the data taking advantage of the JSONq library.
Currently, there is not refactoring and the "chains" are not saved
because they need additionnal computation which will come in a later
commit.
