Commit Graph

99 Commits

Author SHA1 Message Date
Christopher Talib e8f114edef Fix duplicated nodes in matcher 2020-08-28 15:55:43 +02:00
Christopher Talib 1528c28d7e Changing Type to NodeType to avoid issues 2020-08-28 15:55:18 +02:00
Christopher Talib 84e4937f85 Major version update
This new work implements the server and the loader in two different
binaries allowing the code while updating the IOC list.

It updates also the documentation to reflect the new changes.
2020-08-24 17:20:07 +02:00
Christopher Talib 65ad547860 Reimplementation of kafka with the docker compose + connection ok 2020-07-06 12:14:59 +02:00
Christopher Talib f161d60d2d Adding a small abstract to README 2020-06-12 12:04:59 +02:00
Christopher Talib 7f83f720d8 save 2020-06-10 12:31:07 +02:00
Christopher Talib 5d223c3886 Adding pastebin matching + demo IOC 2020-06-10 11:32:56 +02:00
Christopher Talib fe01a9240f Parsing shodan, not droppping DB
* Adding Certstream and Shodan matchers
* Insert or skip for new matchers (working without having to drop the DB
and not more duplicate matchers)
* Closing files after using them
* Adding Match model to schema and Node (for unmarshalling purposes)
2020-06-10 10:48:47 +02:00
Christopher Talib 0234a4f50b Adding *.swp to gitignore" 2020-06-10 10:47:39 +02:00
Christopher Talib 44bc03b51d Adding certstream matching on CN 2020-06-08 12:58:07 +02:00
Christopher Talib 929222aff4 Parallel matching on multiple targets work 2020-06-08 10:49:19 +02:00
Christopher Talib b26cc60d39 multiple search works (kinda) 2020-06-03 16:20:40 +02:00
Christopher Talib ba0b011ce4 save 2020-06-02 15:44:24 +02:00
Christopher Talib e533c2c335 updating matcher nodes works! 2020-06-02 10:58:31 +02:00
Christopher Talib 3961e79062 Matcher logic and IOCs
This work starts to build the matcher logic into styx. For the moment,
the goal is to define IOCs and load them when the Matcher plugin is
activated.

To implement: Then, the matcher will run periodic queries to different
types of nodes and index them to its one Matcher Dgraph Node. So be
targetting a specific IOCs, the user will be able to list the
observation that have been made to it.
2020-05-29 11:32:55 +02:00
Christopher Talib 9547aeea3f basic matching on simple target, no sending to dgraph 2020-05-27 12:05:53 +02:00
Christopher Talib 9fa5d13bf6 Full text search and indexing some keywords
Some of keywords are indexed and open for full text search, please refer
to the README for more details.

CertStream, Pastebin and Shodan are running as services and can be
searched.

Next steps: building the matcher and creating edges.
2020-05-20 10:03:28 +02:00
Christopher Talib 7163147a4f Pastebin nodes simple
Pastebin data is also sent to Dgraph and can be queried.
2020-05-19 10:10:42 +02:00
Christopher Talib b1ca4b3c5f Shodan in Dgraph, first part
Implementing first version for shodan node, missing yet some models, but
the overal approach works and can be queried in Ratel.
2020-05-18 16:09:04 +02:00
Christopher Talib cbdca52ab2 Simple linked model on certstream + better install instructions 2020-05-18 10:22:08 +02:00
Christopher Talib 5dca0a0472 adding Dgraph explanantions 2020-05-13 14:52:38 +02:00
Christopher Talib f0aa10a9d2 Adding docker-compose for dgraph 2020-05-13 11:51:54 +02:00
Christopher Talib d0c8deae99 saving 2020-03-19 09:27:15 +01:00
Christopher Talib fb270a1b66 Mutating data without needing to get it back 2020-03-04 16:34:14 +01:00
Christopher Talib f61fe566a5 Basic connection to Dgraph DB
The first work and input to the graph db is set up in this work. It's
for the moment very basic and doesn't cover relations and only works for
certstream data.
2020-03-04 15:16:59 +01:00
Christopher Talib b72e82071d saving 2020-03-02 17:06:28 +01:00
Christopher Talib cd43194873 First work for the dgraph connection + fixing some error logging 2020-03-02 16:27:51 +01:00
Christopher Talib d761e824f3 Changing the plugin architecture for a modular architecture. 2020-02-25 10:05:31 +01:00
Christopher Talib b2da64a9d7 Enh/modular arch 2020-02-25 10:05:31 +01:00
Christopher Talib e87856b9a1 Removing print statements 2020-02-20 14:53:18 +01:00
Christopher Talib 0f25d6d81c Prototype to domain filtering with Pharos filters 2020-02-20 14:32:06 +01:00
Christopher Talib e7421931c2 Couple of quickfixes to allow run a dry installion (deactivating ES and balboa) 2020-02-19 15:26:47 +01:00
Christopher Talib 56e0e52bb5 Cleaning up example config 2020-02-19 10:03:49 +01:00
Christopher Talib 5bd0b8090b Update dependencies 2020-02-18 12:22:23 +01:00
Christopher Talib 3a3637c847 Move RunIPFilters() to filters/main.go 2020-02-18 12:21:39 +01:00
Christopher Talib f0a6715acd Adding elasticsearch configuration on README 2020-02-17 12:10:27 +01:00
Christopher Talib f4f82d9e1c Adding code and setting up elastic search 2020-02-17 12:08:49 +01:00
Christopher Talib 5b1bfbc195 Filters for IP and shodan 2020-02-14 20:46:09 +01:00
Christopher Talib 1fc11e7a4e Adding all CIDR files in filters/data 2020-02-14 12:01:50 +01:00
Christopher Talib 02a014262a Adding Akamai filters for data recieved in Shodan + allow deactivating Balboa search 2020-02-14 11:36:47 +01:00
Christopher Talib e963633c89 Allowing the possiblity to activate or deactivate modules 2020-02-14 11:30:59 +01:00
Christopher Talib bd3f108d12 Shodan ports in config 2020-02-12 16:54:14 +01:00
Christopher Talib 638216e8fa Implementing config variables in the application 2020-02-10 16:11:25 +01:00
Christopher Talib 7c5b2b714a Adding configuration documentation and the config file in the gitignore 2020-02-10 14:40:33 +01:00
Christopher Talib e634636768 Fix readme 2020-02-10 10:36:36 +01:00
Christopher Talib 40bfc4b01d Beautify readme 2020-02-07 17:50:07 +01:00
Christopher Talib 2991f830bb Update README with info about Balboa queries 2020-02-07 17:45:37 +01:00
Christopher Talib afd99c0a4d Adding balboa enrichment for domains and hostnames + documentation 2020-02-07 17:39:33 +01:00
Christopher Talib 787e2c3d02 Creating balboa package 2020-02-07 17:38:43 +01:00
Christopher Talib 01f7ae70b9 Renaming CertStreamWrapper to CertStreamRaw for more clarity 2020-02-07 15:50:21 +01:00