Commit d305ccb7 authored by Jun Matsushita's avatar Jun Matsushita

Added @borys architecture document

parent da39caeb
# Open Integrity Architecture
## Collection Backend
- APIs.
-
- parsing download pages for binaries, and checksums, maybe creating an agent that downloads and runs a checksum computation.
- building an wrapper around the Libraries.io API, or reimplementing some of their package management parsing tools (https://github.com/librariesio/package-managers/blob/master/package-managers.json) and dependency parsing tools (https://github.com/librariesio/mix-deps-json
TOSBack (https://github.com/pde/tosback2) and I think it could be used (or adapted) to track a number of metrics that depend on documents (like terms of services, but also documentation, transparency reports, download pages…). I’m also interested in what https://morph.io/ can do. Particularly the multi-language capabilities, the available free backend (and open source) to run scrapers. I have some ideas about how we could leverage this to help Ranking Digital Rights track the document they use in their “Sources” sheets here: https://rankingdigitalrights.org/index2015/download/ I’ve written some messy notes about this here: http://openintegrity.github.io/openintegrity.org/metrics/1_sources/ranking_digital_rights/
## Data Backend
- Why not use CKAN, what can we learn from it?
## API
- Tyk, why not.
The work currently could be split up between these various aspects with part time involvement:
Tech Architecture/Data Modeling (possibly team management)
Working on collection agents (CVE parsing, dependency tracking, API wrapping, endpoint watching and probing, document change notification and scraping,…)
Managing a Rancher based Docker infrastructure to spin up and run
This diff is collapsed.
# Scale
## Metric space
The effort needed to update and maintain the metric space should be managed. Considerations include:
- Existing sources: storage space for historical data or mirroring.
- New automated collection: Maintenance and monitoring of agents, traffic, processing, storage.
- Qualitative reviews: verification.
Scale sensitivity is whether adding new aspects to OII create significant scaling challenges.
- Adding new metrics
+
- Adding new tools
+ Deployment types have different metric spaces.
| | SMS | Skype | Signal | Telegram | ChatSecure |
|--------------|-----|--------------|----------------------|----------|------------|
| Dependencies | | Read Credits | Gradle / Maven + iOS | | |
| | Node project | Android
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment