Commit Graph

14 Commits

Author SHA1 Message Date
Syfaro
1eb5fba3f7 Add support for Twitter webhooks. 2021-08-25 13:22:53 -04:00
Syfaro
15ab9563e0
Add worker for refreshing submission data (#11)
* Initial attempt at a worker to refresh old data.

* Build refresh image on non-master branches.

* Allow specifying missing batch sizes.

* Store more data for deleted submissions.

* Add refresh to builds.

* Update furaffinity-rs dependency.

* Update furaffinity-rs, again.

* Update refresh Dockerfile to avoid extra build.

* Update faktory dependency.

* Update deleted flag migration order.
2021-08-21 22:48:05 -07:00
Syfaro
914a9b24ba Add Weasyl metrics, rework metric names. 2021-08-22 00:09:44 -04:00
Syfaro
4c0fb5ac2e Setting for ingesters to download files. 2021-08-21 18:28:54 -04:00
Syfaro
25e7ef3109 Fix clippy lints, update Rust version. 2021-08-10 21:41:11 -04:00
Syfaro
5773cc03f5
Break apart into multiple microservices (#10)
* Update CI.

* Bump versions.

* Fix dependencies.

* Unify tracing and metrics export.

* Create service to hash image.

* Updates for hashing service.

* Fix missing file changes.

* Old changes I don't remember.

* Update dependencies, improve Docker images.

* Use BKApi instead of in-memory tree.

* Include health endpoint with metrics.

* Avoid some unwraps.
2021-08-10 18:28:32 -07:00
Syfaro
47703694d1 More logging updates. 2021-04-21 21:46:10 -04:00
Syfaro
3974f85ce8
JSON log output (#9)
* Allow outputting JSON logs from ingesters.

* Skip instrument for faktory.

* Use more specific timestamp format.

* Update logger for webhooks too.
2021-04-21 19:26:15 -04:00
Syfaro
dd7805d052
Webhooks (#8)
* Rough initial progress on webhooks.

* Better webhook error handling, use all ingesters.

* Include hash in webhook data.

* Always pull Rust image.

* Fix missing features.
2021-04-21 16:58:32 -04:00
Syfaro
17a4d39a75
Improve FurAffinity ingester (#5)
* Improve FurAffinity retry logic, add timeout.

* Only build images when pushing to master.

* Use tracing instead of directly printing messages.

* Make case consistent on messages.

* Use tracing instead of panics directly.

* Record users online.

* Extract submission handling to new function.
2021-02-27 22:26:53 -05:00
Syfaro
4ad577cee5 Update everything to Tokio v1. 2021-02-20 21:16:58 -05:00
Syfaro
8914227f23 Merge branch 'master' into unify 2021-02-20 18:05:04 -05:00
Syfaro
348352f23e Some updates for FurAffinity ingester. 2020-12-07 17:57:13 -06:00
Syfaro
43e5c834a9 Add 'fuzzysearch-ingest-furaffinity/' from commit 'c44b877f70bf478e599110b10e17d152f89c8d61'
git-subtree-dir: fuzzysearch-ingest-furaffinity
git-subtree-mainline: 59da1e99a8
git-subtree-split: c44b877f70
2020-12-07 17:45:37 -06:00