Standardized API for Plugs #2

New Issue

lennard.brinkhaus · 2023-12-07T21:10:56Z

lennard.brinkhaus commented

2023-12-07 21:10:56 +00:00

The e621-Plug is the first plug of many, so we need to define a protocol, so the Orchestrator can talk with the different Plugs.

Possible Layer:

Comunication Layer / Endpoints

PlugStatus:
Return the Status of the Plug and possible some information about the threads and jobs

CreateScrapeJob:
Info from orchestrator to Plug to scrape a specific user

JobStatus:
Return the Information / Status about the job (if he is running, canceled, finished)

JobFinished:
Send an Information from plug to orchestrator that the job is finished

Transport Layer / Protocol

MQ (MQTT / AMQP):
Create different topics to send and receive information

Pub/Sub (Redis):
Create different topics to send and receive information

RPC (gRPC):
Define a protobuf protocol and send and receive with example bidirectional streams the information

Websockets:
Create websocket connection and send information in both directions (JSON serialized)

REST:
Define some rest endpoints and get information by checking regular times if something changes

IRC:
Parse objects to JSON → Hastebin (because of max length in IRC message) → IRC server. It is a custom pub sub model

The e621-Plug is the first plug of many, so we need to define a protocol, so the Orchestrator can talk with the different Plugs. Possible Layer: **Comunication Layer / Endpoints** _PlugStatus:_ Return the Status of the Plug and possible some information about the threads and jobs _CreateScrapeJob:_ Info from orchestrator to Plug to scrape a specific user _JobStatus:_ Return the Information / Status about the job (if he is running, canceled, finished) _JobFinished:_ Send an Information from plug to orchestrator that the job is finished **Transport Layer / Protocol** _MQ (MQTT / AMQP):_ Create different topics to send and receive information _Pub/Sub (Redis):_ Create different topics to send and receive information _RPC (gRPC):_ Define a protobuf protocol and send and receive with example bidirectional streams the information _Websockets:_ Create websocket connection and send information in both directions (JSON serialized) _REST:_ Define some rest endpoints and get information by checking regular times if something changes _IRC:_ Parse objects to JSON → Hastebin (because of max length in IRC message) → IRC server. It is a custom pub sub model

lennard.brinkhaus commented

2023-12-07 21:22:49 +00:00

I've put together a few rough ideas just to mention them. As far as the communication layer is concerned, I don't think we need any more. If others are needed, please feel free to mention them.

As far as the transport layer is concerned, something like Redis is interesting, as it makes it very easy to reach all plugs, especially when scrape jobs are commissioned. However, it becomes more difficult if you expect answers. So if we agree that plugs only need to receive jobs and no further communication is needed, I would suggest using a pub/sub system.

If that is not the case, we should use either gRPC or REST. REST because it is simple and rudimentary. It is also a common protocol that is supported everywhere.
gRPC has the advantage that it is very performant and also because of the .proto files it is a versioned protocol, which necessarily also reflects the documentation of this protocol.

Unfortunately, I don't know enough about MQTT to say anything about it.
Would you like to share your thoughts on this @daskadse

I've put together a few rough ideas just to mention them. As far as the communication layer is concerned, I don't think we need any more. If others are needed, please feel free to mention them. As far as the transport layer is concerned, something like Redis is interesting, as it makes it very easy to reach all plugs, especially when scrape jobs are commissioned. However, it becomes more difficult if you expect answers. So if we agree that plugs only need to receive jobs and no further communication is needed, I would suggest using a pub/sub system. If that is not the case, we should use either gRPC or REST. REST because it is simple and rudimentary. It is also a common protocol that is supported everywhere. gRPC has the advantage that it is very performant and also because of the .proto files it is a versioned protocol, which necessarily also reflects the documentation of this protocol. Unfortunately, I don't know enough about MQTT to say anything about it. Would you like to share your thoughts on this @daskadse

lennard.brinkhaus commented

2023-12-16 15:22:50 +00:00

Wrong Repository

lennard.brinkhaus closed this issue

2023-12-16 15:22:50 +00:00

This repo is archived. You cannot comment on issues.