This repository has been archived on 2024-07-22. You can view files and clone it, but cannot push or open issues or pull requests.
Go to file
2023-06-22 10:39:13 +02:00
api added DB Interface for later dgraph DB backend 2023-06-20 10:38:36 +02:00
database removed dgraph 2023-06-21 13:30:28 +02:00
e621 using new logging provider 2023-06-21 13:29:23 +02:00
services fixed issue #2 2023-06-22 09:04:52 +02:00
utils changed env for DB to be more DB agnostic 2023-06-22 10:39:13 +02:00
.gitignore update 2023-06-21 13:32:27 +02:00
build.bat added simple build script 2023-06-21 13:30:39 +02:00
docker-compose.yml added redis 2023-05-24 23:11:00 +02:00
go.mod go mod tidy 2023-06-21 13:32:22 +02:00
go.sum go mod tidy 2023-06-21 13:32:22 +02:00
main.go using new logging provider 2023-06-21 13:29:23 +02:00
README.MD changed env for DB to be more DB agnostic 2023-06-22 10:39:13 +02:00

e621 to Graph

Scrapes users Favorite Posts and Upload them to a graph based database.

Prerequisites

Before running the program, ensure that you have the following installed on your system:

  • Go (version 1.20.2 or higher)
  • Docker (optional, if you plan to use Docker)

Installation

To install the project, follow these steps:

  1. Clone the repository:

    git clone https://git.dragse.it/fenpaws/e621-to-graph
    
  2. Change into the project's directory:

    cd e621-to-graph
    
  3. Install the required dependencies:

    go mod download
    go mod verify
    

Setting Up Environment Variables

The program requires certain environment variables to be set.

E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
DB_PASSWORD=
DB_USERNAME=

Running the Program

To run the program, execute the following command:

go run main.go

Building the Program

If you want to build the program into an executable binary, use the following command:

go build

Docker Support

The program can also be run using Docker. To use Docker, follow these steps:

  1. Create a .env file in the project's root directory with the following content:

    E621_API_KEY=
    E621_USERNAME=
    DB_TYPE=neo4j
    DB_URL=
    DB_PASSWORD=
    DB_USERNAME=
    
  2. Build the Docker image:

    docker compose build
    
  3. Run the Docker container:

    docker compose up -d
    

Database Support

The program supports the following databases.

Neo4j

To improve performance, it is recommended to create indices on different nodes in Neo4j.

Run the following commands to create the required indices:

neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);

Memgraph:

To improve performance, it is recommended to create indices on different nodes in Memgraph.

Run the following commands to create the required indices:

memgraph> CREATE INDEX ON :e621Tag(e621Tag);
memgraph> CREATE INDEX ON :e621Post(e621PostID);
memgraph> CREATE INDEX ON :Source(URL);
memgraph> CREATE INDEX ON :e621User(e621ID);