2.4 KiB
e621 to Graph
Scrapes users Favorite Posts and Upload them to a graph based database.
Prerequisites
Before running the program, ensure that you have the following installed on your system:
- Go (version 1.20.2 or higher)
- Docker (optional, if you plan to use Docker)
Installation
To install the project, follow these steps:
-
Clone the repository:
git clone https://git.dragse.it/fenpaws/e621-to-graph -
Change into the project's directory:
cd e621-to-graph -
Install the required dependencies:
go mod download go mod verify
Setting Up Environment Variables
The program requires certain environment variables to be set.
E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
DB_PASSWORD=
DB_USERNAME=
Running the Program
To run the program, execute the following command:
go run main.go
Building the Program
If you want to build the program into an executable binary, use the following command:
go build
Docker Support
The program can also be run using Docker. To use Docker, follow these steps:
-
Create a .env file in the project's root directory with the following content:
E621_API_KEY= E621_USERNAME= DB_TYPE=neo4j DB_URL= DB_PASSWORD= DB_USERNAME= -
Build the Docker image:
docker compose build -
Run the Docker container:
docker compose up -d
Database Support
The program supports the following databases.
Neo4j
To improve performance, it is recommended to create indices on different nodes in Neo4j.
Run the following commands to create the required indices:
neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);
Memgraph:
To improve performance, it is recommended to create indices on different nodes in Memgraph.
Run the following commands to create the required indices:
memgraph> CREATE INDEX ON :e621Tag(e621Tag);
memgraph> CREATE INDEX ON :e621Post(e621PostID);
memgraph> CREATE INDEX ON :Source(URL);
memgraph> CREATE INDEX ON :e621User(e621ID);