e621 to Graph

Scrapes users Favorite Posts and Upload them to a graph based database.

Prerequisites

Before running the program, ensure that you have the following installed on your system:

• Go (version 1.20.2 or higher)
• Docker (optional, if you plan to use Docker)

Installation

To install the project, follow these steps:

1. Clone the repository:

   git clone https://git.dragse.it/fenpaws/e621-to-graph
   

2. Change into the project's directory:

   cd e621-to-graph
   

3. Install the required dependencies:

   go mod download
   go mod verify
   

Setting Up Environment Variables

The program requires certain environment variables to be set.

E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
DB_PASSWORD=
DB_USERNAME=
# Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
LOG_LEVEL=
# Allowed values are PLAIN, JSON (default: PLAIN)
LOG_FORMAT=
# Allowed values are TRUE, FALSE (default: FALSE)
NEO4J_DEBUG=

Running the Program

To run the program, execute the following command:

go run main.go

Building the Program

If you want to build the program into an executable binary, use the following command:

go build

Docker Support

The program can also be run using Docker. To use Docker, follow these steps:

1. Create a .env file in the project's root directory with the following content:

   E621_API_KEY=
   E621_USERNAME=
   DB_TYPE=neo4j
   DB_URL=
   DB_PASSWORD=
   DB_USERNAME=
   # Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
   LOG_LEVEL=
   # Allowed values are PLAIN, JSON (default: PLAIN)
   LOG_FORMAT=
   # Allowed values are TRUE, FALSE (default: FALSE)
   NEO4J_DEBUG=
   

2. Build the Docker image:

   docker compose build
   

3. Run the Docker container:

   docker compose up -d
   

Database Support

The program supports the following databases.

Neo4j

To improve performance, it is recommended to create indices on different nodes in Neo4j.

Run the following commands to create the required indices:

neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);

Memgraph:

To improve performance, it is recommended to create indices on different nodes in Memgraph.

Run the following commands to create the required indices:

memgraph> CREATE INDEX ON :e621Tag(e621Tag);
memgraph> CREATE INDEX ON :e621Post(e621PostID);
memgraph> CREATE INDEX ON :Source(URL);
memgraph> CREATE INDEX ON :e621User(e621ID);