e621-sdk-go/README.MD

# e621 to Graph

Scrapes users Favorite Posts and Upload them to a graph based database.

## Prerequisites

Before running the program, ensure that you have the following installed on your system:

- Go (version 1.20.2 or higher)
- Docker (optional, if you plan to use Docker)

## Installation

To install the project, follow these steps:

1. Clone the repository:

   ```shell
   git clone https://git.dragse.it/fenpaws/e621-to-graph
   ```

2. Change into the project's directory:

   ```shell
   cd e621-to-graph
   ```

3. Install the required dependencies:

   ```shell
   go mod download
   go mod verify
   ```

## Setting Up Environment Variables

The program requires certain environment variables to be set.

```plaintext
E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
DB_PASSWORD=
DB_USERNAME=
# Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
LOG_LEVEL=
# Allowed values are PLAIN, JSON (default: PLAIN)
LOG_FORMAT=
# Allowed values are TRUE, FALSE (default: FALSE)
NEO4J_DEBUG=
```

## Running the Program

To run the program, execute the following command:

```shell
go run main.go
```

## Building the Program

If you want to build the program into an executable binary, use the following command:

```shell
go build
```

## Docker Support

The program can also be run using Docker. To use Docker, follow these steps:

1. Create a .env file in the project's root directory with the following content:

   ```plaintext
   E621_API_KEY=
   E621_USERNAME=
   DB_TYPE=neo4j
   DB_URL=
   DB_PASSWORD=
   DB_USERNAME=
   # Allowed values are FATAL, ERROR, WARN, INFO, DEBUG, TRACE (default: INFO)
   LOG_LEVEL=
   # Allowed values are PLAIN, JSON (default: PLAIN)
   LOG_FORMAT=
   # Allowed values are TRUE, FALSE (default: FALSE)
   NEO4J_DEBUG=
   ```

2. Build the Docker image:

   ```shell
   docker compose build
   ```

3. Run the Docker container:

   ```shell
   docker compose up -d
   ```

## Database Support

The program supports the following databases.

### Neo4j

To improve performance, it is recommended to create indices on different nodes in Neo4j.

Run the following commands to create the required indices:

```shell
neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);
```

### Memgraph:

To improve performance, it is recommended to create indices on different nodes in Memgraph.

Run the following commands to create the required indices:

```shell
memgraph> CREATE INDEX ON :e621Tag(e621Tag);
memgraph> CREATE INDEX ON :e621Post(e621PostID);
memgraph> CREATE INDEX ON :Source(URL);
memgraph> CREATE INDEX ON :e621User(e621ID);
```