e621-sdk-go/README.MD

# e621 to Graph

Scrapes users Favorite Posts and Upload them to a graph based database.

## Prerequisites

Before running the program, ensure that you have the following installed on your system:

- Go (version 1.20.2 or higher)
- Docker (optional, if you plan to use Docker)

## Installation

To install the project, follow these steps:

1. Clone the repository:

   ```shell
   git clone https://git.dragse.it/fenpaws/e621-to-graph
   ```

2. Change into the project's directory:

   ```shell
   cd e621-to-graph
   ```

3. Install the required dependencies:

   ```shell
   go mod download
   go mod verify
   ```

## Setting Up Environment Variables

The program requires certain environment variables to be set. 
```plaintext
E621_API_KEY=
E621_USERNAME=
DB_TYPE=neo4j
DB_URL=
NEO4J_PASSWORD=
NEO4J_USERNAME=
```

## Running the Program

To run the program, execute the following command:

```shell
go run main.go
```

## Building the Program

If you want to build the program into an executable binary, use the following command:

```shell
go build
```

## Docker Support

The program can also be run using Docker. To use Docker, follow these steps:

1. Create a .env file in the project's root directory with the following content:

    ````plaintext
    E621_API_KEY=
    E621_USERNAME=
    DB_TYPE=neo4j
    DB_URL=
    NEO4J_PASSWORD=
    NEO4J_USERNAME=
    ````

2. Build the Docker image:

   ```shell
   docker compose build
   ```

3. Run the Docker container:

   ```shell
   docker compose up -d
   ```

## Database Support

The program supports the following databases.

### Neo4j

To improve performance, it is recommended to create indices on different nodes in Neo4j. 

Run the following commands to create the required indices:

```shell
neo4j$ CREATE INDEX tagIndex FOR (t:e621Tag) ON (t.e621Tag);
neo4j$ CREATE INDEX postIndex FOR (p:e621Post) ON (p.e621PostID);
neo4j$ CREATE INDEX sourceUrlIndex FOR (s:Source) ON (s.URL);
neo4j$ CREATE INDEX userIdIndex FOR (u:e621User) ON (u.e621ID);
```

### Memgraph:

To improve performance, it is recommended to create indices on different nodes in Memgraph. 

Run the following commands to create the required indices:

```shell
memgraph> CREATE INDEX ON :e621Tag(e621Tag);
memgraph> CREATE INDEX ON :e621Post(e621PostID);
memgraph> CREATE INDEX ON :Source(URL);
memgraph> CREATE INDEX ON :e621User(e621ID);
```