0.54

/.vscode

/eloparser_docker.code-workspace

eloparser_public.code-workspace

/composer.phar

/CHANGELOG.md

/src/php_app/app.json

/src/php_app/data_response.json

/src/php_app/log.txt

/src/php_app/supervisord.log

settings.json.prod

settings.json.devbucket

src/php_app/dev/settings.json

config_dev.env

config_prod.env

# eloparser

## Fetches data and writes to a InfluxDB database

- Meter readings from Eloverblik (Power consumption per hour)

- Spot prices from Energidataservice

>**Version: v0.54**

---

### JSON Output from update.php

Various info will be available, for use in Annotations in Grafana or for whatever you want, see [example](#example-of-output-updatephp).

### Manual installation or Docker?

- You can use the PHP code on its own if you already have a working PHP 8.1 environment or you can use the Docker version. [Manual installation](#manual-installation)

- The [Docker version](#install-using-docker-compose-recommended) includes InfluxDB and Grafana. Grafana is setup with automatic provisioning of datasource and a template dashboard.

## [Requirements](#requirements)

- PHP 8.1

- InfluxDB 2.0+

- (Optional - for JSON source in Grafana) Infinity plugin - <https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource/>

## [Getting Started](#getting-started)

### [Install using Docker Compose (Recommended)](#install-using-docker-compose-recommended)

#### For the full stack - PHP app, webserver (nginx), database (InfluxDB) and data visualization (Grafana):

1. Insert your refresh token as a environment variable in docker-compose.yml

2. Run Docker Compose on docker-compose.yml

- Grafana is available on <http://localhost:3000> (username admin, password admin)

- InfluxDB is available on <http://localhost:8086>

- JSON output from PHP app is available on <http://localhost:8181>

##### Already got InfluxDB and Grafana?

##### docker-compose_nginxphp.yml provides a stack with only nginx and the PHP app and has environment variables set up ready for use in Portainer.

### [Manual installation](#manual-installation)

1. Setup settings.json (inside src/php_app), see [example](#example-of-correct-settingsjson).[^first]

2. Run get_ancient.php to get "ancient" data (all years before current) - preferably in CLI to avoid problems with timeouts \

Please note that you only have to run get_ancient.php 1 time. Unless you have a time machine its unlikely that your power usage back in etc 2015 changes :D

3. Run update.php *(etc every 6 hours in a cronjob for continuously data retrieval)*

---

## [Examples](#examples)

### [Example of correct settings.json](#example-of-correct-settingsjson)

#### Not relevant if you use Docker.

```json

{

    "refresh_token": "Your-RefreshToken-from-Eloverblik",

    "date_from": "",

    "date_to": "",

    "meter_id": "",

    "influxdb_url": "http://192.168.1.2:8086",

    "influxdb_token": "xxxxxxxxxxxxxxxxxxxxxxxxxxx",

    "influxdb_bucket": "YourBucket",

    "influxdb_org": "YourOrganization"

```

### [Example of output (update.php)](#example-of-output-updatephp)

```json

  "info": {

    "nextupdate_readings_in": 16207,

    "nextupdate_readings_dt": "2022-11-15T00:35:19Z",

    "lastupdate_readings_dt": "2022-11-14T16:35:19Z",

    "nextupdate_prices_in": 16262,

    "nextupdate_prices_dt": "2022-11-15T00:36:14Z",

    "lastupdate_prices_dt": "2022-11-14T16:36:14Z",

    "settings_date_from": "2012-07-17",

    "settings_date_to": "2022-11-14",

    "query_date_from": "2022-11-12",

    "query_date_to": "2022-11-14",

    "meters": "521234567890123450"

  }

```

| Key | Description |

| --- | ----------- |

| nextupdate_readings_in | Seconds to next attempt at fetching data from Eloverblik |

| nextupdate_readings_dt | Same as above but in ISO-8601 format (UTC) |

| lastupdate_readings_dt | Last time data was successful retrieved(and stored) from Eloverblik | ISO-8601 format (UTC)

| nextupdate_prices_in | Seconds to next attempt at fetching data from Energidataservice |

| nextupdate_prices_dt | Same as above but in ISO-8601 format (UTC) |

| lastupdate_prices_dt | Last time data was successful retrieved(and stored) from Energidataservice | ISO-8601 format (UTC)

| settings_date_from | Wil be set after first run to the date of the first available data entry. You can also manually set this to any date *after* this date.

| settings_date_to | Changes automatically to the current date.

| query_date_from & query_date_to | The time period of last query to Eloverblik. (Changes automatically)

## Notes

[^first]: update.php will attempt to fetch new data from Eloverblik every 8 hours (*updateinterval* in Configuration class), for the latest entry in the InfluxDB database to the current date. Please note that Eloverblik will always provide data with 2~ days delay. \

You can call this code every few seconds if you like to, it will only do calls to Eloverblik after 8 hours has passed since last successful request. \

Docker version runs update.php every minute (a bit overkill yes).

- The code has been tested on Linux and Windows, on Windows you need to escape the paths specified in update.php, with \ or just use / instead, example: ```D:\\aFolder\anotherOne\\test1\\``` and ```D:/aFolder/anotherOne/test1/```should work.

## Grafana and InfluxDB (Suggestion for manual installation)

If you want to use InfluxQL with InfluxDB 2.0+ you may need to create a DBRP mapping. \

List buckets with: ```influx v1 dbrp list``` \

Create bucket mapping with: ```influx v1 dbrp create --bucket-id yourBucketID --db yourBucket --rp autogen``` \

<https://docs.influxdata.com/influxdb/v2.4/query-data/influxql/dbrp/#create-dbrp-mappings> \

This only applies to InfluxQL, for Flux queries you don't need to do anything.

 No newline at end of file

services:

    #### nginx (Webserver - Optional)

    web-http:

        build:

            context: ./src

            dockerfile: ./ct_nginx/Dockerfile

        working_dir: /app

        volumes:

            - vol_eloparser:/app

        networks:

            - elostack

        ports:

            - "8181:80"

    #### PHP App (Required)

    php-fpm:

        pull_policy: build

        build: 

            context: ./src

            dockerfile: ./ct_php-fpm/Dockerfile

        volumes:

            - vol_eloparser:/app

        networks:

            - elostack

        environment:

            REFRESH_TOKEN: "yourRefreshTokenfromEloverblik" #For Portainer: ${REFRESH_TOKEN}

            INFLUXDB_URL: "http://influxdb:8086" #For Portainer: ${INFLUXDB_URL}

            INFLUXDB_TOKEN: "thisWillbeUsedinSettings-PleaseCreateaNewTokenOnlyForThisBucket" #For Portainer: ${INFLUXDB_TOKEN}

            INFLUXDB_BUCKET: "myBucket" #For Portainer: ${INFLUXDB_BUCKET}

            INFLUXDB_ORG: "myOrganization" #For Portainer: ${INFLUXDB_ORG}

    #### InfluxDB (Optional)

    influxdb:

        build: 

            context: ./src

            dockerfile: ./ct_influxdb/Dockerfile

        volumes:

            - vol_influxdb_config:/etc/influxdb2

            - vol_influxdb_data:/var/lib/influxdb2

        networks:

            - elostack

        environment:

            INFLUXD_LOG_LEVEL: "info"

            INFLUXD_REPORTING_DISABLED: "true"

            DOCKER_INFLUXDB_INIT_MODE: "setup"

            DOCKER_INFLUXDB_INIT_USERNAME: "myUsername"

            DOCKER_INFLUXDB_INIT_PASSWORD: "changeThis"

            DOCKER_INFLUXDB_INIT_ORG: "myOrganization"

            DOCKER_INFLUXDB_INIT_BUCKET: "myBucket"

            DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: "thisWillbeUsedinSettings-PleaseCreateaNewTokenOnlyForThisBucket"

            V1_DB_NAME: "myBucket" #For InfluxQL in Grafana

            V1_RP_NAME: "forever"

        ports:

            - "8086:8086"

    #### Grafana (Optional)

    # If you change DOCKER_INFLUXDB_INIT_ADMIN_TOKEN or DOCKER_INFLUXDB_INIT_BUCKET, update the values in ct_grafana/provision_datasource.yml aswell.

    # provision_datasource.yml automatically adds a datasource to the InfluxDB container.

    grafana:

        build: 

            context: ./src

            dockerfile: ./ct_grafana/Dockerfile

        volumes:

            - vol_grafana:/var/lib/grafana

        networks:

            - elostack

        environment:

            GF_ANALYTICS_REPORTING_ENABLED: "false"

            GF_AUTH_ANONYMOUS_ENABLED: "false" #You can switch this to true if you want to enable anonymous access (no login required).

            # But before you do, please read https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/#implications-of-enabling-anonymous-access-to-dashboards

            GF_AUTH_ANONYMOUS_ORG_NAME: "Main Org."

            GF_AUTH_ANONYMOUS_ORG_ROLE: "Viewer"

            #GF_LOG_LEVEL: "info" # default info; debug,info,warn,error,critical

        ports:

            - "3000:3000"

        depends_on:

        - influxdb

volumes:

    vol_eloparser:

    vol_influxdb_config:

    vol_influxdb_data:

    vol_grafana:

networks:

    elostack:

 No newline at end of file

services:

    #### nginx (Webserver - Optional)

    web-http:

        build:

            context: ./src

            dockerfile: ./ct_nginx/Dockerfile

        working_dir: /app

        volumes:

            - vol_eloparser:/app

        networks:

            - elostack

        ports:

            - "8181:80"

    #### PHP App (Required)

    php-fpm:

        pull_policy: build

        build: 

            context: ./src

            dockerfile: ./ct_php-fpm/Dockerfile

        volumes:

            - vol_eloparser:/app

        networks:

            - elostack

        environment:

            REFRESH_TOKEN: ${REFRESH_TOKEN} # Portainer uses this to replace it with values from webinterface

            INFLUXDB_URL: ${INFLUXDB_URL} # If you don't want that see docker-compose.yml for a example of setting the values directly in here

            INFLUXDB_TOKEN: ${INFLUXDB_TOKEN}

            INFLUXDB_BUCKET: ${INFLUXDB_BUCKET}

            INFLUXDB_ORG: ${INFLUXDB_ORG}

volumes:

    vol_eloparser:

networks:

    elostack:

 No newline at end of file

FROM grafana/grafana:9.2.3

COPY ./ct_grafana/provision_datasource.yml /etc/grafana/provisioning/datasources/influx.yml

COPY ./ct_grafana/provision_dashboard.yml /etc/grafana/provisioning/dashboards/eloverblik.yml

COPY ./ct_grafana/template_dashboard.json /var/lib/grafana/dashboards/powerusage.json

 No newline at end of file