Energy monitoring system based on REST API Smart Meter.
Iot | Electrical Power Supply Parameters | RestAPI | JSON | Aggregation | SQL Database | Grafana | Real-time Dashboards
In this illustrative solution, InDriver collects and processes electricity parameters from the Shelly 3EM smart meter. Through InDriver's tasks, the data is meticulously completed, synchronized with the clock, and aggregated into 1-minute, 15-minute, 1-hour, and 1-day series, before being stored in the SQL database. Leveraging the Time Series API (TSAPI), visualizing the data in Grafana becomes seamless, allowing easy sharing on the Internet through web browsers
Data flow diagram
Task 1 - RestAPI Data Acquisition, Processing, and Logging into an SQL Table.
Upon starting InDriver's task, the REST API data request is defined.
OnStartup:
InDriver.import("RestApi");
RestApi.defineRequest('shelly','{"url":"https://shelly-56-eu.shelly.cloud/device/status","data":"id={id}&auth_key={key}","timeout":5000, "type":"post","headers":{"ContentTypeHeader":"application/json"}}');
InDriver.installHook(30000);
LOOP
InDriver sends REST API data requests at regular intervals, such as every configured period, for example, every 30 seconds, synchronized to the clock - 00:00:00, 00:00:30, 00:01:00, and so on.
OnHook:
RestApi.sendRequest('shelly');
if (RestApi.isSucceeded()) {
let ts= InDriver.hookTs();
const text = RestApi.getData('shelly');
const obj = JSON.parse(text);
let energy1 = Math.abs(obj.data.device_status.emeters[0].total_returned);
let energy2 = Math.abs(obj.data.device_status.emeters[1].total_returned);
let energy3 = Math.abs(obj.data.device_status.emeters[2].total_returned);
let power1 = Math.abs(obj.data.device_status.emeters[0].power);
let power2 = Math.abs(obj.data.device_status.emeters[1].power);
let power3 = Math.abs(obj.data.device_status.emeters[2].power);
let data = {
power1 : power1,
power2 : power2,
power3 : power3,
voltage1 : Math.abs(obj.data.device_status.emeters[0].voltage),
voltage2 : Math.abs(obj.data.device_status.emeters[1].voltage),
voltage3 : Math.abs(obj.data.device_status.emeters[2].voltage),
current1 : Math.abs(obj.data.device_status.emeters[0].current),
current2 : Math.abs(obj.data.device_status.emeters[1].current),
current3 : Math.abs(obj.data.device_status.emeters[2].current),
energy1 : energy1,
energy2 : energy2,
energy3 : energy3,
energy_total: energy1 + energy2 + energy3,
power_total: power1 + power2 + power3
}
let shelly = {Shelly:data};
let list = [];
list.push(shelly);
InDriver.debug(JSON.stringify(list));
InDriver.sendMessage(ts, 'DeviceData', JSON.stringify(list));
InDriver.sqlExecute("azureserver",
"insert into public.shelly (source, ts, data )
values ( 'Shelly','"+ts.toISOString()+"',$$"+JSON.stringify(list)+"$$);");
}
obj = {"isok":true,"data":{"online":true,"device_status":{"ram_free":32692,"mac":"34945477F146","fs_size":233681,"fs_free":154616,"getinfo":{"fw_info":{"device":"shellyem3-34945477F146","fw":"20221027-110030\/v1.12.1-ga9117d3"}}, "ram_total":49928, "emeters":[{"power":-312.8,"pf":-0.93, "current":1.42, "voltage":238.72, "is_valid":true, "total":0, "total_returned":1699355},{"power":-281.56,"pf":-0.8, "current":1.46, "voltage":240.94, "is_valid":true, "total":20.5, "total_returned":1734052.1},{"power":-10.16,"pf":-0.26, "current":0.16, "voltage":240.05, "is_valid":true,"total":0,"total_returned":542968.2}],"serial":17383,"has_update":true,"unixtime":1699182633,"cloud":{"enabled":true,"connected":true},"actions_stats":{"skipped":0},"v_data":1,"total_power":-604.52, "uptime":171876,"mqtt":{"connected":false},"_updated":"2023-11-05 13:11:49","cfg_changed_cnt":0,"relays":[{"ison":false,"has_timer":false,"timer_started":0,"timer_duration":0,"timer_remaining":0,"overpower":false,"is_valid":true,"source":"i...ip":"192.168.0.111","rssi":-74},"fs_mounted":true,"update":{"status":"pending", "has_update":true, "new_version":"20230913-114244\/v1.14.0-gcb84623","old_version":"20221027-110030\/v1.12.1-ga9117d3"}}}}
Key energy parameters, such as power, voltage, current, and energy, are extracted from the data. Additionally, specific calculations are executed, particularly in correcting the current direction, a crucial step in cases of incorrect current transformer connections.
The resulting processed data is stored in a JSON object, as depicted below:
data = {"power1": 312.8,"power2": 281.56, "power3":10.16, "voltage1":238.72, "voltage2":240.94, "voltage3":240.05, "current1":1.42, "current2":1.46, "current3":0.16, "energy1":1699355, "energy2":1734052.1, "energy3":542968.2,"energy_total":3976375.3, "power_total':604.45}
Here are the energy parameters received in JSON format
from a smart meter via a REST API:
Finally, the processed energy parameters are stored in an SQL table, illustrated below:
select * from public.shelly order by ts desc limit 10
Task 2 - Data aggregation
In the second task, the Data Aggregation Algorithm from the Time Series API (TsApi) Library is executed, as presented below. The TsApi is available with InDriver.
onStartup:
InDriver.import("TsApi");
InDriver.installHook(60000);
TsApi.defineAggregator("ShellyAggregation","azureserver","shelly","Europe/Warsaw",'["Shelly"]');
In this example, aggregation is performed at a frequency of every 1 minute.
onHook:
TsApi.aggregate("ShellyAggregation");
After applying the aggregation algorithm, SQL tables are generated for specific intervals, including default options of 1 minute, 15 minutes, 1 hour, and 1 day. For the 'shelly' input table, corresponding tables emerge: 'shelly_1minute,' 'shelly_15minutes,' 'shelly_1hour,' and 'shelly_1day.
select * from public.shelly_1hour order by ts desc limit 10
Additionally, each aggregation table incorporates an 'extras' column, capturing statistical values from the previous interval. This includes minimum (min), maximum (max), average (avg), and delta values. The 'extras' column also indicates the status, distinguishing between values derived from real data ('real') and those estimated/interpolated ('estimated').
With continually calculated complete aggregation tables for JSON input from any source, effortlessly create visualizations, and dashboards, perform additional calculations, or generate reports.
Example row from the 'shelly_1hour' aggregated table:
source
ts
data
extras
Shelly
2023-11-06 18:00:00+00
Thanks to the described data flow implemented with InDriver, creating visualizations becomes straightforward.
The aggregation and delta calculations for each interval yield instant energy consumption values. InDriver transforms JSON objects from the REST API into real-time tables, creating data that's dashboard-ready.
select ts, (extras->'statistics'->0->'Shelly'->'energy_total'->>'delta')::double precision as energy_consumption
from public."shelly_1hour" where "source" = 'Shelly' and ts >= date_trunc('month', now() ) and ts <= now()
