How to build CGate, CGateWeb, MQTT, Home Assistant and Home Bridge

Discussion in 'Third-Party Solutions' started by theboyg, Nov 30, 2021.

  1. theboyg

    theboyg

    Joined:
    Apr 3, 2005
    Messages:
    210
    Likes Received:
    9
    How to build CGate, CGateWeb, MQTT, Home Assistant and Home Bridge

    After pulling together all the disparate information I could and, along with the learning curve of Home Assistant, YAML, MQTT and more, here's my guide on how to start from scratch to get you up and running. It took a lot of trial and error with curveballs along the way, but this is a reproducible installation.

    You'll need two Raspberry Pis plus a suitable serial to USB or RS232 if using a USB or RS232 CNI, or ignore this if you have an Ethernet CNI (which I don't have).

    The steps will be:
    1. On Pi #1, Install Home Bridge (optional, if you need it. I use it to surface Unifi Protect cameras into Home Assistant, plus Siri)
    2. Install Java
    3. Install CGate
    4. Install CGateweb
    5. Install Home Assistant on Pi #2
    6. Configure the MQTT broker in Home Assistant
    7. Configure Home Assistant for MQTT and C-BUS
    8. Configure Home Assistant scripts for C-BUS devices

    1. Home Bridge

    First grab Home Bridge v1.0.17 from https://github.com/homebridge/homebridge-raspbian-image/releases

    V1.0.25 doesn’t work as it breaks MQTT in NPM with Node 16. There maybe a release later than 1.0.17 that is viable, but this works with Node 14 and you can update Home Bridge once it’s installed

    2. Install Java

    Code:
             sudo apt update
             sudo apt upgrade
             sudo apt install openjdk-8-jdk openjdk-8-jre
             java -version
    3. Install cgate using these instructions
    https://blog.addictedtopi.com/2017/05/21/installing-c-gate-on-a-raspberry-pi/
    • note : use the Debian Jessie rules for the Pi install, and select pi as the user instead of root in the ser2sock.service file (if you’re using a serial connection)

    As of writing the latest cgate is at https://updates.clipsal.com/Clipsal...ite/cis/technical/CGate/cgate-2.11.4_3251.zip


    Once installed:
    -Configure /usr/local/bin/cgate/config/access.txt

    • i.e. if you’re on a 10.2.x.x network, and 101 is your PC for provisioning C-BUS

    Code:
    remote 10.2.0.255 Program
    remote 10.2.0.101 Program
    interface 127.0.0.1 Program
    - Configure /usr/local/bin/cgate/config/C-gateConfig.txt
    • Ensure project.default= and project.start= is your project file
    • Don’t forget to transfer your project file to the /usr/local/bin/cgate/tag directory

    4. Install CGateWeb

    Review these instructions https://addictedtopi.tumblr.com/post/101165284373/setting-up-cgateweb-c-bus-to-mqtt-on-linux

    But don’t follow them as outlined, do this instead

    Code:
    cd /usr/local/bin
    sudo git clone https://github.com/the1laz/cgateweb.git
    sudo chown -R pi:pi cgateweb
    cd cgateweb
    sudo npm install
    
    (Take npm advice on screen)
    
    sudo npm install forever -g
    mkdir log
    nano settings.js
    -Now edit the settings, uncomment the credentials and specify what you will use in Home Assistant for MQTT. As an example, specify the username and password as mqtt and mqtt, you will specify this later in the MQTT broker in Home Assistant.
    -
    Set the project name to your bus project name, and set the IP of your Home Assistant Pi
    Ensure the following are set:
    Code:
    exports.cbusip = '127.0.0.1';
    exports.cbusname = "(your project name here)";
    exports.mqtt = 'IPofyourHomeAssistantPC:1883';

    -Now do the following
    Code:
    sudo nano /etc/systemd/system/cgateweb.service
    Code:
    [Unit]
    Description=cgateweb
    [Service]
    ExecStart=node /usr/local/bin/cgateweb/index.js
    Restart=always
    User=pi
    Group=pi
    Environment=PATH=/usr/bin:/usr/local/bin
    Environment=NODE_ENV=production
    WorkingDirectory=/usr/local/bin/cgateweb
    [Install]
    WantedBy=multi-user.target
    - Then run:


    Code:
    sudo systemctl enable cgateweb.service

    We move to Pi #2


    5. Install Home Assistant
    Follow the instructions on their website

    6. Install the MQTT Broker

    Once installed, specify the username and password in the config such as mqtt/mqtt

    Your MQTT config should look something like this:

    Code:
    logins:
      - username: mqtt
        password: mqtt
    customize:
      active: false
      folder: mosquitto
    certfile: fullchain.pem
    keyfile: privkey.pem
    require_certificate: false

    7. Configure Home Assistant for MQTT and CBUS

    Install the File Editor Home Assistant plugin so we can now edit the .yaml files

    Ensure these are listed in the configuration.yaml

    Code:
    automation: !include_dir_merge_list automation/
    light: !include light.yaml
    • -Create a folder called ‘automation’
    • -Create a new file called light.yaml
    • -Create a new file cbusautomation.yaml in a folder called automation
    In the light.yaml you can start to specify your C-BUS devices.

    An example for a dimmer

    Code:
      - platform: mqtt
        name: Den
        state_topic: "cbus/read/254/56/0/state"
        command_topic: "cbus/write/254/56/0/switch"
        brightness_state_topic: "cbus/read/254/56/0/level"
        brightness_command_topic: "cbus/write/254/56/0/ramp"
        payload_off: "OFF"
        on_command_type: "brightness"
        unique_id: mq_den
        brightness_scale: "100"
    or a switch

    Code:
      - platform: mqtt
        state_topic: "cbus/read/254/56/16/state"
        command_topic: "cbus/write/254/56/16/switch"
        payload_off: "OFF"
        payload_on: "ON"
        name: Cinema
        unique_id: mq_cinema
    8. Do some Home Assistant logic based on C-BUS activity

    In your new cbusautomation.yaml , in the automation folder, you can create script to respond to C-CBUS commands.

    This one toggles a C-BUS switch that already controls a garage door

    Code:
    - id: cbus_garage_door_control_toggle
      alias: Garage Door
      trigger:
      - entity_id: light.garage_double_door_control
        platform: state
      action:
      - service: light.turn_off
        target:
          entity_id: light.garage_double_door_control
        data:
          transition: 1
      mode: single 
    This one grabs the dim level of the C-BUS ID, and tells a Shelly RGBW LED to set the LED to the same level

    Code:
    - id: cbus_shelly_led
      alias: cbus_shelly_led
      trigger:
      - entity_id: light.shelly.led1
        platform: state
        attribute: brightness
      action:
      - service: light.turn_on
        target:
          entity_id: light.shelly.led1
        data:
          brightness: "{{ state_attr('cbus_shelly_led,'brightness') |int(0) }}"
      mode: single

    And lastly, reboot both Pis.

    Big thanks to @Damaxx for the inspiration!
     
    Last edited: Nov 30, 2021
    theboyg, Nov 30, 2021
    #1
    Ian, zei20t, Damaxx and 1 other person like this.
  2. theboyg

    zei20t

    Joined:
    Aug 18, 2010
    Messages:
    130
    Likes Received:
    1
    Location:
    Sydney, Australia
    ive found using node-red as a logic engine is amazing. and easy.

    im mostly using HA as the GUI, NR does all logic and interfacing between device types.

    saves so much on yaml code
     
    zei20t, Dec 14, 2021
    #2
  3. theboyg

    theboyg

    Joined:
    Apr 3, 2005
    Messages:
    210
    Likes Received:
    9
    I've been running into a lot of issues where the system becomes unresponsive, from what I can tell cgateweb can't read changes to C-BUS, but can write. I'm now using this fork on GitHub and if you want to try, swap out section 4 code as follows:
    Code:
    cd /usr/local/bin
    sudo git clone https://github.com/rbhr/docker-cgate-mqtt.git
    mv docker-cgate-mqtt cgateweb
    sudo chown -R pi:pi cgateweb
    cd cgateweb
    sudo npm install
    
    (Take npm advice on screen)
    
    sudo npm install forever -g
    mkdir log
    nano settings.js
    
     
    theboyg, Jan 9, 2022
    #3
  4. theboyg

    MichaelD

    Joined:
    Apr 3, 2016
    Messages:
    46
    Likes Received:
    0
    Location:
    UK
    I'm using cgateweb to send and receive messages through my MQTT broker, it does become unresponsive, and can eventually choke C-Bus altogether so even the lightswitches don't work.

    I'm using a second instance of cgate to check the C-Bus log through a CNI, and what appears to happen is HomeSeer3 will send a single 'on' for a c-bus group, but somewhere between the HS3 and the C-Bus network, that single on command becomes 30+ on commands. I have a night-time HS3 housekeeping event that sets a whole series of groups to their default value, so about 40 groups being set one after the other, and that can take over 5 minutes, with the whole c-bus network being overwhelmed by hundreds of repeated messages.

    The only way I've found to fix this is to unplug the Pi then plug it back in again, then it will all work OK for a few weeks, then by about 4 weeks or so, it will be back babbling repeated messages onto C-Bus and the Pi needs powercycling again.,
     
    MichaelD, Jan 9, 2022
    #4
  5. theboyg

    Pian

    Joined:
    Jan 24, 2022
    Messages:
    2
    Likes Received:
    0
    Hello
    I have been using the wonderful cgateweb for several years now on various platforms, and though I'd chip in with a few thoughts.

    (As an aside, I was one of the users who persuaded Home Assistant to change theor mqtt light implementation to work with cbus, back in 2017.)

    I don't think there is a need for multiple devices - one pi should be able to cope with all of this if you don't have heavy Home Assistant requirements. That said, i migrated to an i3 NUC a couple of years ago and haven't regretted it.

    I strongly suggest that you look into using Docker to run each of these programs. It keeps them separate from the base operating system, and makes things easier to debug and maintain. Personally, I have individual Docker containers running (among others)
    - ser2sock - to interface between the USB on the NUC and the serial port on the CNI
    - cgate - to manage the c-bus system
    - cgateweb - to interface with mqtt
    - an mqtt broker
    - of course, home assistant
    - (I'd also suggest that you look at including Dozzle to see logs, and maybe Portainer to help with container restarts)

    Aside from the usual fun and games when HA does an update, this setup has proved to be as stable as you like for me. And certainly easier to manage and see what is going on that my previous collection of pi's

    I am happy to offer assistance and share my docker-compose setup if it is felt useful - just holler
     
    Pian, Jan 24, 2022
    #5
  6. theboyg

    theboyg

    Joined:
    Apr 3, 2005
    Messages:
    210
    Likes Received:
    9
    Hey, @Pian

    I'd happily run all of these as Dockers. I did try the cgate docker I found, but it was very unreliable using an old cgate build and Toolkit connecting to it from a Windows machine caused all sorts of errors to pop up which I'd never seen before, either running locally or to the Pis natively. So I deleted that and have not found another.

    If you have a list of the docker images you run, or share your compose script, please let me know, I'm up for testing them out. Home Assistant has it's own Pi 4 and I don't run any other containers on it, I'll leave that alone.

    Thanks!
     
    Last edited: Jan 24, 2022
    theboyg, Jan 24, 2022
    #6
  7. theboyg

    Pian

    Joined:
    Jan 24, 2022
    Messages:
    2
    Likes Received:
    0
    Hi @theboyg

    Although it is now two years old, the docker image steppinghat/cgate-server:latest works just great for me. It is using v2.11.4 of c-gate. I have it using host networking, as I was being lazy when I started using it and have never gotten around to making it more secure.

    For the other two (ser2sock and cgateweb) I built my own containers. The Dockerfile for cgateweb (once the cgateweb.service and index.js file for cgateweb are in the same folder) is simple enough:

    Code:
    FROM node:8-alpine
    WORKDIR /app
    COPY . /app
    RUN npm install
    CMD node index.js
    For ser2sock is it ever so slightly more complicated. I use a entrypoint.sh file which is
    Code:
    #!/bin/sh
    set -e
    
    # Exit with error if SERIAL_DEVICE wasn't provided
    if [ -z "${SERIAL_DEVICE}" ]; then
      echo >&2 "You must specify the SERIAL_DEVICE environment variable."
      exit 1
    fi
    
    exec ser2sock -f -p ${LISTENER_PORT} -s ${SERIAL_DEVICE} -i 0.0.0.0 -b ${BAUD_RATE}
    
    and a Dockerfile which is
    Code:
    FROM alpine:latest
    
    # Set environment variables.
    ENV \
      PAGER=more \
      LISTENER_PORT=10001 \
      BAUD_RATE=9600
    
    # Install packages.
    RUN \
      apk --update add \
        build-base \
        git \
        libressl-dev && \
      rm -rf /var/cache/apk/*
    
    # Install the ser2sock application.
    RUN \
      cd /tmp && \
      git clone https://github.com/nutechsoftware/ser2sock.git && \
      cd ser2sock && \
      ./configure && \
      make && \
      cp ser2sock /usr/local/bin/ && \
      cp -R etc/ser2sock /etc/ && \
      cd .. && \
      rm -rf ser2sock
    
    # Add files to the container.
    COPY ./entrypoint.sh /docker-entrypoint
    
    # Set the entrypoint script.
    ENTRYPOINT ["/docker-entrypoint"]
    
    # Expose the listener port
    EXPOSE ${LISTENER_PORT}
    
    Let me know if you meed any help with configuring, but it should be pretty straightforward
     
    Pian, Jan 25, 2022
    #7
  8. theboyg

    cheekymonkey

    Joined:
    Mar 10, 2013
    Messages:
    19
    Likes Received:
    1
    Location:
    Perth, WA
    Just a note if someone wants to build their own version I found index.js needed to be updated to work with the newer mqtt.js node version

    The fix is around line 36 comment out the original
    //var client = mqtt.createClient(mqtt_url.port, mqtt_url.hostname,OPTIONS);
    and add this
    var client = mqtt.connect({port: mqtt_url.port, host: mqtt_url.hostname , keepalive: 10000});

    though in the end I bit the bullet and dipped my toe into the docker pond and grabbed https://hub.docker.com/r/rjmidau/cgate-mqtt. as it seems to have some modifications to work with Home Assistant which is somethings else I need to get across
     
    cheekymonkey, Apr 15, 2022
    #8
  9. theboyg

    cheekymonkey

    Joined:
    Mar 10, 2013
    Messages:
    19
    Likes Received:
    1
    Location:
    Perth, WA
    Just found and fixed a gotcha that was missed when the retain flag was added to
    https://github.com/the1laz/cgateweb.git and doesnt seem to have been picked up and fixed in the docker container I used

    In it line 66 needs to be changed from

    client.publish(msg.topic, msg.payload)
    to
    client.publish(msg.topic, msg.payload, options)

    Not knowing anything about node and why the existing code wasnt working I may have done something that is less that perfect but it works

    This error meant messages learnt actually being published with the retain flag set so everytime HomeAssitant was restarted the cgateweb instance needed restarting
     
    Last edited: Apr 21, 2022
    cheekymonkey, Apr 21, 2022
    #9
  10. theboyg

    Grant

    Joined:
    Jun 13, 2022
    Messages:
    1
    Likes Received:
    0
    I have tried to add the username and password back to the update above but just don't quite know enough, is any one able to help and have a go???
     
    Grant, Jun 13, 2022
    #10
  11. theboyg

    MtSamsonite

    Joined:
    Jul 28, 2017
    Messages:
    6
    Likes Received:
    8
    Location:
    Brisbane
    Another option for getting your cbus devices into Home Assistant is to use cmqttd instead of CGATE. I made a few minor changes to the existing cmqttd docker containers so it could run as an add-on under Home Assistant Operating system. I did this for a friend so he doesn't need to manage any hypervisors / docker hosts etc. as all HA OS and App updates are via the browser and what happens underneath the hood he need not concern himself with.

    If anyone is interested in testing the add-on for HA and taking it for a spin you can find it here: https://github.com/MtSamsonite/ha-cbus2mqtt
     
    MtSamsonite, Jul 18, 2022
    #11
    BrianD, Damaxx and theboyg like this.
  12. theboyg

    BrianD

    Joined:
    Jul 31, 2021
    Messages:
    2
    Likes Received:
    0
    I have a config up and running as a home assistant add on using cgateweb, ser2sock and cgate (all in the one add on) with a serial interface into the c-bus. @MtSamsonite solution with cmqttd looks interesting, so giving it a try, but getting an error `
    PermissionError: [Errno 1] Operation not permitted:` on my USB-Serial_Controller, which I'll raise as an isse over on github.
     
    BrianD, Aug 19, 2022
    #12
  13. theboyg

    MtSamsonite

    Joined:
    Jul 28, 2017
    Messages:
    6
    Likes Received:
    8
    Location:
    Brisbane
    Thanks Brian. I have only had feedback from CNI users so your is the first serial reported to me. I will investigate to see if I can fix it, please post the startup log for me with the issue in github. I'm not a developer so cant promise I will get far.

    For you and others to be aware the feedback has been otherwise positive however the cmqttd process does have a problem with many devices turning on and off. For this reason I was also looking at cgate and cgatweb as addons.

    I am however preferring to run them as separate containers given they are low overhead. It means I don't have to fail the whole container if one process dies, easier for users to configure, each is logging separately in home assistant and easier to diagnose which is not working. I have built them, just getting them tested first as local builds. If anyone wants to test them for me pm me. I started the ser2sock one also and will finish it in the next little bit when I get some available time. Your method with one container sounds good which if I was running for myself I likely have done also.

    I will look into the serial interface issue with cmttd add-on and see if I have done something wrong (most likely). In the interim if you still want to test the underlying cmqttd process you could maybe use your ser2sock instance if you can turn off the cgate process.
     
    MtSamsonite, Aug 19, 2022
    #13
  14. theboyg

    MtSamsonite

    Joined:
    Jul 28, 2017
    Messages:
    6
    Likes Received:
    8
    Location:
    Brisbane
    @BrianD and for anyone interested. I think it was a rookie mistake with my first addon not enabling device access to the container properly. I have made a couple of changes which from what I can read should allow the serial access now. (unfortunately I can't test). I have published an update if you want to try again, understand if you don't.
    Thanks again for giving it a go and providing the feedback.
     
    MtSamsonite, Aug 23, 2022
    #14
  15. theboyg

    MtSamsonite

    Joined:
    Jul 28, 2017
    Messages:
    6
    Likes Received:
    8
    Location:
    Brisbane
    @BrianD and anyone interested, just to advise that the add-on has now been tested with serial interface and it is working now. My apologies for the error.
     
    MtSamsonite, Aug 25, 2022
    #15
  16. theboyg

    towbar

    Joined:
    Mar 2, 2006
    Messages:
    37
    Likes Received:
    0
    towbar, Sep 1, 2022
    #16
  17. theboyg

    towbar

    Joined:
    Mar 2, 2006
    Messages:
    37
    Likes Received:
    0
    towbar, Sep 1, 2022
    #17
  18. theboyg

    BrianD

    Joined:
    Jul 31, 2021
    Messages:
    2
    Likes Received:
    0
    Hi Everyone,

    I have put together a Home Assistant Add-On that uses C-Gate, CGateWeb and Ser2Sock. I'm running on a RPi with a USB to Serial converter.

    I've been using it for several months now and it has been rock solid.

    My installation controls C-Bus connected blinds, lights, exhaust fans and light level sensors. About 60 devices in total.

    I can offer limited support, but no warranty or guarantees of course, but I'd like to put it out there for collaboration to hopefully make some improvements and at least share what is working for me. I know I've got a lot to learn about making HA Add-ons and C-Bus, so as I said, not guarantees.

    You can install it (at your own risk) by adding my repository to your add-on store. https://github.com/bdnstn/HA-Addons

    I hope someone can find this useful. It can no doubt be improved upon, but it is working for me.

    I also created a simple python program to create Home Assistant entity (and switch and light) YAML files from a C-Bus project file. It can save typing and makes it easier to keep changes on your C-Bus in sync with Home Assistant. It's a simple program, you'll need to edit in your project file name and edit the filters used to distinguish lights from switches. It's not a HA Add-on, but a standalone program. You can get it from here: https://github.com/bdnstn/cbusExtractEntities

    Good luck.
    Brian
     
    BrianD, Jan 16, 2023
    #18
  19. theboyg

    Ingo

    Joined:
    Dec 2, 2006
    Messages:
    290
    Likes Received:
    1
    Location:
    South Africa
    Hi All,

    I installed @MtSamsonite 's HA add-on and it seems to work fine for the lighting application and Switches.

    Unfortunately I cannot Ramp over X seconds to 0 using an external application that sends a 'Ramp to 0% over 4s' command. It sets the Brightness down to 0 but keeps the light State On until I Switch it Off.

    Would it be possible to show the MQTT object as Off when a Ramp to 0 command over X seconds is received? Most of my lights are Ramp in 4s meaning after 4s of sending the command the light object should show Off and not just Brightness 0 but still On.

    Secondly, I would love to have the Security Application supported on this as all my PIR's are using the Security App. and then I can monitor motion and all door and window states. Would that be possible to include?

    Ingo
     
    Ingo, Jan 21, 2023
    #19
  20. theboyg

    damian.flynn

    Joined:
    Jan 21, 2008
    Messages:
    26
    Likes Received:
    1
    Location:
    Ireland
    Took the dust off, and started first getting my hands dirty on the old cgateweb application. I ended up with a big refactor

    some Features
    * Build containers for all the parts
    * Refector the C-Gate to MQTT bridge
    * Supports the C-Bus project file
    * Discovers all Lighting Groups and Publishes for MQTT Discovery
    * Only Defined Groups Published
    * Home Assistant configuration automatically for
    * Group Names from Project
    * EntityID locked to network ID
    * Relays, Dimmers and Phantoms Auto-Detected and Configured
    * Extended Attributes for Outputs -
    * Unit Type, Index, Model, and Channel number
    * Trigger Application
    * Published as new Home Assistant MQTT Event
    * Trigger names and groups published

    Bug fixes, etc.

    https://github.com/DamianFlynn/cgate-mqtt

    some additional details here: https://community.home-assistant.io/t/clipsal-c-bus-add-on/96034/64?u=damian.flynn
     
    damian.flynn, Aug 22, 2023
    #20
    Damaxx likes this.
Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.