Category Archives: HMI

Tool Design Concept

I tend to stick to a single, simple concept as I design tools regardless if this is Compiler, Assembler, HMI Designer or something else – I create a tool, a separate Repository and separate output generators as illustrated below:

This means that the tools (like HMI Designer) will store it’s output in a repository database which is in XML format. As this can be read and manipulated by several tools it allow us to easily create a larger, integrated tool package with multiple design tools and code generators working together. This is how modern CASE (Computer Aided Software Engineering) Tool should work.

A second, key prinsiple is that source code is stored in ASCII format editable by simple editors. In this case the repository is stored as XML. The reason is simply that you need ways to recover your project manually if files or tools screw up.

Some years ago I worked on a new tool that also stored it’s work as XML files. One of my projects was a total of ca 1300 files with cryptic content. And as this was a new, unstable tool created in Eclipse it crashed ca every 2-4 hour of usage destroyed it’s XML files. Recovering this manually was possible, but not realistic so I ended up manually backing up everything every 2nd hour. A file system is an excellent undo system and it cost very little to add a backup folder where the last n versions are saved. This way, if the project screw up you can chose to pick up the last saved version. Tools can easily do this automated, something that saves the day if disaster strikes.

Graphical Programming Languages

One of the objectives with creating a diagramming framework in Qt is to be able to create some tools that can support classic desktop look & feel and be re-deployed on Web, tablet’s & phones. I want to take advantage of some experiences from Web based technology, but as mentioned before – I develop network centric control systems, not stand-alone devices.

Creating a HMI designer that makes it easy for anyone to create advanced user interfaces with little or no specialized GUI/Web development experience is the first step. The next is to mature this and build our own tools to support Plain/easyIPC.

One of my objectives is however a fully graphical programming language to sit on top of Plain. Obviously we need something like UML class diagrams or classic ERD to model data, but I also want to program logic using graphics. Tools like LabView or classic CASE tools already does so with some success.

The challenges with any graphical programming language are however often a surprise to many.

A graphic diagram makes things look easy and can easily create a delusion that using the tool also is easy. As most programmers will tell you even the simplest text editor is far faster than any graphic editor in creating logic. Your programming speed will initially drop by a factor of five, meaning that before you can benefit from increased productivity factors you need to compensate for this major disadvantage.

Choice of graphic language is often poorly made. What I will do is to introduce a new language that actually is not new at all, it’s one of the first languages that actually was created. A language that is instantly understood by anyone that see it. Classic flowcharts have standard limitations, so we need to create our own standard to achieve our objectives.

Readability of executable code is seldom addressed. Having a diagram is nice, but if you end up clicking in and out of screens to see the actual executable logic you have introduced yet another limitation that will slow down an experienced developer. I don’t claim to have a verified solution to this one, but I have ideas and awarness of the challenge.

Integration with a text based version is often not covered. Graphical programming tools are experimental and can get in your way because of of unintended limitations. Access to a text editor is often a life saver. The solution is to support Plain blocks directly as well as allowing the user to edit the actual XML used in the repository if need be. This simple trick will ensure that the graphic parts extend – not limit – a text based IDE. Also – we should not under-estimate the combined effect from hybrid tools.

As for the popularity of CASE tools in the past – I believe that the main reasons we see so little of them are lack of standards and lack of affordable tools.

HMI Designer Annotated

This premature screenshot shows an early version of the HMI Designer. The designer itself is quite simple, you use the menu/commands on top to create a “diagram” and the design tools at left will come up. In this case we create a HMI, but I will re-use the framework for other diagrams later.

An HMI (or GUI screen if you like) is straight forward. You just draw the HMI components you want into the screen and assign names to them. This example only show a few rectangles and ellipses. To edit the properties you click on a symbol and change the properties using the property editor at right.

A tab field on top (not shown) will allow you to select between HMI screens as you design a project.

This concept should be well known for anyone with experience from Delphi, Visual Basic, C# etc. I want to just start with a simple HMI Designer for now. The objective is to create the HMI framework, XML format, HMI Designer and HMI Browser to an usable stage.

I will use Plain as a scripting engine capable of processing HMI events locally. In a later version I will add support for experimental diagram logic and see how that goes. I will write a separate entry about the ups- and downs- using a graphical programming language later.

HMI Browser will in reality be an app that starts up with either a file- or communication path. It can either retrieve the XML from a file or have it sent up from a device.

As for time-line an application of this nature will take some hours to get right. As it also have to be done as I teach myself Qt and are occupied in very tight projects elsewhere it will be something we need to focus on for a while. I do however have plenty of unfinished electronics that we can play with if I need a break.

My 1st Qt App

Voila – it’s not much to write home about, but it’s a Qt OpenGL app displaying 4 empty toolbar areas (colors for fun) and some 1000 symbols.

The black area (left) will be the toolbar, the yellow at top will be common tools, red (right) will be property editor allowing editing of selected symbol. The green(bottom) – console output. The area in the middle is the drawing area for the forms.

Not bad for someone who started with Qt 3 days ago!

Qt 5.9.3

Qt & Qt Creator have evolved over the years. I remember giving the tools a go some years ago and ditching them because they did not install & work properly. This is still the issue with some of the packages, so you need a bit of time & patience to get started on Qt.

One of the stronger concepts in Qt is Qt Quick/QML. It allows you to create stunning graphics with little work. This concept is not so unlike the one I am planning myself and well integrated into QtCreator. The strongest part of Qt is however it’s integration into GPU technology that allow you to create high performance graphics applications with ease. Again – usage here requires a bit of time, patience and willingness to learn. That said, if you have done graphics on other platforms you will soon embrace the easy and power that comes with Qt. It is worth the time invested.

HMI Designer

One of the applications I want to create is a “HMI Designer”. This is technically very easy as I have done similar applications before. What I will do is to create a graphical application that can design square objects on a form. Each object will have a name that we can use for IO through a serial line. To make the designer a bit more WYSIWYG we animate the objects as if they where “furniture” like Text Edit, Graphic Plot, Grid, Button etc. Each object will have a list of properties that are set/get design and run-time. The result is stored as an XML file and executed by a “HMI Browser”, a special application that download the XML and interact with the system.

This way we can design config HMI’s stored in the embedded devices as XML files and we only need to re-create the HMI Browser to support HMI on Windows, Linux, OSX, Android etc. The concept is not so unlike a Web Browser with the exception that the server here can be small embedded system in a closed loop. Nextion are selling displays build on a similar concept, so in time we can do our own embedded graphics displays as well. I do however want to do tings a bit different from Nextion.

One of the differences is usage of Plain as “scripting” language and interfacing using easyIPC and RSX as well as Ethernet/Wifi. The other difference is that I want a significantly larger object library and take advantage of GPU’s on Raspberry PI etc. I also want this to be a portable software package, not locked to proprietary hardware.

The most important difference is however system thinking. I want HMI Designer & HMI Browser to be building blocks in the CASE (Computer Aided Software Engineering) Tool I want to create on top of Plain. For now we will use Plain and the HMI Designer, but I do plan a fully graphical programming concept later.

As mentioned before with my Plain blogs I want to evolve the way we do software based system development. The network I build here have plenty of fun for C programmers and electronic hobbyists, but we need to create infrastructure and tools enabling this for non-specialists.

PLC/Home Automation HMI

I have been thinking a bit about creating a stand-alone HMI component. Basically a terminal with a touch display and optional keyboard, mouse and customized input equipment. The key principle would be that we use a Ethernet/RSX as bacbone for data traffic, while the HMI takes care of graphics, input etc.

This is just some random GIF I found on the net, but an HMI app could consist of a set of standard components where layout is controlled by XML files that are uploaded from the device. Communication could be Ethernet/RSX. Equiped with a Plain VM this could be an interesting concept where HMI logic is executed on the HMI unit and only events/data transferred to/from the rest of the system.

Making the app above might seem complicated, but it’s nothing I have not done before from scratch in C/C++. I am thinking more and more about creating a standard HMI app in Qt since that can run on Windows, Linux, Android, OsX etc. I need to check around for options.


Since we are talking about a network of stand-alone units we could create a full control center as well.


SDL (Simple DirectMedia Layer) is a 3D graphics engine that uses OpenGL or DirectX depending on platform. To use it is straight forward. You will do a few rectangles on screen with little work.

I am also working with GDI+ on Windows, so I was stunned as to how slow SDL is to start in comparison. I was even more stunned as I was looking for functions to draw circles, arcs and the finer graphics. They simply don’t exist. All I find is a function to do rectangles, point collections, pictures etc.

In simple words you will have to create a proper 2D library on top of this from scratch if you want to use this as a HMI framework. I have actually done similar tasks back in the DOS days so I know that this is a large task to take on.

Don’t take me wrong! Drawing a rude circle is done in a few lines of code. Drawing a high quality graphics one require a bit more work, but doing this with high performance will require a lot of hours.

It will not be my first option!

10 Mhz Logic Analyzer

One of my interests  for some time have been how to create a MSO (Mixed Signal Oscilloscope). I have several scopes, but I lack a Logic Analyzer on any of them.

My first thought was to bit-bang the GPIO of Raspberry PI directly, but as I looked into this I realized that I would not get much speed out of it. The issue is that Linux can’t really sit around bit-banging as it got other tasks as well. To actually do this you would be better of without the OS running.

They have created a 14 channel, 100Mhz Logic Analyzer on Beaglebone Black, but that makes heavy usage of the 2 embedded PRU’s. It would make sense if we could use one core to bit-bang, but I am not sure how to do that without replacing Linux itself.

One channel is typically 2kb of display data for a frame on a 1920×1024 screen. To be ideal we need a minimum of 25 frames per second (fps) and preferable 50 fps. That is ca 50-100Kb per second eight channels. Or more exactly 200Kb/s for 16 channels which again is ca 1.6Mbps and well within what we can expect from the SPI if we use a Hat.

A STM32 F4 can sample rather fast. Clocking at 180Mhz it should be able to do intelligent sampling of 16 channels up to 10M samples per second if the write the code correctly. This will obviously include some logic to trigger a sample run and upload it. As uploading is done by SPI from a DMA it don’t lake MCU time and we just need to fill a buffer and send it. This Application will need to be tight.

10Mhz will be good, but the hard fact is that most of my needs are covered with 1Mhz. The additional win here is that we get to program the analyzer logic so we can add recognition of things like CAN, SPI, I2C, UART etc.  I am considering two MCU’s for this job:

STM32F405 ticks at 180Mhz and is a M4. Cypress 5LP is a ARM M3 ticking at 80Mhz, but it contains programmable logic that might significantly increase the sampling frequency if used correctly.

User HMI Applications


The drawing above illustrate the different components used to achieve a complete application with HMI. We have briefly discussed the internals of the Hat (left) consisting of hardware interfaces or communication drivers, a IO configuration and the SPI driver communicating with Raspberry PI.

RPI have the master SPI driver and two new components that are part of it’s easyIPC server. One is a Data Repository. Basically a database over available features and data associated with it. This database will be in memory.

API Server allows easyIPC to be connected to multiple clients either on the same Raspberry PI or from a different computer. This connect your HMI application to easyIPC as if it was a database of information.

HMI framework is basically the GUI used to visualize real-time data and add user Interfaces.

This is a very brief drawing sop we will consolidate it as we finish it.