Jekyll2019-07-10T00:36:49+00:00https://embeddeddatasolutions.com/Embedded Data SolutionsMake your data useful{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.comWalk-through From CAN Bus to InfluxDB with Grafana2018-02-13T11:00:00+00:002018-02-14T18:25:52+00:00https://embeddeddatasolutions.com/update/Walkthrough-from-can-to-influx-grafana<p><img src="/assets/images/can_pi_pre_layout.jpg" alt="Hardware" /></p>
<p><em>Preliminary…</em></p>
<p>As electronics have gotten more powerful, the need for embedded devices to communicate in a simple, robust, and scalable way has gotten more and more important. The go to network for this has become the CAN (Controller Area Network) Bus. Car manufacturers have been one of the first to adopt the CAN Bus as it is one of the most resilient communication interfaces out there for operating in such a demanding environment. Now we are starting to see CAN crop up in more and more things as the advantages of using a CAN start to win out.</p>
<!--
Historically embedded communications have been handled via either low level embedded interfaces that are hard to interface with at a high level and susceptible to noise such as SPI or I2C, analog buses such as a 4-20mA that require special analog hardware and processing or the SCADA standard RS-485 with MODBUS which can be fussy with many network topologies.
-->
<p>We are going to walk through how to interact with this low level network traffic and bring it out to some higher level storage and visualization tools. This will all be done with free and open source software and reasonably priced hardware for a professional, configurable and widely scalable real-time monitoring and analytics system.</p>
<!-- [CAN IoT Example Project](https://github.com/embeddeddata/caniot) -->
<h1 id="sensor">Sensor</h1>
<p>For this demo we are going to monitor a temperature sensor over a CAN Bus. This is certainly not the recommended or most economical way to monitor the temperature of your house. But if this were part of a vehicle system with multiple other devices interacting and sensors transmitting data, this same setup should suffice to monitor all the additional traffic that would be present.</p>
<p>To get started we need a sensor and there are basically two different approaches now on how to add sensing to your CAN Bus.</p>
<h2 id="custom-hardware">Custom Hardware</h2>
<p>The most economical way to add CAN bus sensing to a project is to add it into a microcontroller that is already going to be in the system. You can get up and going to experiment with an <a href="https://www.olimex.com/Products/Duino/STM32/OLIMEXINO-STM32/open-source-hardware">OLIMEXINO-STM32</a> which already has an on-board CAN Bus transceiver to interface with the integrated CAN peripheral on an STM32. However the CAN Bus does tend to be one of the more complicated peripherals on these types of controllers so we will leave those details for another article. I would suggest using one of the integrated sensors below to get up and going quickly.</p>
<h2 id="integrated-sensor">Integrated Sensor</h2>
<p>These kinds of sensors are a bit more expensive as they have the hardware and MCU built in, but they are very handy and could be perfect for many applications.</p>
<ul>
<li><a href="http://gps4net.com/images/gps4net/products/g4n03rht/g4n03rht_en.pdf">G4N03RHT</a></li>
<li><a href="https://www.stw-technic.com/wp-content/uploads/2015/08/T01_ManualJ1939.pdf">STW Temperature Transmitter T01-CAN</a></li>
<li><a href="http://www.cmr-group.com/products/industrial-engine-sensors/smart-sensor-j1939-connect-can">J1939 Connect Smart Sensor</a></li>
<li><a href="http://www.axiomatic.com/TDTC4.pdf">TDAXTC4 - Axiomatic</a></li>
<li>Others?</li>
</ul>
<p>For this setup we will be using the G4N03RHT from GPS4NET. We will get get into how this sensor works a bit further down.</p>
<h1 id="can-bus">CAN Bus</h1>
<p>A CAN is a <strong>differential</strong> bus interface to an embedded device. It operates similar to USB in order to handle environmental noise, however arbitration on the bus is built in at the signal level. This means CAN Bus requires no other hardware to add a device on the bus, just wire it into your bus and you are ready to go!</p>
<h2 id="typical-bit-rate-vs-bus-length">Typical Bit Rate vs Bus Length</h2>
<p>Here is a table of bit rates and their corresponding bus lengths. This can vary depending on the noise and traffic in the system, but can be a good guideline. Most common bus speeds tend to be 250k, 500k and 1M bit/s.</p>
<table>
<thead>
<tr>
<th>Bit rate</th>
<th style="text-align: right">Bus length</th>
</tr>
</thead>
<tbody>
<tr>
<td>1 Mbit/s</td>
<td style="text-align: right">25 m</td>
</tr>
<tr>
<td>800 kbit/s</td>
<td style="text-align: right">50 m</td>
</tr>
<tr>
<td>500 kbit/s</td>
<td style="text-align: right">100 m</td>
</tr>
<tr>
<td>250 kbit/s</td>
<td style="text-align: right">250 m</td>
</tr>
<tr>
<td>125 kbit/s</td>
<td style="text-align: right">500 m</td>
</tr>
<tr>
<td>50 kbit/s</td>
<td style="text-align: right">1000 m</td>
</tr>
<tr>
<td>20 kbit/s</td>
<td style="text-align: right">2500 m</td>
</tr>
<tr>
<td>10 kbit/s</td>
<td style="text-align: right">5000 m</td>
</tr>
</tbody>
</table>
<p>As this sensor operates by default at 250kbps, we will stick with that.</p>
<h1 id="gateway-device-setup">Gateway Device Setup</h1>
<p>We will be using a Rapberry Pi 3 as the main monitoring device or Gateway Controller. Since the Rasperry Pi cannot directly interface to the CAN Bus on it’s own we will need to use an adapter board. One decent option we will be using for this setup is the <a href="http://skpang.co.uk/catalog/pican2-canbus-board-for-raspberry-pi-2-p-1475.html">PiCAN</a> adapter board that uses an onboard SPI interface to talk to a CAN transciever.</p>
<h2 id="setup-with-raspberry-pi-and-can-board">Setup with Raspberry Pi and CAN board:</h2>
<p>To get started with the Raspberry Pi as a CAN Bus Gateway you need an operating system. <a href="https://www.raspberrypi.org/downloads/raspbian/">Raspian Lite</a> is recommended as you shouldn’t need a terribly complex setup, however a desktop version would work just as well. There are plenty of <a href="https://www.raspberrypi.org/documentation/installation/installing-images/README.md">guides</a> on getting an SD card setup with the image for your OS.</p>
<p>Next you will need something to interface with the CAN Bus. One powerful and rather easy option is to use the <a href="http://copperhilltech.com/pican-2-can-interface-for-raspberry-pi-2-3/">PiCAN2</a> adapter board. There are even versions available that have an <a href="http://copperhilltech.com/pican2-can-interface-for-raspberry-pi-2-3-with-smps/">integrated power supply</a> to power the device from the CAN Bus itself, as well as dual CAN adapters if that is something you might need.</p>
<h3 id="hardware-install">Hardware install</h3>
<p><a href="http://skpang.co.uk/catalog/images/raspberrypi/pi_2/PICAN2SMPSUGB.pdf">Here</a> is a document to walk you through the setup of the adapter board. Basically just press it onto the header, install the standoffs, and screw it together. <strong>MOST IMPORTANT</strong>, be sure to set the solder jumpers as needed, you will likely need to jumper them to work with CAN Cables unless you would like to use this in an OBDII setup.</p>
<h3 id="os-config">OS Config</h3>
<p>Of course now we need to setup Raspbian to detect our freshly installed adapter board!</p>
<p>First we need to update the OS with the typical:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get update
sudo apt-get upgrade
</code></pre></div></div>
<p>Next we need to add in the overlay to interface with the CAN bus via the SPI interface that is driving the CANPi2. Open up an editor to <code class="highlighter-rouge">/boot/config.txt</code> and add the following at the end:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dtparam=spi=on
dtoverlay=mcp2515-can0-overlay,oscillator=16000000,interrupt=25
dtoverlay=spi-bcm2835-overlay
</code></pre></div></div>
<p>That’s it! Save your changes and reboot the Raspberry Pi: <code class="highlighter-rouge">sudo reboot</code></p>
<h3 id="wiring">Wiring</h3>
<p>Of course we need to wire this system up. Each sensing device may be a little different, but typically you will need +12V DC, Ground, and CAN High and CAN Low. Sometimes you will have a separate CAN GND which you should use if available.</p>
<p><em>schematic image</em></p>
<p>The Raspberry Pi will need a matching connections on the PiCAN board for CAN High, CAN Low and CAN Ground (use the sensor ground if no dedicated CAN Ground). The Raspberry Pi itself will just need USB power if it isn’t powered from the CAN Bus, and Ethernet unless you would like to setup a Wifi interface.</p>
<p><em>actual wired image</em></p>
<h1 id="testing-your-can-interface">Testing Your CAN interface</h1>
<h2 id="command-line">Command line</h2>
<ul>
<li>http://www.skpang.co.uk/dl/can-test_pi2.zip
<ul>
<li>(Is this special or different from can-utils??)</li>
</ul>
</li>
<li>via <a href="https://github.com/linux-can/can-utils">can-utils</a>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/linux-can/can-utils.git
cd can-utils
make
sudo make install
</code></pre></div> </div>
</li>
</ul>
<h2 id="can-bus-testing-with-python-can">CAN Bus Testing with <a href="https://github.com/hardbyte/python-can">python-can</a></h2>
<p>This is an optional step and depends on your application. For just this single sensor and some minimal visualizations the python-can library and a simple web based plotter or a terminal interface may be a very usable option on the Raspberry Pi. But for doing some serious data crunching and visualizations, it doesn’t take too much to overwhelm the Raspberry Pi using the tools in this article. Below we will take the data processing requirements off of the Raspberry Pi and let a more powerful computer do the work.</p>
<p><em>Code snippet based on examples to get raw data</em></p>
<h1 id="socketcand-setup-and-configuration-for-raspberry-pi">SocketCANd setup and configuration for Raspberry Pi:</h1>
<p><a href="https://en.wikipedia.org/wiki/SocketCAN">SocketCAN</a> is a standard networking stack that is used to allow access to a CAN bus. The <a href="https://github.com/dschanoeh/socketcand">socketcand</a> project packages this all up and gives us an easy way to setup the interface and run it in the background as a daemon. This will allow us to get the raw data off of the CAN Bus and onto our network.</p>
<ul>
<li>Steps to clone, compile, make, execute</li>
<li>Set to run on boot</li>
<li>SAVE!</li>
</ul>
<p>If everything is working here, now would be a great time to make a backup image of your SD card so you can always come and reuse or restore this working image.</p>
<p>At this point it is possible to stop here and use something like <a href="https://dschanoeh.github.io/Kayak/">Kayak</a>, which is a very powerful open source CAN interface utility. Or we can continue on with an even more flexible and web friendly solution!</p>
<h1 id="setting-up-a-simple-socket-interface-with-python-from-another-device">Setting up a simple socket interface with Python from another device</h1>
<p><em>coming soon</em></p>
<h1 id="scaling-raw-can-data">Scaling raw CAN data</h1>
<p>There are plenty of CAN data formats out there from CANOpen, J1939, and other used for cars such as the OBD protocol stack. Some devices may also define thier own protocol structure or just use a very simple implementation over the raw CAN bus. This is just fine as the general idea is usually the same: Figure out where the data is coming from with the address field then get the raw data bytes and scale as needed.</p>
<ul>
<li>CAN Packets</li>
</ul>
<p>Extended ID vs Standard ID</p>
<table>
<thead>
<tr>
<th>Packet</th>
<th>ID1</th>
<th>ID2</th>
<th>ID3</th>
<th>ID4</th>
<th>D1</th>
<th>D2</th>
<th>D3</th>
<th>D4</th>
<th>D5</th>
<th>D6</th>
<th>D7</th>
<th style="text-align: right">D8</th>
</tr>
</thead>
<tbody>
<tr>
<td> </td>
<td>Temperature</td>
<td><em>define/get protocol</em></td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td style="text-align: right"> </td>
</tr>
<tr>
<td> </td>
<td>Humidity</td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td style="text-align: right"> </td>
</tr>
</tbody>
</table>
<ul>
<li>Sensor Specific setup</li>
<li>J1939 Generic setup</li>
</ul>
<h1 id="sending-data-to-a-tsdb-like-influxdb-and-basic-queries">Sending data to a TSDB like InfluxDB and basic queries</h1>
<p>Now it is time to install InfluxDB on your server or PC!</p>
<p><a href="https://www.influxdata.com/">InfluxDB</a></p>
<h1 id="setting-up-grafana-and-linking-to-influxdb">Setting up Grafana and linking to InfluxDB</h1>
<p><a href="https://grafana.com/">Grafana</a></p>
<ul>
<li>Installation instructions <a href="https://grafana.com/grafana/download?platform=linux">here</a></li>
<li>Once setup and running, goto localhost:3000</li>
<li>Setup an account</li>
<li>Add a datasource</li>
</ul>
<h1 id="cost-breakdown">Cost Breakdown</h1>
<p>For an Industrial Grade CAN Bus Environmental Monitoring setup</p>
<ul>
<li>CAN Bus Sensor
<ul>
<li>Integrated CAN Device: $85-$150</li>
<li>Custom: $30 or less</li>
</ul>
</li>
<li>Raspberry Pi: $30
<ul>
<li>SD Card: $10</li>
<li>Power USB (optional): $5</li>
</ul>
</li>
<li>CAN Adapter Board w/Enclosure: $85</li>
<li>Total : ~$155->$215</li>
</ul>
<h1 id="potential-optimizations">Potential Optimizations</h1>
<ul>
<li>Read-only OS for power safe operation</li>
<li>Larger CAN bus for multi point sensing</li>
<li>Sensor Meta Data?? Location/type/ranges</li>
<li>J1939 detail abstraction with SQLite</li>
<li>CAN Bus Writing and simulation</li>
<li>InfluxDB Data optimization, retention</li>
<li>Filter levels and sampling with Grafana</li>
</ul>{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.comPython in the Middle Serial Port Monitor How to2018-02-10T14:37:44+00:002018-02-10T14:37:44+00:00https://embeddeddatasolutions.com/update/python_in_the_middle_serial_port_monitor<p><em>preliminary article</em></p>
<h1 id="making-a-quick-python-based-serial-monitor">Making a quick Python based serial monitor</h1>
<p>For some reason everyone has seemingly forgotten about the poor old serial port! While you don’t see it very often in consumer electronics, it’s still a very important dependecy in developing most embedded systems. Even if you are just using serial data over Bluetooth and/or USB, it is still important to be able to monitor and/or log that data for making new tools or fixing existing bugs.</p>
<p>There used to be support in Windows for such things with PortMon and there were even a couple free to use programs that got the job done. Now none of those things seem to work with modern operating systems or they want to charge you an arm and a leg for something you only need to use occasionally. Sure, we could do this in Linux with socat or something else, but in the embedded world we are often hamstrung to Windows. So what to do??</p>
<h1 id="pyserial-and-com0com-to-the-rescue">PySerial and com0com to the rescue!</h1>
<p>With a simple and very useful feature hidden away in PySerial and a virtual set of serial ports using com0com, we can get a Python based logging/monitoring application up and running without much impact on the interface. This example will be quick and crude, but is functional and should certainly get you going if you want to make something more robust with threading and such.</p>
<h1 id="setup-com0com-and-operation">Setup com0com and operation</h1>
<p>Com0com essentially gives us 2 ‘virtual’ serial ports that are tied together. Why on earth would we want something like that? This allows us to put a ‘spy’ program in the middle of our serial line without any extra hardware. We can send the serial port from the device to our PC, monitor that stream with our Python script while forwarding out everything we read to the first side of the virtual COM port and have our application connect to the other side of our virtual COM port to operate as normal.</p>
<ul>
<li>Here is the link: <a href="http://com0com.sourceforge.net/">com0com</a> - ewww, yes it is on sourceforge…</li>
</ul>
<p>Seems a little convoluted, maybe a chart will help!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+---------+ +---------------+ +-----+
| Device |---[PC COM]----| PySerial App |--[V COM A]--[V COM B]--| App |
+---------+ +---------------+ +-----+
</code></pre></div></div>
<p>’’’
Code coming soon!!
‘’’</p>{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.compreliminary articlePython Plotting For Embedded Data and Realtime UIs2018-01-10T14:37:44+00:002018-01-10T14:37:44+00:00https://embeddeddatasolutions.com/update/python_plotting_realtime<p>So we have our data file or stream and now we need to visualize it somehow. We are going to focus on using Python to do this as it is one of the faster ways to go, however there are tons of plotting tools out there that could do the job as well. To get started, we need to select a plotting library to use. There are many out there even just in the Python world, but based purely on widespread use and performance we can narrow this down to…</p>
<h1 id="pyqtgraph-or-matplotlib">PyQtGraph or Matplotlib?</h1>
<p>Most people looking to use Python to do any sort of plotting are typically pointed at Matplotlib first. This graphical plotting suite is very power and reasonably straight forward to use, and works great with generating static plots and some light weight dynamic stuff. But it doesn’t seem to hold up well when it is put under heavy load or is expected to stream real-time data. For heavy or real-time lifting PyQtGraph seems to be of the few solutions out there when working with Python.</p>
<p>PyQtGraph certainly has it’s quirks and isn’t always the most intuitive to get up and going. However it is very powerful and can efficiently stream and plot large amounts of data. Also as it’s name implies it is build upon Qt which makes it quite easy to integrate into an existing GUI if you are using PyQt or PySide.</p>
<h1 id="examples">Examples</h1>
<p>(to be continued…)</p>{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.comSo we have our data file or stream and now we need to visualize it somehow. We are going to focus on using Python to do this as it is one of the faster ways to go, however there are tons of plotting tools out there that could do the job as well. To get started, we need to select a plotting library to use. There are many out there even just in the Python world, but based purely on widespread use and performance we can narrow this down to…SQL when you need it2017-10-22T12:37:44+00:002017-10-22T12:37:44+00:00https://embeddeddatasolutions.com/update/SQL-When-You-Need-it<!-- (Excel v SQL) <img src="/images/pic01.jpg" class="fit image">-->
<p>SQL and databases are something that not many engineers tend to play with. But a database can be a very powerful tool when used proplerly. Often it can get the job done much easier and faster than everyones favorite tool… Excel!</p>
<h2 id="using-excel-as-a-hammer">Using Excel as a hammer</h2>
<p>For many Excel is the goto tool in thier toolbox and understandably so. It is easy to pick up and learn and is flexible enough to handle even some of the most complicated number crunching. However the time will inevitably come where it has gone too far!</p>
<h2 id="databases">Databases</h2>
<p>(show what can be done with SQL)</p>
<h2 id="then-what-tools-should-i-use">Then what tools should I use?</h2>
<h3 id="sqlite">SQLite</h3>
<h3 id="python">Python</h3>
<p>(simple example)
(IDE with Visual Studio - link)</p>{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.comSQL and databases are something that not many engineers tend to play with. But a database can be a very powerful tool when used proplerly. Often it can get the job done much easier and faster than everyones favorite tool… Excel!Python for Electrical Engineers2017-08-22T11:30:00+00:002017-08-22T11:30:00+00:00https://embeddeddatasolutions.com/update/Python-for-electrical-engineers<p>Many developers working close to the metal tend to stick to their knowledge of C/C++ to get things done even when not restricted by their platform, but there are much more powerful tools out there to handle some of the other tasks that come up.</p>
<p>One of the most versatile languages in this domain is Python. It can be viewed as the the go-to tool for what could be considered DevOps for Electrical Engineers. Here are a few ways Python can help with hardware development.</p>
<h1 id="device-interfacing">Device Interfacing</h1>
<p>Serial interfaces ( like RS-232 and MODBUS), CAN buses, Ethernet and others are so much easier to setup compared to writing something in native C/C++ for the specific operating system you are using. Network interfacing is also very straightforward to use if you need to open up a socket and talk to a device over a network, such as a SocketCAN interface or a networked power supply.</p>
<h1 id="programming-bench-tools">Programming Bench Tools</h1>
<p>Programmable tools such as power supplies, DMMs, O-Scopes, Function Generators, and others are simple to set up and run. There are already many libraries out there provided either by open-source projects or by the tool vendors to allow you to automate the data capture or device configurations allowing you to do some really neat things such as Automated Testing.</p>
<h1 id="automated-testing">Automated Testing</h1>
<p>It is much cheaper and faster to automate testing with Python than with expensive and inflexible systems like Labview or ATEasy. With a simple serial interface to change data on the controller, programmable test equipment and a creative harness with feedbacks or another sensing device, the integrity of the hardware can be verified. Many production test system don’t need the in-depth verification levels that you can get with a Labview setup. You can easily get 80–90% of the test coverage for about 5–10% of the cost. Of course, if you really need that extra 10%, then that extra cost might be justified.</p>
<h1 id="software-testing-and-simulation">Software Testing and Simulation</h1>
<p>If your code base is reasonably modular and decoupled from the hardware to the point you can build and run parts of it on on your computer, ctypes might be an option. Ctypes allows you to interact with compiled C code using Python! This opens up the possibilities of mocking your hardware, setting up automated unit tests, or modeling the system interfaces using higher level tools. While this setup has the potential to be quite powerful, it can become overly complicated to setup and maintain for smaller projects.</p>
<h1 id="embedded-data">Embedded Data</h1>
<p>Embedded data itself is also something that is often overlooked. It requires serious tools to parse and visualize at millisecond resolution from hundreds or thousands of sources on a device. Some data sampling can easily rival the amount of data generated by “Big Data” companies/apps. Often times this data just gets dumped to a CSV file, Excel, or some other cryptic text format.</p>
<p>Don’t be scared to give SQL a try… Consider using a SQLite database to get started and see how powerful it can be to analyze and inspect your data. Even some simple SELECT statements can pull out some useful insights.</p>
<p>If you want to get really ambitious you can even send that data up to a Time Series Database (TSDB) for real time streaming and visualizations. There are tons of databases out there that can be a very useful tool in analyzing and visualizing large sets of static or real-time data.</p>
<h1 id="analytics">Analytics</h1>
<p>Libraries like SciPy, Numpy, and Pandas are very useful when you need to crunch large data sets to pull useful information out. You can look to the Big Data world for more details on this, but Python has certainly carved out its place in the world of data science.</p>
<h1 id="visualization">Visualization</h1>
<p>Matplotlib works great for quick static plots of moderate data and complements the analytics tools. There is also PyQtGraph which works great for high resolution or real-time plotting. Even implementing a time-series database with tools traditionally used by DevOps, such as Grafana, can be very useful for doing ad-hoc queries or setting up a monitoring system.</p>
<h1 id="day-to-day-tasks">Day to Day tasks</h1>
<p>There are also plenty of other day to day uses for Python such as code generation, automating builds, and pulling data from Excel. Using a Python template library like Jinja makes automating code generation for repetitive or dynamically configured code sections very simple.</p>
<p>Most engineer’s tool of choice whether you like it or not is Excel, and Python makes it very easy to pull data in and out of these sheets. Not every engineer is going to be a programmer, so Excel becomes a very important part of some organization’s workflow.</p>
<p>Putting all of this together can give some very powerful potential when in the right hands! While there are plenty of other languages that can do some of these things faster, better, or possibly easier, I’d challenge someone to find a language that can do all of these things as well as Python.</p>{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.comMany developers working close to the metal tend to stick to their knowledge of C/C++ to get things done even when not restricted by their platform, but there are much more powerful tools out there to handle some of the other tasks that come up.Is Embedded Data Big Data?2017-04-22T14:37:44+00:002017-04-22T14:37:44+00:00https://embeddeddatasolutions.com/Is-embedded-data-big-data<p>So are embedded systems considered <strong>Big Data</strong>? Typically you see <strong>Big Data</strong> used as a term for a large amount of user or business data that is acquired via some sort of web portal and processed to get more useful information and/or money out of it. This data can span Peta-bytes of storage and can be gathered from millions of sources and or users.</p>
<!--<img src="/images/pic01.jpg" class="fit image">-->
<p>But what about your you hardware? Could the data coming out of your little embedded controller be consider “Big Data”? It may not seem like much, but what an embedded controller may lack in data sources it can more than make up for in resolution. A single device can easily generate more data than we know what to do with even with a single data stream. While there may not be quite the large, widespread sampling of data sources that a distributed website/network may have, we do have very high speed data sources that we need to see and track in real-time to find any bugs or glitches in the system. That could mean we would have sample rates of well under 1ms. With even 100 sources at about a 1ms sample rate we can very quickly start to run into some of the similar issues you may see with Big Data although with some slightly different problems to consider.</p>
<p>Things like SQL start to become increasingly important as Excel will start to fall apart with these large growing data sources. Some things like a time series database (TSDB) may even be worth investigating.</p>
<h1 id="resolution">Resolution</h1>
<p>Typically resolution will be limited by the interface you decide to use to gather your data.</p>
<ul>
<li>Serial Port</li>
</ul>
<p>Typically see these ports used at around 57.6k baud, but you can certainly run them well over 200k baud with the right hardware. These are often used via an RS-232 interface for updating firmware or running a GUI and can be run over a USB converter that is either external or integrated into the controller. You often see MODBUS implementations using a similar RS-485 interface for industrial controls applications.</p>
<ul>
<li>CAN Bus</li>
</ul>
<p>Has many similar uses to RS-485, but is much more robust in the hardware implementation although may be slower. Typically limited to about 1Mbps or less depending on the application and wiring distances. These interface have similar cards and device interface as RS-232 and RS-485, however they are not quite as standardize on how to access the data. See [CAN Bus CAN Do] for more information.</p>
<ul>
<li>JTAG</li>
</ul>
<p>May have the highest resolution of all the methods, however is also likely the hardest data to get at. Every microcontroller, JTAG variant and development tool can have it’s own intricacies to handle in gathering the data. However you can’t beat talking directly to the diagnostic bus for speed when high resolution is required. Sometimes it may be best just use the development tools provided depending on what is being done.</p>
<!-- Ethernet -->
<h1 id="storage-and-retention">Storage and Retention</h1>
<p>There are many ways and places to dump your stream of data, however your application will likely define how to do this.</p>
<ul>
<li>Raw CSV Files</li>
</ul>
<p>Very simple and straightforward. You can even dump raw data to a csv file and figure out what to do with it later if you don’t have the tools in place to handle it.</p>
<ul>
<li>SQLite Database</li>
</ul>
<p>Putting your data into a database format without the need for a server can be very useful. SQLite is pretty much just a storage format, but you get all the SQL goodness that will let you query up your data and search quickly.</p>
<ul>
<li>SQL Database</li>
</ul>
<p>A proper database may be overkill for many low level endeavors, but if you want to share your data or you are truly dump a lot of data, a proper SQL database server will be very useful. There are many choices in this area, but that could be it’s own write up.</p>
<ul>
<li>Time-series Database</li>
</ul>
<p>A time series database is a rather new option, but it does fit very well into the embedded systems world as it’s all time series data. There a many tools out there that can make handling data very easy once you manage to get the data into a TSDB and it may be seriously worth considering depending on your application and orgainzation’s needs.</p>
<h1 id="analytics-and-visualization">Analytics and Visualization</h1>
<p>Coming soon!!</p>
<ul>
<li>Testing and verification</li>
<li>Excel, Grafana and others</li>
</ul>{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.comSo are embedded systems considered Big Data? Typically you see Big Data used as a term for a large amount of user or business data that is acquired via some sort of web portal and processed to get more useful information and/or money out of it. This data can span Peta-bytes of storage and can be gathered from millions of sources and or users.Web or Desktop Application2016-09-22T14:37:44+00:002016-09-22T14:37:44+00:00https://embeddeddatasolutions.com/update/web-or-desktop<!--<img src="/images/pic01.jpg" class="fit image">-->
<p>Should your application be a web based solution or a desktop application?</p>
<p>So you have a data interface up and going to your embedded and system and now it is time to build a front-end, what do we use? That all depends on what you are going to do with your system and how you would like to monitor it.</p>
<p>One of the biggest things to consider is likely: What do you consider real-time data? If you need to see the lots of data on the screen update as soon as it changes in the system, a web based UI might not be the best way to go. If you don’t care about instantaneous data and are looking to do a bunch of historical data analytics and parsing through a user friendly interface, there may already be some tools out there to do the job for you. If you need something else, it can be a little blurry on what is best. <!-- Grafana and other devops tools link--></p>
<p>If you really need to plot things very quickly and ‘live’ then Qt may be the right tool to use. There are plenty of libraries
out there that can give you a very responsive interface and plotting with minimal amount of hassle. There are even quite
a few Python plotting libraries that are more than capable as well.</p>
<p>If you are looking to handle large amounts of data from multiple sources a web based approach may be a good route to go,
even if you don’t plan on doing an ‘Internet of Things’ type of setup or storing your data in the cloud. Just having a
local server setup that can crunch and store data into a database to be streamed, logged, and analyzed as needed can be
very useful.</p>
<p>Deploying your application to the customer can get interesting depending on the route you take. The PC application is much more traditional and easier to
package, but more and more companies have been going to the cloud with a hosted solution and this can certainly work
with the right system in place. But if you just need to get an application to your customer and don’t want to worry
about the overhead of managing users and servers, maybe just a PC app is the way to go. There are also ways of
distributing hosted solutions that could be feasible, but you don’t see too many out there. Example would be running the
server and/or back-end on the customer’s PC essentially decoupling the UI form the processing, this could be useful for other
reasons as well. <!--[redis and decoupling link]. --></p>{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.comShould your application be a web based solution or a desktop application?Hardware in the loop… What loop??2016-09-19T14:37:44+00:002016-09-19T14:37:44+00:00https://embeddeddatasolutions.com/update/Hardware-in-the-loop-what-loop<h2 id="hardware-in-the-loop">Hardware in the Loop</h2>
<p>You may have heard of <a href="https://en.wikipedia.org/wiki/Hardware-in-the-loop_simulation">Hardware In the Loop</a> (HIL), but at first glance it looks like something that is just reserved for fancy math nerds
or engineers in big companies with access to Matlab and the crazy expensive packages that are required to do HIL.</p>
<h1 id="so-what-is-it-exactly">So what is it exactly?</h1>
<p>Hardware in the loop is quite literally putting your controller inside another control loop. This
allows an external system to adjust and stimulate your control to optimize it’s performance. Say you have some sort of
PID loop that needs a lot of finicky manual calibration that often needs to be done by a human. Once this device is able to be
automatically controlled by an external system with more resources, that system can find those calibration constants to give
you the best response possible. This can often be done by trying hundreds or thousands of combinations of variables to
find the right ones.</p>
<h1 id="why-should-i-bother-to-implement-it">Why should I bother to implement it?</h1>
<p>Of course this path may not be the way to go for every project… One offs or things that are primarily digital like a display may not be terribly useful to put into an outer control loop. But even if you have a moderate volume on a device that may have lots of different systems it could be deployed on, you could certainly save some significant time and effort by getting some form of a HIL system up and running.</p>
<h2 id="getting-started">Getting started</h2>
<p>Coming soon!!</p>
<ul>
<li>Setup and interface</li>
</ul>
<h2 id="override-your-control-system">Override your control system</h2>
<ul>
<li>Interfacing</li>
<li>Protocol
<h2 id="send-a-stimulus">Send a stimulus</h2>
</li>
<li>Change an output on the controller</li>
</ul>
<h2 id="using-more-powerful-tools">Using more powerful tools</h2>
<ul>
<li>Scipy</li>
</ul>
<h2 id="monitor-feedback">Monitor feedback</h2>
<p>Hardware in the loop or HIL is a great feedback setup that allows you to directly control your hardware and verify functionality without having to write a ton of low level code to get things going.</p>{"name"=>nil, "avatar"=>nil, "bio"=>nil, "location"=>nil, "email"=>"info@embeddeddatasolutions.com", "uri"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>nil, "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>nil, "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>nil, "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}info@embeddeddatasolutions.comHardware in the Loop You may have heard of Hardware In the Loop (HIL), but at first glance it looks like something that is just reserved for fancy math nerds or engineers in big companies with access to Matlab and the crazy expensive packages that are required to do HIL.