Connection pools and the cloud...how it can go wrong!

I will tell you the story starting from the very beginning...

In a project I'm working on in Voverc, we are running a bunch of microservices on Docker Swarm and at the beginning we decided to use Rexray (vs dead-end state project Flocker) to manage Docker volumes and their orchestration. As you certainly know, Docker Swarm is very powerful to orchestrate stateless containers but when you have a storage attached to your container (e.g. a MySQL database) and you want to move it to another machine, you have to manage the volume replication, mounting and failover by yourself. There is no integrated automated mechanism to solve this kind of issue. For this reason we have chosen a volume orchestrator like Rexray; it looked promising at the beginning, given the size of the community and the large support for many cloud storage providers (AWS, Google, Azure, etc). Shortly speaking, after some months of production use we faced a lot issues, in my opinion mainly due to Azure cloud and its integration still not ready for production use. For example we had some long downtime caused by Docker Swarm and Rexray blocked and unable to move a container after a machine restart.
After long troubleshooting sessions and community help we decided to give up with this solution and put in place a MySQL Galera cluster to manage our MySQL databases; instead of having one MySQL container per microservice we put one schema per microservice on the same MySQL cluster (read this post if you want to know more). We wanted to actuate a fast transition to the new database so we provisioned a three node cluster on the same Azure region, accessible via public IP to avoid VLAN networking setup; we don't have a huge load so we thought that performance loss due to the use of public interfaces would be minimal. Actually, the new architecture was performing very well, we stress-tested it, simulated failovers and monitoring load-balancing. Good, ready for production then!

Here the pain begins...

After some days of production use we started to face strange, apparently random, receive timeouts on upstream services calling microservices in the Docker Swarm. When I say strange, I say that some requests, when debugging with Postman, took exactly 15 minutes and few seconds (in production we obviously use low timeouts to fail-fast). At first, we investigated to exclude Docker Swarm routing issues or Galera cluster replication misconfigurations. At some point, I was wondering if the problem could be coming from the connection pool and I switched from Tomcat to Hikari that has better debug logging. I noticed these stranges logs:

DEBUG com.zaxxer.hikari.pool.PoolBase - Closing connection com.mysql.jdbc.JDBC4Connection@229960c: (connection is evicted or dead)

Well, dropped connections! Seen that, I started to do some tcpdump on both sides to check if there was something strange with MySQL TCP connections and it's precisely there that I've got the inspiration...what if everything here is fucked up by the NAT keep-alive timeout? So I checked Azure NAT keep-alive timeout (4 minutes) and reduced CP maxLifeTime and idleTimeout below this number: This in order to tell the CP to kill connections before they were cut for inactivity on the NAT. Magically everything was good with these settings!

Lesson learned: never access a MySQL database on the cloud via public IP or at least consider that there is NAT and keep-alive timeouts!

How to open your gate with your smartphone

How many times do you forget your external gate remote, especially when you go out for running or cycling? Well, for me it happens so frequently that I decided to find a way to open the gate using my iPhone, considering that it is way less probable that I forget it compared to the remote :)

Wow, I have a new Home Automation project to implement...let's do it!

Basically the my conditions / requirements are the following:

• I don't want to tweak my gate board, considering my limited knowledge on its circuitry
• I have WIFI network coverage at the gate
• I want to integrate this system with my existing HA webapp (SmartHome)
• Everything should be cheaper of every other existing solutions 
• Bonus point: I would like to also use Siri on my iPhone and say "Hey Siri, open the gate"

Given the previous points, the first idea came to my mind is to use the existing remote I have and control its activation using a relay, driven by a tiny Raspberry PI Zero W (yes, W stands for wireless, cool!). Activating a relay with a Raspberry PI is quite straightforward using Python and you can find a lot of tutorials on the internet. The only part that requires a bit of effort is soldering the cables to bring out the battery from the remote in order to enable it using the relay. The gate button can be easy blocked on pushed position using a nylon cable tie.

Once you are able to toggle the relay, you can expose this operation using Python Flask web framework and you can integrate the API in your HA platform. In my case I can open the gate using the following shell command:

curl -X POST localhost:5000/toggle-relay?delay=1

As you can see I also added the delay parameter to control the duration of the relay activation (see the code here).

At this point the project is almost complete but, wait, we have the bonus point, I want it :) After googling a bit I've found the promising HomeBridge project. Essentially it allows you integrate whatever device you want with iOS HomeKit. I've looked a bit at the code and writing a custom accessory / platform is really easy, in my case, apart the boilerplate code, it's less than 10 lines of JS (see the code here)!

You can build this smart gate opener with less than 10 euros (price doesn't include the remote and its battery cost :D)...very easy and useful project! See you at the next one!

How to build a mobile robot with less than 200 €

Many of you already know the revolution that Raspberry PI has brought to us in last years, allowing everyone to develop low-cost pervasive computing applications. In this post I would like to show you how to build a mobile robot with less than 200€, presenting the personal project I'm working on called ARPA (Autonomous Raspberry Pi Agent). You can fork the repo here: https://github.com/cecchisandrone/raspberrypi.

Demo video

The main features this robot has are:

• RaspberryPi powered :)
• Four wheels chassis
• Manual control through Xbox 360 wireless controller
• Simple obstacle avoiding using sonar sensors and magnetometer
• Front camera with pan-tilt, led lights, face tracking and remote web interface
• Microphone and active speaker to allow both voice recognition and TTS functionalities

Parts list

I've bought the majority of the robot parts from Amazon and eBay, spending a total of more or less 200€. This is the complete list:

Chassis	26.1	€
RaspberryPI B+	34.99	€
Sonar	1.78	€
Pantilt Camera mount	2.99	€
Servo	6	€
Wifi USB	7.9	€
PiCamera	26.19	€
Motor Driver	4.48	€
MicroSD	4.5	€
Audio cable	1.88	€
Breakout kit	8	€
Speaker	5	€
USB Microphone	1.5	€
Magnetometer	1.41	€
DSI cable	7	€
Mount	6	€
USB battery	15	€
Various Cable & components	5	€
Total		165.72.€

Implementations details

My background is mainly oriented in Java and Web technologies so first I wanted to reuse my knowledge to do something fast. Indeed that's the case since most part of the code is written in Java with a little part done for the face tracking written in Python (because of the better performance of the OpenCV bindings respect to the Java ones). I will not describe in detail all my design and implementation decisions (you can always have a look on the code :P) but I will comment the main choices I've done to implement the robot.

Manual control

When I've received the motor driver and the chassis, the first thing I wanted to try was to implement a RC car, powered by a RPi. I had a XBox 360 wireless controller for that purpose and after a quick test I've seen that it's really well supported on  Raspian thanks to xboxdrv userspace driver. Once the driver is correctly loaded, I've read the device file from /dev/js0 in Java and abstracted the necessary API to manage buttons, analogs, etc. Basically for every new event (e.g. button pressed) a string is written to the device file, containing the values of all inputs; see https://github.com/cecchisandrone/raspberrypi/blob/master/raspio/src/main/java/com/github/cecchisandrone/raspio/input/JoypadController.java for more details about this.
Regarding the motor interface, I've used Pi4j with PWM, you can see the details here but it was quite easy to implement, thanks to the library. After this part, ARPA was capable of being remotely controlled; yes, it's not a robot but this could be very useful when his autonomous decisions are not good :)

Camera pan-tilt control

In order to correctly implement face tracking, I wanted to install a pan-tilt mount for the PiCamera, moved by a couple of servo motors. These motors can be driven with PWM and even if they are really cheap, they are quite precise for this kind of applications. I preferred using ServoBlaster here because of better performance in software-managed PWM (Raspberry Pi has one physical PWM but it can cause conflicts with 3.5mm audio output). ServoBlaster is started at the startup of the RPi (using /etc/rc.local) and I send values to the motor using string commands like echo 0=50% > /dev/servoblaster.
I have wrapped also this device control with Java, you can check the code here.

Very cheap servo motor I've used

Face tracking

When I started to think about this feature, I immediately remembered a library used at the university during some experiments in Computer Vision, OpenCV. It is a C++ library but it has also good porting for Python and Java. I decided to use the Python implementation mainly for its better performances; doing CV on Raspberry Pi is not trivial, mainly because of its low clock frequency and memory. For the tracking algorithm, I've taken inspiration from many examples I've found on the web, based on Haar cascade classifiers. You can have a look at the Python code here.

ARPA face tracking web interface

As a plus, I've implemented a web interface with Flask and jQuery to show the face tracking bounding box and to toggle it (to save some CPU power when not needed). It is also useful as remote camera streaming interface to watch what the robot is seeing :)

Autonomous navigation

When you talk about mobile robot the first need is to make it move autonomously. Obviously it's a tricky task since it depends a lot on the sensors and hardware you have. The chassis I bought is not so big and also I quickly realized that the weight of all the devices, batteries and sensors is too much for it. Also encoders on this type of wheels are not really precise and not enough to build a navigation map. So I've chosen to use mainly two sensors, the sonar HC-SR04 and the magnetometer HMC5883L (connected on I2C pins), the first to measure the distance in centimeters from the head of the robot to obstacles and the second to have its heading in degrees. The algorithm I've implemented is pretty straightforward, it's just an obstacle avoidance algorithm that takes a random angle when it finds an obstacle; really dumb but enough for my tests. You can find the code for both here.

VTT and TTS engines

You can't consider a robot complete if it doesn't interact with humans using their natural language. Implementing from scratch VTT (voice-to-text) and TTS (text-to-speech) algorithms is really complex and there are decades of studies for that. So, first, I wanted to find affordable solution that could fit the performances available on the RPi. 

VTT
I have tested first Google Voice engine that is really good but it allows you for free to have not more than 50 requests per day. Then I've discovered http://wit.ai that is a startup acquired recently by Facebook that provides you a rich API to implement voice recognition. In fact it doesn't transform only your voice into text but it creates an abstraction layer above; you can link every sentence with an intent (for example 'Turn off the light') and put some intent attributes on it (in this case 'off' is the parameter). Also, if you say 'Switch off the light', that is the same intent and so could be matched with the previous one. This learning process is managed with the WIT console, in which you have the log of all the recordings and you can associate them to intents and extract variables. Every recognition result has a confidence factor that can help you to decide if the sentence has been correctly recognized. The good thing of WIT is that with an automatic learning process you don't have to map every different sentence to an intent since the platform will do that for you. It's really impressive.
Once you have trained a bit the platform, you can query it with a REST API, in order to send recordings and get back the recognition results in JSON. Simple, right?

TTS
There are a lot of implementations for TTS softwares, both online and offline; you can find a list here. I've tested Google, eSpeak and PicoTTS and I've preferred the latest since it works offline and the voice is not so metallic and also the performances seemed better globally on the RPi. The engine takes as parameter the sentence and the language (it supports the 6 major languages) and writes in output a .WAV file containing the synthesized voice.

Some photos of ARPA

Overview of ARPA

Top view - Raspberry PI and motor driver

Rear view - Active mini speaker and battery compartment

Front view - Sonar sensor and pan-tilt camera

XBOX 360 controller used for remote commands to the robot

 

Water pump control using Raspberry PI

Hello guys, today I want to share a great experience I had with the Raspberry PI.
Since I have a basement and the foundations needs to be emptied sometimes, especially during the rainy seasons, I've developed a web application that runs on Tomcat + Java 7 that allows the user to check the water level (using Ultrasonic Sensor HC-SR04) and activate remotely the pump (using Relay board). Also some sort of push notifications notifies the user if some given limit has been trespassed.

2 channel relay board

HC-SR04 Sonar sensor 

You can fork the project on Github: https://github.com/cecchisandrone/raspberrypi.

Also see the demo in the following video:

Please feel free to comment if you have questions on this project!

Bye

Tomcat + Apache HTTP cluster in 4 steps!

In this post I want to describe an easy procedure to set up a Tomcat cluster. I assume that Apache Tomcat 6.0.x and HTTPD 2.2.x servers are already installed.

1. Using Eclipse WTP, create two Tomcat instances and assign them different ports for Admin port, HTTP and AJP
2. Go into server.xml of each server:
1. Add the jvmRoute attribute at the element and specify a value (e.g. primary, secondary)
2. Uncomment the line where  is present
3. In the application web.xml add the <distributable/> element, directly under the web-app tag
4. Configure the file httpd.conf as follows:
1. Uncomment the following modules: 
1. LoadModule proxy_module modules/mod_proxy.so
2. LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
3. LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
4. LoadModule proxy_connect_module modules/mod_proxy_connect.so
5. LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
6. LoadModule proxy_http_module modules/mod_proxy_http.so
2. Add the following VirtualHost element:

 <VirtualHost *:80>  
     ServerName localhost  
      ServerAlias localhost  
     ProxyRequests Off  
     <Proxy *>  
      Order deny,allow  
      Allow from all  
     </Proxy>  
     ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid scolonpathdelim=On  
     ProxyPassReverse / balancer://mycluster/  
     <Proxy balancer://mycluster>  
      BalancerMember ajp://localhost:8009 route=primary loadfactor=10       
      BalancerMember ajp://localhost:8011 route=secondary loadfactor=10       
     </Proxy>  
     
 </VirtualHost>  

All is now ready for the cluster!

Here you can download the files:
https://dl.dropbox.com/u/1580227/blogspot/httpd.conf
https://dl.dropbox.com/u/1580227/blogspot/server1.xml
https://dl.dropbox.com/u/1580227/blogspot/server2.xml

Vaadin + Spring + MongoDB...on the cloud!

Hi all, in this post I want to publicize a little experiment I had with the cloud platform from VMWare, CloudFoundry http://www.cloudfoundry.com. The application deployed as example is an evolution of Vaadin Address Book (http://demo.vaadin.com/AddressBook) that is a CRUD application based on Vaadin and JPA. This is the list of modifications I've done on the original application:

• Spring 3.1 integration
• JPA implementation switched to Hibernate 4 (original was EclipseLink)
• Added a document manager feature (upload/download/delete), that stores files with MongoDB GridFS implementation
• Automatic configuration switch (development / cloud) using Spring 3.1 profiles
• Integration with Vaadin addon vaadin-lazyquerycontainer https://vaadin.com/directory#addon/lazy-query-container

I found CloudFoundry very easy to use, especially with the Eclipse plugin (http://docs.cloudfoundry.com/tools/STS/configuring-STS.html#installing-the-cloud-foundry-integration-extension-in-eclipse). You can run your app in the same way you do with a generic web application in Eclipse.

I have shared the project on GitHub (https://github.com/cecchisandrone/cloudfoundry). Feel free to download it and try on the cloud! A guide to deploy the app on CloudFoundry can be found here: http://docs.cloudfoundry.com/tools/deploying-apps.html

Note: before starting the project as webapp (in the cloud or locally), please edit the services.properties file. It contains configurations parameters, especially for services on cloud and databases.

Fix Facebook badge snippet

Sometimes, when generating Facebook profile badges (http://www.facebook.com/facebook-widgets/profilebadges.php) that the resulting badge doesn't work correctly (broken link). To fix the error change the HTML generated in the following manner: substitute the http://badge.facebook.com/badge/1613451344..........png tag with http://www.facebook.com/badge/1613451344..........png and you are done.