[SOLVED] Serial problem

jkandasa

@skywatch ok, The corrupted data only reports in node id right?

jkandasa

@skywatch I have added a workaround for number issue, Can you check it in the SNAPSHOT version?

skywatch

@jkandasa Yes - oddly it is only the node ID that seems to be corrupted with junk characters...

skywatch

I have the snapshot version now, thank you - I want to see how it goes for a day first as I have modified the pi slightly and want to see what effect, if any, that has first.

As always I will let you know how it goes in a few days!

skywatch

Just a quick update - 25 hours and no corrupted node id's, so I am optimistic on the mod.......But, at 6.30AM all data stopped flowing from all nodes - a disconnect and re-connect of the GW pro mini and all came back....hmmm.

jkandasa

@skywatch

Just a quick update - 25 hours and no corrupted node id's, so I am optimistic on the mod.......

Can you share what change needs in hardware to resolve this issue?

But, at 6.30AM all data stopped flowing from all nodes - a disconnect and re-connect of the GW pro mini and all came back....hmmm.

Did gateway reconnect automatically? Can you email me the mycontroller.log file?

skywatch

@jkandasa - Gateway reconnected automatically.

It all went bad today - I tried a new 5v pro mini GW (with level shifter for data lines) and that failed miserably. So I made a new 3.3V pro mini GW and a new 3.3v regulator board, and have spent the last 7 hours trying to get things working again.

Sadly I think this will require a full reinstall, losing all data from the hot summer and taking more time than I have available to put into it. Just as it was looking hopeful too......

I'm gutted.

P.S. Hardware (Rpi3). Add a 100n capacitor across the 5VDC input on the back of the pi (points to attach to are PP2 and PP5). It will filter out spikes on the power input to the pi. An 0805 SMD ceramic multilayer cap is what I used and I suggest the same for anyone else - but I take no resoposibility if you break a pi - do it at your own risk and only if competent to do so.

skywatch

At 21.30 I got it back to working again with a different 3.3v pro mini as GW, at 22.50 I got the first corrupted node data and at 23.45 all data from remote nodes stopped again and it remained in this state until half an hour ago when I powered down/ powered up the pi.

jkandasa

@skywatch Can you share your hardware(serial gateway) and power supply details?

skywatch

@jkandasa Sure!

Gateway using mysensors 2.3.0.......

/**
* The MySensors Arduino library handles the wireless radio link and protocol
* between your home built sensors/actuators and HA controller of choice.
* The sensors forms a self healing radio network with optional repeaters. Each
* repeater and gateway builds a routing tables in EEPROM which keeps track of the
* network topology allowing messages to be routed to nodes.
*
* Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
* Copyright (C) 2013-2015 Sensnology AB
* Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
*
* Documentation: http://www.mysensors.org
* Support Forum: http://forum.mysensors.org
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* version 2 as published by the Free Software Foundation.
*
*******************************
*
* DESCRIPTION
* The ArduinoGateway prints data received from sensors on the serial link.
* The gateway accepts input on seral which will be sent out on radio network.
*
* The GW code is designed for Arduino Nano 328p / 16MHz
*
* Wire connections (OPTIONAL):
* - Inclusion button should be connected between digital pin 3 and GND
* - RX/TX/ERR leds need to be connected between +5V (anode) and digital pin 6/5/4 with resistor 270-330R in a series
*
* LEDs (OPTIONAL):
* - To use the feature, uncomment any of the MY_DEFAULT_xx_LED_PINs
* - RX (green) - blink fast on radio message recieved. In inclusion mode will blink fast only on presentation recieved
* - TX (yellow) - blink fast on radio message transmitted. In inclusion mode will blink slowly
* - ERR (red) - fast blink on error during transmission error or recieve crc error
*
*/

// Enable debug prints to serial monitor
//#define MY_DEBUG


// Enable and select radio type attached
#define MY_RADIO_NRF24

//#ifndef MY_RF24_CHANNEL
#define MY_RF24_CHANNEL (97)
//#endif
#define MY_RX_MESSAGE_BUFFER_FEATURE
#define MY_RF24_IRQ_PIN 2
// Set LOW transmit power level as default, if you have an amplified NRF-module and
// power your radio separately with a good regulator you can turn up PA level.
#define MY_RF24_PA_LEVEL RF24_PA_HIGH

// Enable serial gateway
#define MY_GATEWAY_SERIAL

// Define a lower baud rate for Arduino's running on 8 MHz (Arduino Pro Mini 3.3V & SenseBender)
#if F_CPU == 8000000L
#define MY_BAUD_RATE 38400
#endif

// Enable inclusion mode
//#define MY_INCLUSION_MODE_FEATURE
// Enable Inclusion mode button on gateway
//#define MY_INCLUSION_BUTTON_FEATURE

// Inverses behavior of inclusion button (if using external pullup)
//#define MY_INCLUSION_BUTTON_EXTERNAL_PULLUP

// Set inclusion mode duration (in seconds)
//#define MY_INCLUSION_MODE_DURATION 60
// Digital pin used for inclusion mode button
//#define MY_INCLUSION_MODE_BUTTON_PIN  3

// Set blinking period
//#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Inverses the behavior of leds
//#define MY_WITH_LEDS_BLINKING_INVERSE

// Flash leds on rx/tx/err
// Uncomment to override default HW configurations
//#define MY_DEFAULT_ERR_LED_PIN 4  // Error led pin
//#define MY_DEFAULT_RX_LED_PIN  6  // Receive led pin
//#define MY_DEFAULT_TX_LED_PIN  5  // the PCB, on board LED

#include <MySensors.h>

void setup()
{
	// Setup locally attached sensors
}

void presentation()
{
	// Present locally attached sensors
}

void loop()
{
	// Send locally attached sensor data here
}

Power is from official raspberry pi power supply - I'll check the rating when I power it down later on tonight.

Tag

@skywatch

you are using a 3.3v promini as a GW.... 2 questions...

how do you get the data from the promini into the RPI, how is that wired since the promini does not have an USB port...
the GW sketch you posted is designed for an arduino nano 16Mhz, if you use a 3.3v promini it needs run at 8Mhz... Try to make this change in the GW sketch to hardcode the 8Mhz feature. It might be that the cpu freq in the if statement is not picked up bt the compiler, so try with the if statements removed.

// Define a lower baud rate for Arduino's running on 8 MHz (Arduino Pro Mini 3.3V & SenseBender)
#if F_CPU == 8000000L
#define MY_BAUD_RATE 38400
#endif

to

// Define a lower baud rate for Arduino's running on 8 MHz (Arduino Pro Mini 3.3V & SenseBender)
//#if F_CPU == 8000000L
#define MY_BAUD_RATE 38400
//#endif

Can you try an arduino nano + NRF24L01 as gateway, and connect it to USB (if you have one lying around)

Just as background info, my 1st mycontroller setup was an RPI 2b with an arduino nano on usb that acted as a GW... had this setup for years and it worked flawless, other versions of the SW, but still....

skywatch

@Tag

Thanks for the reply. In answer to your questions.....

The pro mini is connected via serial connection to the pi (tx/rx on pro mini to tx/rx on pi). I chose 3.3v pro mini as then there is no level shifter needed.
It is definitely being picked up by the compiler as the serial rate between the pi and promini works at 38400. If this were not picked up I would expect 115200. I don't have a nano anywhere, but could try a uno or mega.

I now am running on a 'Pi Hut' 5V 2A supply and have had 2 junk node id's already this morning (But this time both from the same node I think, so will look into that as well). I am also going to see what good power supplies RS Components has and probably add a mains filter as well.

Tag

Ah okay!,
Well, it might work this way, but sometime ago I ran into issues with serial comms where the TTL level between high and low was different between systems, and had to use a pull down resistor to fix this...

Another solution is to remove the arduino from the equation... it is possible to use plug the radio directly on the GPIO header of the RPI.. For a howto look here

Got this working for over a year now on and OPI zero and it works as expected.....

skywatch

@tag I originally had the nrf attached directly to the pi and moved to an arduino GW as I had issues with the directly connected setup (also I realised that HW signing is only supported for arduino GW and not for NRF attached directly to the pi).

However, I have discovered something.

I am testing it since yesterday lunchtime. At the momtent it looks good, but I have been at this point before.

Give me a few days to test and I will report back then.

skywatch

I seem to have found the root cause of this, now working for 4 days with no corrupted data. But I have seen this overnight in the log, anything to worry about?

2018-08-24 03:43:20,485 ERROR [mc-th-pool-7] [org.mycontroller.standalone.db.dao.NodeDaoImpl:66] unable to get Node
org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-194]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
	at org.h2.message.DbException.get(DbException.java:168)
	at org.h2.message.DbException.convert(DbException.java:295)
	at org.h2.message.DbException.toSQLException(DbException.java:268)
	at org.h2.message.TraceObject.logAndConvert(TraceObject.java:352)
	at org.h2.jdbc.JdbcResultSet.findColumn(JdbcResultSet.java:188)
	at com.j256.ormlite.jdbc.JdbcDatabaseResults.findColumn(JdbcDatabaseResults.java:103)
	at com.j256.ormlite.field.FieldType.resultToJava(FieldType.java:826)
	at com.j256.ormlite.stmt.mapped.BaseMappedQuery.mapRow(BaseMappedQuery.java:61)
	at com.j256.ormlite.stmt.StatementExecutor.queryForFirst(StatementExecutor.java:109)
	at com.j256.ormlite.dao.BaseDaoImpl.queryForFirst(BaseDaoImpl.java:239)
	at com.j256.ormlite.stmt.QueryBuilder.queryForFirst(QueryBuilder.java:380)
	at org.mycontroller.standalone.db.dao.NodeDaoImpl.get(NodeDaoImpl.java:64)
	at org.mycontroller.standalone.provider.ExecuterAbstract.getNode(ExecuterAbstract.java:473)
	at org.mycontroller.standalone.provider.ExecuterAbstract.isNodeRegistered(ExecuterAbstract.java:610)
	at org.mycontroller.standalone.provider.ExecuterAbstract.execute(ExecuterAbstract.java:98)
	at org.mycontroller.standalone.provider.EngineAbstract.auditQueue(EngineAbstract.java:260)
	at org.mycontroller.standalone.provider.EngineAbstract.run(EngineAbstract.java:131)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
	at org.h2.util.StringUtils.toUpperEnglish(StringUtils.java:106)
	at org.h2.jdbc.JdbcResultSet.getColumnIndex(JdbcResultSet.java:3129)
	at org.h2.jdbc.JdbcResultSet.findColumn(JdbcResultSet.java:186)
	... 15 common frames omitted
2018-08-24 03:43:20,517 ERROR [mc-th-pool-7] [org.mycontroller.standalone.db.dao.BaseAbstractDaoImpl:247] unable to add new item:[Node(id=null, eui=1, gatewayTable=GatewayTable(id=1, enabled=null, name=null, networkType=null, timestamp=null, state=UNAVAILABLE, statusMessage=null, statusSince=null, type=null, properties=null), name=null, version=null, type=null, libVersion=null, batteryLevel=null, eraseConfig=null, firmware=null, state=UP, lastSeen=1535078600491, rssi=null, properties={}, parentNodeEui=null, registrationState=REGISTERED, smartSleepEnabled=false)]
java.sql.SQLException: Unable to run insert stmt on object Node(id=null, eui=1, gatewayTable=GatewayTable(id=1, enabled=null, name=null, networkType=null, timestamp=null, state=UNAVAILABLE, statusMessage=null, statusSince=null, type=null, properties=null), name=null, version=null, type=null, libVersion=null, batteryLevel=null, eraseConfig=null, firmware=null, state=UP, lastSeen=1535078600491, rssi=null, properties={}, parentNodeEui=null, registrationState=REGISTERED, smartSleepEnabled=false): INSERT INTO `node` (`id` ,`eui` ,`gatewayId` ,`name` ,`version` ,`type` ,`libVersion` ,`batteryLevel` ,`eraseConfig` ,`firmwareId` ,`state` ,`lastSeen` ,`rssi` ,`properties` ,`parentNodeEui` ,`registrationState` ,`smartSleepEnabled` ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
	at com.j256.ormlite.misc.SqlExceptionUtil.create(SqlExceptionUtil.java:25)
	at com.j256.ormlite.stmt.mapped.MappedCreate.insert(MappedCreate.java:137)
	at com.j256.ormlite.stmt.StatementExecutor.create(StatementExecutor.java:458)
	at com.j256.ormlite.dao.BaseDaoImpl.create(BaseDaoImpl.java:328)
	at org.mycontroller.standalone.db.dao.BaseAbstractDaoImpl.create(BaseAbstractDaoImpl.java:244)
	at org.mycontroller.standalone.provider.ExecuterAbstract.getNode(ExecuterAbstract.java:486)
	at org.mycontroller.standalone.provider.ExecuterAbstract.isNodeRegistered(ExecuterAbstract.java:610)
	at org.mycontroller.standalone.provider.ExecuterAbstract.execute(ExecuterAbstract.java:98)
	at org.mycontroller.standalone.provider.EngineAbstract.auditQueue(EngineAbstract.java:260)
	at org.mycontroller.standalone.provider.EngineAbstract.run(EngineAbstract.java:131)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "CONSTRAINT_24_INDEX_2 ON PUBLIC.NODE(EUI, GATEWAYID) VALUES ('1', 1, 61)"; SQL statement:
INSERT INTO `node` (`id` ,`eui` ,`gatewayId` ,`name` ,`version` ,`type` ,`libVersion` ,`batteryLevel` ,`eraseConfig` ,`firmwareId` ,`state` ,`lastSeen` ,`rssi` ,`properties` ,`parentNodeEui` ,`registrationState` ,`smartSleepEnabled` ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) [23505-194]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
	at org.h2.message.DbException.get(DbException.java:179)
	at org.h2.message.DbException.get(DbException.java:155)
	at org.h2.index.BaseIndex.getDuplicateKeyException(BaseIndex.java:103)
	at org.h2.mvstore.db.MVSecondaryIndex.checkUnique(MVSecondaryIndex.java:231)
	at org.h2.mvstore.db.MVSecondaryIndex.add(MVSecondaryIndex.java:190)
	at org.h2.mvstore.db.MVTable.addRow(MVTable.java:707)
	at org.h2.command.dml.Insert.insertRows(Insert.java:156)
	at org.h2.command.dml.Insert.update(Insert.java:114)
	at org.h2.command.CommandContainer.update(CommandContainer.java:101)
	at org.h2.command.Command.executeUpdate(Command.java:258)
	at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:160)
	at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:146)
	at com.j256.ormlite.jdbc.JdbcDatabaseConnection.insert(JdbcDatabaseConnection.java:183)
	at com.j256.ormlite.stmt.mapped.MappedCreate.insert(MappedCreate.java:91)
	... 11 common frames omitted

It is a holiday weekend here, but I will post the story of how I found and fixed the issue. You can all guess if you like. I bet nobody will get it!

jkandasa

@skywatch

I seem to have found the root cause of this, now working for 4 days with no corrupted data.

Great news! :thumbs_up:

But I have seen this overnight in the log, anything to worry about?

How often are you seeing this log? If you see this error very frequently kindly report a bug/issue. I do not get any clue why failed to retrieve a node from the database. A particular message will be dropped(without any action) when you face this issue.

It is a holiday weekend here, but I will post the story of how I found and fixed the issue. You can all guess if you like. I bet nobody will get it!

Enjoy your holiday weekend! I am eager to see your fix

jkandasa

@skywatch Can you break the surprise?

skywatch

@jkandasa OK - it's a long story though......

So for the last 2 months after upgrading to mysensors 2.3.0 I get the library update pop-up with arduino IDE - It sometimes tells me to upgrade to a lower version of the library than I already have, so I figured it was a bit buggy.

Recently it was telling me I needed to upgrade to the latest mysensors 2.2.0 - But since I already had 2.3.0 installed I ignored it and carried on. After all, all the nodes reported as 2.3.0 and there was only 1 mysensors folder inside the libraries folder.

But when I went into the mysensors examples to clear eeprom, I found 2x mysensors options. They were different. I was baffled. I rechecked the libraries folder, only one mysensors folder. I checked app data, program files, temp files and everywhere I could think of. No other mysensors folders present.

After a lot of searching I found something interesting - In the arduino libraries folder there were a few 'arduino xxxxxx' folders (where xxxxxx was 6 seemingly random digits). Inside one of them were files relating to mysensors 2.1.1. - I deleted this folder and tested the IDE which compiled mysensors sketches with no issues.

With this rouge folder removed I decided to re-flashed the gateway and all nodes. This was the point that all the problems went away (and now even ack works properly)!

So it appears that this was the issue - it explains why the issue was only affecting me, but I still can't figure out what was happening fully as the nodes all reported as 2.3.0 and yet somehow having an older mysensors.h and mysensorscore.h inside a folder that wasn't mysensors in any way seemed to be causing the issue.

Still I found the problem and now have a stable system. It just goes to show that if you don't give up, eventualy you'll make progress.

So a big THANK YOU to you all for the help with this and sorry for taking your time on this one.

It's a learning curve for sure!

jkandasa

@skywatch Interesting story

So a big THANK YOU to you all for the help with this and sorry for taking your time on this one.

No worries at all. I have fixed couple of other issues(when working on this) and we have improved MyController quality further

skywatch

@jkandasa Thank you! - I have no idea what created those arduinoxxxxxx folders, there were 4 or 5 of them. A casual look and it's easy to think they are part of the internal arduino ide or something.

Still, at least I may make more progress now.

But not a time for relaxing, I still have FOTA, SWsinging, HWsigning and low power battery nodes to learn! - You have been warned! Hahahaha.....:)

[SOLVED] Serial problem

1

547

524

3.4k