Frequent errors related to Ethernet Gateways
-
In the last week I often receive errors like this:
2020-04-03 09:03:30,572 ERROR [mc-th-pool-7] [org.mycontroller.standalone.gateway.ethernet.EthernetDriver:107] Exception, java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:143) at org.mycontroller.standalone.gateway.ethernet.EthernetDriver.write(EthernetDriver.java:104) at org.mycontroller.standalone.gateway.ethernet.GatewayEthernet.write(GatewayEthernet.java:44) at org.mycontroller.standalone.provider.EngineAbstract.auditQueue(EngineAbstract.java:229) at org.mycontroller.standalone.provider.EngineAbstract.run(EngineAbstract.java:133) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)I have 4 ethernet gateways, 3 based on Arduino Uno + W5100 Shield and 1 with Arduino Nano + W5500.
They never gave me a problem for more than 1 year, and now they're all suddenly showing connection problems.
I tried changing the ethernet cables and moving them to a different ethernet port: same errors.
They're connected to the same swtich I use for my laptop and I don't see issues, so I don't expect to be the switch.Any idea?
Thanks
Daniele -
@Daniele
Have you tried to restart MyController OS once?
Do you remember anything you did recently on your router, something like block ports?
any firmware upgrade happened recently on your router? -
@jkandasa I restart MyController Raspberry regularly (scheduled once per week), I tried doing that manually but nothing changed.
No changes on the router, but you made me remember I recently re-uploaded the sketches of my gateways and in this way I upgraded their mysensors library version.
I'll try downgrade one of them and see if this helps. -
I made many tests in these days.
On the arduino side:
- downgrading mysensors library
- reducing and than disabling the RF24 message buffer feature
- Soft SPI
- adding capacitors to get a more stable power
On the RPI side:
- moved the ethernet connection from a switch to the main router
- moved from ethernet to wifi (to understand if there could be any issue on the RPI ethernet port)
- upgrading to mycontroller 1.5
- new SD card
I still see ethernet related errors, but now the message is slightly different, it talks about connection reset instead of broken pipe:
2020-04-14 10:04:42,138 ERROR [mc-th-pool-4] [org.mycontroller.standalone.gateway.ethernet.EthernetDriver:107] Exception, java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115) at java.net.SocketOutputStream.write(SocketOutputStream.java:143) at org.mycontroller.standalone.gateway.ethernet.EthernetDriver.write(EthernetDriver.java:104) at org.mycontroller.standalone.gateway.ethernet.GatewayEthernet.write(GatewayEthernet.java:44) at org.mycontroller.standalone.provider.EngineAbstract.auditQueue(EngineAbstract.java:229) at org.mycontroller.standalone.provider.EngineAbstract.run(EngineAbstract.java:133) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)@jkandasa could you please give me more details about the meaning of the 2 error messages?
Thanks
Daniele -
@Daniele The error says,
Your connection between MyController and ethernet gateway reset.Possible causes may be:
- MySensors ethernet gateway hardware issue
Try to replicate your old stable firmware to check the hardware.
Update: After this error, the connection will be reestablished automatically. But it will take couple of moments(you lose your data on this period)
-
@jkandasa I've not been clear: could you please explain what's the difference between "Broken pipe" and "Connection reset" errors?
I yet tried with the old firmware, but I get the same errors, so the issue is somewhere else. =(
-
I guess you have a problem with your MySensors gateway ethernet device attached with Arduino
If you have an additional ethernet hardware test with it.-
java.net.SocketException: Broken pipe (Write failed): Writing to a connection when the other end(your gateway) has already closed it -
java.net.SocketException: Connection reset: on the server side when the client(your gateway) closed the socket connection before the response could be returned over the socket
-
-
@jkandasa said in Frequent errors related to Ethernet Gateways:
our MySensors gateway ethernet device attached with Arduino
That's the same thing I thought, but having 4 of them all failing sounds a bit odd....
That's why I started looking also on the RPI side.
I tried also moving the ethernet cables from the swtich to the main router, to understand if the switch is having some issue, but still getting the same issues. -
@Daniele said in Frequent errors related to Ethernet Gateways:
That's the same thing I thought, but having 4 of them all failing sounds a bit odd....
ah, really a tough situation.
Can you power down all the 4 gateways?
Power up at a time only one gateway and test with it, if it looks good, test the next gateway -
That's the same thing I thought, but having 4 of them all failing sounds a bit odd....
Maybe the gateways' DHCP lease runs out, they lose their IP and don't ask it back soon enough?
-
@Avamander That's not a bad suggeation!
@Daniele Try re-flashing the gateways and see how that works.
-
@skywatch done multiple times during my tests, but nothing changed!
-
@Daniele When you said before you had tried a 'new SD' card, did you flash that card from an image or use a fresh install?
-
@skywatch said in Frequent errors related to Ethernet Gateways:
d tried a 'new SD' card, did you flash tha
fresh install
-
@Daniele Then I am out of ideas at the moment, sorry! But I do know the pain you have now. It happened to me a few days ago that a GW just stopped without any real reason. I re-plugged into the pi and all came back up again.
There is discussion on MS forum about using optiboot bootloader to get full benefit from watchdog timer. There it is said that watchdog is not working in standard 'Chinese' bootloader.
Maybe worth looking into - its going to be raining here after today (think summer just came and went for us!).... So I should have some time to 'fiddle' a little....
-
I can confirm, chinese nano bootloader does not support watchdog.
All my nanos have MySBootloader (for this reason and to be able to use OTA update), but I don't want to use it on GWs since it takes 10-15 seconds to start because it searches for a GW in order to chech new OTA updates.
What I tried is to enable watchdog on a Uno (which in my experiments seems to support watchdog), but till now I haven't found a good way to check ethernet connectivity without interfering with GW functioning.
Do you have any idea on that?
Thanks
Daniele -
@Daniele Have you tried monitoring it with wireshark?
-
@skywatch said in Frequent errors related to Ethernet Gateways:
There it is said that watchdog is not working in standard 'Chinese' bootloader.
Yes, some products ship with the old bootloader.
You can either use an existing Arduino or an USBASP as an ISP programmer to flash a new bootloader, not tremendously difficult if you just wire it up and press the button in Arduino IDE.
@Daniele
I still have this question though, instead of watchdog, did you check that your ethernet module/code just renews its DHCP lease? I forgot to do that on one of my projects and had very similar issues. You should also maybe add a piece of code that sends ICMP ping to your gateway every 1h to make sure ARP cache doesn't time out - avoids that issue too. -
@Avamander could you please post me an example of DHCP renewal?
I investigated the ICMP ping idea but I've not been able to get something working, since the ping seems to interfere with GW code. Did you manage to use it in a mysensors GW?
