Scarlet: Nerian Stereo ROS2 utilizes 100% CPU and freezes

[SvenL]

Hello,

We are facing an issue trying to use the Scarlet 3D Depth Camera in ROS2. We are running ROS 2 Galactic on Ubuntu 20.04. We are using the latest nerian_stereo_ros2 code from here https://github.com/RaminYT/nerian_stereo_ros2.

Once we are launching the node with the provided launch file, the node initially runs, outputting data at around 10Hz. However, after a few seconds the nerian_stereo_node's CPU usage jumps to 100% and the node freezes/becomes stuck and does not output any data anymore. We can kill and relaunch the node, but the same behaviour occurs.

We are facing an additional issue, when trying to subscribe to the stereo_camera_info topic. Subscribing to this topic causes the node to immediately crash with an error stating that two timestamps of different sources have been subtracted.

We would appreciate any help or insights into why this might be happening. Unfortunately, the camera is currently unusable for us in this state.

Thanks.

[RaminYT]

Hi, I will take a look at this problem.

At first look, the current default branch on my GitHub seems to work normally and in a stable fashion with the devices here, in terms of data transport. (Incidentally, the 'officially released' version would be the one in the Nerian company repo https://github.com/nerian-vision/nerian_stereo_ros2).

Does this happen when just the Nerian node is active and idling, i.e. no ROS recipients attached? Could you confirm that there are no other consumers for the camera data running in parallel? (NVCom or the web interface preview/calibration views)?

stereo_camera_info was likely insufficiently tested in the recent past, I will do that now.

Kindly attach an exported device settings archive (Maintenance -> Download Configuration -> Download) so we can best try to replicate your setup for the observed image transfer / CPU issue!

Best regards,
Ramin

[RaminYT]

Update: the second issue (stereo_camera_info) should be fixed now (would be glad to hear feedback). -- EDIT: At this time, this fix is in the RaminYT/... repo only.

[SvenL]

Hi Ramin,

Thanks for the quick reply! I am offsite today, but will have access to the camera again tomorrow. I will test the stereo_camera_info fix and will also get back to you with the exported device settings archive.

During our tests, we had no other ROS recipients/subscribers attached to the node. It seems to us that the async frame request starts timing out.

Thanks again for your help and I will follow up tomorrow!

Best,
Sven

[SvenL]

Hi Ramin,

Please find the device settings archive of the camera attached.

Best,
Sven

[SvenL]

Hi, quick additional update.

We did make some changes to the configuration file and enables Jumbo packets for the 1G connection, which we are using. The node seems to be running fine for a longer time now, but after a while, it still stops working.

Specifically in the function "processOneImageSet()" of nerian_stereo_node.cpp, asyncTransfer->collectReceivedImageSet(imageSet,0.1) starts returning false indefinitely.

[RaminYT]

OK, I will continue the investigation first thing tomorrow.

Probably I will push some code with more debug output, this problem is definitely not typical. If I see it correctly, the archive you sent was likely exported after a clean restart; if this is the case, it would be very helpful if you could provide another one exported after the problem occurred, as I would possibly contain more of the transient device-side logging info. I am sorry to trouble you here because I forgot to mention this before.

One more thing you could test right away is to see whether switching the image transfer protocol to TCP changes the situation.

[SvenL]

I've changed the settings back to TCP, but that seems to have made the problem worse. The ros node is now not publishing anything on its topics right from the beginning. I have attached an exported archive from after when this happened here. Please note, that I had to switch the network connection of the camera after I was done with testing to get the archive - this might be reflected in the log messages but is unrelated to the issue we are facing.

Thanks again for your help! I believe a git push of the ros code with additional debug messages would be very helpful.

[RaminYT]

I will prepare and test a debug build for both the client software and the Scarlet firmware. This might take the better part of the day at least...

One more thing that is a bit spurious is that some stray WebSocket information is present in the log (they are used for the browser preview windows). If at all possible, for comparison, could you do another run with config export, but first completely close all browser instances running on machines attached to the device? I want to exclude that there is a bug related to websocket reconnection / persistence that might be present for some (new) browsers (just guessing though).
Thanks a lot.

EDIT: I forgot to ask an important question - does connecting to the device with NVCom instead of the ROS2 node exhibit the same issues? Thanks again.

[RaminYT]

Please note that to test the ROS2 node with TCP, you also have to expicitly set the node to use TCP, e.g. with ros2 run:

ros2 run nerian_stereo nerian_stereo_node --ros-args -p remote_host:=192.168.xxxxxx -p use_tcp:=True

or analogous modification of your launch setup. It is unfortunately not detected automatically at this time; and you get no images otherwise.