ros图像处理相关

1. image_transport

ros规定了多种基本的数据结构，用于node之间的传输。图像也是一种常用的数据，image_transport package就是用来处理图像数据传输的，它本身是个框架，只提供最基本的原始图像数据(raw)，如果需要降低传输时的带宽，还需要压缩格式，由框架下集成的各种插件来完成。

image_transport 会发布 sensor_msgs/Image 格式的数据，其他压缩格式由插件提供。

---

ROS中原始图像格式"sensor_msgs/Image"如下：

                        # This message contains an uncompressed image
                        # (0, 0) is at top-left corner of image
                        #
Header header           # std_msgs/Header

uint32 height           # image height, that is, number of rows
uint32 width            # image width, that is, number of columns

                        # The legal values for encoding are in file src/image_encodings.cpp
                        # If you want to standardize a new string format, join
                        # ros-users@lists.sourceforge.net and send an email proposing a new encoding.
string encoding         # Encoding of pixels -- channel meaning, ordering, size
                        # taken from the list of strings in include/sensor_msgs/image_encodings.h

uint8 is_bigendian      # is this data bigendian?
uint32 step             # Full row length in bytes
uint8[] data            # actual matrix data, size is (step * rows)

其中 header 是标准结构:

                        # Standard metadata for higher-level stamped data types.
                        # This is generally used to communicate timestamped data
                        # in a particular coordinate frame.
                        #
                        # sequence ID: consecutively increasing ID
uint32 seq

                        #Two-integer timestamp that is expressed as:
                        # * stamp.sec: seconds (stamp_secs) since epoch (in Python the variable is called 'secs')
                        # * stamp.nsec: nanoseconds since stamp_secs (in Python the variable is called 'nsecs')
                        # time-handling sugar is provided by the client library
time stamp

                        #Frame this data is associated with
string frame_id

一个具体例子:

root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw --noarr
header:
  seq: 293              # 当前帧序号
  stamp:                # 获取图像的时间戳
    secs: 1636868180
    nsecs:  53476744
  frame_id: "camera"    # frame_id
height: 480             # 行数
 640              # 列数
encoding: "bgr8"        # 像素点的压缩格式，通道数3，顺序是BGR，深度是24。按照这个可以知道data数组中值的意义
is_bigendian: 0         # 是否大端排序
step: 1920              # 步长，一行的字节数，宽度(640) x 像素点字节数(3) = 1920bytes。图像总共 480行，也就是 1920x480=921600字节，也是下边data数组长度
data: "<array type: uint8, length: 921600>"

---

使用上的例子
不使用 image_transport 的订阅发布：

// Do not communicate images this way!
#include <ros/ros.h>

void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
  // ...
}

ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe("in_image_topic", 1, imageCallback);
ros::Publisher pub = nh.advertise<sensor_msgs::Image>("out_image_topic", 1);

使用 image_transport 的订阅发布:

// Use the image_transport classes instead.
#include <ros/ros.h>
#include <image_transport/image_transport.h>

void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
  // ...
}

ros::NodeHandle nh;
image_transport::ImageTransport it(nh);  // 相当于将原始的nh封装了一次
image_transport::Subscriber sub = it.subscribe("in_image_base_topic", 1, imageCallback);
image_transport::Publisher pub = it.advertise("out_image_base_topic", 1);

---

提供的东西:

API

publisher

image_transport::Publisher
image_transport本身只在基础topic: <base topic> 上发布 sensor_msgs/Image 原始格式的图像，如果有其他插件，会自动在基础topic下的其他topic发布:<base topic>/<transport name>，比如基础topic是 "/cv_camera",
image_transport 会发布 "/cv_camera/image_raw" 原始格式的topic，插件compressed_image_transport 会在 "/cv_camera/image_raw/compressed" 发布压缩格式的图像。

subscriber
image_transport::Subscriber
订阅基础topic后，会自动订阅其他相关topic。

parameter

image_transport 本身不发布参数，但插件会发布各种参数，比如帧率、压缩等级等，需要看插件本身。
发布者应该使用动态参数机制，让参数更容易被订阅者使用。
参数格式一般是: <base topic>/<transport name>/<parameter name>
比如: /cv_camera/image_raw/compressed/parameter_descriptions

node

$ rosrun image_transport republish [in_transport] in:=<in_base_topic> [out_transport] out:=<out_base_topic>

比如将 theora 格式的topic图像解压，重新发布到另一个topic，可以这样写:

$ rosrun image_transport republish theora in:=camera/image raw out:=camera/image_decompressed

如果一个node只发布原始的图像格式，可以用 image_transport 重新发布成多种格式的topic。

命令行工具

$ rosrun image_transport list_transports

可以查看当前所有的package，看看是否有可用的图像处理插件，以及插件状态。

---

下边的插件介绍
使用 image_transport 提供的一个node命令： list_transports，可以查看当前有的topic和对应的插件：

root@suntus:~# rosrun image_transport list_transports
Declared transports:
image_transport/compressed
image_transport/compressedDepth
image_transport/ffmpeg (*): Not available. Try 'catkin_make --pkg ffmpeg_image_transport'.
image_transport/raw
image_transport/theora

Details:
----------
"image_transport/compressed"
 - Provided by package: compressed_image_transport
 - Publisher:
      This plugin publishes a CompressedImage using either JPEG or PNG compression.

 - Subscriber:
      This plugin decompresses a CompressedImage topic.

----------
"image_transport/compressedDepth"
 - Provided by package: compressed_depth_image_transport
 - Publisher:
      This plugin publishes a compressed depth images using PNG compression.

 - Subscriber:
      This plugin decodes a compressed depth images.


----------
"image_transport/ffmpeg"
*** Plugins are not built. ***
 - Provided by package: ffmpeg_image_transport
 - Publisher:
      This plugin encodes frame into ffmpeg compressed packets

 - Subscriber:
      This plugin decodes frame from ffmpeg compressed packets

----------
"image_transport/raw"
 - Provided by package: image_transport
 - Publisher:
      This is the default publisher. It publishes the Image as-is on the base topic.

 - Subscriber:
      This is the default pass-through subscriber for topics of type sensor_msgs/Image.

----------
"image_transport/theora"
 - Provided by package: theora_image_transport
 - Publisher:
      This plugin publishes a video packet stream encoded using Theora.

 - Subscriber:
      This plugin decodes a video packet stream encoded using Theora.

---

compressed_image_transport("compressed"):
发布和订阅 JPEG 或 PNG 压缩格式的图像，支持运行时改变图片质量。

压缩格式:

                    # This message contains a compressed image
Header header       # std_msgs/Header
string format       # Specifies the format of the data
                    #   Acceptable values:
                    #     jpeg, png
uint8[] data        # Compressed image buffer

一个例子:

root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw/compressed --noarr
header:
  seq: 0
  stamp:
    secs: 1636880329
    nsecs: 751462995
  frame_id: "camera"
format: "bgr8; jpeg compressed bgr8"
data: "<array type: uint8, length: 43548>"
---

提供的东西：

发布插件

发布topic:

<base_topic>/compressed (sensor_msgs/CompressedImage)

参数

• <base_topic>/compressed/format (string, default: jpeg) Compression format to use, "jpeg" or "png".
• <base_topic>/compressed/jpeg_quality (int, default: 80) JPEG quality percentile, in the range [1, 100]. Lower values trade image quality for space savings.
• <base_topic>/compressed/png_level (int, default: 9) PNG compression level, in the range [1, 9]. Higher values trade computation time for space savings.

订阅插件

订阅topic:
<base_topic>/compressed (sensor_msgs/CompressedImage)

---

compressed_depth_image_transport
提供深度图像(depth image),主要用于3D领域，带距离的图像格式。

---

theora_image_transport
发布成视频流
theora视频编码格式的视频流数据
theora是个视频编码格式，类似H264这种
对应的声音编码技术是vorbis
对应的文件封装格式是ogg

发布topic
<base topic>/theora (theora_image_transport/Packet)

参数

• <base topic>/theora/optimize_for (int, default: Quality (1))
  Controls whether to use constant bitrate (CBR) encoding, aiming for ~theora/target_bitrate; or variable bitrate (VBR) encoding, aiming for ~theora/quality. Values are Bitrate (0) and Quality (1).
• <base topic>/theora/target_bitrate (int, default: 800000)
  Target bitrate. Used if optimize_for is set to Bitrate.
• <base topic>/theora/quality (int, default: 31)
  Encoding quality level, in the range [0, 63]. Used if optimize_for is set to Quality.
• <base topic>/theora/keyframe_frequency (int, default: 64)
  Maximum distance between key frames. If set to 1, every frame is a keyframe.

订阅topic:
<base topic>/theora (theora_image_transport/Packet)

提供有一个node: ogg_saver，可以保存成 .ogv 格式的封装文件，用其他播放器播放。

---

ffmpeg_image_transport
使用ffmpe将图像转换成h264或h265格式，并且支持nvidia GPU硬件加速。

https://github.com/daniilidis-group/ffmpeg_image_transport

---

cv_bridge
在ROS Image messages 和 OpenCV images 格式之间进行转换。

ros图像格式基础的是 sensor_msgs/Image, 还有 sensor_msgs/CompressedImage, opencv中用到的格式为 cv::Mat 矩阵，需要进行转换，才能放到opencv中使用。

opencv中支持的像素编码格式有：
8UC[1-4]
8SC[1-4]
16UC[1-4]
16SC[1-4]
32SC[1-4]
32FC[1-4]
64FC[1-4]

cv_bridge有时候会进行必要的像素格式转换，可以使用如下的字符串来表示目标格式：

• mono8: CV_8UC1, grayscale image
• mono16: CV_16UC1, 16-bit grayscale image
• bgr8: CV_8UC3, color image with blue-green-red color order
• rgb8: CV_8UC3, color image with red-green-blue color order
• bgra8: CV_8UC4, BGR color image with an alpha channel
• rgba8: CV_8UC4, RGB color image with an alpha channel

其中 mono8和bgr8是opencv中常用的格式。

---

roscpp

初始化和停止
ros::init(argc, argv, "my_node_name");
ros::shutdown();
ros::ok() 用于检查node是否shutdown

timer

// 定时器，周期性执行回调。
ros::Timer ros::NodeHandle::createTimer(ros::Duration period, <callback>, bool oneshot = false);
ros::Timer timer = nh.createTimer(ros::Duration(0.1), timerCallback); // 创建timer

// 回调签名:
void callback(const ros::TimerEvent&);
// 其中 ros::TimerEvent:
//      ros::Time last_expected     -- 理想中前一个回调应该被执行的时间
//      ros::Time last_real         -- 实际上前一个回调执行的时间
//      ros::Time current_expected  -- 理想中当前回调应该被执行的时间
//      ros::Time current_real      -- 实际上当前回调执行的时间
//      ros::WallTime profile.last_duration -- 前一个回调执行的耗时

线程

单线程

// 单线程调用
ros::init(argc, argv, "my_node");
ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe(...);
...
ros::spin();

// 自己实现 spin()
#include <ros/callback_queue.h>
ros::NodeHandle n;
while (ros::ok())
{
  ros::getGlobalCallbackQueue()->callAvailable(ros::WallDuration(0.1));
}


// 另一种形式的单线程调用
ros::Rate r(10); // 10 hz
while (should_continue)
{
  ... do some work, publish some messages, etc. ...
  ros::spinOnce();
  r.sleep();
}

// 自己实现 spinOnce()
#include <ros/callback_queue.h>
ros::getGlobalCallbackQueue()->callAvailable(ros::WallDuration(0));

多线程

// 同步spin，阻塞
ros::MultiThreadedSpinner spinner(4); // Use 4 threads
spinner.spin(); // spin() will not return until the node has been shutdown

// 异步spin
ros::AsyncSpinner spinner(4); // Use 4 threads
spinner.start();
ros::waitForShutdown(); // 这里会阻塞

调用队列

// ros有个全局默认队列, 所有回调都会挂到这个上边
ros::getGlobalCallbackQueue()

// 创建自己的调用队列，有两层次:
#include <ros/callback_queue.h>
ros::CallbackQueue my_queue;

// 层次1: 每个 subscribe(), advertise(), advertiseService()
// 使用 Options 结构体传入自己的队列

// 层次2: 每个 NodeHandle，更常用. ros::spin() 不会自动触发这些回调，需要手动去触发
ros::NodeHandle nh;
nh.setCallbackQueue(&my_callback_queue);

// 一次调用所有的回调
callAvailable()

// 一次调用一个回调
callOne()