1. 前言
本文是webrtc拥塞控制的下文，主要介绍的是从cc-controller获取码率之后，如何将码率设置到PacingController控制发送速率，同时如何将码率分配调整到各个stream，各个stream的layer, simulcast，fec中
2. 正文
2.1 整体码控结构
webrtc中是会同时存在多个stream，但所有的stream都会共用一个码率预估和平滑发送，这很符合逻辑(虽然gcc保障带宽公平性)，发送数据的流程如上图数字所标，不同的stream最终发包的时候都是发送到RtpControllerSend的PacketSender中，然后在PacingController进行码率控制发送，发送包经过PacketRouter路由到其所属的RTP_RTCP进行缓存记录以便重传，然后通过RtpSenderEgress将包发送到Transport层，由Transport做Dtls，通过ice发到网络。
在拥塞控制(上)中已经介绍完了整个CcController通过cc-feedback计算目标码率，由cc-controller计算出来的码率最后会被放到PacingController中控制发送速度，同时计算出来的码率也会由RtpControllerSend通知观察者回调至上层做码率调整
整个调节过程如上图所示，会将cc-controller估算出来的码率分配给所有的streams，每个stream有设置好的最小码率(min_bitrate)和最大码率(max_bitrate)通过它们进行分配，本文没有介绍音频部分的分配和处理，只介绍了视频部分的，对于每个VideoStream，码率有两个部分构成，一个是media(媒体)，一个是Protection(抗丢包)
media(媒体码率)由Overhead(Rtp头，UDP头开销), Packetization(编码媒体帧封装开销)，Encoder(编码码率)构成；
而Encoder(编码码率)由可以根据simulcast进行细分，每个simulcast下还有不同的Temporal
Protection(抗丢包码率)主要由Fec码率和Rtx(nack重传码率)构成
下面逐渐对以上内容进行细述
2.2 RtpTransportController 反馈码率结果
在拥塞控制(上)中已经介绍完了整个CcController通过cc-feedback计算目标码率，RtpTransportControllerSend从CcController获取目标码率后会使用PostUpdates()进行更新
void RtpTransportControllerSend::OnTransportFeedback(
const rtcp::TransportFeedback& feedback) {
feedback_demuxer_.OnTransportFeedback(feedback);
auto feedback_time = Timestamp::Millis(clock_->TimeInMilliseconds());
task_queue_.PostTask([this, feedback, feedback_time]() {
RTC_DCHECK_RUN_ON(&task_queue_);
absl::optional<TransportPacketsFeedback> feedback_msg =
transport_feedback_adapter_.ProcessTransportFeedback(feedback,
feedback_time);
if (feedback_msg && controller_) {
//从cc-controller中获取目标码率进行设置
PostUpdates(controller_->OnTransportPacketsFeedback(*feedback_msg));
}
pacer()->UpdateOutstandingData(
transport_feedback_adapter_.GetOutstandingData());
});
}
PostUpdates()的过程如下所示:
void RtpTransportControllerSend::PostUpdates(NetworkControlUpdate update) {
if (update.congestion_window) {
// 设置拥塞窗口大小
pacer()->SetCongestionWindow(*update.congestion_window);
}
if (update.pacer_config) {
// 设置平滑发送速率
pacer()->SetPacingRates(update.pacer_config->data_rate(),
update.pacer_config->pad_rate());
}
for (const auto& probe : update.probe_cluster_configs) {
// 按照probe_cluster_config生成探测簇
pacer()->CreateProbeCluster(probe.target_data_rate, probe.id);
}
if (update.target_rate) {
// 目标码率更新了，生成目标码率
control_handler_->SetTargetRate(*update.target_rate);
// 更新码率分配
UpdateControlState();
}
}
RtpTransportControllerSend::PostUpdates()中:
-
调用SetCongestionWindow()设置pacer的拥塞窗口大小, 详见2.3.1
-
调用SetPacingRates()设置pacer的发送速率，详见2.3.2
-
调用CreateProbeCluster()设置pacer的探测簇, 详见2.3.3
-
调用UpdateControlState()将码率反馈给上层源端，去做编码，simulcast, svc，fec的码率控制。
2.3 PacingController 控制发送码率
2.3.1 拥塞窗口
PacingController使用了一个拥塞窗口来辅助平滑发送，这个拥塞窗口的值在拥塞控制(上)-2.8中提到过，其内涵是一个使用目标码率 * rtt计算的值。
PacingController使用该拥塞窗口来控制一个rtt内的数据发送，当检测到当前的发送中的数据(已发送但是未得到ack的数据)大于该窗口值得时候，就处于一个congested状态。
bool PacingController::Congested() const {
if (congestion_window_size_.IsFinite()) {
// 发送中的数据(outstanding_data_)大于当前窗口值，当前处于拥塞状态
return outstanding_data_ >= congestion_window_size_;
}
return false;
}
2.3.1.1 outstanding_data的维护
发送中的数据(outstanding_data)大小的更新过程在两个地方:
outstanding_data第一个更新在底层udp socket成功发送数据的时候会执行回调OnPacketSent()通知发送了多少数据
/**
* @description: packet成功从udp发送出去的回调，包此时未到达接收端
* @param {packet_size} 发送的数据大小
* @param {send_time} 发送的时间
* @return {void}
*/
void PacingController::OnPacketSent(RtpPacketMediaType packet_type,
DataSize packet_size,
Timestamp send_time) {
if (!first_sent_packet_time_) {
first_sent_packet_time_ = send_time;
}
bool audio_packet = packet_type == RtpPacketMediaType::kAudio;
if (!audio_packet || account_for_audio_) {
// 更新已发送的数据大小
UpdateBudgetWithSentData(packet_size);
}
last_send_time_ = send_time;
last_process_time_ = send_time;
}
其中调用的UpdateBudgetWithSentData()如下：
void PacingController::UpdateBudgetWithSentData(DataSize size) {
// 增加发送中的数据量
outstanding_data_ += size;
if (mode_ == ProcessMode::kPeriodic) {
// 数据已发送,降低发送budget
media_budget_.UseBudget(size.bytes());
padding_budget_.UseBudget(size.bytes());
} else {
media_debt_ += size;
media_debt_ = std::min(media_debt_, media_rate_ * kMaxDebtInTime);
padding_debt_ += size;
padding_debt_ = std::min(padding_debt_, padding_rate_ * kMaxDebtInTime);
}
}
其中:
- 主要关注最上面的outstang_data的更新
- 其次，pacing controller使用了周期(KPeriodic)和动态(KDynamic)两种方式管理媒体发送速度，后文再详述
outstanding_data第二个更新主要是通过PacingController::UpdateOutstandingData()直接做数值上的更新的
void PacingController::UpdateOutstandingData(DataSize outstanding_data) {
const bool was_congested = Congested();
outstanding_data_ = outstanding_data;//直接更新
if (was_congested && !Congested()) {
TimeDelta elapsed_time = UpdateTimeAndGetElapsed(CurrentTime());
UpdateBudgetWithElapsedTime(elapsed_time);
}
}
这个调用会在socket层探测到当前网络的发生变化的时候，会直接对outstand_data重置为0
void RtpTransportControllerSend::OnNetworkRouteChanged(
....
task_queue_.PostTask([this, msg, network_route] {
....
// 网络路由发生了改变，将发送中的数据重设为0
pacer()->UpdateOutstandingData(DataSize::Zero());
});
}
}
void RtpTransportControllerSend::OnNetworkAvailability(bool network_available) {
....
task_queue_.PostTask([this, msg]() {
....
// 网络的可用性发生改变，对发送中的数据大小进行重新初始化为0
pacer()->UpdateOutstandingData(DataSize::Zero());
....
});
....
}
或者在收到了cc-feedback或者RR-report的时候，在这里它们它们充当了一个ack的作用，通过它们就可以将发送中的数据更新降低, 举cc-feedback为例：
void RtpTransportControllerSend::OnTransportFeedback(
const rtcp::TransportFeedback& feedback) {
....
task_queue_.PostTask([this, feedback, feedback_time]() {
// 解析cc-feedback获得feedback_msg, 更新transport_feedback_adapter_
absl::optional<TransportPacketsFeedback> feedback_msg =
transport_feedback_adapter_.ProcessTransportFeedback(feedback,
feedback_time);
// 更新发送中的数据大小
pacer()->UpdateOutstandingData(
transport_feedback_adapter_.GetOutstandingData());
});
}
其中transport_feedback_adapter使用变量in_filght对发送数据进行统计维护
DataSize TransportFeedbackAdapter::GetOutstandingData() const {
return in_flight_.GetOutstandingData(network_route_);
}
in_flight详细维护过程如下，发送数据的时候会在TransportFeedbackAdapter::ProcessSentPacket()增加in_filght的大小
absl::optional<SentPacket> TransportFeedbackAdapter::ProcessSentPacket(
const rtc::SentPacket& sent_packet) {
....
if (sent_packet.info.included_in_feedback || sent_packet.packet_id != -1) {
....
if (!packet_retransmit) {
if (it->second.sent.sequence_number > last_ack_seq_num_)
in_flight_.AddInFlightPacketBytes(it->second);// 新增已发送数据
}
}
}
....
}
收到feedback的时候会调用TransportFeedbackAdapter::ProcessTransportFeedbackInner()，对当前已经ack到的seq从in_flight中删除
TransportFeedbackAdapter::ProcessTransportFeedbackInner(
const rtcp::TransportFeedback& feedback,
Timestamp feedback_receive_time) {
for (const auto& packet : feedback.GetAllPackets()) {
int64_t seq_num = seq_num_unwrapper_.Unwrap(packet.sequence_number());
if (seq_num > last_ack_seq_num_) {
// Starts at history_.begin() if last_ack_seq_num_ < 0, since any valid
// sequence number is >= 0.
for (auto it = history_.upper_bound(last_ack_seq_num_);
it != history_.upper_bound(seq_num); ++it) {
// 将上一个包到已经ack的当前包的一个in_flight删除
in_flight_.RemoveInFlightPacketBytes(it->second);
}
last_ack_seq_num_ = seq_num;
}
}
2.3.1.2 拥塞状态的使用
从congested()获取到的拥塞状态主要在PacingController控制下次发送时间的时候起作用，NextSendTime()是PacingController给PacedSender的建议下次发包时间，还未到这个时间PacedSender陷入睡眠，当到达这个时间后会让PacingController起来工作，处理队列中的包。
其中就看到了当判断到当前处于一个拥塞状态的时候，就会返回拥塞沉睡间隔，暂时不处理队列中的包。
Timestamp PacingController::NextSendTime() const {
....
if (Congested() || packet_counter_ == 0) {
// We need to at least send keep-alive packets with some interval.
return last_send_time_ + kCongestedPacketInterval;
}
....
}
此外在发送的时候，也会判断拥塞窗口是否已经使用完毕，如果是，则不发包，详见2.3.2.2
2.3.2 PacingRate
2.3.2.1 PacingRate的更新
Pacing相关码率的更新会通过PacingController::SetPacingRates()进行
void PacingController::SetPacingRates(DataRate pacing_rate,
DataRate padding_rate) {
RTC_DCHECK_GT(pacing_rate, DataRate::Zero());
media_rate_ = pacing_rate;
padding_rate_ = padding_rate;
pacing_bitrate_ = pacing_rate;
padding_budget_.set_target_rate_kbps(padding_rate.kbps());
RTC_LOG(LS_VERBOSE) << "bwe:pacer_updated pacing_kbps="
<< pacing_bitrate_.kbps()
<< " padding_budget_kbps=" << padding_rate.kbps();
}
PacingController::SetPacingRates()中主要设置的是pacing_rate和padding_rate，这两个值在GoogCcNetworkController::GetPacingRates()设置的，前者是cc-controller最终给出的目标码率 * 系数, 而后者是一个设置值，这个值还没细看不知道是否动态更新，但只有OnStreamConfig()的时候才设置了一次，感觉是应该不是动态更新
PacerConfig GoogCcNetworkController::GetPacingRates(Timestamp at_time) const {
// Pacing rate is based on target rate before congestion window pushback,
// because we don't want to build queues in the pacer when pushback occurs.
// 此处的pacing rate使用的是last_loss_based_target_rate_, 这个值没有经过拥塞窗口的更新处理
// 但是没太看懂注释，"当退避产生的时候不想在pacer创建队列"，因为pacer有两种,一种是有queue的
// 一种是无queue的，可能想要表达的是congestion push back不应用在有queue的队列上?
DataRate pacing_rate =
std::max(min_total_allocated_bitrate_, last_loss_based_target_rate_) *
pacing_factor_;
// padding_rate 主要的值还是max_padding_rate_，这是一个来自于外部(bitrateAllocation)计算的一个值
// 其次，它肯定不能大于窗口控制的码率(last_pushback_target_rate_)
DataRate padding_rate =
std::min(max_padding_rate_, last_pushback_target_rate_);
PacerConfig msg;
msg.at_time = at_time;
msg.time_window = TimeDelta::Seconds(1);//1s
msg.data_window = pacing_rate * msg.time_window;
msg.pad_window = padding_rate * msg.time_window;
return msg;
}
2.3.2.2 PacingRate的使用
PacingRate是用来控制发送速度的，但PacingController并没有直接使用它，PacingController中基于PacingRate为了控制方便，封装了两个类用来控制发送速度，一个是(class IntervalBudget)，一个是media_debt(DataSize)，前者使用在PacingController的Periodic模式中，在该模式下，PacingController会周期间隔(5ms)的去处理包队列，后者使用在是Dynamic模式，在该模式下，PacingController处理包队列的间隔是任意的，media_budget和media_debt的值会随着时间的流逝而变化；下面以media_budget为例介绍
PacingRate在PacingController::ProcessPackets()中被设置到media_budget上, 并在下面的while循环中使用GetPendingPacket()从队列获取包去发送的时候会使用到:
void PacingController::ProcessPackets() {
....
if (elapsed_time > TimeDelta::Zero()) {
// 获取pacing rate
DataRate target_rate = pacing_bitrate_;
....
if (mode_ == ProcessMode::kPeriodic) {
// 将pacing rate设置到media_budget中
media_budget_.set_target_rate_kbps(target_rate.kbps());
// 基于流逝的时间更新budget
UpdateBudgetWithElapsedTime(elapsed_time);
} else {
media_rate_ = target_rate;
}
}
....
while (!paused_) {
....
// 根据budget和发送时间从queue中获取下一个要发的包
std::unique_ptr<RtpPacketToSend> rtp_packet =
GetPendingPacket(pacing_info, target_send_time, now);
....
}
....
}
PacingController::ProcessPackets() 中
-
会调用
set_target_rate_kbps()将pacing rate更新到media_budget的目标码率中 -
基于流逝的时间使用
UpdateBudgetWithElapsedTime()去增大media_budget 的值void PacingController::UpdateBudgetWithElapsedTime(TimeDelta delta) { if (mode_ == ProcessMode::kPeriodic) { delta = std::min(kMaxProcessingInterval, delta); media_budget_.IncreaseBudget(delta.ms()); // old + target_bitrate * delate_ms padding_budget_.IncreaseBudget(delta.ms()); } else { media_debt_ -= std::min(media_debt_, media_rate_ * delta); padding_debt_ -= std::min(padding_debt_, padding_rate_ * delta); } } -
然后在发包的时候调用PacingController::GetPendingPacket()队列中待发送包的时候会通过media_budget判断当前是否还有足够的发送预算
std::unique_ptr<RtpPacketToSend> PacingController::GetPendingPacket( const PacedPacketInfo& pacing_info, Timestamp target_send_time, Timestamp now) { if (packet_queue_.Empty()) { return nullptr; } bool unpaced_audio_packet = !pace_audio_ && packet_queue_.LeadingAudioPacketEnqueueTime().has_value(); bool is_probe = pacing_info.probe_cluster_id != PacedPacketInfo::kNotAProbe; if (!unpaced_audio_packet && !is_probe) { if (Congested()) { // 拥塞窗口满了，不发包 return nullptr; } // 没有budget了，不发包 if (mode_ == ProcessMode::kPeriodic) { if (media_budget_.bytes_remaining() <= 0) { // Not enough budget. return nullptr; } } else { // Dynamic processing mode. if (now <= target_send_time) { // target_send_time是基于当前队列大小/目标发送码率(media_rate)得到的一个目标发送 // 时间值 // We allow sending slightly early if we think that we would actually // had been able to, had we been right on time - i.e. the current debt // is not more than would be reduced to zero at the target sent time. TimeDelta flush_time = media_debt_ / media_rate_; if (now + flush_time > target_send_time) { return nullptr; } } } } return packet_queue_.Pop(); }
而media_budget的减少在之前的PacingController::UpdateBudgetWithSentData()中已经提到过，在 packet成功发送的时候会调用onPacketSent，则会被消耗，最终调用UpdateBudgetWithSentData()去减少
void PacingController::UpdateBudgetWithSentData(DataSize size) {
outstanding_data_ += size;
if (mode_ == ProcessMode::kPeriodic) {
// 数据已发送,降低发送budget
media_budget_.UseBudget(size.bytes());
padding_budget_.UseBudget(size.bytes());
} else {
media_debt_ += size;
media_debt_ = std::min(media_debt_, media_rate_ * kMaxDebtInTime);
padding_debt_ += size;
padding_debt_ = std::min(padding_debt_, padding_rate_ * kMaxDebtInTime);
}
}
2.3.3 ProbeCluster
PacingController除了控制发送速率，同时还肩负另一个功能，支持码率探测，其使用了一个叫做BitratePorber 的类以用来响应cc-controller的探测需求，BitratePorber在拥塞控制(上)-码率预估的2.4.2已经提到过了，此处不再赘述。
PacingController部分的内容到此结束了，这是cc-controller估算出来的码率对下游最终发送上的应用，接下来介绍估算出来的码率对上游编码和fec等的影响
2.4 将码率分配到源
void RtpTransportControllerSend::PostUpdates(NetworkControlUpdate update) {
....
if (update.target_rate) {
// 通知上层
control_handler_->SetTargetRate(*update.target_rate);
UpdateControlState();
}
}
在RtpTransportControllerSend::PostUpdates()的最后：
-
将目标码率设置到了control_handler_
-
调用UpdateControlState()，通过control_handler获取目标码率，并将最新码率回调给观察者上层
void RtpTransportControllerSend::UpdateControlState() { absl::optional<TargetTransferRate> update = control_handler_->GetUpdate(); if (!update) return; retransmission_rate_limiter_.SetMaxRate(update->target_rate.bps()); RTC_DCHECK(observer_ != nullptr); observer_->OnTargetTransferRate(*update); }
上层观察者(call)接收码率变换通知的是Call::OnTargetTransferRate()
void Call::OnTargetTransferRate(TargetTransferRate msg) {
RTC_DCHECK_RUN_ON(send_transport_queue());
uint32_t target_bitrate_bps = msg.target_rate.bps();
// For controlling the rate of feedback messages.
// 控制feedback的开销在5%左右
receive_side_cc_.OnBitrateChanged(target_bitrate_bps);
// 重新分配到各个stream中的码流
bitrate_allocator_->OnNetworkEstimateChanged(msg);
// 更新统计直方图
worker_thread_->PostTask(
ToQueuedTask(task_safety_, [this, target_bitrate_bps]() {
RTC_DCHECK_RUN_ON(worker_thread_);
last_bandwidth_bps_ = target_bitrate_bps;
// Ignore updates if bitrate is zero (the aggregate network state is
// down) or if we're not sending video.
if (target_bitrate_bps == 0 || video_send_streams_.empty()) {
estimated_send_bitrate_kbps_counter_.ProcessAndPause();
pacer_bitrate_kbps_counter_.ProcessAndPause();
return;
}
estimated_send_bitrate_kbps_counter_.Add(target_bitrate_bps / 1000);
// Pacer bitrate may be higher than bitrate estimate if enforcing min
// bitrate.
uint32_t pacer_bitrate_bps =
std::max(target_bitrate_bps, min_allocated_send_bitrate_bps_);
pacer_bitrate_kbps_counter_.Add(pacer_bitrate_bps / 1000);
}));
}
Call::OnTargetTransferRate()中:
-
调用 ReceiveSideCongestionController::OnBitrateChanged(), 用于更新cc-report部分的发送时间间隔，将这一部分的发送码率控制在目标码率的5%。
void RemoteEstimatorProxy::OnBitrateChanged(int bitrate_bps) { // TwccReportSize = Ipv4(20B) + UDP(8B) + SRTP(10B) + // AverageTwccReport(30B) // TwccReport size at 50ms interval is 24 byte. // TwccReport size at 250ms interval is 36 byte. // AverageTwccReport = (TwccReport(50ms) + TwccReport(250ms)) / 2 constexpr int kTwccReportSize = 20 + 8 + 10 + 30; const double kMinTwccRate = kTwccReportSize * 8.0 * 1000.0 / send_config_.max_interval->ms(); const double kMaxTwccRate = kTwccReportSize * 8.0 * 1000.0 / send_config_.min_interval->ms(); // Let TWCC reports occupy 5% of total bandwidth. // bandwidth_fraction 为 0.05 MutexLock lock(&lock_); send_interval_ms_ = static_cast<int>( 0.5 + kTwccReportSize * 8.0 * 1000.0 / rtc::SafeClamp(send_config_.bandwidth_fraction * bitrate_bps, kMinTwccRate, kMaxTwccRate)); } -
调用BitrateAllocator::OnNetworkEstimateChanged()重新将码率分配到各个流中, BitrateAllocator中的内容比较多，会在2.5展开介绍
-
最后还Post一个任务，这个任务会将发送码率更新到统计直方图
2.5 BitrateAllocator 分配码率
BitrateAllocator是对码率进行重新分配的分配器，逻辑入口为BitrateAllocator::OnNetworkEstimateChanged()
其会把码率分配到不同的stream上
void BitrateAllocator::OnNetworkEstimateChanged(TargetTransferRate msg) {
RTC_DCHECK_RUN_ON(&sequenced_checker_);
// 获取目标码率和保守码率
last_target_bps_ = msg.target_rate.bps();
last_stable_target_bps_ = msg.stable_target_rate.bps();
last_non_zero_bitrate_bps_ =
last_target_bps_ > 0 ? last_target_bps_ : last_non_zero_bitrate_bps_;
// loss_ration 放大255
int loss_ratio_255 = msg.network_estimate.loss_rate_ratio * 255;
last_fraction_loss_ =
rtc::dchecked_cast<uint8_t>(rtc::SafeClamp(loss_ratio_255, 0, 255));
last_rtt_ = msg.network_estimate.round_trip_time.ms();
last_bwe_period_ms_ = msg.network_estimate.bwe_period.ms();
// Periodically log the incoming BWE.
int64_t now = msg.at_time.ms();
if (now > last_bwe_log_time_ + kBweLogIntervalMs) {
RTC_LOG(LS_INFO) << "Current BWE " << last_target_bps_;
last_bwe_log_time_ = now;
}
// 按照目标码率(target_bitrate)为所有的stream分配码率
auto allocation = AllocateBitrates(allocatable_tracks_, last_target_bps_);
// 按照保守目标码率(stable_target)为所有的stream分配码率
auto stable_bitrate_allocation =
AllocateBitrates(allocatable_tracks_, last_stable_target_bps_);
for (auto& config : allocatable_tracks_) {
uint32_t allocated_bitrate = allocation[config.observer];
uint32_t allocated_stable_target_rate =
stable_bitrate_allocation[config.observer];
BitrateAllocationUpdate update;
update.target_bitrate = DataRate::BitsPerSec(allocated_bitrate);
update.stable_target_bitrate =
DataRate::BitsPerSec(allocated_stable_target_rate);
update.packet_loss_ratio = last_fraction_loss_ / 256.0;
update.round_trip_time = TimeDelta::Millis(last_rtt_);
update.bwe_period = TimeDelta::Millis(last_bwe_period_ms_);
update.cwnd_reduce_ratio = msg.cwnd_reduce_ratio;
// 更新流上的码率
uint32_t protection_bitrate = config.observer->OnBitrateUpdated(update);
if (allocated_bitrate == 0 && config.allocated_bitrate_bps > 0) {
if (last_target_bps_ > 0)
++num_pause_events_;
// The protection bitrate is an estimate based on the ratio between media
// and protection used before this observer was muted.
uint32_t predicted_protection_bps =
(1.0 - config.media_ratio) * config.config.min_bitrate_bps;
RTC_LOG(LS_INFO) << "Pausing observer " << config.observer
<< " with configured min bitrate "
<< config.config.min_bitrate_bps
<< " and current estimate of " << last_target_bps_
<< " and protection bitrate "
<< predicted_protection_bps;
} else if (allocated_bitrate > 0 && config.allocated_bitrate_bps == 0) {
if (last_target_bps_ > 0)
++num_pause_events_;
RTC_LOG(LS_INFO) << "Resuming observer " << config.observer
<< ", configured min bitrate "
<< config.config.min_bitrate_bps
<< ", current allocation " << allocated_bitrate
<< " and protection bitrate " << protection_bitrate;
}
// Only update the media ratio if the observer got an allocation.
if (allocated_bitrate > 0)
config.media_ratio = MediaRatio(allocated_bitrate, protection_bitrate);
config.allocated_bitrate_bps = allocated_bitrate;
}
// 更新limit,编码对一些特地分辨率的设置的预设值，当编码器发生调整的
// 时候，limit是会发生改变的
UpdateAllocationLimits();
}
BitrateAllocator::OnNetworkEstimateChanged()中:
- 首先从TargetTransferRate获取目标码率和保守码率，然后调用AllocateBitrates()将码率分配到各个stream中， AllocateBitrates()在2.5.1中会详细介绍
- 然后调用observer->OnBitrateUpdated()将分配的结果发送到各个流中，音频的处理函数为AudioSendStream::OnBitrateUpdated()， 视频的处理函数为VideoSendStreamImpl::OnBitrateUpdated()，这一块在2.5.2中展开详细介绍
- 给流分配的码率的也会导致帧率和码率的调整，从而影响到每个流的min_bitrate和max_bitrate(对于编码的min_bitrate和max_bitrate，webrtc中两种方式设置，一种是从trial中获得，一种是对于一些分辨率的mo人设置，详细的可参考： encoder_info_setting.cc)，最终会调用UpdateAllocationLimits()计算总的能分配到的min_bitrate和max_bitrate，通过GoogCcNetworkController::OnStreamsConfig()传到cc-controller中去更新SendSideBandwidthEstimation，让其再次做码率预估，看之前预估的码率是否需要修改，重新通知整个系统进行更改，这一块将会在2.5.3中详细介绍
2.5.1 AllocateBitrates()
AllocateBitrates()会将码率分配到所有的stream(audio stream, video stream)，主要是根据每个流config中的min_bitrate_bps和max_bitrate_bps进行分配。
std::map<BitrateAllocatorObserver*, int> AllocateBitrates(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate) {
if (allocatable_tracks.empty())
return std::map<BitrateAllocatorObserver*, int>();
if (bitrate == 0)
return ZeroRateAllocation(allocatable_tracks);
// 统计所有track设定的最大最小码率
uint32_t sum_min_bitrates = 0;
uint32_t sum_max_bitrates = 0;
for (const auto& observer_config : allocatable_tracks) {
sum_min_bitrates += observer_config.config.min_bitrate_bps;
sum_max_bitrates += observer_config.config.max_bitrate_bps;
}
// Not enough for all observers to get an allocation, allocate according to:
// enforced min bitrate -> allocated bitrate previous round -> restart paused
// streams.
// 码率不够，按照最低分配的策略，给每个流分配码率
if (!EnoughBitrateForAllObservers(allocatable_tracks, bitrate,
sum_min_bitrates))
return LowRateAllocation(allocatable_tracks, bitrate);
// All observers will get their min bitrate plus a share of the rest. This
// share is allocated to each observer based on its bitrate_priority.
if (bitrate <= sum_max_bitrates)
return NormalRateAllocation(allocatable_tracks, bitrate, sum_min_bitrates);
// All observers will get up to transmission_max_bitrate_multiplier_ x max.
return MaxRateAllocation(allocatable_tracks, bitrate, sum_max_bitrates);
}
在AllocateBitrates()中:
- 首先遍历所有的track，统计出sum_min_bitrates和sum_max_bitrates
- 调用EnoughBitrateForAllObservers()判断目标码率是否能够满足分配到所有的流，如果不是则调用LowRateAllocation()进行最小码率分配，否则的话检查当前的码率是否小于最大所需码率，是的话调用NormalRateAllocation()进行分配，当超过了最大所需码率后，则调用MaxRateAllocation()进行分配。
EnoughBitrateForAllObservers()如下：
bool EnoughBitrateForAllObservers(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate,
uint32_t sum_min_bitrates) {
if (bitrate < sum_min_bitrates)
return false;
// 计算出多余的码流可以平均摊给每个track的为多少
uint32_t extra_bitrate_per_observer =
(bitrate - sum_min_bitrates) /
static_cast<uint32_t>(allocatable_tracks.size());
// 算上多余均摊之后，无法满足每个track所需的码率(考虑上了Hysteresis即暂停滞后过所需要的增益码率),则false
for (const auto& observer_config : allocatable_tracks) {
if (observer_config.config.min_bitrate_bps + extra_bitrate_per_observer <
observer_config.MinBitrateWithHysteresis()) {
return false;
}
}
return true;
}
EnoughBitrateForAllObservers()中:
-
先按照每个流的最低码率进行分配，然后将多余的码率平均分配到每个流上，但如果在这种分配模式下流的码率仍然无法达到最低的滞后码率，则认为当前的码率不足；流的滞后(Hysteresis)是该率在上一次分配中由于码率有限被分配到了0码率而导致暂停，此刻它的最低滞后码率将会在最低码率的基础上增加10%
/** * @description: 获取最小的码率(会考虑过流是否暂停过，如果是，增益10%) * @param {*} * @return {*} */ uint32_t bitrate_allocator_impl::AllocatableTrack::MinBitrateWithHysteresis() const { uint32_t min_bitrate = config.min_bitrate_bps; if (LastAllocatedBitrate() == 0) { // 被暂停发送了要追加一个10%的系数提升 min_bitrate += std::max(static_cast<uint32_t>(kToggleFactor * min_bitrate), kMinToggleBitrateBps); } // Account for protection bitrate used by this observer in the previous // allocation. // Note: the ratio will only be updated when the stream is active, meaning a // paused stream won't get any ratio updates. This might lead to waiting a bit // longer than necessary if the network condition improves, but this is to // avoid too much toggling. // 启用了fec，需要通过media_ratio 把fec的码率也计算进去，其中media_ratio就是媒体所占分配码率部分 // media_ratio = allocated_bitrate - protection_bitrate / allocated_bitrate if (media_ratio > 0.0 && media_ratio < 1.0) min_bitrate += min_bitrate * (1.0 - media_ratio); return min_bitrate; }
码率不足的时候会调用LowRateAllocation()进行码率分配
std::map<BitrateAllocatorObserver*, int> LowRateAllocation(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate) {
std::map<BitrateAllocatorObserver*, int> allocation;
// Start by allocating bitrate to observers enforcing a min bitrate, hence
// remaining_bitrate might turn negative.
// 先每个流分配一个强制最小码率(enforce_min_bitrate)
int64_t remaining_bitrate = bitrate;
for (const auto& observer_config : allocatable_tracks) {
int32_t allocated_bitrate = 0;
if (observer_config.config.enforce_min_bitrate)
allocated_bitrate = observer_config.config.min_bitrate_bps;
allocation[observer_config.observer] = allocated_bitrate;
remaining_bitrate -= allocated_bitrate;
}
// Allocate bitrate to all previously active streams.
// 如果还有码率，则为活跃(不暂停)的流分配码率
if (remaining_bitrate > 0) {
for (const auto& observer_config : allocatable_tracks) {
// 配置了enforce_min_bitrate或者当前处于暂停的流，先跳过
if (observer_config.config.enforce_min_bitrate ||
observer_config.LastAllocatedBitrate() == 0)
continue;
// 获取需要的码率(包含fec)
uint32_t required_bitrate = observer_config.MinBitrateWithHysteresis();
if (remaining_bitrate >= required_bitrate) {
allocation[observer_config.observer] = required_bitrate;
remaining_bitrate -= required_bitrate;
}
}
}
// Allocate bitrate to previously paused streams.
// 如果还有码率，则为暂停的流分配码率
if (remaining_bitrate > 0) {
for (const auto& observer_config : allocatable_tracks) {
if (observer_config.LastAllocatedBitrate() != 0)
continue;
// Add a hysteresis to avoid toggling.
uint32_t required_bitrate = observer_config.MinBitrateWithHysteresis();
if (remaining_bitrate >= required_bitrate) {
allocation[observer_config.observer] = required_bitrate;
remaining_bitrate -= required_bitrate;
}
}
}
// Split a possible remainder evenly on all streams with an allocation.
// 如果还有剩余的码率，则均分给所有的流
if (remaining_bitrate > 0)
DistributeBitrateEvenly(allocatable_tracks, remaining_bitrate, false, 1,
&allocation);
RTC_DCHECK_EQ(allocation.size(), allocatable_tracks.size());
return allocation;
}
LowRateAllocation()中：
- 先每个流分配一个强制最小码率(enforce_min_bitrate)
- 如果还有码率，则为活跃(LastAllocatedBitrate不为0)的流分配码率
- 如果还有码率，则为暂停的流分配码率
- 如果还有码率，再均分给所有的流
关于这个策略的第二点要非常注意，这种分配模式下将可能导致码率低的时候大部分流勉强正常工作，而某部分流持续异常，出现这种情况，可以检查一下带宽是否太低了。
码率充足的时候会调用NormalRateAllocation()进行分配
std::map<BitrateAllocatorObserver*, int> NormalRateAllocation(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate,
uint32_t sum_min_bitrates) {
// 为每个流分配最小满足码率，并计算剩余容量
std::map<BitrateAllocatorObserver*, int> allocation;
std::map<BitrateAllocatorObserver*, int> observers_capacities;
for (const auto& observer_config : allocatable_tracks) {
// 分配满足最小码率
allocation[observer_config.observer] =
observer_config.config.min_bitrate_bps;
// 每个stream的剩余能被分配的容量
observers_capacities[observer_config.observer] =
observer_config.config.max_bitrate_bps -
observer_config.config.min_bitrate_bps;
}
bitrate -= sum_min_bitrates;
// TODO(srte): Implement fair sharing between prioritized streams, currently
// they are treated on a first come first serve basis.
// 所有的流都会有一个优先分配码率，此处是先将码率分配给有优先分配码率的流
for (const auto& observer_config : allocatable_tracks) {
// 计算剩余的码率
int64_t priority_margin = observer_config.config.priority_bitrate_bps -
allocation[observer_config.observer];
if (priority_margin > 0 && bitrate > 0) {
int64_t extra_bitrate = std::min<int64_t>(priority_margin, bitrate);
allocation[observer_config.observer] +=
rtc::dchecked_cast<int>(extra_bitrate);
observers_capacities[observer_config.observer] -= extra_bitrate;
bitrate -= extra_bitrate;
}
}
// From the remaining bitrate, allocate a proportional amount to each observer
// above the min bitrate already allocated.
// 基于各个流的优先码率比例对对剩余的码率进行分配
if (bitrate > 0)
DistributeBitrateRelatively(allocatable_tracks, bitrate,
observers_capacities, &allocation);
return allocation;
}
NormalRateAllocation()中:
- 首先会满足所有track的最小码率(min_bitrate_bps)，然后通过max_bitrate_bps - min_bitrate_bps计算了一个值capacities，表示该流还能够被分配的容量
- 如果还有码率能被分配，遍历所有的流，将分配给流的码率提升到priority_bitrate，这个值目前仅在音频中见到会通过field trial的方式设置，视频中为0，我把它称之为流的优先码率，相对于没有设置该值的流，设置了的能够优先获得更高的码率
- 如果还有码率剩余，则调用DistributeBitrateRelatively()，将剩余的码率基于各个流的优先码率的比进行分配
当能被分配的码率超过最大的时候则直接调用MaxRateAllocation()按照最大的分配
std::map<BitrateAllocatorObserver*, int> MaxRateAllocation(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate,
uint32_t sum_max_bitrates) {
std::map<BitrateAllocatorObserver*, int> allocation;
// 按照max_bitrate_bps进行分配
for (const auto& observer_config : allocatable_tracks) {
allocation[observer_config.observer] =
observer_config.config.max_bitrate_bps;
bitrate -= observer_config.config.max_bitrate_bps;
}
// 将剩余得码率均分给所有流，但不超过流最大码率得2倍
DistributeBitrateEvenly(allocatable_tracks, bitrate, true,
kTransmissionMaxBitrateMultiplier, &allocation);
return allocation;
}
void DistributeBitrateEvenly(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate,
bool include_zero_allocations,
int max_multiplier,
std::map<BitrateAllocatorObserver*, int>* allocation) {
RTC_DCHECK_EQ(allocation->size(), allocatable_tracks.size());
// 统计所有的流
std::multimap<uint32_t, const AllocatableTrack*> list_max_bitrates;
for (const auto& observer_config : allocatable_tracks) {
if (include_zero_allocations ||
allocation->at(observer_config.observer) != 0) {
list_max_bitrates.insert(
{observer_config.config.max_bitrate_bps, &observer_config});
}
}
auto it = list_max_bitrates.begin();
while (it != list_max_bitrates.end()) {
RTC_DCHECK_GT(bitrate, 0);
// 对剩余码率进行均分
uint32_t extra_allocation =
bitrate / static_cast<uint32_t>(list_max_bitrates.size());
// 求得均分后得码率
uint32_t total_allocation =
extra_allocation + allocation->at(it->second->observer);
bitrate -= extra_allocation;
if (total_allocation > max_multiplier * it->first) {
// There is more than we can fit for this observer, carry over to the
// remaining observers.
// 如果均分后得码率大于最大码率得两倍，则最多给某个流分配两倍得最大码率
bitrate += total_allocation - max_multiplier * it->first;
total_allocation = max_multiplier * it->first;
}
// Finally, update the allocation for this observer.
allocation->at(it->second->observer) = total_allocation;
it = list_max_bitrates.erase(it);
}
}
MaxRateAllocation()中:
- 首先将码率满足每个流的最大码率
- 然后将多出来的码率调用DistributeBitrateEvenly()均分给所有流，但同时不能超过流最大码率的两倍
2.5.2 视频流码率分配更新
当从BitratteAllocator中计算出每个stream的码率之后，视频流的码率会通过VideoSendStreamImpl::OnBitrateUpdated()进行更新分配
uint32_t VideoSendStreamImpl::OnBitrateUpdated(BitrateAllocationUpdate update) {
RTC_DCHECK_RUN_ON(worker_queue_);
RTC_DCHECK(rtp_video_sender_->IsActive())
<< "VideoSendStream::Start has not been called.";
// When the BWE algorithm doesn't pass a stable estimate, we'll use the
// unstable one instead.
if (update.stable_target_bitrate.IsZero()) {
update.stable_target_bitrate = update.target_bitrate;
}
// 通过目标码率，计算编码码率，保护码率(fec, rtx等)分配
rtp_video_sender_->OnBitrateUpdated(update, stats_proxy_->GetSendFrameRate());
// 获取更新后的目标编码码率
encoder_target_rate_bps_ = rtp_video_sender_->GetPayloadBitrateBps();
// 获取更新后的目标fec码率
const uint32_t protection_bitrate_bps =
rtp_video_sender_->GetProtectionBitrateBps();
// 计算link_allocation
DataRate link_allocation = DataRate::Zero();
if (encoder_target_rate_bps_ > protection_bitrate_bps) {
link_allocation =
DataRate::BitsPerSec(encoder_target_rate_bps_ - protection_bitrate_bps);//???
}
// 计算overhead(一些rtp头什么的)
DataRate overhead =
update.target_bitrate - DataRate::BitsPerSec(encoder_target_rate_bps_);
DataRate encoder_stable_target_rate = update.stable_target_bitrate;
if (encoder_stable_target_rate > overhead) {
// 保守码率比overhead大，则保守编码码率为保守码率 - overhead
encoder_stable_target_rate = encoder_stable_target_rate - overhead;
} else {
// 否则，保守编码码率直接设置为保守码率
encoder_stable_target_rate = DataRate::BitsPerSec(encoder_target_rate_bps_);
}
// encoder_target_rate_bps_不得大于上限
encoder_target_rate_bps_ =
std::min(encoder_max_bitrate_bps_, encoder_target_rate_bps_);
encoder_stable_target_rate =
std::min(DataRate::BitsPerSec(encoder_max_bitrate_bps_),
encoder_stable_target_rate);
DataRate encoder_target_rate = DataRate::BitsPerSec(encoder_target_rate_bps_);
//link_allocation不小于encoder_target_rate
link_allocation = std::max(encoder_target_rate, link_allocation);
// 更新编码器
video_stream_encoder_->OnBitrateUpdated(
encoder_target_rate, encoder_stable_target_rate, link_allocation,
rtc::dchecked_cast<uint8_t>(update.packet_loss_ratio * 256),
update.round_trip_time.ms(), update.cwnd_reduce_ratio);
stats_proxy_->OnSetEncoderTargetRate(encoder_target_rate_bps_);
return protection_bitrate_bps;
}
VideoSendStreamImpl::OnBitrateUpdated()中：
-
首先使用RtpVideoSender::OnBitrateUpdated()，通过分配给该流的目标码率，计算分配的编码码率和抗丢包码率，将在2.5.2.1进行详述
-
然后获取目标编码码率(encoder_target_rate_bps)和目标保护码率(protection_bitrate_bps，分配给fec, nack等), 并且定义了一个变量link_allocation，这个值表示的是当前video stream能使用的网络通道容量（已排除调protection rate），这个值最低会比encoder_target_rate大。但是encoder_target_rate_bps_ - protection_bitrate_bps这一步的计算无法理解，因为encoder_target_rate_bps已经排除了protection_bitrate_bps，很费解。
-
最终会调用VideoStreamEncoder::OnBitrateUpdated()，使用新的编码码率去更新编码器编码码率，调整各simulcast层级和temporal layer的码率以及帧率，在2.5.2.2进行详述
2.5.2.1 编码码率和抗丢包码率的更新
RtpVideoSender::OnBitrateUpdated()中会通过分配给当前流的目标码率和当前的帧率计算新的编码码率和抗丢包码率
void RtpVideoSender::OnBitrateUpdated(BitrateAllocationUpdate update,
int framerate) {
// Substract overhead from bitrate.
// 计算simulcast流的平均overhead, overhead:Rtp扩展头 + rtp头开销
MutexLock lock(&mutex_);
size_t num_active_streams = 0;
size_t overhead_bytes_per_packet = 0;
for (const auto& stream : rtp_streams_) {
if (stream.rtp_rtcp->SendingMedia()) {
// 统计rtp上的扩展头+rtp头开销
overhead_bytes_per_packet += stream.rtp_rtcp->ExpectedPerPacketOverhead();
++num_active_streams;
}
}
if (num_active_streams > 1) {
overhead_bytes_per_packet /= num_active_streams;
}
// media_packet_header + udp header
// 计算每个包的over_head
DataSize packet_overhead = DataSize::Bytes(
overhead_bytes_per_packet + transport_overhead_bytes_per_packet_);
// 计算包的最大大小
DataSize max_total_packet_size = DataSize::Bytes(
rtp_config_.max_packet_size + transport_overhead_bytes_per_packet_);
uint32_t payload_bitrate_bps = update.target_bitrate.bps();
if (send_side_bwe_with_overhead_ && has_packet_feedback_) {
// 如果预估码率是包含over_head_的，则减去overhead这部分
DataRate overhead_rate =
CalculateOverheadRate(update.target_bitrate, max_total_packet_size,
packet_overhead, Frequency::Hertz(framerate));
// TODO(srte): We probably should not accept 0 payload bitrate here.
// 计算payload的码率
payload_bitrate_bps = rtc::saturated_cast<uint32_t>(payload_bitrate_bps -
overhead_rate.bps());
}
// Get the encoder target rate. It is the estimated network rate -
// protection overhead.
// TODO(srte): We should multiply with 255 here.
// 此时的payload_bitrate_bps还是包括fec的，要减去fec部分才能是编码码率
// fec_controller根据目标码率和丢包率，更新fec_rate，同时计算fec需要使用的码率
// 和剩下的给编码器的码率
encoder_target_rate_bps_ = fec_controller_->UpdateFecRates(
payload_bitrate_bps, framerate,
rtc::saturated_cast<uint8_t>(update.packet_loss_ratio * 256),
loss_mask_vector_, update.round_trip_time.ms());
if (!fec_allowed_) {
// 没开启fec，直接设置为encoder_target_bitrate
encoder_target_rate_bps_ = payload_bitrate_bps;
// fec_controller_->UpdateFecRates() was still called so as to allow
// |fec_controller_| to update whatever internal state it might have,
// since |fec_allowed_| may be toggled back on at any moment.
}
// Subtract packetization overhead from the encoder target. If target rate
// is really low, cap the overhead at 50%. This also avoids the case where
// |encoder_target_rate_bps_| is 0 due to encoder pause event while the
// packetization rate is positive since packets are still flowing.
// 如果目标码率实在太低，也要把payload打包的开销压在50%以下，payload打包开销
// 主要是发生在h264和av1，h264在做聚合包的时候需要添加nal进行分割等，详情见RtpPacketizerH264::NextPacket()
// 这是为了防止在编码器暂停时仍在打包而导致的目标编码码率被设置成0的情况,
uint32_t packetization_rate_bps =
std::min(GetPacketizationOverheadRate(), encoder_target_rate_bps_ / 2);
// 编码码率减去rtp打包头
encoder_target_rate_bps_ -= packetization_rate_bps;
loss_mask_vector_.clear();
uint32_t encoder_overhead_rate_bps = 0;
if (send_side_bwe_with_overhead_ && has_packet_feedback_) {
// TODO(srte): The packet size should probably be the same as in the
// CalculateOverheadRate call above (just max_total_packet_size), it doesn't
// make sense to use different packet rates for different overhead
// calculations.
// 通过encoder rate重新计算overhead
DataRate encoder_overhead_rate = CalculateOverheadRate(
DataRate::BitsPerSec(encoder_target_rate_bps_),
max_total_packet_size - DataSize::Bytes(overhead_bytes_per_packet),
packet_overhead, Frequency::Hertz(framerate));
encoder_overhead_rate_bps = std::min(
encoder_overhead_rate.bps<uint32_t>(),
update.target_bitrate.bps<uint32_t>() - encoder_target_rate_bps_);
}
// When the field trial "WebRTC-SendSideBwe-WithOverhead" is enabled
// protection_bitrate includes overhead.
const uint32_t media_rate = encoder_target_rate_bps_ +
encoder_overhead_rate_bps +
packetization_rate_bps;
RTC_DCHECK_GE(update.target_bitrate, DataRate::BitsPerSec(media_rate));
protection_bitrate_bps_ = update.target_bitrate.bps() - media_rate;
}
RtpVideoSender::OnBitrateUpdated()中:
-
整体的码率构成是: 分配的总码率 = 媒体码率 + 保护码率
媒体码率 = 目标编码码率 + 媒体打包码率 + 封装(rtp头等)
保护码率 = fec码率 + nack_rate(重传码率)
-
首先计算当前stream下所有stream(simulcast)的平均overhead，这个平均overhead指的是rtp扩展头+rtp头
-
然后计算每个包的开销 packet_overhead, 这个开销是在rtp扩展头+rtp头的基础上增加了transport层的开销(srtp, udp)
-
如果预估的码率已经考虑了overhead(send_side_bwe_with_overhead)，则将目标码率减去overhead_rate，得到payload_bitrate_bps(有效负载码率)
-
将payload_bitrate_bps(有效负载码率)， 帧率，丢包率，fec掩码表，rtt传到FecControllerDefault::UpdateFecRates()中，通过FecController重新计算分配FEC和Nack等的码率，更新fec_rate，同时根据更新后的的video_rate和fec_rate, nack_rate的比例，计算出当前payload_bitrate_bps下的新的有效载荷码率
-
接着把载荷码率减去媒体打包开销(H264中的NALU，或者聚合包等会有额外的打包开销，不是直接)，就得到目标编码码率了
-
最终通过修正过的编码码率和当前帧率和包大小，使用CalculateOverheadRate()重新计算了一次overhead，然后把encoder_target_rate_bps（目标编码码率） + encoder_overhead_rate_bps (修正过的overhead,rtp头的) + packetization_rate_bps(打包开销) 就得到media_rate(媒体的码率), 将目标编码码率减去media_rate就得到保护码率(protection_bitrate_bps)，
整个过程可以总结为:
目标码率减去平均额外开销得到有效负载码率
有效负载码率经过FEC得到包含了打包的目标编码码率
打包的目标编码码率减去打包开销得到目标编码码率
基于目标编码码率和包平均大小，帧率重新计算overhead，将编码码率 + overhead + 打包开销就得到了媒体码率(media_rate)
目标码率 - 媒体码率(meida_rate) 得到保护码率(protection_bitrate_bps)
其中FecControllerDefault::UpdateFecRates()计算有效载荷码率的详情如下:
uint32_t FecControllerDefault::UpdateFecRates(
uint32_t estimated_bitrate_bps,
int actual_framerate_fps,
uint8_t fraction_lost,
std::vector<bool> loss_mask_vector,
int64_t round_trip_time_ms) {
float target_bitrate_kbps =
static_cast<float>(estimated_bitrate_bps) / 1000.0f;
// Sanity check.
if (actual_framerate_fps < 1.0) {
actual_framerate_fps = 1.0;
}
// 通过丢包率，码率等更新I帧和P帧的fec_rate
FecProtectionParams delta_fec_params;
FecProtectionParams key_fec_params;
{
MutexLock lock(&mutex_);
loss_prot_logic_->UpdateBitRate(target_bitrate_kbps);
loss_prot_logic_->UpdateRtt(round_trip_time_ms);
// Update frame rate for the loss protection logic class: frame rate should
// be the actual/sent rate.
loss_prot_logic_->UpdateFrameRate(actual_framerate_fps);
// Returns the filtered packet loss, used for the protection setting.
// The filtered loss may be the received loss (no filter), or some
// filtered value (average or max window filter).
// Use max window filter for now.
media_optimization::FilterPacketLossMode filter_mode =
media_optimization::kMaxFilter;
uint8_t packet_loss_enc = loss_prot_logic_->FilteredLoss(
clock_->TimeInMilliseconds(), filter_mode, fraction_lost);
// For now use the filtered loss for computing the robustness settings.
loss_prot_logic_->UpdateFilteredLossPr(packet_loss_enc);
if (loss_prot_logic_->SelectedType() == media_optimization::kNone) {
return estimated_bitrate_bps;
}
// Update method will compute the robustness settings for the given
// protection method and the overhead cost
// the protection method is set by the user via SetVideoProtection.
loss_prot_logic_->UpdateMethod();
// Get the bit cost of protection method, based on the amount of
// overhead data actually transmitted (including headers) the last
// second.
// Get the FEC code rate for Key frames (set to 0 when NA).
key_fec_params.fec_rate =
loss_prot_logic_->SelectedMethod()->RequiredProtectionFactorK();
// Get the FEC code rate for Delta frames (set to 0 when NA).
delta_fec_params.fec_rate =
loss_prot_logic_->SelectedMethod()->RequiredProtectionFactorD();
// The RTP module currently requires the same |max_fec_frames| for both
// key and delta frames.
delta_fec_params.max_fec_frames =
loss_prot_logic_->SelectedMethod()->MaxFramesFec();
key_fec_params.max_fec_frames =
loss_prot_logic_->SelectedMethod()->MaxFramesFec();
}
// Set the FEC packet mask type. |kFecMaskBursty| is more effective for
// consecutive losses and little/no packet re-ordering. As we currently
// do not have feedback data on the degree of correlated losses and packet
// re-ordering, we keep default setting to |kFecMaskRandom| for now.
delta_fec_params.fec_mask_type = kFecMaskRandom;
key_fec_params.fec_mask_type = kFecMaskRandom;
// Update protection callback with protection settings.
uint32_t sent_video_rate_bps = 0;
uint32_t sent_nack_rate_bps = 0;
uint32_t sent_fec_rate_bps = 0;
// Rate cost of the protection methods.
float protection_overhead_rate = 0.0f;
// TODO(Marco): Pass FEC protection values per layer.
// 将nack，fec的码率更新,
// 获取更新后的sent_video_rate_bps(视频发送码率), nack_rate(nack码率)
// fec_rate(fec码率)
protection_callback_->ProtectionRequest(
&delta_fec_params, &key_fec_params, &sent_video_rate_bps,
&sent_nack_rate_bps, &sent_fec_rate_bps);
// 计算总发送码率
uint32_t sent_total_rate_bps =
sent_video_rate_bps + sent_nack_rate_bps + sent_fec_rate_bps;
// Estimate the overhead costs of the next second as staying the same
// wrt the source bitrate.
if (sent_total_rate_bps > 0) {
// 计算protection_rate
protection_overhead_rate =
static_cast<float>(sent_nack_rate_bps + sent_fec_rate_bps) /
sent_total_rate_bps;
}
// Cap the overhead estimate to a threshold, default is 50%.
protection_overhead_rate =
std::min(protection_overhead_rate, overhead_threshold_);
// Source coding rate: total rate - protection overhead.
// 计算media_rate
return estimated_bitrate_bps * (1.0 - protection_overhead_rate);
}
在FecControllerDefault::UpdateFecRates()中
-
首先会将丢包率， 帧率等输入到loss_prot_logic，其最终会生成给关键帧和非关键帧的fec_rate，fec调整这一块可参见webrtc源码分析-fec
-
在获取更新完成后的码率后调用RtpVideoSender::ProtectionRequest()进行计算更新完成后的video_rate, fec_rate, nack_rate
int RtpVideoSender::ProtectionRequest(const FecProtectionParams* delta_params, const FecProtectionParams* key_params, uint32_t* sent_video_rate_bps, uint32_t* sent_nack_rate_bps, uint32_t* sent_fec_rate_bps) { *sent_video_rate_bps = 0; *sent_nack_rate_bps = 0; *sent_fec_rate_bps = 0; for (const RtpStreamSender& stream : rtp_streams_) { // 设置该rtp_rtcp上的fec_rate stream.rtp_rtcp->SetFecProtectionParams(*delta_params, *key_params); // 统计设置完之后的video_rate, fec_rate, nack_rate auto send_bitrate = stream.rtp_rtcp->GetSendRates(); *sent_video_rate_bps += send_bitrate[RtpPacketMediaType::kVideo].bps(); *sent_fec_rate_bps += send_bitrate[RtpPacketMediaType::kForwardErrorCorrection].bps(); *sent_nack_rate_bps += send_bitrate[RtpPacketMediaType::kRetransmission].bps(); } return 0; } -
基于更新后的fec_rate和nack_rate, 计算protection_rate(fec_rate加nack_rate)，protection_rate会被限制在50%以内，最后，通过estimated_bitrate_bps * (1- protection_rate)得到video_bitrate,
而使用CalculateOverheadRate()计算overhead的方式如下，原理上很简单，通过目标码率/包大小得到包数量，然后将包数量 * overhead就得到overhead了
/**
* @description: 求overhead的每秒码率，计算方式是通过目标码率和帧率得到帧大小
* 计算的方式很奇怪：data_rate / framerate / packet_size * framerate * overhead_per_packet
* 明显是可以看到中间的framerate可以消掉更符合逻辑： data_rate / packet_size * overhead_per_packet, 难道只是为了中间的ceil()操作？
* @param {data_rate} 分配的码率
* @param {packet_size} 包的大小
* @param {overhead_per_packet} 每个包的overhead
* @param {framerate} 当前帧率
* @return {*}
*/
DataRate RtpVideoSender::CalculateOverheadRate(DataRate data_rate,
DataSize packet_size,
DataSize overhead_per_packet,
Frequency framerate) const {
Frequency packet_rate = data_rate / packet_size;
if (use_frame_rate_for_overhead_) {
framerate = std::max(framerate, Frequency::Hertz(1));// 获取帧率
DataSize frame_size = data_rate / framerate; // 计算帧大小
int packets_per_frame = ceil(frame_size / packet_size); // 计算每帧有多少包
packet_rate = packets_per_frame * framerate; // 计算包率
}
return packet_rate.RoundUpTo(Frequency::Hertz(1)) * overhead_per_packet;
}
2.5.2.2 更新编码器
VideoStream的分配的码率计算出来后，编码器的会通过VideoStreamEncoder::OnBitrateUpdated()进行编码码率的分配更新，会重新调整simulcast 和 temporal layer各层级的码率，以及帧率大小。
/**
* @description: 更新编码码率，更新当前stream下分配给各个simulcast和
* temporal layer的码率，调整帧率
* @param {target_bitrate} 目标编码码率
* @param {stable_target_bitrate} 目标保守编码码率
* @param {link_allocation} 能使用的网络通道容量
* @param {fraction_lost} 丢包率
* @param {round_trip_time_ms} rtt
* @param {cwnd_reduce_ratio}
* @return {*}
*/
void VideoStreamEncoder::OnBitrateUpdated(DataRate target_bitrate,
DataRate stable_target_bitrate,
DataRate link_allocation,
uint8_t fraction_lost,
int64_t round_trip_time_ms,
double cwnd_reduce_ratio) {
// 线程检查
RTC_DCHECK_GE(link_allocation, target_bitrate);
if (!encoder_queue_.IsCurrent()) {
encoder_queue_.PostTask([this, target_bitrate, stable_target_bitrate,
link_allocation, fraction_lost, round_trip_time_ms,
cwnd_reduce_ratio] {
DataRate updated_target_bitrate =
UpdateTargetBitrate(target_bitrate, cwnd_reduce_ratio);
OnBitrateUpdated(updated_target_bitrate, stable_target_bitrate,
link_allocation, fraction_lost, round_trip_time_ms,
cwnd_reduce_ratio);
});
return;
}
RTC_DCHECK_RUN_ON(&encoder_queue_);
// target_bitrate == 0 -> video suspended(暂停)
const bool video_is_suspended = target_bitrate == DataRate::Zero();
const bool video_suspension_changed = video_is_suspended != EncoderPaused();
if (!video_is_suspended && settings_.encoder_switch_request_callback &&
encoder_selector_) {
// 竟然还能根据链路容量进行编码器的切换，但是目前没在其中看到encoder_selector的实现
if (auto encoder = encoder_selector_->OnAvailableBitrate(link_allocation)) {
QueueRequestEncoderSwitch(*encoder);
}
}
RTC_DCHECK(sink_) << "sink_ must be set before the encoder is active.";
RTC_LOG(LS_VERBOSE) << "OnBitrateUpdated, bitrate " << target_bitrate.bps()
<< " stable bitrate = " << stable_target_bitrate.bps()
<< " link allocation bitrate = " << link_allocation.bps()
<< " packet loss " << static_cast<int>(fraction_lost)
<< " rtt " << round_trip_time_ms;
if (encoder_) {
// 丢包率和rtt更新到encoder中并没有实际使用
encoder_->OnPacketLossRateUpdate(static_cast<float>(fraction_lost) / 256.f);
encoder_->OnRttUpdate(round_trip_time_ms);
}
uint32_t framerate_fps = GetInputFramerateFps();
frame_dropper_.SetRates((target_bitrate.bps() + 500) / 1000, framerate_fps);
// 根据目标码率设置到不同的layer中，然后设置到编码器中，并告诉rtp_video_sender
EncoderRateSettings new_rate_settings{
VideoBitrateAllocation(), static_cast<double>(framerate_fps),
link_allocation, target_bitrate, stable_target_bitrate};
SetEncoderRates(UpdateBitrateAllocation(new_rate_settings));
if (target_bitrate.bps() != 0)
encoder_target_bitrate_bps_ = target_bitrate.bps();
stream_resource_manager_.SetTargetBitrate(target_bitrate);
if (video_suspension_changed) {
RTC_LOG(LS_INFO) << "Video suspend state changed to: "
<< (video_is_suspended ? "suspended" : "not suspended");
encoder_stats_observer_->OnSuspendChange(video_is_suspended);
}
if (video_suspension_changed && !video_is_suspended && pending_frame_ &&
!DropDueToSize(pending_frame_->size())) {
// 此时码率可能，不再suspended了, 把pending_frame放入编码队列
int64_t pending_time_us =
clock_->CurrentTime().us() - pending_frame_post_time_us_;
if (pending_time_us < kPendingFrameTimeoutMs * 1000)
EncodeVideoFrame(*pending_frame_, pending_frame_post_time_us_);
pending_frame_.reset();
}
}
VideoStreamEncoder::OnBitrateUpdated()中:
- 首先进行执行任务线程检查是否为encoder_queue，如果不在，则切换运行到该线程
- 根据link_allocation做编码器的切换，在windows端没有找到实现，源码中有ios和安卓的实现，和移动端平台的编码器有关系，目前还未深入了解，创建完成的编码器会调用QueueRequestEncoderSwitch()对stream的编码器重新创建初始化，并修改相关编码参数，从这里能看出，webrtc是能够支持中途变更编码器的
- 把丢包率和rtt放到编码器中，可能是想依靠他们做什么码率控制，但目前没有实现，OnPacketLossRateUpdate()和OnRttUpdate()两个都是空函数
- 调用GetInputFramerateFps()获取当前帧率，把当前的帧率和码率放到frame_dropper中，让frame_dropper在MaybeEncodeVideoFrame()进行帧编码的时候计算是否需要丢帧以适应当前的码率。
- 调用UpdateBitrateAllocation()，计算分配给当前stream下的simulcast和temporal的码率，使用SetEncoderRates()将新分配的码率设置到编码器，同时告知下层RTCP模块，可能发生了layer或者分辨率的变化，让其发送RTCP报文告知对方
- 检查在更新码率后是否从suspend(分配码率无法达到min_bitrate而导致的挂起状态)状态脱离，如果是，则将当前因为suspend而Pending 的Frame送入编码
其中，将码率分配到不同的simulcast和temporal layer的UpdateBitrateAllocation()见下：
VideoStreamEncoder::EncoderRateSettings
VideoStreamEncoder::UpdateBitrateAllocation(
const EncoderRateSettings& rate_settings) {
VideoBitrateAllocation new_allocation;
// Only call allocators if bitrate > 0 (ie, not suspended), otherwise they
// might cap the bitrate to the min bitrate configured.
// 通过rate_allocator重新分配码率
if (rate_allocator_ && rate_settings.encoder_target > DataRate::Zero()) {
// 重新计算分配到simulcast和temporal下的码率
// -> SimulcastRateAllocator::Allocate()
new_allocation = rate_allocator_->Allocate(VideoBitrateAllocationParameters(
rate_settings.encoder_target, rate_settings.stable_encoder_target,
rate_settings.rate_control.framerate_fps));
}
EncoderRateSettings new_rate_settings = rate_settings;
new_rate_settings.rate_control.target_bitrate = new_allocation;
new_rate_settings.rate_control.bitrate = new_allocation;
// VideoBitrateAllocator subclasses may allocate a bitrate higher than the
// target in order to sustain the min bitrate of the video codec. In this
// case, make sure the bandwidth allocation is at least equal the allocation
// as that is part of the document contract for that field.
new_rate_settings.rate_control.bandwidth_allocation =
std::max(new_rate_settings.rate_control.bandwidth_allocation,
DataRate::BitsPerSec(
new_rate_settings.rate_control.bitrate.get_sum_bps()));
if (bitrate_adjuster_) {
// 对码率进行再调节，以平滑码率调整的过程
VideoBitrateAllocation adjusted_allocation =
bitrate_adjuster_->AdjustRateAllocation(new_rate_settings.rate_control);
RTC_LOG(LS_VERBOSE) << "Adjusting allocation, fps = "
<< rate_settings.rate_control.framerate_fps << ", from "
<< new_allocation.ToString() << ", to "
<< adjusted_allocation.ToString();
new_rate_settings.rate_control.bitrate = adjusted_allocation;
}
return new_rate_settings;
}
UpdateBitrateAllocation()中:
-
调用rate_allocator_->Allocate()分配码率到simulcast 和 temporal layer，实则调用的为SimulcastRateAllocator::Allocate(), simulcast的码率分配策略是，先满足max_bitrate的最低的layer的min_bitrate，如果有剩余的码率，从低到高去满足他们的target_bitrate;
VideoBitrateAllocation SimulcastRateAllocator::Allocate( VideoBitrateAllocationParameters parameters) { VideoBitrateAllocation allocated_bitrates; DataRate stable_rate = parameters.total_bitrate; if (stable_rate_settings_.IsEnabled() && parameters.stable_bitrate > DataRate::Zero()) { stable_rate = std::min(parameters.stable_bitrate, parameters.total_bitrate); } // 将码率分配到不同的simulcast DistributeAllocationToSimulcastLayers(parameters.total_bitrate, stable_rate, &allocated_bitrates); // 通过分配好的simulcat码率再分配到不同的temporalayers DistributeAllocationToTemporalLayers(&allocated_bitrates); return allocated_bitrates; }而分配给temporal的码率，主要是根据SimulcastRateAllocator::GetTemporalRateAllocation()函数中选取的模型，一种是BaseHeavy, 对应的3个temporal 的值是(60%, 20%, 20%), 而另一种则是变化层数的，见下kLayerRateAllocation
static const float kLayerRateAllocation[kMaxTemporalStreams][kMaxTemporalStreams] = { {1.0f, 1.0f, 1.0f, 1.0f}, // 1 layer {0.6f, 1.0f, 1.0f, 1.0f}, // 2 layers {60%, 40%} {0.4f, 0.6f, 1.0f, 1.0f}, // 3 layers {40%, 20%, 40%} {0.25f, 0.4f, 0.6f, 1.0f} // 4 layers {25%, 15%, 20%, 40%} }; static const float kBaseHeavy3TlRateAllocation[kMaxTemporalStreams] = { 0.6f, 0.8f, 1.0f, 1.0f // 3 layers {60%, 20%, 20%} }; float SimulcastRateAllocator::GetTemporalRateAllocation( int num_layers, int temporal_id, bool base_heavy_tl3_alloc) { if (num_layers == 3 && base_heavy_tl3_alloc) { // 层数为3，并且选择了heavy_base return kBaseHeavy3TlRateAllocation[temporal_id]; } return kLayerRateAllocation[num_layers - 1][temporal_id]; } -
调用bitrate_adjuster_->AdjustRateAllocation()使上面分配后的码率趋向于平滑
用以设置新的码率SetEncoderRates()如下:
void VideoStreamEncoder::SetEncoderRates(
const EncoderRateSettings& rate_settings) {
RTC_DCHECK_GT(rate_settings.rate_control.framerate_fps, 0.0);
// 检测(fps, bitrate)是否发生改变
bool rate_control_changed =
(!last_encoder_rate_settings_.has_value() ||
last_encoder_rate_settings_->rate_control != rate_settings.rate_control);
// For layer allocation signal we care only about the target bitrate (not the
// adjusted one) and the target fps.
// 对于layer, 只是考虑target_bitrate和framerate_fps
bool layer_allocation_changed =
!last_encoder_rate_settings_.has_value() ||
last_encoder_rate_settings_->rate_control.target_bitrate !=
rate_settings.rate_control.target_bitrate ||
last_encoder_rate_settings_->rate_control.framerate_fps !=
rate_settings.rate_control.framerate_fps;
if (last_encoder_rate_settings_ != rate_settings) {
last_encoder_rate_settings_ = rate_settings;
}
if (!encoder_) {
return;
}
// |bitrate_allocation| is 0 it means that the network is down or the send
// pacer is full. We currently only report this if the encoder has an internal
// source. If the encoder does not have an internal source, higher levels
// are expected to not call AddVideoFrame. We do this since its unclear
// how current encoder implementations behave when given a zero target
// bitrate.
// TODO(perkj): Make sure all known encoder implementations handle zero
// target bitrate and remove this check.
if (!HasInternalSource() &&
rate_settings.rate_control.bitrate.get_sum_bps() == 0) {
return;
}
//fps和bitrate发生了改变，告知编码器
if (rate_control_changed) {
// EncoderSimulcastProxy::SetRates()
// ->H264EncoderImpl::SetRates()
encoder_->SetRates(rate_settings.rate_control);
encoder_stats_observer_->OnBitrateAllocationUpdated(
send_codec_, rate_settings.rate_control.bitrate);
frame_encode_metadata_writer_.OnSetRates(
rate_settings.rate_control.bitrate,
static_cast<uint32_t>(rate_settings.rate_control.framerate_fps + 0.5));
stream_resource_manager_.SetEncoderRates(rate_settings.rate_control);
// 不同layer的流的码率发生了变化，告知rtp_rtcp，在发送关键帧的时候，通过extension把最新的layer
// 分辨率带上，详见RtpSenderVideo::SetVideoLayersAllocation()和RtpSenderVideo::SetVideoLayersAllocationInternal()
if (layer_allocation_changed &&
allocation_cb_type_ ==
BitrateAllocationCallbackType::kVideoLayersAllocation) {
sink_->OnVideoLayersAllocationUpdated(CreateVideoLayersAllocation(
send_codec_, rate_settings.rate_control, encoder_->GetEncoderInfo()));
}
}
// 通知rtp_video_sender，由它通知rtp_rtcp等模块，码率是否发生了改变
if ((allocation_cb_type_ ==
BitrateAllocationCallbackType::kVideoBitrateAllocation) ||
(encoder_config_.content_type ==
VideoEncoderConfig::ContentType::kScreen &&
allocation_cb_type_ == BitrateAllocationCallbackType::
kVideoBitrateAllocationWhenScreenSharing)) {
// rtp_rtcp检测到layer发生了变化后
sink_->OnBitrateAllocationUpdated(
// Update allocation according to info from encoder. An encoder may
// choose to not use all layers due to for example HW.
UpdateAllocationFromEncoderInfo(
rate_settings.rate_control.target_bitrate,
encoder_->GetEncoderInfo()));
}
}
VideoStreamEncoder::SetEncoderRates()中：
-
首先检测FPS和Bitrate是否发生改变，如果是则告知编码器
-
不同Layer的流的码率发生了变化，告知RTP_RTCP，RtpSenderVideo有可能通过RTP-EXTENSION将Layer的码率带上
-
告知RTP_RTCP Layer的变化，层数可能发生改变，要发送相应的RTCP-XR
void RTCPSender::SetVideoBitrateAllocation( const VideoBitrateAllocation& bitrate) { MutexLock lock(&mutex_rtcp_sender_); // Check if this allocation is first ever, or has a different set of // spatial/temporal layers signaled and enabled, if so trigger an rtcp report // as soon as possible. // 检测这个码流分配器是否包含了新的时域/空域层，有的话尽快发送xr-rtcp告知对端 absl::optional<VideoBitrateAllocation> new_bitrate = CheckAndUpdateLayerStructure(bitrate); if (new_bitrate) { video_bitrate_allocation_ = *new_bitrate; RTC_LOG(LS_INFO) << "Emitting TargetBitrate XR for SSRC " << ssrc_ << " with new layers enabled/disabled: " << video_bitrate_allocation_.ToString(); next_time_to_send_rtcp_ = clock_->TimeInMilliseconds(); } else { video_bitrate_allocation_ = bitrate; } send_video_bitrate_allocation_ = true; SetFlag(kRtcpAnyExtendedReports, true); }这个XR-RTCP Report的内容详见RFC3611， 也可参考RTCPSender::BuildExtendedReports()函数，可以看到包含了simulcast layer和temporal layer的码率信息，这些信息可用于在服务端做simulcast和temporal转发决策。
void RTCPSender::BuildExtendedReports(const RtcpContext& ctx, PacketSender& sender) { rtcp::ExtendedReports xr; xr.SetSenderSsrc(ssrc_); if (!sending_ && xr_send_receiver_reference_time_enabled_) { rtcp::Rrtr rrtr; rrtr.SetNtp(TimeMicrosToNtp(ctx.now_us_)); xr.SetRrtr(rrtr); } for (const rtcp::ReceiveTimeInfo& rti : ctx.feedback_state_.last_xr_rtis) { xr.AddDlrrItem(rti); } if (send_video_bitrate_allocation_) { rtcp::TargetBitrate target_bitrate; // 将每个simulcast layer和temporal layer的信息码率提取出来放到report中 for (int sl = 0; sl < kMaxSpatialLayers; ++sl) { for (int tl = 0; tl < kMaxTemporalStreams; ++tl) { if (video_bitrate_allocation_.HasBitrate(sl, tl)) { target_bitrate.AddTargetBitrate( sl, tl, video_bitrate_allocation_.GetBitrate(sl, tl) / 1000); } } } xr.SetTargetBitrate(target_bitrate); send_video_bitrate_allocation_ = false; } sender.AppendPacket(xr); }
2.5.3 UpdateAllocationLimits()
BitrateAllocator::OnNetworkEstimateChanged()中将码率分发各个视频流中并更新编码器码率后，stream的limit(min_allocatable_rate, max_padding_rate,max_allocatable_rate)会发生改变，此时需要将变更后的limit重新统计，告知cc-controller变更后的limit，调用的是BitrateAllocator::UpdateAllocationLimits()
void BitrateAllocator::UpdateAllocationLimits() {
BitrateAllocationLimits limits;
for (const auto& config : allocatable_tracks_) {
uint32_t stream_padding = config.config.pad_up_bitrate_bps;
if (config.config.enforce_min_bitrate) {
limits.min_allocatable_rate +=
DataRate::BitsPerSec(config.config.min_bitrate_bps);
} else if (config.allocated_bitrate_bps == 0) {
stream_padding =
std::max(config.MinBitrateWithHysteresis(), stream_padding);
}
limits.max_padding_rate += DataRate::BitsPerSec(stream_padding);
limits.max_allocatable_rate +=
DataRate::BitsPerSec(config.config.max_bitrate_bps);
}
if (limits.min_allocatable_rate == current_limits_.min_allocatable_rate &&
limits.max_allocatable_rate == current_limits_.max_allocatable_rate &&
limits.max_padding_rate == current_limits_.max_padding_rate) {
return;
}
current_limits_ = limits;
RTC_LOG(LS_INFO) << "UpdateAllocationLimits : total_requested_min_bitrate: "
<< ToString(limits.min_allocatable_rate)
<< ", total_requested_padding_bitrate: "
<< ToString(limits.max_padding_rate)
<< ", total_requested_max_bitrate: "
<< ToString(limits.max_allocatable_rate);
limit_observer_->OnAllocationLimitsChanged(limits);
}
其中limit_observer_->OnAllocationLimitsChanged()会通过 RtpTransportControllerSend::UpdateStreamsConfig()会调用到cc-controller中
void RtpTransportControllerSend::UpdateStreamsConfig() {
streams_config_.at_time = Timestamp::Millis(clock_->TimeInMilliseconds());
if (controller_)
PostUpdates(controller_->OnStreamsConfig(streams_config_));
}
NetworkControlUpdate GoogCcNetworkController::OnStreamsConfig(
StreamsConfig msg) {
NetworkControlUpdate update;
if (msg.requests_alr_probing) {
probe_controller_->EnablePeriodicAlrProbing(*msg.requests_alr_probing);
}
if (msg.max_total_allocated_bitrate &&
*msg.max_total_allocated_bitrate != max_total_allocated_bitrate_) {
if (rate_control_settings_.TriggerProbeOnMaxAllocatedBitrateChange()) {
// 重设probe_controller的max_bitrate
update.probe_cluster_configs =
probe_controller_->OnMaxTotalAllocatedBitrate(
msg.max_total_allocated_bitrate->bps(), msg.at_time.ms());
} else {
probe_controller_->SetMaxBitrate(msg.max_total_allocated_bitrate->bps());
}
max_total_allocated_bitrate_ = *msg.max_total_allocated_bitrate;
}
bool pacing_changed = false;
if (msg.pacing_factor && *msg.pacing_factor != pacing_factor_) {
pacing_factor_ = *msg.pacing_factor;
pacing_changed = true;
}
if (msg.min_total_allocated_bitrate &&
*msg.min_total_allocated_bitrate != min_total_allocated_bitrate_) {
min_total_allocated_bitrate_ = *msg.min_total_allocated_bitrate;
pacing_changed = true;
if (use_min_allocatable_as_lower_bound_) {
ClampConstraints();
// 重设delay_based_bwe和bandwidth_estimation的limit
delay_based_bwe_->SetMinBitrate(min_data_rate_);
bandwidth_estimation_->SetMinMaxBitrate(min_data_rate_, max_data_rate_);
}
}
if (msg.max_padding_rate && *msg.max_padding_rate != max_padding_rate_) {
max_padding_rate_ = *msg.max_padding_rate;
pacing_changed = true;
}
if (pacing_changed)
update.pacer_config = GetPacingRates(msg.at_time);
return update;
}
在GoogCcNetworkController::OnStreamsConfig()中:
- 检测到max_total_allocated_bitrate发生变更的时候，会调用probe_controller_->OnMaxTotalAllocatedBitrate()更新max_total_allocated_bitrate，同时获取新的探测进行探测
- 重设delay_based_bwe和bandwidth_estimation的limit
- 检测到limit发生改变的时候，更新update.pacer_config，重新设置pacing_rate
- 返回update，调用PostUpdates()从头开始进行码率调整