Bit allocation between texture and depth

• Bit allocation for multiview image compression using cubic synthesized view distortion model [1]
• Asymmetric coding of multi-view video plus depth based 3-D video for view rendering [2]
• View and rate scalable multiview image coding with depth-image-based rendering [3]
• Video and depth bitrate allocation in multiview compression [4]

References

[1] V. Velisavljevic, G. Cheung, and J. Chakareski, “Bit allocation for multiview image compression using cubic synthesized view distortion model,” in Proc. IEEE Int. Conf. Multimedia Expo., Jul. 2011, pp. 1–6.

[2] F. Shao, G. Jiang, M. Yu, K. Chen, and Y. S. Ho, “Asymmetric coding of multi-view video plus depth based 3-D video for view rendering,” IEEE Trans. Multimedia, vol. 14, no. 1, pp. 157–167, Feb. 2012.

[3] V. Velisavljevic, V. Stankovic, J. Chakareski, and G. Cheung, “View and rate scalable multiview image coding with depth-image-based rendering,” in Proc. IEEE Int. Conf. Digit. Signal Process., Jul. 2011, pp. 1–8.

[4]  K. Klimaszewski, K. Wegner, and M. Domanski, “Video and depth bitrate allocation in multiview compression,” in Proc. IEEE Int. Conf. Syst. Signals Image Process., May 2014, pp. 207–210.

How to code multiview depth maps?

Multiview depth map sequences can be also encoded using H.264/MVC. However, MVC is optimized for natural video sequences and does not consider the depth map characteristics. Current depth coding methods that handle depth map characteristics include:

• Adaptive wavelet coding of the depth map for stereoscopic view synthesis [1]
• Mesh-based depth coding for 3D video using hierarchical decomposition of depth maps [2]
• Depth reconstruction filter and down/up sampling for depth coding in 3-D video [3]
• The effect of depth compression on multiview rendering quality [4]

References

[1] I. Daribo, C. Tillier, and B. Pesquet-Popescu, “Adaptive wavelet coding of the depth map for stereoscopic view synthesis,” in Proc. IEEE Int. Workshop Multimedia Signal Process., Oct. 2008, pp. 413–417.

[2] S.-Y. Kim and Y.-S. Ho, “Mesh-based depth coding for 3D video using hierarchical decomposition of depth maps,” in Proc. IEEE Int. Conf. Image Process., Sep. 2007, pp. V-117–V-120.

[3] K.-J. Oh, S. Yea, A. Vetro, and Y.-S. Ho, “Depth reconstruction filter and down/up sampling for depth coding in 3-D video,” IEEE Signal Process. Lett., vol. 16, no. 9, pp. 747–750, Sep. 2009.

[4] P. Merkle et al., “The effect of depth compression on multiview rendering quality,” in Proc. 3DTV Conf., May 2008, pp. 245–248.

How to use an MDP to solve an optimization problem?

First, we need to define a set of states, actions, rewards and transition probabilities. From the definition of the states, we must observe that the Markov property exists (all of these states depend on their immediately previous state only). Then, we can use dynamic programming under the reinforcement learning framework [1] to find the optimal state-value function which maximizes the long term rewards.


References
[1] R. Sutton and A. Barto. Reinforcement learning: An introduction. Cambridge Univ Press, 1998.

How to model the variation of wireless channel conditions?

Finite-state Markov model can be used to describe the variation of wireless channel conditions.

• Finite-state Markov channel – a useful model for radio communication channels [1]
• A packet-level model for UWB channel with people shadowing process based on angular spectrum analysis [2]

References

[1] H. S. Wang and N. Moayeri. Finite-state Markov channel – a useful model for radio communication channels. IEEE Trans on Veh. Tech., 44(1):163–171, 1995.

[2] R. Zhang and L. Cai. A packet-level model for uwb channel with people shadowing process based on angular spectrum analysis. IEEE Trans. on Wireless Comm., 8(8):4048–55, Aug. 2009.

How to encode a video in a scalable manner?

• The open source SVC coder (JSVM) [1] can be used to encode a video into multiple layers.
• A SVC-enabled video player can be implemented using an open source SVC decoder [2]. 
• A scalable video streaming testbed was prototyped in [3]. 

References
[1] J. Reichel, H. Schwarz, and M. Wien. Joint scalable video model 11 (JSVM 11). Joint Video Team, Doc. JVT- X, 2007.
[2] M. Blestel and M. Raulet. Open SVC decoder: a flexible SVC library. ACM MM ’10, pages 1463–1466, New York, NY, USA, 2010.
[3] S. Xiang. Scalable streaming. https://sites.google.com/site/svchttpstreaming/.

Adaptive Video Streaming Products

• Adobe HTTP Dynamic Streaming (HDS) [1]
• Apple HTTP Live Streaming (HLS) [2]
• Microsoft Live Smooth Streaming [3]
• Adaptive Scalable Video Streaming over HTTP [4]

For an experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP, refer to [5].

References

[1] Adobe Systems: HTTP Dynamic Streaming (HDS), February 2015, Available: http://tiny.cc/HDS.

[2] Apple: HTTP Live Streaming (HLS), February 2015, Available: http://tiny.cc/HLS2015.

[3] Microsoft: Live Smooth Streaming, February 2015, Available: http://tiny.cc/MSSmooth.
[4] S. Xiang. Scalable streaming. https://sites.google.com/site/svchttpstreaming/.
[5] S. Akhshabi, A. C. Begen, and C. Dovrolis. An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP. In ACM MMSys’11, pages 157–168, New York, NY, USA, 2011.

Optimization Problem Solvers

• Integer Programming
• CPLEX [1]
• GLPK [2]
• Decentralized POMDP
• Multi-Agent Decision Process (MADP) Toolbox [3]

For more information about optimization problems, refer to [4].

References

[1] CPLEX: IBM ILOG Optimizer, July 2009, Available: http://tiny.cc/CPLEX.

[2] GLPK: GNU Linear Programming Kit, June. 2012, http://tiny.cc/GLPK.

[3] F. A. Oliehoek, M. T. J. Spaan, and P. Robbel, “MultiAgent Decision Process (MADP) Toolbox 0.3,” 2014.

[4] Practical Optimization: A Gentle Introduction, http://www.sce.carleton.ca/faculty/chinneck/po.html

Streaming methods in cloud gaming systems

• Video Streaming
• A game attention model for efficient bit rate allocation in cloud gaming [1]
• Graphics Streaming
• Collaborative Rendering
• High-quality mobile gaming using gpu offload. [2]

References

[1] Ahmadi, Hamed, et al. "A game attention model for efficient bit rate allocation in cloud gaming." Multimedia Systems 20.5 (2014): 485-501.
[2] E. Cuervo, A. Wolman, L. P. Cox, K. Lebeck, A. Razeen, S. Saroiu, and M. Musuvathi. Kahawai: High-quality mobile gaming using gpu offload. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys ’15, pages 121–135, New York, NY, USA, 2015. ACM.

Open source cloud gaming frameworks

• GamingAnywhere [1]
• Video Streaming
• Cloud Gaming Testbed [2]
• Video Streaming
• Games@large [3]
• Graphics Streaming

References

[1] Huang, Chun-Ying, et al. "GamingAnywhere: an open cloud gaming system." Proceedings of the 4th ACM multimedia systems conference. ACM, 2013.

[2] Ahmadi, Hamed, et al. "A game attention model for efficient bit rate allocation in cloud gaming." Multimedia Systems 20.5 (2014): 485-501.
[3] I. Nave, H. David, A. Shani, Y. Tzruya, A. Laikari, P. Eisert, and P. Fechteler. Games@large graphics streaming architecture. In 2008 IEEE International Symposium on Consumer Electronics, pages 1–4, April 2008.

Applying Visual Attention in Video Coding

• A game attention model for efficient bit rate allocation in cloud gaming [1]
• H.264/AVC Standard
• Cloud Gaming Application
• Proposed a hybrid bottom-up and top-down attention model
• Assigning QPs based on attention map
• Extension of [2]
• Fast Mode Decision in the HEVC Video Coding Standard by Exploiting Region with Dominated Motion and Saliency Features [3]
• HEVC
• Deciding on modes based on attention map
• Used GBVS (a bottom-up attention model)

References
[1] Ahmadi, Hamed, et al. "A game attention model for efficient bit rate allocation in cloud gaming." Multimedia Systems 20.5 (2014): 485-501.
[2] Ahmadi, Hamed, et al. "Efficient bitrate reduction using a game attention model in cloud gaming." Haptic Audio Visual Environments and Games (HAVE), 2013 IEEE International Symposium on. IEEE, 2013.
[3] Podder, Pallab Kanti, Manoranjan Paul, and Manzur Murshed. "Fast Mode Decision in the HEVC Video Coding Standard by Exploiting Region with Dominated Motion and Saliency Features." PloS one 11.3 (2016): e0150673.

How to compare two rate-distortion curves?

Rate-distortion (RD) curves are two-dimensional, therefore comparing them is not straightforward. Bjøntegaard Delta-rate (BD-rate) criterion [1] can be used to compare rate-distortion curves.


References
[1] G. Bjøntegaard. “Calculation of average PSNR differences between RD-curves”. ITU-T Video Coding Experts group document VCEG-M33, April. 2001.

VQM: Video Quality Metric

VQM [1] is an objective video quality metric which has a good correlation with human perception. The VQM value is number between 0 and 1. A lower VQM value indicates a better video quality.


References
[1] Pinson, Margaret H., and Stephen Wolf. "A new standardized method for objectively measuring video quality." IEEE Transactions on broadcasting50.3 (2004): 312-322.

Encoding Settings For Streaming Vancouver Olympics

Level	Bitrate (kbps)	Resolution	Frame rate
1	400	312x176	15
2	600	400x224	15
3	900	512x288	15
4	950	544x304	15
5	1250	640x360	25
6	1600	736x416	25
7	1950	848x480	25
8	3450	1280x720	30

References
[1] Jan Ozer, “Adaptive Streaming in the Field”, in Streaming Media Magazine, 2011.
[2] Liu, Yao, et al. "User experience modeling for DASH video." 2013 20th International Packet Video Workshop. IEEE, 2013.

When to use Proportional Fairness?

Proportional fairness performs well when all users have the same utility function [1].


References
[1] Mu, Mu, et al. "User-level fairness delivered: network resource allocation for adaptive video streaming." 2015 IEEE 23rd International Symposium on Quality of Service (IWQoS). IEEE, 2015.

What affects the quality of adaptive video streams?

• Quality Switches
• Amplitude
• Frequency
• Stallings
• Duration 
• Frequency
• Initial Startup Delay

There are some surveys on the quality of experience for adaptive video streaming:

• Quality of Experience and HTTP adaptive streaming: A review of subjective studies [1]
• A Survey on Quality of Experience of HTTP Adaptive Streaming [2]
• User experience modeling for DASH video [3]

QoE is affected by spatial and temporal quality of the video stream. Initial delay, total stall duration, and number of stalls affect the temporal quality. Average video quality, number of switches, and average switch magnitude affect the spatial quality. In [3], the effect of each of these factors has been quantified.

Some important notes:

• Gradual multiple variations are preferred over abrupt variations.
• Constant quality is usually preferred to varying quality.
• In general, providing a bitrate as high as possible does not necessarily lead to the highest QoE.
• All is end that ends well: the end quality of the video has a definite impact on the perceived quality.
• The effect of spatial and temporal switching varies depending on the content type. 
• A single long stalling is preferred over multiple short freezes.
• Regular freezes are preferred over irregular freezes.
• Tolerable startup delay is dependent on the type of the application.
• Users prefer to wait longer if they can get less video stalling.       

Metrics

• Number of Quality Changes (NoC) [5]
• Number of Interruptions (NoI) [5]
• Percentage of Interruptions(PoI) [5]
• Impairment due to initial delay [3]
• Impairment due to stall [3]
• Impairment due to level fluctuations [3]
• Impairment due to low level video quality [3]
• Average playback Quality (APQ) [4]
• Playback Smoothness (PS) [6]
• Interruption Ratio [4]

References

[1] Garcia, M-N., et al. "Quality of experience and HTTP adaptive streaming: A review of subjective studies." Quality of Multimedia Experience (QoMEX), 2014 Sixth International Workshop on. IEEE, 2014.

[2] Seufert, Michael, et al. "A survey on quality of experience of http adaptive streaming." IEEE Communications Surveys & Tutorials 17.1 (2015): 469-492.
[3] Liu, Yao, et al. "User experience modeling for DASH video." 2013 20th International Packet Video Workshop. IEEE, 2013.
[4] S. Xiang, L. Cai, and J. Pan, “Adaptive scalable video streaming in wireless networks,” in Proc. of ACM MMSys, Feb. 2012, pp. 167–172.
[5] Yan, Zhisheng, Jingteng Xue, and Chang Wen Chen. "QoE continuum driven HTTP adaptive streaming over multi-client wireless networks." 2014 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2014.
[6] S. Nelakuditi, R. Harinath, E. Kusmierek, and Z. Zhang. Providing smoother quality layered video stream. In ACM NOSSDAV’00, June 2000.

What parameters affect the bitrate of a video?

• Encoding Settings
• Spatial Settings (such as resolution)
• Temporal Settings (such as frame rate)

should users receive the same bitrate or the same quality?

Since there is no linear correlation between the bitrate of a video stream and its perceptual quality [1], the fairness algorithm should assure that users get similar qualities rather than similar bitrates.



References  
[1] G. Cermak, M. Pinson, and S. Wolf. The relationship among video quality, screen resolution, and bit rate. Broadcasting, IEEE Transactions on, 57(2):258–262, 2011.

Finding the optimal values for discrete parameters

One way, used in [1], is use this heuristic: Solve the problem in continuous space. Conduct a bi-dimensional search for the nearest valid (discrete) values. If there were N parameters, there would be at most 2^N possible options. Apply the same utility function or a different one to find the best option. 

Another way, used in [2], is a branch and bound algorithm [3] that uses heuristic evaluation, which allows to optimise over a discrete data set in linear time. In [2], the utilized two versions (Promote and Boost), but the results were similar. 


References
[1] Mu, Mu, et al. "User-level fairness delivered: network resource allocation for adaptive video streaming." 2015 IEEE 23rd International Symposium on Quality of Service (IWQoS). IEEE, 2015.
[2] Georgopoulos, Panagiotis, et al. "Towards network-wide QoE fairness using openflow-assisted adaptive video streaming." Proceedings of the 2013 ACM SIGCOMM workshop on Future human-centric multimedia networking. ACM, 2013.
[3] R. J. Dakin. A Tree-search Algorithm for Mixed Integer Programming Problems. Comput. J., 8(3):250{255, 1965.

Forgiveness Effect

Refer to these papers:

• Layerp2p: Using layered video chunks in p2p live streaming [1]
• Forgiveness effect in subjective assessment of packet video [2]
• Temporal characterization of forgiveness effect [3]
• User-level fairness delivered: network resource allocation for adaptive video streaming [4]

References
[1] Z. Liu, Y. Shen, K. W. Ross, S. S. Panwar, and Y. Wang. Layerp2p: Using layered video chunks in p2p live streaming. IEEE Transactions on Multimedia, 11(7):1340–1352, 2009.
[2] V. Seferidis, M. Ghanbari, and D. Pearson. Forgiveness effect in subjective assessment of packet video. Electronics Letters, 28(21):2013–2014, 1992.
[3] D. Hands. Temporal characterization of forgiveness effect. Electronics Letters, 37, 2002.
[4] Mu, Mu, et al. "User-level fairness delivered: network resource allocation for adaptive video streaming." 2015 IEEE 23rd International Symposium on Quality of Service (IWQoS). IEEE, 2015.

Metrics for Adaptive Video Streaming

Metrics can be divided into user-level and network-level groups.

• User-level Fairness Delivered: Network Resource Allocation for Adaptive Video Streaming [1]
• They used three metrics: video quality, switching impact, and network cost (or utility)
• They solved the problem just based on video quality metrics, and then adjusted the results based on a weighted average of the three metrics.
• They considered that clients might have devices with varying resolutions, but didn't consider that the representations on the server might also have different resolutions.
• To simulate wireless conditioned they randomly changed the available bandwidth between 500 kbps and 8 Mbps.
• In switching impact metric, they considered forgiveness effect [2][3][4].
• To implement fairness, they minimized the Relative Standard Deviation (RSD) of the metrics.  
• 
  

References

[1] Mu, Mu, et al. "User-level fairness delivered: network resource allocation for adaptive video streaming." 2015 IEEE 23rd International Symposium on Quality of Service (IWQoS). IEEE, 2015.

[2] Z. Liu, Y. Shen, K. W. Ross, S. S. Panwar, and Y. Wang. Layerp2p: Using layered video chunks in p2p live streaming. IEEE Transactions on Multimedia, 11(7):1340–1352, 2009.

[3] V. Seferidis, M. Ghanbari, and D. Pearson. Forgiveness effect in subjective assessment of packet video. Electronics Letters, 28(21):2013–2014, 1992.

[4] D. Hands. Temporal characterization of forgiveness effect. Electronics Letters, 37, 2002.

How to solve Nonlinear Integer Programming problems?

A nonlinear integer programming problem in general is NP-hard. If the objective function has some special form (e.g., convexity), the Lagrangian relaxation [1] or dynamic programming [2] can be used. If the objective function has a concave or log-concave property, the steepest ascent algorithm [3] can be used.


References
[1] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge Univ. Press, 2004.
[2] T. Ibaraki and N. Katoh, Resource Allocation Problems: Algorithmic Approaches. The MIT Press, 1988.
[3] G. P. Akilov, L. V. Kantorovich, Functional Analysis, 2nd edition. Pergamon Press, 1982.