Linked e-resources
Details
Table of Contents
Cover
Title Page
Copyright Page
Contents
About the Author
Acknowledgements
About the Companion Website
Chapter 1 Introduction
1.1 Why Write This Book?
1.2 What Is in the Book?
1.3 How Should You Use This Book?
References
Chapter 2 Video Coding and Video Quality
2.1 Introduction
2.2 An Overview of Video Coding
2.2.1 Just How Much Data Are We Talking About?
2.2.2 Bandwidth Is Increasing Every Year, But...
2.3 Inputs and Outputs
2.3.1 Digital Video
2.3.2 Uncompressed Video Formats
2.3.2.1 Frames, Fields and Pixels
2.3.2.2 Colour and Colour Spaces
2.3.3 Side Information
2.3.4 Coded Video
2.4 Structural Elements
2.5 Prediction
2.5.1 Prediction in a Video Encoder
2.5.2 Prediction in a Video Decoder
2.5.3 Prediction Example
2.5.4 Choosing a Prediction
2.5.5 Inter and Intra Prediction
2.6 Transform and Quantisation
2.6.1 Transform
2.6.2 Quantisation
2.7 Bitstream Coding
2.7.1 Pre-processing
2.7.2 Entropy Encoding
2.8 The Coded Bitstream
2.9 Storing and Transmitting the Coded Bitstream
2.10 The Decoder
2.10.1 Entropy Decoding
2.10.2 Rescaling and Inverse Transform
2.10.3 Reconstruction
2.10.4 Decoder Output
2.11 The Video Codec Model
2.12 Video Codec Performance
2.12.1 Video Quality
2.12.2 Subjective Quality Measurement
2.12.2.1 Factors Influencing Subjective Quality
2.12.2.2 Subjective Quality Assessment Methods
2.12.3 Objective Quality Measurement
2.12.3.1 PSNR
2.12.3.2 Other Objective Quality Metrics
2.13 Conclusion
References
Chapter 3 A History of Video Coding and Video Coding Standards
3.1 Introduction
3.2 The Foundations of Video Coding, 1950-1990
3.2.1 Entropy Coding: 1952-1987
3.2.2 Frame Differencing: 1956-1972
3.2.3 The Discrete Cosine Transform: 1974-1984.
3.2.4 The Development of the Video Codec Model: 1977-1985
3.2.5 Early Advances in Motion-Compensated Prediction: 1989-1995
3.2.5.1 Bidirectional Prediction
3.2.5.2 Variable Block Size Motion Compensation
3.2.5.3 Multiple Reference Frames
3.2.5.4 Motion Compensation with Fractional Pixel Accuracy
3.3 Video Coding Standards and Formats: 1990-2021
3.3.1 How Are the Standards Developed?
3.3.1.1 MPEG and VCEG
3.3.1.2 The Development Process
3.3.1.3 Standards and Intellectual Property Rights
3.3.2 H.120, H.261 and MPEG-1: Early Video Coding Standards
3.3.3 MPEG-2: Digital TV and DVD-Video
3.3.4 H.263 and MPEG-4 Part 2: Internet and Mobile Video
3.3.5 H.264: Multi-Purpose Video Coding
3.3.6 H.265/HEVC
3.3.7 H.266/VVC
3.3.8 Other Standards and Formats
3.3.8.1 VC-1
3.3.8.2 VP8, VP9 and AV1
3.3.8.3 Essential Video Coding and Low-ComplexityEnhancement Video Coding
3.4 Comparing Video Coding Standards
3.5 Conclusions
References
Chapter 4 Structures
4.1 Introduction
4.1.1 How Does a Video Codec Use Structures?
4.1.2 Separating a Sequence into Processing Units
4.1.3 Regular or Irregular Structures
4.1.4 Transmission and Access
4.2 Coded Video: Sequence to Picture
4.2.1 Coded Sequence
4.2.2 Coded Frame or Picture
4.2.3 Multi-picture Structures
4.3 Coded Video: Picture to Basic Unit
4.3.1 Basic Units
4.3.2 Slices
4.3.3 Tiles
4.4 Coded Video: Basic Unit to Block
4.4.1 Coding Structures
4.4.2 Prediction Structures
4.4.3 Transform Structures
4.5 HEVC Coding Structures
4.5.1 Coded HEVC Sequence
4.5.2 Parameter Sets
4.5.3 Profiles and Levels
4.5.4 Reference Picture Sets
4.5.5 HEVC Picture Types and Random Access
4.5.5.1 Random Access Picture
4.5.5.2 Examples: RAPs, Trailing and Leading Pictures.
4.5.5.3 Random Access and Error Recovery
4.5.5.4 Bitstream Splicing Operations
4.5.6 HEVC-Coded Pictures
4.5.7 Slices and Slice Segments
4.5.8 Tiles
4.5.9 Combining Slices and Tiles in HEVC
4.5.10 HEVC Structures and Parallel Processing
4.5.10.1 Parallel Decoding Using Tiles
4.5.10.2 Parallel Decoding Using Slices
4.5.10.3 Wavefront Parallel Processing/Parallelising CTU Rows
4.5.11 Coding Tree Units and Coding Units
4.5.11.1 Quantisation Groups
4.5.12 Prediction Units and Transform Units
4.5.12.1 Prediction Unit Structures
4.5.12.2 Transform Unit, Transform Block and Transform Sub-Block
4.5.13 HEVC Structures Example
4.6 Structures in Versatile Video Coding/H.266
4.6.1 Sequence Down to Picture
4.6.2 Picture Down to Coding Unit
4.7 Conclusion
Reference
Chapter 5 Intra Prediction
5.1 Introduction
5.2 The Intra Prediction Process
5.3 Intra Prediction Modes
5.3.1 DC Prediction
5.3.2 Planar Prediction
5.3.3 Directional Prediction
5.3.4 Sample Filtering
5.4 Prediction Block Sizes
5.5 Signalling Intra Prediction Choices
5.6 Choosing a Prediction
5.7 HEVC Intra Prediction
5.7.1 HEVC Intra Prediction Modes
5.7.2 HEVC Intra Prediction Filtering
5.7.3 Signalling HEVC Intra Prediction Modes
5.7.3.1 HEVC Luma Prediction Mode Signalling
5.7.3.2 HEVC Chroma Prediction Mode Signalling
5.7.4 HEVC Intra Prediction Examples
5.8 VVC Intra Prediction
5.8.1 Intra Prediction Modes
5.8.2 Intra Prediction of Rectangular Blocks
5.8.3 Intra Interpolation Filters
5.8.4 Cross-Component Prediction
5.8.5 Extra Reference Lines
5.8.6 Intra Sub-Partitions
5.8.7 VVC Intra-coding Performance
5.9 Conclusions
References
Chapter 6 Inter Prediction
6.1 Introduction
6.2 Inter Prediction
the Basics.
6.2.1 Motion Vectors for Inter Prediction: The Terminology Problem
6.3 Forward, Backward and Biprediction
6.3.1 Picture or Slice Types
6.3.2 Forward Prediction from Past Frames
6.3.3 Backward Prediction from Future Frames
6.3.4 Biprediction from Two Frames
6.4 Inter Prediction Block Sizes
6.5 Motion Vectors
6.5.1 How Does the Encoder Choose Each Motion Vector?
6.6 Sub-Pixel Interpolation
6.6.1 What Does a Sub-Pixel Interpolated Reference Frame Look Like?
6.6.2 Interpolation Filters
6.6.3 Trade-Offs in Sub-Pixel Interpolation
6.7 Reference Pictures
6.7.1 Reference Pictures for Inter Prediction
6.7.2 Reference Picture Structures
6.7.2.1 Low-Delay Picture Structures
6.7.2.2 Random Access Picture Structures
6.7.3 Reference Picture Buffers
6.7.3.1 Adding Pictures to the Reference Picture Buffer
6.7.3.2 Removing Pictures from the Reference Picture Buffer
6.7.3.3 Ordering Reference Pictures
6.7.3.4 Communicating Reference Picture Structures
6.8 Signalling Inter Prediction Choices
6.8.1 Inter Prediction Parameters
6.8.2 Signalling the Base Motion Vector Parameters
6.8.3 Signalling the Delta Motion Vector Parameters
6.8.4 Signalling Reference Picture Choices
6.9 Skip Mode
6.10 Loop Filter
6.11 When Inter Prediction Does Not Find a Good Match
6.12 HEVC Inter Prediction
6.12.1 Overview
6.12.1.1 HEVC Inter Prediction Example
6.12.2 HEVC Inter prediction Unit Partitioning
6.12.2.1 Slice Types and PU Types
6.12.2.2 Symmetric PU Partitions
6.12.2.3 Asymmetric PU Partitions
6.12.2.4 CU Partitionings into Inter-PUs: Examples
6.12.3 HEVC Inter-PU Coding
6.12.3.1 Skip or Merge Coding
6.12.3.2 Inter-PU Coding: Advanced Motion Vector Prediction
6.12.3.3 HEVC Inter-PU Coding: Examples
6.12.4 HEVC Inter Prediction Sources.
6.12.5 HEVC Biprediction and Weighted Prediction
6.12.6 HEVC Sub-Pixel Interpolation
6.12.6.1 Luma Sample Interpolation
6.12.6.2 Chroma Sample Interpolation
6.12.7 HEVC Reference Pictures
6.13 Inter Prediction in VVC
6.13.1 Inter Prediction Partitions
6.13.2 Extended Merge Mode
6.13.3 Affine Motion-Compensated Prediction
6.13.4 Biprediction Enhancements
6.13.5 Sub-Pixel Interpolated Motion Compensation
6.14 Conclusions
References
Chapter 7 Transform and Quantisation
7.1 Introduction
7.1.1 Transform: An Overview
7.1.2 Quantisation: An Overview
7.2 Residual Blocks
7.3 Block Transforms
7.3.1 What Is the Purpose of a Block Transform?
7.3.2 Spatial Frequencies and Basis Patterns
7.3.3 What Does the Transform Do?
7.3.4 Types of Block Transform
7.3.4.1 The Karhunen-Loeve Transform
7.3.5 The Discrete Cosine Transform
7.3.6 The Hadamard Transform
7.3.7 Discrete Sine Transform
7.3.8 The Discrete Cosine Transform in Detail
7.3.9 The Discrete Cosine Transform: What It Means
7.3.10 The Discrete Cosine Transform: Significant Coefficients
7.4 Quantisation
7.4.1 Combining Transform and Quantisation
7.4.2 Designing the Quantiser
7.5 Transform and Quantisation in Practice
7.6 HEVC Transform and Quantisation
7.6.1 Overview of HEVC Transform and Quantisation
7.6.2 The HEVC Transforms
7.6.2.1 DCT-Based Transforms
7.6.2.2 DST-Based Transform
7.6.2.3 The HEVC Forward and Inverse Transform and Quantisation Processes
7.6.3 HEVC Quantisation and Rescaling
7.6.3.1 HEVC QP and Qstep
7.6.3.2 HEVC Forward Quantiser
7.6.3.3 HEVC Rescaling
7.6.3.4 HEVC Quantisation Matrices
7.6.4 Example: The Complete HEVC Transform and Quantise Processes
7.6.4.1 Forward Transform and Quantise
7.6.4.2 Rescale and Inverse Transform.
Title Page
Copyright Page
Contents
About the Author
Acknowledgements
About the Companion Website
Chapter 1 Introduction
1.1 Why Write This Book?
1.2 What Is in the Book?
1.3 How Should You Use This Book?
References
Chapter 2 Video Coding and Video Quality
2.1 Introduction
2.2 An Overview of Video Coding
2.2.1 Just How Much Data Are We Talking About?
2.2.2 Bandwidth Is Increasing Every Year, But...
2.3 Inputs and Outputs
2.3.1 Digital Video
2.3.2 Uncompressed Video Formats
2.3.2.1 Frames, Fields and Pixels
2.3.2.2 Colour and Colour Spaces
2.3.3 Side Information
2.3.4 Coded Video
2.4 Structural Elements
2.5 Prediction
2.5.1 Prediction in a Video Encoder
2.5.2 Prediction in a Video Decoder
2.5.3 Prediction Example
2.5.4 Choosing a Prediction
2.5.5 Inter and Intra Prediction
2.6 Transform and Quantisation
2.6.1 Transform
2.6.2 Quantisation
2.7 Bitstream Coding
2.7.1 Pre-processing
2.7.2 Entropy Encoding
2.8 The Coded Bitstream
2.9 Storing and Transmitting the Coded Bitstream
2.10 The Decoder
2.10.1 Entropy Decoding
2.10.2 Rescaling and Inverse Transform
2.10.3 Reconstruction
2.10.4 Decoder Output
2.11 The Video Codec Model
2.12 Video Codec Performance
2.12.1 Video Quality
2.12.2 Subjective Quality Measurement
2.12.2.1 Factors Influencing Subjective Quality
2.12.2.2 Subjective Quality Assessment Methods
2.12.3 Objective Quality Measurement
2.12.3.1 PSNR
2.12.3.2 Other Objective Quality Metrics
2.13 Conclusion
References
Chapter 3 A History of Video Coding and Video Coding Standards
3.1 Introduction
3.2 The Foundations of Video Coding, 1950-1990
3.2.1 Entropy Coding: 1952-1987
3.2.2 Frame Differencing: 1956-1972
3.2.3 The Discrete Cosine Transform: 1974-1984.
3.2.4 The Development of the Video Codec Model: 1977-1985
3.2.5 Early Advances in Motion-Compensated Prediction: 1989-1995
3.2.5.1 Bidirectional Prediction
3.2.5.2 Variable Block Size Motion Compensation
3.2.5.3 Multiple Reference Frames
3.2.5.4 Motion Compensation with Fractional Pixel Accuracy
3.3 Video Coding Standards and Formats: 1990-2021
3.3.1 How Are the Standards Developed?
3.3.1.1 MPEG and VCEG
3.3.1.2 The Development Process
3.3.1.3 Standards and Intellectual Property Rights
3.3.2 H.120, H.261 and MPEG-1: Early Video Coding Standards
3.3.3 MPEG-2: Digital TV and DVD-Video
3.3.4 H.263 and MPEG-4 Part 2: Internet and Mobile Video
3.3.5 H.264: Multi-Purpose Video Coding
3.3.6 H.265/HEVC
3.3.7 H.266/VVC
3.3.8 Other Standards and Formats
3.3.8.1 VC-1
3.3.8.2 VP8, VP9 and AV1
3.3.8.3 Essential Video Coding and Low-ComplexityEnhancement Video Coding
3.4 Comparing Video Coding Standards
3.5 Conclusions
References
Chapter 4 Structures
4.1 Introduction
4.1.1 How Does a Video Codec Use Structures?
4.1.2 Separating a Sequence into Processing Units
4.1.3 Regular or Irregular Structures
4.1.4 Transmission and Access
4.2 Coded Video: Sequence to Picture
4.2.1 Coded Sequence
4.2.2 Coded Frame or Picture
4.2.3 Multi-picture Structures
4.3 Coded Video: Picture to Basic Unit
4.3.1 Basic Units
4.3.2 Slices
4.3.3 Tiles
4.4 Coded Video: Basic Unit to Block
4.4.1 Coding Structures
4.4.2 Prediction Structures
4.4.3 Transform Structures
4.5 HEVC Coding Structures
4.5.1 Coded HEVC Sequence
4.5.2 Parameter Sets
4.5.3 Profiles and Levels
4.5.4 Reference Picture Sets
4.5.5 HEVC Picture Types and Random Access
4.5.5.1 Random Access Picture
4.5.5.2 Examples: RAPs, Trailing and Leading Pictures.
4.5.5.3 Random Access and Error Recovery
4.5.5.4 Bitstream Splicing Operations
4.5.6 HEVC-Coded Pictures
4.5.7 Slices and Slice Segments
4.5.8 Tiles
4.5.9 Combining Slices and Tiles in HEVC
4.5.10 HEVC Structures and Parallel Processing
4.5.10.1 Parallel Decoding Using Tiles
4.5.10.2 Parallel Decoding Using Slices
4.5.10.3 Wavefront Parallel Processing/Parallelising CTU Rows
4.5.11 Coding Tree Units and Coding Units
4.5.11.1 Quantisation Groups
4.5.12 Prediction Units and Transform Units
4.5.12.1 Prediction Unit Structures
4.5.12.2 Transform Unit, Transform Block and Transform Sub-Block
4.5.13 HEVC Structures Example
4.6 Structures in Versatile Video Coding/H.266
4.6.1 Sequence Down to Picture
4.6.2 Picture Down to Coding Unit
4.7 Conclusion
Reference
Chapter 5 Intra Prediction
5.1 Introduction
5.2 The Intra Prediction Process
5.3 Intra Prediction Modes
5.3.1 DC Prediction
5.3.2 Planar Prediction
5.3.3 Directional Prediction
5.3.4 Sample Filtering
5.4 Prediction Block Sizes
5.5 Signalling Intra Prediction Choices
5.6 Choosing a Prediction
5.7 HEVC Intra Prediction
5.7.1 HEVC Intra Prediction Modes
5.7.2 HEVC Intra Prediction Filtering
5.7.3 Signalling HEVC Intra Prediction Modes
5.7.3.1 HEVC Luma Prediction Mode Signalling
5.7.3.2 HEVC Chroma Prediction Mode Signalling
5.7.4 HEVC Intra Prediction Examples
5.8 VVC Intra Prediction
5.8.1 Intra Prediction Modes
5.8.2 Intra Prediction of Rectangular Blocks
5.8.3 Intra Interpolation Filters
5.8.4 Cross-Component Prediction
5.8.5 Extra Reference Lines
5.8.6 Intra Sub-Partitions
5.8.7 VVC Intra-coding Performance
5.9 Conclusions
References
Chapter 6 Inter Prediction
6.1 Introduction
6.2 Inter Prediction
the Basics.
6.2.1 Motion Vectors for Inter Prediction: The Terminology Problem
6.3 Forward, Backward and Biprediction
6.3.1 Picture or Slice Types
6.3.2 Forward Prediction from Past Frames
6.3.3 Backward Prediction from Future Frames
6.3.4 Biprediction from Two Frames
6.4 Inter Prediction Block Sizes
6.5 Motion Vectors
6.5.1 How Does the Encoder Choose Each Motion Vector?
6.6 Sub-Pixel Interpolation
6.6.1 What Does a Sub-Pixel Interpolated Reference Frame Look Like?
6.6.2 Interpolation Filters
6.6.3 Trade-Offs in Sub-Pixel Interpolation
6.7 Reference Pictures
6.7.1 Reference Pictures for Inter Prediction
6.7.2 Reference Picture Structures
6.7.2.1 Low-Delay Picture Structures
6.7.2.2 Random Access Picture Structures
6.7.3 Reference Picture Buffers
6.7.3.1 Adding Pictures to the Reference Picture Buffer
6.7.3.2 Removing Pictures from the Reference Picture Buffer
6.7.3.3 Ordering Reference Pictures
6.7.3.4 Communicating Reference Picture Structures
6.8 Signalling Inter Prediction Choices
6.8.1 Inter Prediction Parameters
6.8.2 Signalling the Base Motion Vector Parameters
6.8.3 Signalling the Delta Motion Vector Parameters
6.8.4 Signalling Reference Picture Choices
6.9 Skip Mode
6.10 Loop Filter
6.11 When Inter Prediction Does Not Find a Good Match
6.12 HEVC Inter Prediction
6.12.1 Overview
6.12.1.1 HEVC Inter Prediction Example
6.12.2 HEVC Inter prediction Unit Partitioning
6.12.2.1 Slice Types and PU Types
6.12.2.2 Symmetric PU Partitions
6.12.2.3 Asymmetric PU Partitions
6.12.2.4 CU Partitionings into Inter-PUs: Examples
6.12.3 HEVC Inter-PU Coding
6.12.3.1 Skip or Merge Coding
6.12.3.2 Inter-PU Coding: Advanced Motion Vector Prediction
6.12.3.3 HEVC Inter-PU Coding: Examples
6.12.4 HEVC Inter Prediction Sources.
6.12.5 HEVC Biprediction and Weighted Prediction
6.12.6 HEVC Sub-Pixel Interpolation
6.12.6.1 Luma Sample Interpolation
6.12.6.2 Chroma Sample Interpolation
6.12.7 HEVC Reference Pictures
6.13 Inter Prediction in VVC
6.13.1 Inter Prediction Partitions
6.13.2 Extended Merge Mode
6.13.3 Affine Motion-Compensated Prediction
6.13.4 Biprediction Enhancements
6.13.5 Sub-Pixel Interpolated Motion Compensation
6.14 Conclusions
References
Chapter 7 Transform and Quantisation
7.1 Introduction
7.1.1 Transform: An Overview
7.1.2 Quantisation: An Overview
7.2 Residual Blocks
7.3 Block Transforms
7.3.1 What Is the Purpose of a Block Transform?
7.3.2 Spatial Frequencies and Basis Patterns
7.3.3 What Does the Transform Do?
7.3.4 Types of Block Transform
7.3.4.1 The Karhunen-Loeve Transform
7.3.5 The Discrete Cosine Transform
7.3.6 The Hadamard Transform
7.3.7 Discrete Sine Transform
7.3.8 The Discrete Cosine Transform in Detail
7.3.9 The Discrete Cosine Transform: What It Means
7.3.10 The Discrete Cosine Transform: Significant Coefficients
7.4 Quantisation
7.4.1 Combining Transform and Quantisation
7.4.2 Designing the Quantiser
7.5 Transform and Quantisation in Practice
7.6 HEVC Transform and Quantisation
7.6.1 Overview of HEVC Transform and Quantisation
7.6.2 The HEVC Transforms
7.6.2.1 DCT-Based Transforms
7.6.2.2 DST-Based Transform
7.6.2.3 The HEVC Forward and Inverse Transform and Quantisation Processes
7.6.3 HEVC Quantisation and Rescaling
7.6.3.1 HEVC QP and Qstep
7.6.3.2 HEVC Forward Quantiser
7.6.3.3 HEVC Rescaling
7.6.3.4 HEVC Quantisation Matrices
7.6.4 Example: The Complete HEVC Transform and Quantise Processes
7.6.4.1 Forward Transform and Quantise
7.6.4.2 Rescale and Inverse Transform.