Assignment 1-a Specification

1   Key Information

• The mark of "COMP3419 Assignment 1-a Motion Estimation and Visualization" will be marked based on canvas submission.  Due Time:  before 23:59, Sunday of Week 8 (25-Sep-2022).

• This individual assignment is worth 8% of your final assessment.

• Submission Deliverable: Students are asked to create a zip file of all deliverable, including all source code and a demo video. A README txt file (to describe the steps/instructions regarding how to get their source code running to derive the expected outputs) should be included within the zip file. Please be aware of the following submission restrictions that (1) this zip file should be named as "SIDxxx_Ass1a.zip" where xxx denotes the student ID

(e.g., "SID450003419_Ass1a"); (2) the format of source code is restricted to py or ipynb; (3) the demo video should be named as "SIDxxx_Ass1a" and the format of the demo video is restricted to mp4.

• Student’s assignment will only be marked if all deliverable can be accessed from the Canvas System, and they can be runnable following instructions provided in README txt file. Failing to follow these restrictions or missing of demo video would cause a deduction of 4 marks. Once plagiarism detected by the Canvas system, the student will receive no mark immediately, as well as other related penalties from university.

2   Demo Video

• Students are required to use the video provided as the input source for this assignment (monkey.avi).

• Students are asked to record a demo video including their output video and demonstration of their implementation/code.

• The length of the demo video recording should not exceed 1 minute and should be sufficient enough to show the motion changes.

3   General Marking Policy

Late Submission & Demonstration Policy

For the late submission cases, penalties will be assigned according to the university wide late penalties for assignment Clause 7A of the Assessment Procedures.

Special Consideration and Arrangements

While you are studying, there may be circumstances or essential commitments that impact your academic performance.  Our special consideration and special arrangements process is there to

support you in these situations.  More information on how to lodge the special consideration application, can be found from this webpage.

4   Motion Estimation and Visualization (Assignment 1-a)

This assignment is to perform the motion estimation with macroblock matching on the video provided. The basic premise of motion estimation is that in most cases, consecutive video frames will be similar except for changes induced by objects moving within the frames. The basic idea of motion estimation is to define grids of block regions on two adjacent frames and find the displacement vector

(a 2D Cartesian vector in 2D videos) between the matched blocks. To describe the meta-algorithm step by step:

1. Iterate the video frames Fi of size (fx , fy); Define a grid block of size (2k + 1, 2k +1), whose size should be odd for convenience in determining the coordinates of centroid point (or, origin point) for each grid block. Each frame Fi results in  grid blocks overall.

2. For each grid block Bx,y) (whose centroid point) at (x,y) in current frame Fi, search for all candidate blocks in next reference frame Fi+1, and then find its matched grid block B at (x\ ,y\ ) in Fi+1 .

• Note:  Bx,y)  is also dubbed as the source block, while B is also called as the destination or target block. Fi represents the current frame, while Fi+1 denotes the next or reference frame. Please see Fig. 1(a) for visual illustration.

• Among all candidates blocks within the search area of Bx,y) in Fi+1, the matched grid block B should produce the minimum sum-of-squared-distances (SSD) to B

• Simplifying Bx,y) and B as Bi and B+1, respectively, their square root of SSD can be computed as

sqrt(SSD(Bi ,B+1)) = /Lllbxk byk b0 [Bi(bx, by, bc) −B+1(bx, by, bc)]2 ,    (1)

where bx, by, and bc denote the inside-block index of pixel location in x-direction, y- direction, and color channels (i.e., RGB), respectively. For example, when bx = by = −k and bc = 0, then Bi(bx, by, bc) denotes the R-channel value of pixel at top-left corner of

Bi .

3. The (centroid-point) displacement vector from source block (Bi) to target block (B+1) can be represented as a 2-D vector, i.e., (x\ −x,y\ −y). Next, save the displacement vectors for all source blocks Bi in frame Fi as a 3D matrix of shape ( , , 2).

4. For better visualization purpose, filter out the displacement vectors computed in Step 3 to remove unexpected noises, whose sqrt(SSD) are outside of a self-defined thresholding range (Tmin , Tmax).

(a) Illustration of the block matching

(b) Example arrow visualization of extracted motion vectors.

Figure 1: The illustration of block matching algorithm and the extracted optical flows.

• In other words, neglect the noisy displacement vectors whose sqrt(SSD)  (Tmin , Tmax) .

• The optimal values of Tmin and Tmax vary on different videos or implementation details, which should be determined by experiments.

5. Draw arrows to visualise these selected (centroid-point) displacement fields on frame Fi .

6. Repeat Step 1-5 for all frames.

R     Tip: To speed up for Step 2, an assumption could be reached that there are high chances that

these matched blocks might appear in positions close to the source block Bi . Therefore, we