Author: Vijay

Introduction

Hello folks 👋

The past 4 months has been pretty exciting and challenging for me . Me and my mentor Andi Bräu together worked on a cool project called Videoodyssee , with GSoC’22 almost coming to an end in this blog post I wanna share all of my accomplishments , results and little bit about the project.

About Videoodyssee

In 2017 we started our video portal for freifunk at media.freifunk.net. To get videos online we have a more or less complicated upload workflow:

1. Upload video file to a processing server.
2. Correct video file settings.
3. Convert video to target formats.
4. upload to CDN and Youtube.
5. Publish the video to the media portal.

The previous Videoosyssee system has some problem like

• With current lambdacd pipeline we can’t have multiple workers to enable parallel video processing.
• The current lambdacd pipeline isn’t maintainable anymore and it has got a lot of dependency issues.
• Currently we don’t have any video verification steps in our publishing workflow which leads to publishing of unwanted videos.

So we decided to re-implement the pipeline using a new CI/CD tool and we chose to use the GoCD tool to build the pipeline as it suits better for the video processing pipeline we are looking to build. Later in the project we also decided to re-implement the Videoodyssee uploader using React.

Systems Involved in the project:

Video processing pipeline

The pipeline is implemented using GoCD , a CI/CD tool as it suits better for the video processing pipeline we are looking to build. The GoCD follows master-slave architecture which consists a Go-Server and multiple Go-Agents.

Go-Server

Go-server acts as a master and is the one that controls everything, assign the jobs to Go-agents and provides the user interface to users of the system.This is where we define the pipelines and trigger the pipelines.Go-server assigns the jobs to the agents which are free and stores the artifacts produced by Go-agents.

Go-Agents

The agents are the ones that do the actual work like processing videos , generating thumbnails and uploading processed videos. We can have N number of agents which we can scale up or scale down according to our needs. So in future if we need to process more number videos at the same time we can increase the number of agents to get the job done.

You can have a look at the video processing pipeline code here. And here is the Videpipeline GitHub repo https://github.com/freifunk/videopipeline .

Tasks accomplished:

• We created a config repo in GitHub to store the pipeline code so that whenever we change the pipeline code GoCD server will automatically pull the changes and builds the new pipeline.
• Implemented the video processing pipeline.
• Modified the pipeline processing scripts to make them work with the new GoCD pipeline.
• Automated the provisioning and deployment of Go-Server and Go-Agents using Ansible. You can have a look at the Ansible playbooks here.

Remaining tasks:

• Automating the pipeline deployment from the GitHub config repo.
• Publish the video to YouTube.
• Notify the admins with an email when a new video is submitted by someone.
• Notify the publisher when his video got approved , processed and published to Voctoweb.

Videoodyssee Frontend

For the users to submit the video details we need a frontend application which takes the data from the user and sends it to the REST API which will eventually trigger the pipeline using the GoCD API to start the video upload workflow.

We chose to use the React to implement the frontend application as it is quick and easy. The frontend application will have a upload form for normal users and admin dashboard for the admins for all administration tasks like approving/rejecting videos , updating video details etc.

Task Accomplished:

• Implemented the video upload form so a user can submit the details of a new video.
• Implemented the Videodyssee admin dashboard where admins can review the videos to accept or reject them.
• Implemented Authentication so that only admins can access the videoodyssee admin dashboard.

Remaining Tasks:

• Implement the functionality to update the details of already published videos.
• Implement the functionality to update video details before approving a video.
• Implement the user management.
• Implement profile management.

Videoodyssee API

We used Node.js Express framework to implement the REST API which will handle the requests from the Videoodyssee frontend. We chose Express as it is quick and easy to implement a REST API using Express framework. We have used systemd as the process manager for the API server which will have some benefits like the server will restart on its own when it crashes.

Tasks Accomplished :

• Implemented a route which will take the video details from the frontend and triggers the GoCD processing pipeline.
• Automated the deployment process of REST API server using Ansible by implementing videoodyssee-api playbook.

Video review process:

1. Admin logs into the Videoodyssee admin dashboard by using username and password.
2. Reviews the video details and if clicks the video title he can open the video in another tab to watch it.
3. If everything is fine he will approve the video.
4. If some of the details are wrong he can change the details and then approves the video.
5. if the admin thinks that the video is a spam video he can reject it.

Conclusion:

Finally the GSoC’22 has been pretty exciting and challenging for me .My mentor Andi Bräu really helped me a lot in the development process. I have learned a lot about open source development workflows, automation using Ansible and unit testing.

And finally , I would like to thank my mentor Andi Bräu for his incredible support and feedback.

If you got interested in my project and wanted to know more about it you can reach out to me on Linkedin.

Hello folks 👋

The first phase of my GSoC is pretty exciting and challenging , together with my mentor I decided to complete the video processing part of the project in the first phase of the project.

Before started working on the project me and my mentor Andi Bräu figured out a video processing workflow so that we get a bird’s-eye view of the project and can figure out systems need to be implemented for the project.

Video Processing Workflow

Systems Involved in the project:

1 . Videoodyssee Frontend : A React frontend application for the users to submit the video data and admin dashboard for the admins.

2. Videoodyssee API : A Node.js REST API implemented using Express framework.

3. Videopipeline : A GoCD server with processing pipeline to process and publish the video.

Video pipeline

After evaluating several CI/CD tools we chose to use the GoCD tool to build the pipeline as it suits better for the video processing pipeline we are looking to build.

Tasks completed:

• We created a config repo in GitHub to store the pipeline code so that whenever we change the pipeline code GoCD server will automatically pull the changes and builds the new pipeline.
• Automated the installation of the GoCD server and GoCD agent in our remote machines using ansible playbooks.
• Implemented processing pipeline upto the video encoding part.
• Changed the previous video processing bash scripts to make them work with the current GoCD pipeline.

Videoodyssee Frontend

For the users to submit the video details we need a frontend application which takes the data from the user and sends it to the REST API which will eventually trigger the pipeline using the GoCD API to start the video upload workflow.

We chose to use the React to implement the frontend application as it is quick and easy. The frontend application will have a upload form for normal users and admin dashboard for the admins for all administration tasks like approving/rejecting videos , updating video details etc.

Tasks Completed:

• Completed the video upload form so that a user can submit the details of a new video .
• Automated the deployment of the frontend application to GitHub pages using GitHub Actions by implementing a deploy workflow.
• UI design of the admin dashboard is also completed.

Below is how the admin dashboard will look like:

Videoodyssee API

We used Node.js Express framework to implement the REST API which will handle the requests from the Videoodyssee frontend. We chose Express as it is quick and easy to implement a REST API using Express framework.

Tasks completed :

• Implemented a route which will take the video details from the frontend and triggers the GoCD processing pipeline.
• Automated the deployment process of the pipeline using Ansible by implementing videoodyssee-api playbook.

Conclusion:

Finally the first phase of GSoC is very exciting and challenging for me and my mentor Andi Bräu really helped me a lot in making design choices and development. I hope the second phase of GSoC would be as exciting as the first phase.

Tasks for the second Phase:

• Completing the remaining part of the processing pipeline upto the publishing step.
• Automating the pipeline deployment process using Ansible .
• Implementing the Unit testing in the REST API.
• Completing the Admin Dashboard frontend and backend.

Introduction

Hello folks 👋,

I’m Vijay. In this summer I will be working on the project ‘Videoodyssee System Refactoring’.In this first blog post I would like about my project and little bit about me.

About me

I’m Vijay Reddy , a Computer Science undergrad from India. I’m interested in Web application security, Computer Architecture and Blockchain Security. I do play CTF’s in my free time to learn about the latest security vulnerabilities and to learn new skills.

I’m new to the open source world and I found it really fascinating to work with communities around the world which are making the world a better place.I’m also fascinated by the idea of writing code which will be used by thousands of people around the world .Open source contributions will allow me to work with new people and make connections with cool people around the world which will be helpful for my career.

About Videoodyssee

In 2017 we started our video portal for freifunk at media.freifunk.net. To get videos online we have a more or less complicated upload workflow:

The following systems are involved:

1. Videoodyssee Uploader – a web portal to upload video files built with Spring Boot, Thymeleaf, CSS, JS, Ansible, and Maven
2. Videoodyssee – the processing server, built with bash scripts and python
3. lambdacd pipeline – controls the workflow, built with Clojure, Leiningen, and LambdaCD
4. mirrorbits – the CDN, built with Go
5. voctoweb – the media portal, built with Ruby, HTML, CSS, JS and puma

As of now, it is possible to upload new videos, but the process is not comfortable. So we need to improve the software.

Problems with the current Videoodyssee System:

1. With current lambdacd pipeline we can’t have multiple workers to enable parallel video processing.
2. The current lambdacd pipeline isn’t maintainable anymore and it has got a lot of dependency issues.
3. Currently we don’t have any video verification steps in our publishing workflow which leads to publishing of unwanted videos.
4. Currently there isn’t any feedback mechanism implemented in the front-end part so whenever a user uploads a video using Videoodyssee uploader, the user will not get any feedback on whether the video is uploaded , processed and published or not.

Project Milestones

• Reimplement the video processing pipeline using a better and maintainable CI/CD tool which supports multiple workers for parallel processing:
  • Replace lambdacd pipeline with a better-maintained work flow system.
  • Update the processing pipeline scripts to make them work with the newest versions of tools like FFmpeg.
  • Reimplement the video processing pipeline with those updated pipeline scripts using the new CI/CD tool.
• Add publishing workflow to prevent the publishing of unwanted videos.

• Implement video processing feedback mechanism in the Videoodyssee uploader so that the user will get notified when each step of the video processing pipeline is finished.

Community Bonding Period:

In the community Bonding Period I have familiarised myself with my mentor Andi Bräu and discussed a lot about the project.

In this 3-week community bonding period , I evaluated different CI/CD tools with my mentor to find out a perfect CI/CD tool that fits best for our video processing pipeline and found out that GoCD and Jenkins will be perfect for our usage.

In the first week of the coding period, I will implement a sample pipeline using both of these tools to evaluate the performance and convenience of these tools , then I’ll choose one for implementing our video processing pipeline.

And finally , I would like to thank my mentor Andi Bräu for his incredible support and feedback.