The latest articles on DEV Community by toshusai (@toshusai). https://dev.to/toshusai https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F760517%2Fba218c9c-51f1-4878-9a84-f310cbdec1a7.png https://dev.to/toshusai en toshusai Sat, 27 Nov 2021 06:47:52 +0000 https://dev.to/toshusai/how-to-create-a-video-editor-that-works-in-your-browser-5akh https://dev.to/toshusai/how-to-create-a-video-editor-that-works-in-your-browser-5akh <h2> Intro </h2> <p>Recently, browser features such as SharedArrayBuffer and WebAssembly are increasing. Video and video converter <a href="https://github.com/Kagami/ffmpeg.js">ffmpeg also works in browsers</a>.</p> <p>Therefore, I tried to create a video editing software that works on a browser experimentally.</p> <p>You can try the demo from here.<br> <a href="https://vega.toshusai.net/">https://vega.toshusai.net/</a><br> source<br> <a href="https://github.com/toshusai/vega">https://github.com/toshusai/vega</a></p> <h2> Basic Idea </h2> <p>Using basic below materials.</p> <ul> <li>images</li> <li>video</li> <li>audio</li> <li>text</li> </ul> <p>In order to display these materials on the screen, a container for information such as playback position, length, and screen position is called a strip.</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--lUPYjhcy--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4vx4f5p239txrfshn5lh.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--lUPYjhcy--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4vx4f5p239txrfshn5lh.png" alt="Strips" width="658" height="344"></a></p> <p>Convert these strips into the final video in a way that suits each material.</p> <p>FFmpeg inputs are <code>video</code>, <code>image sequences</code> or <code>audio</code>.<br> If you want to add text, you have to generate image included text.</p> <h3> Video </h3> <p>So, convert all the images/videos to image sequence once and pass them to ffmpeg. used <a href="https://github.com/mrdoob/three.js/">three.js</a> for the renderer. Three.js can handle image as Texture, also video as VideoTexture.</p> <p>The image capture step uses <a href="https://github.com/spite/ccapture.js/">ccapture</a> to ensure that the playback frame is converted to an image. All frame images to webm video.</p> <h3> Audio </h3> <p>Next, Merge image sequence and audio. This is ffmpeg part.</p> <p>For video, get a specific range of audio from the video.<br> For audio, cut the audio.</p> <p>Finally, use <code>filter_compex</code> to combine all the audio in strip time.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> -i _vega_video.webm -i video.mp3 -i audio.mp4 -filter_complex [2:a]adelay=5715|5715[out2];[1:a]adelay=260|260[out1];[out1][out2]amix=inputs=2[out] -map [out]:a </code></pre> </div> <p>This is the overall flow diagram.</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--cUHGvpqI--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nry9otwunuct3lt5ffrt.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--cUHGvpqI--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nry9otwunuct3lt5ffrt.png" alt="architecture" width="645" height="696"></a></p> <h2> Thanks Open Source Softwares/Libraries </h2> <ul> <li>JavaScript Framework <ul> <li><a href="https://github.com/nuxt/nuxt.js">Nuxt.js</a></li> <li><a href="https://github.com/vuejs/vue">Vue.js</a></li> </ul> </li> <li>Video/Audio Converter <ul> <li><a href="https://github.com/ffmpegwasm/ffmpeg.wasm">ffmpeg.wasm</a></li> <li><a href="https://www.ffmpeg.org/">FFmpeg</a></li> </ul> </li> <li>Capture canvas <ul> <li><a href="https://github.com/spite/ccapture.js/">CCapture.js</a></li> <li><a href="https://github.com/thenickdude/webm-writer-js/">WebM video encoder</a></li> </ul> </li> <li>3D library (as Renderer) <ul> <li><a href="https://github.com/mrdoob/three.js/">three.js</a></li> </ul> </li> <li>Audio visualization <ul> <li><a href="https://github.com/katspaugh/wavesurfer.js/">wavesufer.js</a></li> </ul> </li> <li>Monospace font <ul> <li><a href="https://github.com/edihbrandon/RictyDiminished">RictyDimished</a></li> </ul> </li> <li>Icons <ul> <li><a href="https://github.com/templarian/MaterialDesign">Material Design Icons</a></li> <li><a href="https://github.com/adobe/spectrum-css-workflow-icons">Spectrum CSS Workflow Icons</a></li> </ul> </li> <li>Design System <ul> <li><a href="https://github.com/toshusai/spectrum-vue">Spectrum Vue</a></li> <li><a href="https://github.com/adobe/spectrum-css">Spectrum CSS</a></li> </ul> </li> <li>Documentation <ul> <li><a href="https://github.com/docsifyjs/docsify/">Docsify</a></li> </ul> </li> <li>Test Asset <ul> <li>Audio</li> <li><a href="https://ondoku3.com/">ondoku3</a></li> <li>Video</li> <li><a href="https://www.bigbuckbunny.org">Blender Foundation</a></li> <li>Image</li> <li><a href="https://www.kenney.nl/">Kenney</a></li> </ul> </li> </ul> <p>I also created a UI components to use for this. If you like / are interested, check this out too.<br> <a href="https://github.com/toshusai/spectrum-vue">https://github.com/toshusai/spectrum-vue</a></p> webdev tutorial typescript nuxt