Results 1 to 4 of 4

Thread: Download and merge multiple mp4 files on the fly

  1. #1
    Join Date
    Oct 2020
    Beans
    28

    Download and merge multiple mp4 files on the fly

    Hi,
    I need to download and merge multiple mp4 files on the fly. These are many 1 megabyte mp4 files. I created a script (I'm not an expert in this) to download them individually, but then manually merging them with avidemux fills all the ram of the PC. The script looks like this:

    for i in $ (seq 1000000);
    do
    wget https://address.com/file$i.mp4
    done

    Is there a way to download and merge all these files on the fly?
    Thanks

  2. #2
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Download and merge multiple mp4 files on the fly

    I'm sure there is, but I'd pull the files down 1 at a time and use mkvmerge myself. I have a script that will merge file-001.mkv .... file-999.mkv into a single file.mkv. From the mkvmerge manpage:

    Code:
    SYNOPSIS
           mkvmerge [global options] {-o out} [options1] {file1}
                    [[options2] {file2}] [@optionsfile]
    $ more ~/bin/mkv-merge.sh
    Code:
    #!/bin/bash
    
    if [ "X$1" = "X" ] ; then
      echo "Usage: $0 file-001.mkv"
      exit 1;
    fi
    
    ROOT=`echo $1 | sed -e 's/-001.mkv//g'` 
    CNT=1
    
    OUT="mkvmerge -o \"$ROOT.mkv\" \"$ROOT-00$CNT.mkv\" "
    
    for CNT in {002..999} ; do
      if [ -f "$ROOT-$CNT.mkv" ] ; then
        OUT="$OUT + \"$ROOT-$CNT.mkv\""
      fi
    done
    echo $OUT | sh -
    It is for mkv files, but tweaking it to support .mp4 input files wouldn't be hard. Leading zeros can be moved in the for-loop line. Just change the 002 --> 2.

    Hope this gives you some ideas. Oh - and no spaces (or other odd characters) are allowed in the filenames. Some might work. Who knows?

    Update: this tool will also validate that the input files all match on important muxing parameters. It doesn't transcode anything, so it is as fast as a file copy and because most of the files will likely be in the OS-managed buffers, performance will be about the same as a file copy.
    Last edited by TheFu; January 10th, 2021 at 05:29 PM. Reason: validation of parameters and copy-like performance.

  3. #3
    Join Date
    Apr 2007
    Beans
    2,911
    Distro
    Ubuntu

    Re: Download and merge multiple mp4 files on the fly

    In principle, ffmpeg can take input from an URL, but I am not sure whether you could concatenate this way. I would rather work with a script that 1) downloads the file and 2) concatenates that to the video file that contains the already downloaded parts. See here on how that works. It boils down to creating a text file listing files you want to concatenate ("mylist.txt"), after which you concatenate the files with
    Code:
    ffmpeg -f concat -safe 0 -i mylist.txt -c copy output.mp4
    Actually, to concatenate only a few files, following also works:
    Code:
    ffmpeg -i concat:"input1.mp4|input2.mp4" output.mp4
    You could have your script 1) download a new file 2) add it to the concatenated file of the previous downloads to a new concatenated file 3) move the new file out or delete it 4) delete the contaminated file of the previous downloads and give its name to the new concatenated file to append in the next loop.

    This command transmuxes, i.e., it does not recode and should not need excessive memory. For this to work, the files you download should be exactly the same format (codecs, resolution).

    On a more advanced level, true on the fly concatenation may be possible with a named pipe. You would download to the pipe, and on the other end ffmpeg would read from the pipe. I have no experience whatsoever if and how this would work with ffmpeg.
    Last edited by vanadium; January 10th, 2021 at 05:10 PM.

  4. #4
    Join Date
    Oct 2020
    Beans
    28

    Re: Download and merge multiple mp4 files on the fly

    Thanks, i will try

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •