Bug 317093 - Feature Request: Multithreaded transcoding
Summary: Feature Request: Multithreaded transcoding
Status: CONFIRMED
Alias: None
Product: amarok
Classification: Applications
Component: Transcoding (show other bugs)
Version: 2.7.0
Platform: Arch Linux Linux
: NOR wishlist
Target Milestone: 2.8
Assignee: Amarok Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-03-20 17:27 UTC by Dave
Modified: 2013-07-31 18:37 UTC (History)
6 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dave 2013-03-20 17:27:06 UTC
In my opinion, the greatest limitation of the transcoding feature is its lack of concurrency.  I have over 5000 audio files and most are incompatible with my portable media device.  With modern processors containing upwards of 12 cores, concurrent transcoding could improve performance by an order of magnitude.  Right now, it takes over 24 hours to copy and transcode my music collection with Amarok and only 3 with Gtkpod (which will transcode multiple files concurrently).  I'd much rather use Amarok to manage my media devices.

I'm not familiar with the Amarok source, but here's how I would handle the problem from a generic standpoint:

void copyToDevice(file_list, device) {
  foreach file in file_list {
    if isCompatable(file, device) {
      copyQueue.push(file);
    } else {
      file transcodeQueue.push(file);
    }
  }

  transcodeThreadPool->setMaxThreadCount(getNumberOfCpus());
  foreach file in transcodeQueue {
    transcodeThreadPool->run(new TranscodeThread(file));
  }
}

class TranscodeThread : public Runnable
{
  void run() {
    ffmpeg file output;
    copyQueue.push(output);
  }
}

Sorry for my bad pseudo code.  In the above model, compatible files would begin transferring immediately.  At the same time, incompatible files would begin transcoding and pushed to the back of the queue.  Obviously the queues are thread safe ;)

Reproducible: Always

Steps to Reproduce:
Copy many incompatible files to your media device, watch how many ffmpeg processes are spawned.
Actual Results:  
Media files are transcoded in serial.

Expected Results:  
Media files should be encoded in parallel.

I've posted elsewhere about this, as well as done some testing.  I've determined that it's not possible to use ffmpeg's internal threading capability, that the only good way to do this is to simply launch many ffmpegs.

http://forum.kde.org/viewtopic.php?f=116&t=107234&p=247671&hilit=concurrent#p247671

http://teom.org/blog/kde/a-summer-of-transcoding-for-amarok/
(I'm Dave in the comments section)
Comment 1 Matěj Laitl 2013-03-20 19:43:56 UTC
Confirmed, shouldn't be really hard to do.
Comment 2 Anmol Ahuja 2013-05-05 12:56:05 UTC
I'd like to work on this bug.
Comment 3 Anmol Ahuja 2013-05-05 13:04:45 UTC
How should I go about this?
Start as many TranscodingJobs as the numer of threads the user has selected, starting new ones as old ones finish?
Or should the multithreading logic be in the TranscodingJob itself, starting a new QProcess for every file to be transcoded, and new files added via something like addTrack(const KUrl &src, const KUrl &dest)?
Or should I be using something like QtConcurrent::map?
Comment 4 vedant agarwala 2013-07-23 21:34:31 UTC
What is the status of this bug? Is it up for taking or Anmol you are working on it?
Comment 5 Myriam Schweingruber 2013-07-23 21:48:37 UTC
(In reply to comment #4)
> What is the status of this bug? Is it up for taking or Anmol you are working
> on it?

I think Anmol is busy with GSoC, so feel free to take it. But please take this to the amarok-devel@ ML in any case
Comment 6 vedant agarwala 2013-07-31 18:37:03 UTC
Patch created: https://git.reviewboard.kde.org/r/111824/

I could not test with an iPod because I do not have one, so someone else will have to do that. Also, provide feedback about the time reduction, if any, while transcoding.