This avoids a potential race with the demuxer adding new streams. It is also more efficient, since we no longer do inter-thread transfers of packets that will be just discarded.