Directly push buffers into queue

For the amount these functions get called, std copy is way to expensive
to do. This removes the need for buffer to be copied twice, first to a
std::vector then SmallFixedVector. This also directly copies buffer into
a smallVector without the init overhead smallVector's ctor do. This
result is a massive performance bootst (at least on arm devices).

Also use memcpy since that interestingly seems to be faster on arm
devices, on amd64 they seems to be about the same speed.