Are there any alternatives to using OpenCV's imdecode? It is too slow

I have created a DLL where the user can either read an image from a file name or from a stream as follows:

std::string filePath = "SomeImage.bmp";

// (1) Reading from a file
Image2D img1;
img1.readImage(filePath);

// (2) Reading from a stream
std::ifstream imgStream (filePath.c_str(), std::ios::binary);
Image2D img2;
img2.readImage(imgStream);

The first readImage(filePath) is implemented using cv::imread(filePath) which is reasonably fast (on average 0.001 seconds for a 600 x 900 image). However, the second version readImage(fileStream) is implemented using cv::imdecode which is considerably slower (on average 2.5 seconds for the same image).

Are there any alternatives to cv::imdecode where I can decode an image from a memory buffer without taking such a long time? This is for the core component of an application that is frequently used, so it has to be quick.

Any assistance would be appreciated. Thanks in advance.

EDIT:

I measure the timings using a timer. It didn't make sense to me too. I don't understand why there is such a large disparity in the time. Image2D is just a class that has an OpenCV matrix as a member. The implementation of the readImage functions are simplified as follows:

int Image2D::readImage(std::ifstream& input)
{       
    input.seekg(0, std::ios::end);
    size_t fileSize = input.tellg();
    input.seekg(0, std::ios::beg);

    if (fileSize == 0) {
        return 1;
    }

    std::vector<unsigned char> data(fileSize);
    input.read(reinterpret_cast<char*>(&data[0]), sizeof(unsigned char) * fileSize);

    if (!input) {
        return 1;
    }

    StopWatch stopWatch;
    mImg = cv::imdecode(cv::Mat(data), CV_LOAD_IMAGE_COLOR);
    std::cout << "Time to decode: " << stopWatch.getElapsedTime() << std::endl;

    return 0;
}


int Image2D::readImage(const std::string& fileName)
{
    StopWatch stopWatch;
    mImg = cv::imread(fileName, CV_LOAD_IMAGE_COLOR);

    std::cout << "Time to read image: " << stopWatch.getElapsedTime() << std::endl;
    return 0;
}