d2jsp
Log InRegister
d2jsp Forums > Off-Topic > Computers & IT > Programming & Development > Opencv C++ / Computer Vision Help > Camera Coordinates & Stitching
Add Reply New Topic New Poll
Member
Posts: 37,470
Joined: Feb 27 2009
Gold: 12.95
Sep 10 2014 05:32am
Hi there!
I'm quite new to openCV/C++ and atm I'm a little stuck concerning the computer vision part I want to do.

What I got is multiple (atm 2) cameras and their captured images as well as the cameras' intrinsics and distortion data. Also some kinda warehouse layout as jpg.

Now what I wanna do is to put the warehouse image in the background and add the images from the camera (in realtime) to specific places. Therefore I'd have to somehow get the coordinates of what the camera is showing to match the coordinates of said layout (where (0,0) is just the upper left corner or smth.).
When I got that I want to blend the camera image on top of a specific part (resized to fit right in). The problem here is that I haven't found a good way in openCV to blend images of different sizes and at certain locations.
Lastly I want to do almost the same with the second camera. Only difference: Stitch it to the 1st image manually with the help of those calculated coordinates. (I already got a "working" stitching code which works with keypoints and matching but I think for my problem it would be better if I could somehow "match" it at a certain position).

If anyone feels familiar with one of those problems I have (finding and setting coordinates, blending of differently sized images, stitching) let me know :)


/e: also i'm curious as to how to change the following code to stitch 3 (or more) instead of 2 cameras:

Code

#include <stdio.h>
#include <opencv.hpp>
#include <nonfree/nonfree.hpp>
#include <vector>

using namespace std;
using namespace cv;

Mat TwoInOneOut(Mat Left, Mat Right);

int main()
{
VideoCapture stream1(1);
VideoCapture stream2(2);

if (!stream1.isOpened()) { //check if video device has been initialised
cout << "cannot open camera 1";
return 0;
}

if (!stream2.isOpened()) { //check if video device has been initialised
cout << "cannot open camera 2";
return 0;
}

Mat H;
int mode=0;

while (true) {
Mat cameraFrame1;
stream1.read(cameraFrame1); //get one frame from cam1 (left)


Mat cameraFrame2;
stream2.read(cameraFrame2); //get one frame from cam2 (right)

if(mode == 0)
{
imshow("Left", cameraFrame1);
imshow("Right", cameraFrame2);
}

Mat Left(cameraFrame1.rows, cameraFrame1.cols, CV_8U);
Mat Right(cameraFrame2.rows, cameraFrame2.cols, CV_8U);
cvtColor(cameraFrame1, Left, CV_RGB2GRAY, CV_8U);
cvtColor(cameraFrame2, Right, CV_RGB2GRAY, CV_8U);

if (waitKey(33) == 'p')
{
printf("Homography Matrix Processing\n");
H = TwoInOneOut(Left, Right);
mode=1;
destroyWindow("Left");
destroyWindow("Right");
}

if(waitKey(33) == 'r')
{
printf("normal mode\n");
destroyWindow("Processing");
mode=0;
}

if(H.cols == 3 && H.rows == 3)
{
Mat WarpImg( Left.rows*2, Left.cols*2, cameraFrame1.depth() );
warpPerspective(cameraFrame2, WarpImg, H, Size(WarpImg.cols, WarpImg.rows));
Mat tempWarpImg = WarpImg(Rect(0,0,Left.cols,Left.rows));
cameraFrame1.copyTo(tempWarpImg);

if(mode ==1)
imshow("Processing", WarpImg );

}



if (waitKey(33) == 27)
break;
}

destroyAllWindows();


}


Mat TwoInOneOut(Mat Left, Mat Right)
{
Mat H;

if(Left.channels() != 1 || Right.channels() != 1)
{
printf("Channel Error\n");
return H;
}

/////////////////
//Detect the keypoints using SURF Detector
int minHessian = 300; //1500;
SurfFeatureDetector detector( minHessian );
SurfDescriptorExtractor extractor;

/////////////////
//A
std::vector<KeyPoint> kp_Left;
detector.detect( Left, kp_Left );
Mat des_Left;
extractor.compute( Left, kp_Left, des_Left );

/////////////////
//B
std::vector<KeyPoint> kp_Right;
detector.detect( Right, kp_Right );
Mat des_Right;
extractor.compute( Right, kp_Right, des_Right );

/////////////////
//Match
std::vector<vector<DMatch > > matches;
FlannBasedMatcher matcher;
matcher.knnMatch(des_Left, des_Right, matches, 2);
std::vector<DMatch > good_matches;
good_matches.reserve(matches.size());

for (size_t i = 0; i < matches.size(); ++i)
{
if (matches[i].size() < 2)
continue;

const DMatch &m1 = matches[i][0];
const DMatch &m2 = matches[i][1];

if(m1.distance <= 0.7 * m2.distance)
good_matches.push_back(m1);
}

//Draw only "good" matches
Mat img_matches;
drawMatches( Left, kp_Left, Right, kp_Right, good_matches,
img_matches, Scalar::all(-1), Scalar::all(-1),
vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );
imshow("Match", img_matches);

/////////////////
//Find H
if(good_matches.size() > 20 )
{
std::vector< Point2f > LeftMatchPT;
std::vector< Point2f > RightMatchPT;

for( unsigned int i = 0; i < good_matches.size(); i++ )
{
//-- Get the keypoints from the good matches
LeftMatchPT.push_back( kp_Left[ good_matches[i].queryIdx ].pt );
RightMatchPT.push_back( kp_Right[ good_matches[i].trainIdx ].pt );
}

H = findHomography( RightMatchPT, LeftMatchPT, CV_RANSAC );
}


return H;

return img_matches;
}


I already tried handling it with doing everything twice and Left + Middle and Middle + Right or rather LeftMiddle + Right but I got some errors concerning type changes etc iirc and kinda gave up on it for now.

This post was edited by KillerApfel on Sep 10 2014 05:46am
Member
Posts: 37,470
Joined: Feb 27 2009
Gold: 12.95
Sep 15 2014 02:46am
Just stumbled upon another problem.
I'm trying to stream the camera images in realtime from a socket into a browser.
So far I've found several ways to do the reverse thing but not tooo much on what I want to do.. at least not in c++ & on windows 8. (MPJEG Streamer only working on linux afaik)
Anyone ever did something like that? Haven't worked with sockets/browser stuff so far so I really have no idea.
Go Back To Programming & Development Topic List
Add Reply New Topic New Poll