After a few google searches I found sphider I really love it and it does exactly what I was expecting.

You define the URL's you wish to index, set the level of how deep you wish to follow the links and it gives you a super fast and powerful search engine. Check it out: http://search.sofasurfer.org/