Page 1 of 3 123 LastLast
Results 1 to 10 of 21

Thread: Help Needed: Making My Own Search Engine

  1. #1
    Join Date
    Dec 2007
    Location
    die Welt
    Beans
    635
    Distro
    Ubuntu Gnome 14.04 Trusty Tahr

    Question Help Needed: Making My Own Search Engine

    Hello,

    I'm interested in the possibilty of making my own search engine rather than relying on the existing ones like Google, Bing, and Dogpile. I'm interested in doing this for a variety of reasons; to overcome censorship, geographical and language filtering, and to maximize privacy. Does anyone have any ideas about how to do this?

    I suspect I would probably want to dedicate a machine for this purpose, and of course some kind of web-crawler application would have to be installed to crawl the web, looking for web sites to index. Also, I suspect that I would probably have to let such an application run uninterrupted, pretty much continuously. I've tried searching the web for pre-existing applications or scripts to do this, but I just haven't found anything. Perhaps I don't know the vocabulary and am not entering the correct search terms.

    Any help or suggestions appreciated.


    Thanks, Hannibal
    I love it when a plan comes together. It keeps me "on the jazz."

    Ubuntu Quantum 31.10.03 & Windows 10000 on Dell Inspiron 55200 Infinite-Core
    dual-boot system. 'Kill Bill XIV!'

  2. #2
    Join Date
    Oct 2010
    Location
    Britain
    Beans
    79
    Distro
    Ubuntu 10.10 Maverick Meerkat

    Re: Help Needed: Making My Own Search Engine

    Gosh!!!
    You must be a genius.
    Do you not have to come up with search algorithms etc, ets?

  3. #3
    Join Date
    May 2010
    Location
    uk
    Beans
    9,299
    Distro
    Xubuntu 14.04 Trusty Tahr

    Re: Help Needed: Making My Own Search Engine

    Quote Originally Posted by jennybrew View Post
    Gosh!!!
    You must be a genius.
    Do you not have to come up with search algorithms etc, ets?
    and a big, fat, optimised database....

    Stick to google.
    If you believe everything you read, you better not read. ~ Japanese Proverb

    If you don't read the newspaper, you're uninformed. If you read the newspaper, you're mis-informed. - Mark Twain

    Thinking about becoming an Ubuntu Member?

  4. #4
    Join Date
    Dec 2007
    Location
    die Welt
    Beans
    635
    Distro
    Ubuntu Gnome 14.04 Trusty Tahr

    Red face Re: Help Needed: Making My Own Search Engine

    For the afore mentioned reasons, I'd prefer not to. Also, I suspect much of the things you've mentioned have already been done by others; so I don't think I'd have to completely re-invent the wheel, as you seem to suggest. Additionally, I would expect that this forum would attract individuals of that calibre. So for the moment at least, I don't think I'm in the wrong place. Also, for the moment, this is just a thought experiment...
    why such strong reactions?

    Any 'constructive' replys appreciated.


    Thanks, Hannibal
    Last edited by coljohnhannibalsmith; April 2nd, 2011 at 03:07 AM.
    I love it when a plan comes together. It keeps me "on the jazz."

    Ubuntu Quantum 31.10.03 & Windows 10000 on Dell Inspiron 55200 Infinite-Core
    dual-boot system. 'Kill Bill XIV!'

  5. #5
    Join Date
    May 2010
    Location
    uk
    Beans
    9,299
    Distro
    Xubuntu 14.04 Trusty Tahr

    Re: Help Needed: Making My Own Search Engine

    Hi

    As i said, start off by setting up large, optimised database. MySQL is a good start.

    Read this.

    http://www.webopedia.com/DidYouKnow/...nginesWork.asp

    If you are worried about privacy, have you considered TOR ?

    Can you administer a database and write code ?

    Kind regards
    Last edited by matt_symes; April 2nd, 2011 at 03:07 AM.
    If you believe everything you read, you better not read. ~ Japanese Proverb

    If you don't read the newspaper, you're uninformed. If you read the newspaper, you're mis-informed. - Mark Twain

    Thinking about becoming an Ubuntu Member?

  6. #6
    Join Date
    Mar 2009
    Location
    Buenos Aires, AR
    Beans
    2,325
    Distro
    Ubuntu

    Re: Help Needed: Making My Own Search Engine

    You dont need to do that, theres already open source search engines,
    in example see Apache Lucene and derivatives...

  7. #7
    Join Date
    Feb 2010
    Location
    South of the Maple Trees
    Beans
    9,316
    Distro
    Xubuntu 18.04 Bionic Beaver

    Re: Help Needed: Making My Own Search Engine

    Moved to programming talk.

    Please do not create support threads in the Community Cafe.
    Cheers & Beers, uRock

  8. #8
    Join Date
    Jan 2007
    Location
    Location: Location:
    Beans
    1,246
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Help Needed: Making My Own Search Engine

    The undertaking of creating your own search engine would be ridiculously complex to nigh impossible to implement unless you have tons of dollars and some amazing coders at your disposal.

    Thats as constructive as I can be on this.
    clear && echo paste url and press enter; read paste; (youtube-dl $paste) | zenity --progress --title="" --text "Downloading, please wait" --auto-close --pulsate && ans=$(zenity --file-selection); gnome-terminal -x mplayer "$ans"

  9. #9
    Join Date
    May 2010
    Location
    uk
    Beans
    9,299
    Distro
    Xubuntu 14.04 Trusty Tahr

    Re: Help Needed: Making My Own Search Engine

    Hi

    the moment, this is just a thought experiment...
    why such strong reactions?
    No strong reactions. I think you misunderstand me.

    It is just a huge, huge job and you need more than one skill set to do it.

    Kind regards
    If you believe everything you read, you better not read. ~ Japanese Proverb

    If you don't read the newspaper, you're uninformed. If you read the newspaper, you're mis-informed. - Mark Twain

    Thinking about becoming an Ubuntu Member?

  10. #10
    Join Date
    Dec 2007
    Location
    die Welt
    Beans
    635
    Distro
    Ubuntu Gnome 14.04 Trusty Tahr

    Smile Re: Help Needed: Making My Own Search Engine

    Quote Originally Posted by matt_symes View Post
    Hi



    No strong reactions. I think you misunderstand me.

    It is just a huge, huge job and you need more than one skill set to do it.

    Kind regards

    Thank you Matt,

    Perhaps I did misunderstand You. You're probably very right that this is a very complex undertaking. I am however very interested in how this can be done and I have admitted my naivety about this topic. Also, these kinds of challenges are my 'Raison d'etre,' or something. I've always found the available search engines, not just a potential source of insecurity, but also a massive and distracting generator of advertising, and disappointing in terms of relevent results.

    Thank you for the informative information link you provided; I will study it as soon as I get the chance. BTW the poster from Buenos Aires suggested something close to what I was hoping for:

    "You dont need to do that, there's already open source search engines,
    in example see Apache Lucene and derivatives... "

    I'm sure this is still a complex solution, and I'll try not to underestimate it. BTW, I can code a 'little.' I'm very comfortable compiling code, and somewhat comfortable modifying small errors in other peoples code.

    'uRock,' sorry about posting in the wrong forum. I didn't know where else to post this thread, and I figured that it would be better to 'start' here, and then get moved, than to start the post in a more specific forum. Perhaps 'General Help' would have been a better starting point?


    Thanks, Hannibal
    Last edited by coljohnhannibalsmith; April 2nd, 2011 at 03:34 PM.
    I love it when a plan comes together. It keeps me "on the jazz."

    Ubuntu Quantum 31.10.03 & Windows 10000 on Dell Inspiron 55200 Infinite-Core
    dual-boot system. 'Kill Bill XIV!'

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •