Language design

[Mikeb85]

R and Octave aren't typically used for long-running applications constantly polling data. They're most often used to explore a given dataset to find patterns or create charts / visuals.

R is closer to thinking about datasets and has lots of functionality for loading / saving data from various formats. It's more of a scientists tool.

Octave/Matlab can be used similarly, but they're more geared towards the mathematical side. Courses in machine learning usually use Octave/Matlab because they natively handle basic stats, linear algebra, and calculus.

Ruby would be more geared towards the automation process. It would be a more logical language to set up a server to dispatch the analysis / data mining algorithms at scheduled times and to create a summary of the current state (either web or local file).

Yeah, I was kind of thinking that about R, but it's nice to hear from someone else.

What you described is my end goal, I know it'll probably take me some time to get to that point, but I think with the available tools I can eventually get to that point.

A pipe dream of mine would also be to mine data from Twitter, Reuters, etc..., and with some clever regex scripts try to gather news about various companies and incorporate that into my scripts (if the script could look for positive/negative connotations connected to various numbers that would be even better).

For now I've got a few scripts to gather quotes that are fairly basic, but effective (and gather real-time quotes remarkably well - nice being able to quickly check on things from a console).

[ofnuts]

A pipe dream of mine would also be to mine data from Twitter, Reuters, etc..., and with some clever regex scripts try to gather news about various companies and incorporate that into my scripts (if the script could look for positive/negative connotations connected to various numbers that would be even better).

I have worked with people doing just that... it requires a lot more than just "clever scripts" and regular expressions...

[Mikeb85]

I have worked with people doing just that... it requires a lot more than just "clever scripts" and regular expressions...

I'm sure it does, hence the pipe dream stuff. The difference between basic AI programs and IBM's Watson is hundreds of engineers and millions of lines of code, but it's the same basic idea.

As of right now though, data mining and sorting through small amounts of data is feasible for me, this is just a fun project, maybe in 5-10 years I'll have something useful? I've been making good money trading sorting through everything myself, so I'm in no hurry.

[CptPicard]

basic AI programs and IBM's Watson is hundreds of engineers and millions of lines of code, but it's the same basic idea.

I would say it is the different, much deeper ideas that make AI programs actually work where clever scripts and regexps never could

[Mikeb85]

I would say it is the different, much deeper ideas that make AI programs actually work where clever scripts and regexps never could

Hmmm, it seems to me as though most AI programs involve sorting data into databases, creating associations, searching through terms that come up, and then making decisions based on a statistic associated with said term. Or am I completely off-base? Alot of my understanding of AI comes from what I learned as a kid from my dad (a programmer), reading about/watching a piece on IBM's Watson, and basic game AI examples...

So as a most basic example, if I wanted a program to tell me if I should invest in stock A or B, I'd get the program to pull say, a set of historical quotes for both from the internet, maybe tell me what the % change is over 30 days, and pick the higher one... Or maybe calculate the number of up/down days over a period of time, and pick the stock with the highest probably of being up tomorrow (based on criteria)? Like I said, very basic...

And then it seems as though slightly more complicated AI is based around creating 'rules', based on mathematical equations such as: http://en.wikipedia.org/wiki/Bayesian_inference

Is this how the basic principles of AI work, or am I missing something?

[vexorian]

AI has multiple definitions, but AI algorithms typically are stuff in which a programmer does not really think of the logic. So you would rarely be inventing your own regex for searches (that would mean natural intelligence using a computer). With a neuronal network, associations are rather made automatically and just depend on the input you give to the system rather than your own tweaks. Genetic programming is supposed to create programs automatically. And so and so. So often, AI is about using canned systems of inference that already exist but you spend more time "training" them and tweaking them. In this regards, the simplicity in code is not different than grabbing your Java and making it use a regex class. You can grab your Java and make it use a neuronal network class.

There are usually hybrid systems. Specially in games. Things like pathing may be the product of genetic algorithms but basic sequences of actions might be hard coded. Etc.

But either way, you'd be surprised at how simple it is to code your own neuronal network or genetic algorithm. The real problems lie in finding good applications and in having a lot of patience in performing the training.

So I'm pretty much a newb when it comes to programming (although I've always followed tech and have been around Linux for years), but I'm curious about something from the pros here.

Do you guys believe it is possible to have a 'best' multipurpose language, is there 1 you spend most of your time using? Do you believe languages are inherently limited, or limited only by usage and libraries?

Also, if you do use multiple languages for various purposes, how many and for what?

And note - I don't mean this thread to be a this vs. that thread, but rather a discussion about the nature of programming languages. And if the question is silly tell me too, but no versus arguing...

A language can be used for everything. But if that's the case, then everything you do with the language will look messy and complicated. Hence why I like C++.

Otherwise, if you want a language that would do what you want but very cleanly, then it is better to go with more specialized languages.

[mevun]

A pipe dream of mine would also be to mine data from Twitter, Reuters, etc..., and with some clever regex scripts try to gather news about various companies and incorporate that into my scripts (if the script could look for positive/negative connotations connected to various numbers that would be even better).

You don't have to write the data mining yourself. There's a company called AlchemyAPI which has a web interfaces for doing some of that, including "sentiment analysis". They do provide free "developer" accounts which allows you a limited rate of interface calls. I assume they have competitors, but I haven't really searched for any. (The company is located in my hometown and I have met people from there.)

Of course, hedge funds already do what you suggest. We just don't know how successful they are at it and how much human intelligence is still involved.