Forums  > Software  > Scalalab Fslab, functional/quasi functional languages  
     
Page 2 of 3Goto to page: 1, [2], 3 Prev Next
Display using:  

jslade


Total Posts: 1168
Joined: Feb 2007
 
Posted: 2016-01-11 19:28
Neural nets look very pretty in J, but they run slower than in Matlab and friends, because J doesn't use Lapack natively. This may have changed; Pascal Jasmin has written J bindings for Arrayfire.
https://github.com/Pascal-J/Jfire

I liked Clojure as a language, but my experience with it was that it was abysmally slow on numerics, basically because the JVM is crap at numerics. There was no integrated columnar DB (the thing that makes J a spark-beater on data aggregation). Clojure was was the last time I tried to do a project in the JVM without someone shoveling money at me to deal with JVM problems. If I wanted to play with Lisp again, I'd find one that talks to C binaries. Or maybe Golang.
https://github.com/glycerine/zygomys

"Learning, n. The kind of ignorance distinguishing the studious."

agentq


Total Posts: 28
Joined: Jul 2008
 
Posted: 2016-01-13 06:11
Since jslade is being shy ...
Kerf looks interesting. And is readable by mere mortals Blush

http://www.kerfsoftware.com/

EspressoLover


Total Posts: 364
Joined: Jan 2015
 
Posted: 2016-01-14 00:31
R is already a pretty functional language. Functions are first-class citizens, closures are fully supported and the standard higher-order function set (lambdas/map/reduce/filter/curry/etc.) already exist or are trivially easy to implement.

> I tried building things to replace slowpoke Matlab/R

R's really not slow as long as its vectorized. Neural networks are one of the few things that R is slow at, because they can't be. But you can always throw SNOW on top of your R instance. In most cases adding spinning up more instances in the cloud is probably cheaper than porting code from R to a faster language.

Good questions outrank easy answers. -Paul Samuelson

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-01-15 20:24
With great respect for the huge community R = slow, a lot of things cannot be vectorized. Then you need to write c and link so what is the point.

"amicus Plato sed magis amica Veritas"

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-01-16 12:34
kerf is really interesting thank you for sharing....

"amicus Plato sed magis amica Veritas"

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-02-11 22:28
How difficult is to integrate j with other languages? How would you use it in a trading system, just for array manipulations, sorting, subsetting, resampling etc?

"amicus Plato sed magis amica Veritas"

jslade


Total Posts: 1168
Joined: Feb 2007
 
Posted: 2016-02-12 07:01
J has trivial interop with R and C, and somewhat less trivial with Java. You can always messagepack/zeromq things around. People use it for prop trading: I think that's what keeps jsoftware alive.

If you're good at J and understanding algorithms, it's probably a better choice than Python or something, and will force you to make good choices (since it doesn't have anything like SKlearn). If you're not good at J, well, you will have to enjoy puzzles. Early work was, "write amazing things in one line that takes 10 minutes to write and replaces 40 pages of C++/Java and runs faster, then spend 2 days figuring out how to do a seemingly trivial (four character) operation that is one line in C++ or Java."

"Learning, n. The kind of ignorance distinguishing the studious."

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-02-12 11:08
This is interesting thank you. How would you database something like hdf5 or jd? I would need a columnar db that is readable by other languages I guess.


"amicus Plato sed magis amica Veritas"

jslade


Total Posts: 1168
Joined: Feb 2007
 
Posted: 2016-02-12 19:02
HDF5 is a pretty good data store with some primitives for building databases. Lots of half baked things out there using this idea. I wrote one myself.
You can read jd in R or whatever, but you're doing memory copies and expensive translation all over the place, just like in Postgres or whatever. If you want to use jd to do big data processing, you have to do it in J. Same story with K, Kerf, whatever. Operate directly on the mmaped data structures and you get huge performance benefits. FWIIW, this is one of the performance benefits over Spark as well; most of the data stores need to be translated into something else to be processed.

"Learning, n. The kind of ignorance distinguishing the studious."

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-02-12 23:57
Let me recap:
1) j for data processing, subsetting, reading/writing hdf5 or mmaped files
2) talking to R or whaever
3) talking to C / java for data/trading apis

Does it sound ok for prod trading systems?


"amicus Plato sed magis amica Veritas"

Rashomon


Total Posts: 193
Joined: Mar 2011
 
Posted: 2016-02-13 17:13
jslade: Which ones do you think are half baked? several of the noSQL but probably not all…

"My hands are small, I know, but they're not yours, they are my own. And they're, not yours, they are my own." ~ Jewel

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-02-17 15:11
Ok done some research and figured out that the solution with J+hdf5 is very good if we just do databasing and subsetting, anything else would be out of my league, too difficult.

So back to square one.

Using python pandas for a project these days and found it good and simple. I think python, pandas (optimized), pytables and cython/numba a winner for now, it is all very simple, at least on paper, speed is acceptable with cython. Pandas is good but will need to be rewritten to work fast I believe.

Am I happy with it, probably not, scala is the best bet but for now it is really hard to make it work and I need to learn it too, which is not straightforward even though I like functional prog more than OO. Really is something I will do slowly as it becomes more popular.
The key is to do everything in one platform (research and prod) otherwise there is no point, the bottleneck in productivity is always there.

I also would like to say that .NET is really good and simple. In fact for the slow stuff matlab + C# (for execution) is the winner and I do not have to rewrite anything.






"amicus Plato sed magis amica Veritas"

jslade


Total Posts: 1168
Joined: Feb 2007
 
Posted: 2016-02-17 22:32
Why would you use J + hdf5 when J has two decent database engines that come with it? I mean, one of them costs money for commercial use, but it's pretty cheap compared to writing something in HDF5 if you value your time at all.

Most people should just use R or Pandas and be happy about it.

Rashomon: I thought the Pandas/Pytables<->hdf5 thing was kind of half baked. There have been other things over the years, but I can't remember any of their names either.



"Learning, n. The kind of ignorance distinguishing the studious."

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-02-18 15:26
Why would you use J + hdf5 when J has two decent database engines that come with it? I mean, one of them costs money for commercial use, but it's pretty cheap compared to writing something in HDF5 if you value your time at all.

You are right thank you, in any case J is a no go for me.

Most people should just use R or Pandas and be happy about it.

I really would like to believe it, R even in soft real time does not work (so is any scripting language), even though I used in prod even for pretty serious (but slow) things. Apparently cython should be able to handle it (on paper I will report back when I tried it).

So to clarify I was not really talking about managing large datasets but really doing that and also run it in real time with some adjustments of course but without rewriting the whole thing.

"amicus Plato sed magis amica Veritas"

EspressoLover


Total Posts: 364
Joined: Jan 2015
 
Posted: 2016-02-19 05:30
Why not Haskell? It's blazing fast, easily hooks into C, and the degree of abstraction allows for rapid prototyping in research. Downsides are the learning curve and lack of the great numeric library you get with R/Matlab/Numpy.

Good questions outrank easy answers. -Paul Samuelson

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-02-21 19:06
Thank you another one to add to the queue....

"amicus Plato sed magis amica Veritas"

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-02-22 13:57
Rashomon: I thought the Pandas/Pytables<->hdf5 thing was kind of half baked. There have been other things over the years, but I can't remember any of their names either.

More on this, hdf5-pytables can be used in-memory I am gonna try that and see how it goes. There is a lot of easy functionality compared to talking to the api directly (I do that in matlab still struggling with it).


"amicus Plato sed magis amica Veritas"

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-03-06 15:33
Any thoughts on Julia? It seems to be interesting also it can be natively integrated with C and python.

"amicus Plato sed magis amica Veritas"

Hansi


Total Posts: 303
Joined: Mar 2010
 
Posted: 2016-03-06 16:00
>Any thoughts on Julia?

Too early days. Give it 3-4 years to work out the kinks.

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-03-07 12:30
You are right, it looks so good though....

"amicus Plato sed magis amica Veritas"

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-03-14 16:04
Completely off topic but kind of related to the objective if you like. Matlab C compiler works decently well these days. Some major changes are necessary in the code (no variable size arrays otherwise is slow) but you can really compile the whole application in C/C++, including the parfor loops (I have not tried it yet).
Then you can call it from a C++ main program.

Someone good in C++ will see the upside I think.


"amicus Plato sed magis amica Veritas"

EspressoLover


Total Posts: 364
Joined: Jan 2015
 
Posted: 2016-03-14 20:09
Yeah, but then you have to use Matlab, which is, by far, the worst of the major numerical languages. Not to mention the licensing headaches if you want to cluster anything.

Good questions outrank easy answers. -Paul Samuelson

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-03-15 21:49
Mhh matlab is quite straightforward and neat imo.
Yeah licensing headaches are a major issue, reason why a lot of shops are leaving it. But once it is coded in C you can run it everywhere.

Before the issue was that because it was a narrow subset that can be handled by the coder, there was no real advantage.

I guess this can be done in Cython too and no headaches?

"amicus Plato sed magis amica Veritas"

rickyvic


Total Posts: 164
Joined: Jul 2013
 
Posted: 2016-07-08 11:22
http://news.efinancialcareers.com/uk-en/246387/boost-your-salary-by-learning-the-julia-programming-language/

"amicus Plato sed magis amica Veritas"

Hansi


Total Posts: 303
Joined: Mar 2010
 
Posted: 2016-07-09 13:01
Still haven't met anyone using it for anything other than trying it out or small personal hoppy type stuff but it's definitely going into interesting directions (comments are more interesting than the article):
https://matloff.wordpress.com/2016/07/06/latest-on-the-julia-language-vs-r/

Also: https://www.youtube.com/user/JuliaLanguage/videos
Previous Thread :: Next Thread 
Page 2 of 3Goto to page: 1, [2], 3 Prev Next