Forums  > Software  > Any advancement on compiled vector oriented languages  
     
Page 3 of 3Goto to page: 1, 2, [3] Prev
Display using:  

doomanx


Total Posts: 69
Joined: Jul 2018
 
Posted: 2020-04-29 08:20
In the end tools like pandas/numpy aren't designed to be used for *massive* datasets. If you get to a size where these operations become painful it's a sign you might need to switch to doing things in memory or distribute, for example by using Dask https://pandas.pydata.org/docs/user_guide/scale.html.

did you use VWAP or triple-reinforced GAN execution?

Maggette


Total Posts: 1225
Joined: Jun 2007
 
Posted: 2020-04-29 09:02
@kloc
You are probably right. If I think about it, I actually convert a lot via ".values" while using pandas (probably most of the time when slicing) and hence probably numpy than pandas. In addition I use dask when the data set is large and fall back to the "for looping hell" using numba when my code is too slow. Probably that's why I never felt pandas was slow because I never pushed it to the max.

@doomax I made very nice experiences with dask so far. The PySpark API of Spark is alright as well and I use it in several projects. But dask feels easier to use on a single machine.

and by the way: your signature is awesome!! :)

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

sharpe_machine


Total Posts: 57
Joined: Feb 2018
 
Posted: 2020-04-29 11:29
Getting back to Julia... What are some examples which motivate one to switch from Python (Numpy/Pandas, sometimes Numba) to Julia? Assuming Python is being used to create features from raw data and then feed it into a "black box" algorithm written in C++/Rust (already top performance)

prikolno


Total Posts: 48
Joined: Jul 2018
 
Posted: 2020-04-29 15:18
>What are some examples which motivate one to switch from Python (Numpy/Pandas, sometimes Numba) to Julia?

Only one I can think of is native support for multiple dispatch, which can be really nice for building features.

sharpe_machine


Total Posts: 57
Joined: Feb 2018
 
Posted: 2020-04-29 20:11
Does not Numba's @jit without signature implement sort of multiple dispatch in Python? It starts with some "default" signature and then re-compiles a function given a new unknown signature which then stored for future calls. Not sure how dispatch is implemented in Julia though.

smt1


Total Posts: 5
Joined: Jun 2012
 
Posted: 2020-05-01 06:26
I don't have much experience with numba so I can't compare its dispatch system relative to julia's. I mostly use cppyy w/ cython&pypy depending on what I'm doing.

I think Julia's most clever design choices are combining multiple dispatch and the macro/metaprogramming system with a python-like syntax.

Like this guy: https://flow.byu.edu/posts/julia-c++ I've run into occasional gotchas that absolutely kill performance in Julia if you're trying to write superelegant terse numerical code that matches C++ performance w/o doing a lot of profiling, but the situation seems to be steadily improving w/ each julia version.

I've also found julia's distributed/parallel support really great, as you'd expect from a modern language that didn't have to deal with legacy cruft.

Julia's multiple dispatch system has a lot of interesting consequences. I think it's tooling of libraries will have a very different graphical structure than something like python.
Previous Thread :: Next Thread 
Page 3 of 3Goto to page: 1, 2, [3] Prev