Forums  > Software  > Jd database?  
     
Page 2 of 2Goto to page: 1, [2] Prev
Display using:  

Maggette


Total Posts: 1251
Joined: Jun 2007
 
Posted: 2020-08-06 07:48
I am still too young to be able to paint the 30k foot picture.

It's just that I had several projects in big and medium sized companies that were heavily reliant on open source and that I consider a failure (for example 15 scrum teams + a big DevOps and some practically unmaintainable micro service architecture on kubernetis ...they will never get rid of the 15 teams)

On the other hand I know to well the stories that are told by grizzled veterans. The company licensed some expensive shit...and then a hundred consultants came to configure, trouble shoot and "integrate" the "wunderwaffe".

Maybe the grass is just always greener on the other side of the fence.

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

svisstack


Total Posts: 355
Joined: Feb 2014
 
Posted: 2020-08-06 09:40
@Maggette: I meant the spark development and direction, as you told that contributed.

Success/failure is all about the people and open-source/prop-software don't play any role here as you should consider software only as a tool to accomplish some goal. Software is selected by the people in the first place so in both cases not enough "IQ" on the driver seat.

First Commander of the USS Enterprise

jslade


Total Posts: 1221
Joined: Feb 2007
 
Posted: 2020-08-07 21:09
> I was sort of hoping J/Jd might be vaguely similar - stable, performant, productive despite the small community.

Jd isn't real Erlang like; nobody runs the telephone system on jd. It's low risk for me; I already know it, develop my novel algorithms in J, it serves my needs well, and I know I can get the help I need from the people who wrote it.

APL family is one of those things; if you spend a couple years with it, you really can develop things much faster than in other frameworks, languages, even without the wide variety of libraries available most people are used to. The problem is you really have to spend a couple of years (or a chunk of a year full time) on learning it. Hobby that turned into useful tool for business.


"Learning, n. The kind of ignorance distinguishing the studious."

prikolno


Total Posts: 67
Joined: Jul 2018
 
Posted: 2020-08-08 03:59
@svisstack If it helps qualify the statement, when I say a naive file system, I mean something that comes with a single namespace, and doesn't require you to install expensive hardware like an Infiniband director switch for clustering; write your own custom drivers; design your own tiering code; or use an unofficial clustering extension. Also I keep in mind that single rack setups are usually less dense, e.g. in Equinix these would usually be on a 4.5 kVA circuit.

A naive file system can most definitely support 1 rack of servers in my experience. You can do it with DDN, Vast, Weka, Isilon with little fan fare, and the cost for commercial solutions is becoming more competitive over the years with QLC. You can still beat throughput benchmarks on a third of the EC2 instance types with an open source file system with a single head node, although I've never seen it saturate line rate with a single head node. (Usually the limiting variable is that the access pattern resembles random IO when you have that many clients hitting it at once.)

Some of your views seem localized to a different search space.

I don't feel Spark's community problems are explained by its origin. How about other student projects like xgboost? Likewise, year after year my juniors seem to win the ACM Sigmod and are immediately productive when they join the workforce, and I'd much trust these students to write better DBMS code than myself or half of the frameworks namedropped here.

And I've (fortunately) never encountered a serious bug in a linear algebra library. Even if they exist, production systems usually have trivial domain knowledge safeguards that catch these. On the other hand, I've seen plenty across the kernel, the compilers, Boost, Intel official publications, and so on, with very nuanced and unpredictable behavior that make them potentially more dangerous and harder to catch. Under a consistent world view, you're more justified in refusing to use any of these, but mere mortals like me probably can't get much work done adopting this stance.

svisstack


Total Posts: 355
Joined: Feb 2014
 
Posted: 2020-08-10 09:40
@prikolno: makes sense, but I think you are conservative with the estimates.

We currently run 1 dense rack (2x16A) in one location on a similar setup and I can see this could be doubled (2x dense rack), This is backed by the other 2 racks with 1x CX4 and 1x VNX. The answer highly depends on I/O requirements.

First Commander of the USS Enterprise

prikolno


Total Posts: 67
Joined: Jul 2018
 
Posted: 2020-08-26 23:51
Not an endorsement from me, haven't used it myself, but noticed this YC startup's time series DB demo today. The performance seems rather decent. Might fit some folks' use cases.
Previous Thread :: Next Thread 
Page 2 of 2Goto to page: 1, [2] Prev