|
after spending more time on IT issues, i'd like to have your insights on the following stuff...
1) nodejs. i found the idea nice at start. asyinc programming felt like opening new perscpectives. then i had my ticket to the "callback hell", when you have to jump between async and sequential programming due to real-world realities. then you have to learn "promise/deffered" stuff, and get into libs like Q, or async. at the end, i found myself in the ridiculous situation when you move to a language to leverage a new method, then make your life harder to avoid new problems created this new method. all-in, i found that nodejs's founding idea is great, still not suitable for mainstream problematics.
2) more or less same conclusion with mongoDB. i liked also the schemaless. but after few codes, i discovered the index issue (key and values 1024 bit limit, memory consumption, etc... now, i'm considering using mongo only for a tiny part, still counting for most on good old PostgreSQL.... and by what i recently found on inet, after few months many startups are leaving non-SQL hype to (re-)join SQL dbs.
3) Lua. any experience ? sounds fast & promising.
4) still a big fan of sublime text 2 ! |
/* Trust is good, no trust is better. */ |
|
|
 |
FDAXHunter
|
| Founding Member |
| Total Posts: 8372 |
| Joined: Mar 2004 |
| |
|
1. Node.js
Well, enough has been said here. Lots of attempts have been made to solve something with JavaScript. They usually end up being the wrong approach. So they pile another layer on top to fix the deficiencies of the previous layer and so it goes, turtles all the way down. What should have been done was a different approach in the first place.
2. NoSQL
At the moment there is actually a bit of a push away from NoSQL. In fact, there are several large scale projects in order to bring SQL functionality into NoSQL. No, seriously, all the vendors are working on that. Why is this? Again, it's a similar answer to 1: as always technology that is successful in one area gets applied to another (where it shouldn't be applied). The result is that you end up having to introduce another layer to address the obvious deficiencies. NosQL has it's place, and that place is usually not the realm of highly interconnected, complex data relationship that you have in finance.
3. Lua is primarily used as a scripting language because it's easily mapped into C/C++. You don't really use it to develop stand alone applications.
4. Sublime 3 is out as beta (although I didn't really notice much change). It is da bomb as far as text editors go. But you should not use it write code. Obviously if you are not using an IDE to write your code, you're doing it wrong. |
The Figs Protocol. |
|
 |
|
we're totally inline with 1,2 and 3
regarding 4 i cannot disagree more (except that ST3 is more or less ST2+update_fees).
all paths go to Rome. there's countless of HUGE projects writen out of emacs !
spent far too much time fighting against Eclipse modules management system for instance.
all IDEs tools are available outside IDEs, you just need that multi-window capability on your OS ! |
/* Trust is good, no trust is better. */ |
|
|
|
radikal
|
|
| Total Posts: 267 |
| Joined: Dec 2012 |
| |
|
I like node.js for one-off standalone-ish stuff. In general, I prefer python for web-server stuff, Tornado etc; it plays more easily with various backend DBs + pandas + quant code.
I use mongo for a few things, generally temp-ish data where Redis is probably a better fit, but I'm too lazy to migrate the stuff over. I'm really not a huge believer that there's one DB to rule them all...we use h5/mongo/mysql. I like kx if there's a full time dev supporting it....
I use ST2/iPythonNB for 99% of what I'm working on these days. |
There are no surprising facts, only models that are surprised by facts |
|
|
YukaRedux
|
| Now with added evil |
| Total Posts: 650 |
| Joined: Dec 2004 |
| |
|
I'm struggling with the Mongo issue too right now. Imagine is talking about using it as the back end for their data warehouse solution, and I'm having a hard time wrapping my head around how that's going to work. I hope to know more in the coming weeks.
On a side note, Google goes back to the future with SQL F1 database. |
Patience is just desperation disguised as a virtue. |
|
|
|
MadMax
|
|
| Total Posts: 424 |
| Joined: Feb 2006 |
| |
|
I have been toying with Mongo and CouchDB for a monitoring db. I am leaning towards Mongo.
Currently, we have a large system (by local standards, approx. 1500 CPUs) and it produces thousands of log files with details of execution and performance data, etc.
I currently have a growing body of scripts to collect all kinds of useful info from these files everyday and analyse them (the system runs nearly 20 hrs a day, a couple of batch jobs + some near-real time calculations). Now, I am feeling the need for a db to support this activity.
The choice of Mongo is because I also need to store the full docs as well for the time being since I don't really have a clear picture of all the data that I need to extract. So as I refine and extend, I would like to be able to go back and run new analysis on older runs. Note that I really need to get the best out of any solution with minimal work as I am nearly alone working on a large part of the back-end and no resources are going to be allocated to this. I will have to do it with no support and on my own time, while still implementing the continuously growing requirements and features required by the business.
I would appreciate any suggestions.
An additional piece of info: I cannot log directly to the db, as this project is not sanctioned by my employer. I may be able to do in the future if I first build a really useful tool to convince the hierarchy.
Thanks
|
|
|
|
astar
|
|
| Total Posts: 176 |
| Joined: Mar 2007 |
| |
|
| On mongo, the best way to use it is as a caching solution. A better version of memcached, because you get some searching on values. The lack of schema is nice when you are prototyping, but afterwards, you have to maintain discipline in code, and create indexes for frequently used queries. |
|
|
|
|
|
Yuka: sounds promising F1 stuff
here is the original presentation your article is based on :
google paper on F1
and to finish on node.js, something not that far from truth...
youtube movie |
/* Trust is good, no trust is better. */ |
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
an old one.
was testing this out: wiki to mongo
I'm getting this error when I run the dude's commmand
wikipedia-to-mongodb ./afwiki-latest-pages-articles.xml.bz2
{ Error: connect ECONNREFUSED 127.0.0.1:27017
at Object.exports._errnoException (util.js:1007:11)
at exports._exceptionWithHostPort (util.js:1030:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1080:14)
name: 'MongoError',
message: 'connect ECONNREFUSED 127.0.0.1:27017' }
not an expert but this looks like it's complaining about connecting to a port. I've already downloaded the wiki bz2 file....wtf?
|
Chiral is Tyler Durden |
|
|
|
rdg
|
|
| Total Posts: 26 |
| Joined: Dec 2008 |
| |
|
27017 is the default mongodb port. Do you have a mongodb instance running on localhost?
Since this thread was created, postgres has added super useful support for json. It fit my use case perfectly: mostly structured data mixed with some ad hoc nonsense. |
|
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
seems mongo isn't even running. when I try to run I get this:
Downloads/mongodb-osx-x86_64-3.2.7\ 2/bin/mongo ; exit;
MongoDB shell version: 3.2.7
connecting to: test
2016-06-23T15:44:57.797+0100 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, reason: errno:61 Connection refused
2016-06-23T15:44:57.798+0100 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connectsrc/mongo/shell/mongojs:229:14
@(connect):1:6
exception: connect failed
logout
[Process completed]
I did a netstat -an | grep LISTEN and I don't see 27017
|
Chiral is Tyler Durden |
|
|
|
rdg
|
|
| Total Posts: 26 |
| Joined: Dec 2008 |
| |
|
| you gotta run the daemon bin (mongod). then then shell and that doods script should be able to connect. |
|
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
| Will check later many thanks! |
Chiral is Tyler Durden |
|
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
got this on mongogod
2016-06-24T06:25:26.124+0100 I CONTROL [initandlisten] MongoDB starting : pid=8231 port=27017 dbpath=/data/db 64-bit host=Nonius-Air-2.default
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] db version v3.2.7
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] OpenSSL version: OpenSSL 0.9.8za 5 Jun 2014
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] allocator: system
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] modules: none
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] build environment:
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] distarch: x86_64
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] target_arch: x86_64
2016-06-24T06:25:26.125+0100 I CONTROL [initandlisten] options: {}
2016-06-24T06:25:26.129+0100 I STORAGE [initandlisten] exception in initAndListen: 29 Data directory /data/db not found., terminating
2016-06-24T06:25:26.129+0100 I CONTROL [initandlisten] dbexit: rc: 100
edit: fixed that. |
Chiral is Tyler Durden |
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
| Got it to work! thanks rdg! |
Chiral is Tyler Durden |
|
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
so, got the db. can go into mongo shell and execute commands. now trying C# wrap.
tried this, which doesn't work. (ps I wish copy paste would respect indentation)
public static async void Run(){
IMongoDatabase _database;
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
_database = client.GetDatabase("af_wikipedia");
var collection = _database.GetCollection("Antofagasta");
var filter = new BsonDocument();
var count = 0;
using (var cursor = await collection.FindAsync(filter))
{
while (await cursor.MoveNextAsync())
{
var batch = cursor.Current;
foreach (var document in batch)
{
// process document
count++;
}
}
}
} |
Chiral is Tyler Durden |
|
|
rdg
|
|
| Total Posts: 26 |
| Joined: Dec 2008 |
| |
|
you're welcome Nonious. but you now know 90% of what I do about mongo. i'm not set up to test in c# on this machine but i did try loading some data with that guys script. it looks like all the documents are in the collection 'wikipedia'. i'd guess if you used 'var collection = _database.GetCollection("wikipedia");' you'd start seeing some results.
you can also test it in the shell:
> use af_wikipedia
> db.wikipedia.find()
(shittons of output)
|
|
|
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
| Thanks rdg, I'll be pretty mad at myself if it's that simple yet pleased that, doh, it's that simple. Will try soon |
Chiral is Tyler Durden |
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
| yes, I was able to get results in a shell, but not in C#.....don't know why. |
Chiral is Tyler Durden |
|
|
|
rdg
|
|
| Total Posts: 26 |
| Joined: Dec 2008 |
| |
|
hrm, dunno. this does it for me. it counts ~60k documents.
// edited to add back in the sripped generic type using html escapes
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using MongoDB.Bson;
using MongoDB.Driver;
namespace MongoTest
{
class MainClass
{
public static void Main(string[] args)
{
Task.Run(async () =>
{
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var db = client.GetDatabase("af_wikipedia");
var collection = db.GetCollection<BsonDocument>("wikipedia");
var filter = new BsonDocument();
var count = 0;
using (var cursor = await collection.FindAsync(filter))
{
while (await cursor.MoveNextAsync())
{
var batch = cursor.Current;
foreach (var document in batch)
{
// process document
count++;
}
}
}
Console.WriteLine("Count " + count);
}).Wait();
}
}
}
|
|
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
weird...I get a compilation error on this line
var collection = db.GetCollection("wikipedia");
while I normally run VS .NET in a windows environment, in this case I was running Xamarin on a mac air.
will check
|
Chiral is Tyler Durden |
|
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
got it it to work! but had to replace
with
db.GetCollection("wikipedia");
thanks! |
Chiral is Tyler Durden |
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
| I guess it will take a while to download the English version....it's surprisingly small though... |
Chiral is Tyler Durden |
|
|
|
rdg
|
|
| Total Posts: 26 |
| Joined: Dec 2008 |
| |
|
nice.
yeah, the forum stripped the <BsonDocument> because it looks like markup.
|
|
|
|
Nonius
|
| Founding MemberNonius Unbound |
| Total Posts: 12808 |
| Joined: Mar 2004 |
| |
|
| oh, yeah, noticed that too! |
Chiral is Tyler Durden |
|
|
|
