Storage for Inforgs
| Main Author: | H. Mangalam |
|---|---|
| Format: | info Proceeding |
| Terbitan: |
, 2016
|
| Online Access: |
https://zenodo.org/record/162776 |
Daftar Isi:
- I use ?Inforg? as a portmanteau to describe information-based organizations; what most of our systems are becoming. Coming from a university background, information interchange is perhaps more important than most. We generate and exchange large data sets, both structured and extremely unstructured. Data and documents in both large and tiny sizes, referenced as curated objects, flat and internally hierarchical files, archives of various types, and multimedia streams of reliable and unreliable network data over increasingly large pipes. And a number of these different data sets have hugely variable legal and security and legal requirements. As endpoints, a lot of academics are shamefuly still on 100 Megabits/s, but Gigabits/s (Gbs) is old hat, 10Gbs is fairly common for servers, 40Gbs is uncommon but coming and 100Gbs switch ports are already in service. Most user tools still expect Megabytes over Local Area Networks, but Terabytes (TB) over Wide Area Networks are the new normal, both between individuals and especially among Inforgs. Users expect to be able to use all the ?iTools? of their Generation with thoughtless friction-free abandon. And we, the technology Morlocks have to address the legal and administrative requirements of this in a fairly transparent way. How? This data has to land at some point, so how do we catch all this polyglot data, turn it into semipermanent bytes and make sense of these unruly TB herds, stay ahead of imminent catastrophe, and then back it all up?I propose some some solutions that have worked for us, some that haven?t, some that we?re working on, and some that we?d like to see from vendors that have not yet appeared.