Datacore Play|Import

Allows to manage (CRUD) and find Data Resources and their Models using JSON/HTTP REST calls.

  - Data Resources are handled in JSON-LD-like (with an implicit context) format (see also its playground) and with W3C LDP (Linked Data Platform, see primer and wiki)-like operations (URIs, future collection filtering (1, 2) -inspired finders etc.). Have a look at existing Resources, at /dc/type/pli:city_0 for instance for resources in the pli:city_0 Model type (meaning version 0 of the pli:city Model). In addition to the simpler default native pure JSON format, "true" JSON-LD formats and semantic web formats (such as RDF, see example) are available.

  - Data Models describe what kinds of Data Resources are allowed, in a JSON Schema-like structure (see playground) with string, boolean, int, float, long, double, date, map, list, i18n, resource fields grouped in reusable Mixin types. Have a look at known Models at /dc/type/dcmo:model_0 and Mixins at /dc/type/dcmi:mixin_0, in their own metamodel where they may be introspected (see how), but also drafted and published (upcoming).

See wiki for further examples, FAQ, how to & common use cases, cookbook (writing clients...) and full documentation.

Datacore (NOT INITED) :

See this Playground's user manual below and its tutorial on wiki.

Most common use cases' queries:

NB. if some queries fail, please check Frequently AskedQuestions first.

  - geo_1 - list countries : countries supported by Ozwillo (Portal etc.), or also sorted by name. Or for some use cases any country (actually first 10 countries ; or first 100 countries but beware that limit can't go beyond 100, in which case iteration on ex. id or name should be preferred, see below for organizations).

NB. from now on, all queries are sorted by the most appropriate field.

  - geo_1 - choose city by upper hierarchy ex. Paris or Sofia (София) : having chosen country, NUTS2s by country (or country NUTS2s) then NUTS3s by NUTS2, or depending on your country NUTS3s by country (or country NUTS3s), and finally address cities by NUTS3 ( another example ), or for some use cases geographical cities by NUTS3 (or country geographical cities by NUTS3).

  - geo_1 - choose city using autocompletion field: address city by fulltext name and country ( another example ; however it has been disabled on generic display name to avoid finding its cities when looking for a province), or for some use cases geographical city by fulltext name and country.

  - org_1 - choose org using autocompletion input : orgs by fulltext name in country then also by altName in country (must be done in separate requests), or country org by fulltext name then also by altName (however it has been disabled on generic display name)

  - org_1 - choose business activity using autocompletion input : activity by fulltext name and country or country activity by fulltext name

  - org_1 - downloading all business activities to cache them by iterating on @id (easier than on orgact:code which would need to do >= and merge same-code activities of different countries, rather than merely >) : first query, then iterate on activities coming after last name returned : second query, third query...


  - "reached maxScan (300) before document limit" errors (or no return) on ex. "by geo area" queries : ask model designer to add the right indexes (and first to check that said field has values). Workaround : query in country-specific types.

  - "operation exceeded time limit" errors on ex. geoname "by geoname:id" queries : same as above, ask model designer to add the right indexes (and first to check that said field has values). Workaround : use the URI (@id field) as ID.

  - "Attempting a fulltext search on a field that is not configured so" errors : ask model designer to enable the right fulltext indexes. Workaround : query on odisp:name.

  - No return from a fulltext search : data can't be returned if it hasn't been saved since enabled for fulltext, so try saving some, else ask data provider to do so, possibly in bulk.

  - "Access Denied" or "Forbidden" : see Rights FAQ.

  - "Can't get Resource, more than one with same URI [uri] (in projects [a, b seeing a]): has been forked implicitly, should also be forked explicitly" : if this Resource shouldn't exist in project a, remove it from there. Otherwise, add its URI in b's dcmp:forkedUris. Then you will also be able to this Resource it from b if it should not be there, in which case don't forget to remove said URI from b's dcmp:forkedUris.

  - Requests made in your own code don't work, though in the Playground they do : your code does not do everything it should. Enable Firebug in Firefox and have a look in the "Console" tab at the actual request that the Playground makes: URL, headers, body... and find out where your code's request differs.

  - Query on Resource links don't work, though on primitive values they do : maybe you haven't encoded the Resource link's URI request parameter, as it should be per HTTP specs (so since URIs already have some encoded part, they're actually double encoded). It's actually a subset of the previous FAQ entry. (Reported in #142)

  - "Version of data resource to update is required in PUT" error : to update an existing data, you must PUT (rather than POST) it with the current version number (HTTP Entity Tag, used for optimistic locking to ensure data integrity). And if you don't store said current version locally, you must first GET it from server. You must also handle the case of the version being changed while your request is still reaching the server, by a retrying it a few times.

  - Project & Model troubleshooting & FAQ

  - Wrong or inconsistent data : see known cases in issues tagged "data inconsistency" on github. If not yet there, report it in #62 for geo & org data or as such a new issue tagged by "data inconsistency".

  - All reported FAQ cases : see issues tagged "FAQ" on github.

Adopting the Datacore:

To add your own data using REST calls, you must first define or reuse the appropriate business-specific Models (?) . You can do both using the Import UI (see Master Model, empty template & samples there including OpenElec's, and its documentation on wiki).
However, if you prefer, you can do it using Java (see Provto sample), REST calls, or any other way that still follows the methodology.

To use data (existing or yours) from your own business application, use the JSON/HTTP client of your own business application's platform and / or of your choice to call the Datacore API server using REST JSON/HTTP calls. See README for details on alternatives : the Datacore CXF Java client, any other Java JAXRS-compatible client, swagger.js in javascript, swagger codegen in any other language, any bare HTTP client, or supporting Ozwillo app frameworks if you've chosen to build your app on top of it (oasis-spring-integration, ozwillo-node-client).

API Details

This endpoint has for base URL NOT INITED (URL that must be called by clients using HTTP) and for container URL NOT INITED (URL that must be used to build Datacore Data resource URIs).

URIs that uniquely identify Datacore Data resource are in the form $containerUrl/dc/type/$type/$id, where $containerUrl is unique for a container (ex., $type is among available model types and $id is the resource's business ID (therefore relative to tjee model type). Also note that /dc/type/$type/$id is the Resource's Internal Resource Identifier (IRI).

  - IDs have 3 constraints : they must be valid URL endings, unique (not enforced yet), and if possible readable and representative of the resource.

  - URIs must be valid URLs meaning that all non safe characters (beyond [0-9a-zA-Z]$-_.+!*'() including reserved characters $&+,/:;=?@ outside of their purpose such as '?' within $type) must be URL-encoded, and it's bad practice for model names to contain some.

  - URIs of embedded Resources (as sub Resource within another top level Resource) are by default $containerUrl/dc/type/$subResourceType/$resourceId/$subResourceJsonPath, but can be something else if parsing is defined consistently within $subResourceType.

Values are formatted in JSON as follows :

  - date as ISO8601 such as "2014-01-08T09:31:19.062Z", "2014-01-08T10:31:19.062+01:00". Put your (data or application) timezone in your dates so that they can be shared across the world while staying readable for your local use case.

  - long and double as string

  - i18n as a list of translation objects (like JSONLD), for instance : [{"v": "Torino", "l": "it"}, {"v": "Turin","l": "fr"}]. It allows language-agnostic lookups such as /dc/type/pli:city_0?pli:name_i18n=Torino (or using the exact field path /dc/type/pli:city_0?pli:name_i18n.v=Torino ) so that values are useful even if not untranslated and therefore are not siloed in their language, but also language-scoped lookups such as /dc/type/pli:city_0? (or using the exact field path through $elemMatch /dc/type/pli:city_0?pli:name_i18n=$elemMatch{"v":"Torino","l":"it"} ). Moreover, language can also be specified once for the whole query in a l or @language parameter such as in /dc/type/pli:city_0?pli:name_i18n=Torino&l=it . Beyond such exact lookups, all string operators can be used, and most usefully $fulltext (must be explicitly enabled on the field) and $regex.

Advanced field features are documented in the Models' import wiki and templates, all linked from their Import UI. Here's a quick list : required, defaultValue, queryLimit, aliasedStorageNames, readonly ; string & i18n : fulltext ; i18n : defaultLanguage ; resource : resourceType ; list : listElementField, isSet, keyFieldName ; map : mapFields.

Operations are specified in details below within the Swagger UI playground, and especially :

  - "postAllDataInType" for typed updates (version) and

  - "findDataInType" for typed queries (criteria including $fulltext, sort, ranged query- or iterator-based pagination, debug mode, view).

Note that typed operations where type of data is provided in URL are usually more efficient : '*inType' methods are faster, finder queries with type even more.

  - In production (and integration), Ozwillo Kernel OAuth2 authentication must be used, i.e. HTTP requests' Authorization header must be "Bearer <valid OAuth2 Bearer / Access Token>" (see how to get one). This implies that you must first register your application with the Kernel (see documentation).
  - In dev mode, there is a mock authentication system that supports Basic Auth (i.e. HTTP requests' Authorization header must be "Basic <base 64-encoding of username:password>"), for a few users without password, see available ones in configuration.
  - Datacore API's Swagger UI allows to provide said Authorization header, with a default value that logs as admin in dev mode.

Explore available models and projects

A project is a business-consistent set of models and their resources, and can see those of its visible projects (its dependencies). All interactions occur within the context of the current project, but some may be prevented : for instance, it usually forbidden to write models in another project than the current one. A project can have forked URIs i.e. that cannot be seen in visible projects, which mostly allows to fork models i.e. redefine / override its own version of a model.

To start exploring available models, select your chosen project (i.e. the point of view that you wish to explore from, ex. geo_1 for geographical jurisdictions such as countries and cities) or merely the default oasis.main project in the dropdown above the Playground UI. This will display in the Playground result output its project portal, i.e. the list of storage models (ex. in geo_1 geo:Area_0 ) and for each of them a link to models they store (ex. models stored in geo:Area_0 ), and the list of pure mixins (not mandatorily stored in a single storage model) and for each of them a link to models that use them as mixins, that are local to the selected project. Note that this is why there are none in the geo project, which makes resources and models of geo_1 visible but doesn't have any of its own.

Further, querying models as Resources allows to know in which Model type(s) a given kind of Resources can be queried, for instance :
      - dcmo:globalMixins=pl:place_0 : lists Models that describe a location i.e. have a pl:shape WKS point, such as : co:company_0, whose WKS can then be retrieved, for instance by GET /dc/type/co:company_0 .
      - dcmo:globalFields.dcmf:name=plo:name : lists Models that "officially" have a country (i.e. having the "official" field for a country in Ozwillo Datacore), and that can therefore be queried on that, such as : plo:country_0, pli:city_0, co:company_0, cityareauseit:urbanAreaDestinationOfUse_0, cityarea:cityArea_0 , which can then be queried by country, for instance by GET /dc/type/plo:country_0, GET /dc/type/pli:city_0, GET /dc/type/co:company_0, GET /dc/type/cityareauseit:urbanAreaDestinationOfUse_0 and GET /dc/type/cityarea:cityArea_0.

Here are the generic projects :
  - oasis.main is the default project and shows all current stable versions of projects of public interest,
  - oasis.meta contains the metamodel and is visible by all other projects,
  - oasis.sample contains unit test technical samples,
  - oasis.sandbox can be used by anyone to test the Datacore with its own models and resources, especially in the Import UI.

Here are some of the most often reused projects :
  - geo_1 (major version 1) contains all geographical jurisdictions (such as countries and cities), while geo makes its current stable version visible whatever its number,
  - org_1 (major version 1) contains all public and private organizations as well as persons and also sees geo_1, while org makes its current stable version visible whatever its number.

Here are project-level Rights features :
  - dcmp:securityConstraints : if any, checked firsthand whatever the resource.
  - dcmp:securityDefaults : used when !modelLevelSecurityEnabled or no security found in model hierarchy ; null means global defaults (depends on devmode) ; in case of multi-project storage models (such as dcmi:mixin_0 i.e. the metamodel), is directly used instead of model-level security which makes no sense (because without that, models of any project would only be editable by oasis.meta writers).
  - dcmp:modelLevelSecurityEnabled : allows different security levels in models/mixins (even in same resource), with permissions that are defined at Resource-level allowing the most rights (ex. owner on orgpr:PrivateOrganization_0 fields), and other permissions being allowed by model- or project-level security (ex. reader on org:Organization_0 fields). Defaults to false.
  - dcmp:useModelSecurity : use model security instead of project security as default security ; if disabled, model security has not to be defined and is replaced by project securityDefaults (even and especially if modelLevelSecurityEnabled !) and is the same across all models of this project.
  - dcmp:visibleSecurityConstraints : allows generic conf & constraints on the project to visible project relation without having to manage & store a visibleProjectRelation object ; is checked firsthand if any WHATEVER THE RESOURCE (null dataEntity) when checking rights in a model from another current project. Typically used to prevent from writing from a project in one of its dependency project's models.

Here are more project-level features :
  - dcmp:forkedUris : add a model URI there to allow to fork it i.e. to import it in this project even if it is already in one of its visible projects. This also works with any data resource outside (meta) models, but it shouldn't be abused. In details, resources listed here are handled as if, only for them, their project did not inherit / have any other project visible.
  - dcmp:frozenModelNames : allows to freeze (forbid change on) some of this project's models or all of them using the '*' wildcard, once they are stable and in order to be able to use them (create resources in them) without risking these becoming obsolete at some point.
  - dcmp:allowedModelPrefixes : if any, only models whose name starts by any one of those prefixes ('*' wildcard also allowed, to ease switching it on and off) can be changed in this project, in order to avoid to add models that belong to another project by mistake, such as POSTing an org model while importing geo elements using the master model file.

Project & Model troubleshooting & FAQ :
  - model failed to load : the Model is persisted as a Resource alright (its .../dcmo:model_0/modelName URI works) but getting its Resources fails with a ModelNotFoundException. This means that the Model can't be loaded from its Resource representation. To solve it, try to rePOST the Model's Resource representation (ex. using edit / POST or import playground features), which will trigger Model reload, and if it fails try to patch it according to the error message.
  - can't write Resource ("access denied") : check that you have the rights to, and (using ex. the edit / POST playground feaure) that its Model is not among its Project's dcmp:frozenModelNames or dcmp:allowedModelPrefixes (see above).
  - "Bad URI syntax: Illegal character in path at index n" (last index of the model URI) : check that the mixin name is not written in the "Mixin" column with an (illegal) invisible character such as a space at the end.

Rights FAQ

  - "Access Denied" or "Forbidden" : maybe you're trying to write from another project than this Resource's, or to write a frozen model or with a non-allowed prefix (see its project's dcmp:frozenModelNames and dcmp:allowedModelPrefixes respectively or patch them using the edit / POST playground feature). Or maybe the Datacore can't yet see that a new Kernel organization has been add to those you belong to, in which case log out and in again or wait for 5 minutes (see #124). Otherwise, check that you have the rights to do that or patch them, see below.

  - How to see Resource-level permissions using the Playground : Resource-level permissions are returned when querying as owner (or admin) in debug mode. Therefore to check them in the Playground, go to the said Resource by entering ex. /dc/type/poitour:Geoloc_0/32949 in the Playground URI address bar, then click on the colon (":") of the URI field (@id) to build a single result query such as this one , then go in debug mode by clicking on the "?" button, and finally look in the results for the "ownedEntities" part (that only contains data Resources that you have owner / admin rights on, for data security reasons) and within in the "owners", "writers" and "readers" entity fields. See below how to change them.

  - How to change Resource-level permissions using the Swagger UI : To change Resource-level rights on a Resource, use the Swagger part at the bottom of the Playground, and more precisely operations in the "r : Rights management (add/remove/flush)" group (after selecting the Resource's Project in the top-level dropdown box if not done yet):
    1. do a GET /dc/r/... (click on it to open it up) by providing type, id and current version of the Resource (if you don't know it, look the Resource up in the Playground) and clicking on the "Try it out" button. This returns the Resource's current rights, ex.:
GET /dc/r/orgtr:Organizasyon_0/TR/27408293994/0
  "owners": [ /* have all rights, including reading and changing rights */
    "u_2239bc06-5f33-49dd-af99-b5c87da055ab", /* 2239bc06-5f33-49dd-af99-b5c87da055ab is the id of a user */
    "9056cde5-8135-4fd9-a913-4c3103f19fac" /* id of a Kernel organization */
  "writers": [], /* have write (and delete) rights, in addition to read rights*/
  "readers": [] /* have read rights */

    2. copy and paste the result in the body of a PUT /dc/r/... (click on it to open it up), change it as you wish, then again providing type, id and same current version, and do it (click on the "Try it out" button). Now rights should have changed.
    3. to check that rights have changed, either use debug mode (see above) or do a GET /dc/r/... again BUT with the new current version (should have been incremented by 1).

Playground User Manual

(See also its tutorial on wiki.)

  - click on a URI's id to go to the Resource it is the URI of.

  - click on type in a URI or on an item of the @type list to go to the corresponding model definition.

  - click on a property name to similarly go to the model that defines it.

  - click on : between a property's key and value to query all other resources with the same value for this property in the same model type. Note that the Playground shows said key and value UNencoded in order to make it easier to enter text in the language of the user, but that developers should URI-encode them nonetheless, as Playground actually does (see its HTTP queries in a debugger such as Firebug).

  - click on / between a URI's model type and id to look for resources that refer (link) to it : if it's a metamodel it queries its resources, otherwise it lists all models that define a resource field that links to it through its model type at top level or in a top level list field, and provides links allowing to do the same at further depth below top level and for other mixins besides its model type.

  - project portal : select another project in the dropdown above the Playground (see details on how to explore its available (meta)models).

  - query line : write there the relative URI of a single Resource or of a LDP-like Datacore query (see documentation below in Swagger UI of operation findDataInType for typed queries (criteria, sort, ranged query- or iterator-based pagination).

  - buttons : roll your mouse over them to see what they do.

This server's configuration

## Development mode (also allows to set system property datacore.dropdb=true ex. in maven)
datacore.devmode=NOT INITED

## Shard Endpoint - full URL (for clients...)
datacoreApiServer.baseUrl=NOT INITED

## Container - default URL (for URIs...)
datacoreApiServer.containerUrl=NOT INITED

## Other (or all) known Datacores (comma-separated container URLs)
datacoreApiServer.knownDatacoreContainerUrls=NOT INITED

## Query :
## default maximum number of documents to scan when fulfilling a query, overriden by
## DCFields', themselves limited by DCModel's. 0 means no limit (for tests), else ex.
## 1000 (secure default), 100000 (on query-only nodes using secondary & timeout)...
datacoreApiServer.query.maxScan=NOT INITED
## default maximum start position
datacoreApiServer.query.maxStart=NOT INITED
## default maximum number of documents returned
datacoreApiServer.query.maxLimit=NOT INITED
## default number of documents returned
datacoreApiServer.query.defaultLimit=NOT INITED

## Kernel Configuration
kernel.baseUrl=NOT INITED