> For the complete documentation index, see [llms.txt](https://docs.veilio.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.veilio.xyz/documentation/basics/datasets-uploads.md).

# Datasets Upload: CSV and JSON

Veilio datasets let you upload structured files, tokenize sensitive columns, and share filtered views with role-based access policies.

#### Supported formats

* **CSV** (`.csv`)
* **JSON** (`.json`) — array of objects

**Expected JSON shape**

```json
[
  { "email": "a@example.com", "phone": "+33600000000", "customer_id": "C001" },
  { "email": "b@example.com", "phone": "+33700000000", "customer_id": "C002" }
]
```

***

#### Upload flow

1. Owner or DPO uploads a file from the dashboard **or** via API.
2. Veilio extracts headers / columns from CSV or JSON.
3. `columnPolicies` must include **every** column exactly once.
4. Non-empty cell values are tokenized into the dataset grid.
5. A dataset version is created and available for controlled download.

**Identity column (`entityId`)**

Specify which column holds your business identifier (kept in clear text, links tokens to a profile):

* **Dashboard:** set during upload configuration.
* **API:** `entityIdColumn` in the multipart form (must match a column name).

**Historical migration upload**

For large legacy imports, pass `migration=true` so tokenized cells consume the **migration budget** instead of monthly plan quota.

Requirements:

* Migration mode activated (`POST /api/onboarding/migration`).
* Active Starter / Pro / Enterprise plan (or MIGRATION API key).

***

#### API upload (multipart)

**Endpoint:** `POST /api/datasets`

**Auth:** API key (`Authorization: Bearer` or `X-API-Key`) or dashboard session.

**Role:** Owner or DPO only.

**Content-Type:** `multipart/form-data`

| Form field       | Required | Description                                              |
| ---------------- | -------- | -------------------------------------------------------- |
| `file`           | Yes      | CSV or JSON file                                         |
| `name`           | Yes      | Dataset name (1–200 chars)                               |
| `columnPolicies` | Yes      | JSON **string** — one policy per column                  |
| `format`         | No       | `csv` or `json` (auto-detected from filename if omitted) |
| `description`    | No       | Max 2000 chars                                           |
| `entityIdColumn` | No       | Column name for business ID (plaintext)                  |
| `migration`      | No       | `"true"` to use migration budget                         |

**Example `columnPolicies`**

```json
{
  "customer_id": { "roles": ["OWNER", "DPO", "MEMBER", "VIEWER"] },
  "email": { "roles": ["OWNER", "DPO"] },
  "phone": { "roles": ["OWNER", "DPO", "MEMBER"] }
}
```

* `roles`: which organization roles may see this column on download.
* `userIds`: optional list of specific user IDs (alternative to roles).

Every column in the file **must** have a policy entry. Extra or missing keys return `VALIDATION_ERROR`.

**curl example**

```bash
export VEILIO_API_KEY="veil_live_xxxxxxxx"
export VEILIO_BASE_URL="https://app.veilio.xyz/api"

curl -s -X POST "$VEILIO_BASE_URL/datasets" \
  -H "Authorization: Bearer $VEILIO_API_KEY" \
  -F "file=@contacts.csv" \
  -F "name=CRM Export March 2026" \
  -F 'description=Historical customer export' \
  -F 'entityIdColumn=customer_id' \
  -F 'migration=true' \
  -F 'columnPolicies={
    "customer_id": { "roles": ["OWNER", "DPO", "MEMBER", "VIEWER"] },
    "email": { "roles": ["OWNER", "DPO"] },
    "phone": { "roles": ["OWNER", "DPO", "MEMBER"] }
  }'
```

**JSON file example**

```bash
curl -s -X POST "$VEILIO_BASE_URL/datasets" \
  -H "Authorization: Bearer $VEILIO_API_KEY" \
  -F "file=@contacts.json;type=application/json" \
  -F "name=Contacts JSON" \
  -F 'format=json' \
  -F 'columnPolicies={"email":{"roles":["OWNER","DPO"]},"phone":{"roles":["OWNER","DPO","MEMBER"]}}'
```

**Success response `201`**

```json
{
  "id": "ds_...",
  "name": "CRM Export March 2026",
  "version": 1,
  "byteSize": 1048576,
  "columnCount": 3,
  "storage": "tokenized",
  "shareApiKey": "veil_...",
  "shareApiKeyPrefix": "veil_ab",
  "datasetAccessToken": "dsat_...",
  "datasetAccessTokenExpiresAt": "2026-03-28T10:00:00.000Z",
  "datasetAccessTokenHeader": "x-veilio-dataset-access-token"
}
```

`shareApiKey` and `datasetAccessToken` are returned only for **organization** datasets.

***

#### Validation rules and limits

| Rule             | Default / detail                                           |
| ---------------- | ---------------------------------------------------------- |
| Max file size    | **320 MB** (`VEILIO_DATASET_MAX_BYTES`)                    |
| Max cells        | **200 000** (`headers × rows`, `VEILIO_DATASET_MAX_CELLS`) |
| Empty file       | Rejected                                                   |
| `columnPolicies` | Must match columns exactly                                 |
| JSON shape       | Non-empty array of objects                                 |

#### Common errors

| Message                                    | Cause                           |
| ------------------------------------------ | ------------------------------- |
| `JSON dataset must be an array of objects` | Invalid JSON structure          |
| `JSON dataset is empty`                    | No rows                         |
| `Missing policy for column "..."`          | Incomplete `columnPolicies`     |
| `Policy references unknown column "..."`   | Extra policy key                |
| `Too many cells (max 200000)`              | File too large in cell count    |
| `File exceeds max size (... bytes)`        | File too large in bytes         |
| `Only Owner or DPO can upload datasets`    | Insufficient role               |
| `PLAN_LIMIT`                               | Dataset or token quota exceeded |
| Migration budget exhausted / expired       | When `migration=true`           |

***

#### Download behavior

Downloaded datasets are filtered by role and column policy. Consumers only receive columns they are allowed to see.

For programmatic download, use a Dataset Access Token (`dsat_...`) in header `x-veilio-dataset-access-token`.

**Related links**

* Dataset Access Tokens
* Historical Migration Mode
* API Reference


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.veilio.xyz/documentation/basics/datasets-uploads.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.